PGM.cpp@ 18927

Last change on this file since 18927 was 18927, checked in by vboxsync, 16 years ago
Big step to separate VMM data structures for guest SMP. (pgm, em)
Property svn:eol-style set to `native` Property svn:keywords set to `Id`
File size: 228.6 KB

Line
1	/* $Id: PGM.cpp 18927 2009-04-16 11:41:38Z vboxsync $ */
2	/** @file
3	* PGM - Page Manager and Monitor. (Mixing stuff here, not good?)
4	*/
5
6	/*
7	* Copyright (C) 2006-2007 Sun Microsystems, Inc.
8	*
9	* This file is part of VirtualBox Open Source Edition (OSE), as
10	* available from http://www.virtualbox.org. This file is free software;
11	* you can redistribute it and/or modify it under the terms of the GNU
12	* General Public License (GPL) as published by the Free Software
13	* Foundation, in version 2 as it comes in the "COPYING" file of the
14	* VirtualBox OSE distribution. VirtualBox OSE is distributed in the
15	* hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
16	*
17	* Please contact Sun Microsystems, Inc., 4150 Network Circle, Santa
18	* Clara, CA 95054 USA or visit http://www.sun.com if you need
19	* additional information or have any questions.
20	*/
21
22
23	/** @page pg_pgm PGM - The Page Manager and Monitor
24	*
25	* @see grp_pgm,
26	* @ref pg_pgm_pool,
27	* @ref pg_pgm_phys.
28	*
29	*
30	* @section sec_pgm_modes Paging Modes
31	*
32	* There are three memory contexts: Host Context (HC), Guest Context (GC)
33	* and intermediate context. When talking about paging HC can also be refered to
34	* as "host paging", and GC refered to as "shadow paging".
35	*
36	* We define three basic paging modes: 32-bit, PAE and AMD64. The host paging mode
37	* is defined by the host operating system. The mode used in the shadow paging mode
38	* depends on the host paging mode and what the mode the guest is currently in. The
39	* following relation between the two is defined:
40	*
41	* @verbatim
42	Host > 32-bit \| PAE \| AMD64 \|
43	Guest \| \| \| \|
44	==v================================
45	32-bit 32-bit PAE PAE
46	-------\|--------\|--------\|--------\|
47	PAE PAE PAE PAE
48	-------\|--------\|--------\|--------\|
49	AMD64 AMD64 AMD64 AMD64
50	-------\|--------\|--------\|--------\| @endverbatim
51	*
52	* All configuration except those in the diagonal (upper left) are expected to
53	* require special effort from the switcher (i.e. a bit slower).
54	*
55	*
56	*
57	*
58	* @section sec_pgm_shw The Shadow Memory Context
59	*
60	*
61	* [..]
62	*
63	* Because of guest context mappings requires PDPT and PML4 entries to allow
64	* writing on AMD64, the two upper levels will have fixed flags whatever the
65	* guest is thinking of using there. So, when shadowing the PD level we will
66	* calculate the effective flags of PD and all the higher levels. In legacy
67	* PAE mode this only applies to the PWT and PCD bits (the rest are
68	* ignored/reserved/MBZ). We will ignore those bits for the present.
69	*
70	*
71	*
72	* @section sec_pgm_int The Intermediate Memory Context
73	*
74	* The world switch goes thru an intermediate memory context which purpose it is
75	* to provide different mappings of the switcher code. All guest mappings are also
76	* present in this context.
77	*
78	* The switcher code is mapped at the same location as on the host, at an
79	* identity mapped location (physical equals virtual address), and at the
80	* hypervisor location. The identity mapped location is for when the world
81	* switches that involves disabling paging.
82	*
83	* PGM maintain page tables for 32-bit, PAE and AMD64 paging modes. This
84	* simplifies switching guest CPU mode and consistency at the cost of more
85	* code to do the work. All memory use for those page tables is located below
86	* 4GB (this includes page tables for guest context mappings).
87	*
88	*
89	* @subsection subsec_pgm_int_gc Guest Context Mappings
90	*
91	* During assignment and relocation of a guest context mapping the intermediate
92	* memory context is used to verify the new location.
93	*
94	* Guest context mappings are currently restricted to below 4GB, for reasons
95	* of simplicity. This may change when we implement AMD64 support.
96	*
97	*
98	*
99	*
100	* @section sec_pgm_misc Misc
101	*
102	* @subsection subsec_pgm_misc_diff Differences Between Legacy PAE and Long Mode PAE
103	*
104	* The differences between legacy PAE and long mode PAE are:
105	* -# PDPE bits 1, 2, 5 and 6 are defined differently. In leagcy mode they are
106	* all marked down as must-be-zero, while in long mode 1, 2 and 5 have the
107	* usual meanings while 6 is ignored (AMD). This means that upon switching to
108	* legacy PAE mode we'll have to clear these bits and when going to long mode
109	* they must be set. This applies to both intermediate and shadow contexts,
110	* however we don't need to do it for the intermediate one since we're
111	* executing with CR0.WP at that time.
112	* -# CR3 allows a 32-byte aligned address in legacy mode, while in long mode
113	* a page aligned one is required.
114	*
115	*
116	* @section sec_pgm_handlers Access Handlers
117	*
118	* Placeholder.
119	*
120	*
121	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
122	*
123	* Placeholder.
124	*
125	*
126	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
127	*
128	* We currently implement three types of virtual access handlers: ALL, WRITE
129	* and HYPERVISOR (WRITE). See PGMVIRTHANDLERTYPE for some more details.
130	*
131	* The HYPERVISOR access handlers is kept in a separate tree since it doesn't apply
132	* to physical pages (PGMTREES::HyperVirtHandlers) and only needs to be consulted in
133	* a special \#PF case. The ALL and WRITE are in the PGMTREES::VirtHandlers tree, the
134	* rest of this section is going to be about these handlers.
135	*
136	* We'll go thru the life cycle of a handler and try make sense of it all, don't know
137	* how successfull this is gonna be...
138	*
139	* 1. A handler is registered thru the PGMR3HandlerVirtualRegister and
140	* PGMHandlerVirtualRegisterEx APIs. We check for conflicting virtual handlers
141	* and create a new node that is inserted into the AVL tree (range key). Then
142	* a full PGM resync is flagged (clear pool, sync cr3, update virtual bit of PGMPAGE).
143	*
144	* 2. The following PGMSyncCR3/SyncCR3 operation will first make invoke HandlerVirtualUpdate.
145	*
146	* 2a. HandlerVirtualUpdate will will lookup all the pages covered by virtual handlers
147	* via the current guest CR3 and update the physical page -> virtual handler
148	* translation. Needless to say, this doesn't exactly scale very well. If any changes
149	* are detected, it will flag a virtual bit update just like we did on registration.
150	* PGMPHYS pages with changes will have their virtual handler state reset to NONE.
151	*
152	* 2b. The virtual bit update process will iterate all the pages covered by all the
153	* virtual handlers and update the PGMPAGE virtual handler state to the max of all
154	* virtual handlers on that page.
155	*
156	* 2c. Back in SyncCR3 we will now flush the entire shadow page cache to make sure
157	* we don't miss any alias mappings of the monitored pages.
158	*
159	* 2d. SyncCR3 will then proceed with syncing the CR3 table.
160	*
161	* 3. \#PF(np,read) on a page in the range. This will cause it to be synced
162	* read-only and resumed if it's a WRITE handler. If it's an ALL handler we
163	* will call the handlers like in the next step. If the physical mapping has
164	* changed we will - some time in the future - perform a handler callback
165	* (optional) and update the physical -> virtual handler cache.
166	*
167	* 4. \#PF(,write) on a page in the range. This will cause the handler to
168	* be invoked.
169	*
170	* 5. The guest invalidates the page and changes the physical backing or
171	* unmaps it. This should cause the invalidation callback to be invoked
172	* (it might not yet be 100% perfect). Exactly what happens next... is
173	* this where we mess up and end up out of sync for a while?
174	*
175	* 6. The handler is deregistered by the client via PGMHandlerVirtualDeregister.
176	* We will then set all PGMPAGEs in the physical -> virtual handler cache for
177	* this handler to NONE and trigger a full PGM resync (basically the same
178	* as int step 1). Which means 2 is executed again.
179	*
180	*
181	* @subsubsection sub_sec_pgm_handler_virt_todo TODOs
182	*
183	* There is a bunch of things that needs to be done to make the virtual handlers
184	* work 100% correctly and work more efficiently.
185	*
186	* The first bit hasn't been implemented yet because it's going to slow the
187	* whole mess down even more, and besides it seems to be working reliably for
188	* our current uses. OTOH, some of the optimizations might end up more or less
189	* implementing the missing bits, so we'll see.
190	*
191	* On the optimization side, the first thing to do is to try avoid unnecessary
192	* cache flushing. Then try team up with the shadowing code to track changes
193	* in mappings by means of access to them (shadow in), updates to shadows pages,
194	* invlpg, and shadow PT discarding (perhaps).
195	*
196	* Some idea that have popped up for optimization for current and new features:
197	* - bitmap indicating where there are virtual handlers installed.
198	* (4KB => 220 pages, page 212 => covers 32-bit address space 1:1!)
199	* - Further optimize this by min/max (needs min/max avl getters).
200	* - Shadow page table entry bit (if any left)?
201	*
202	*/
203
204
205	/** @page pg_pgm_phys PGM Physical Guest Memory Management
206	*
207	*
208	* Objectives:
209	* - Guest RAM over-commitment using memory ballooning,
210	* zero pages and general page sharing.
211	* - Moving or mirroring a VM onto a different physical machine.
212	*
213	*
214	* @subsection subsec_pgmPhys_Definitions Definitions
215	*
216	* Allocation chunk - A RTR0MemObjAllocPhysNC object and the tracking
217	* machinery assoicated with it.
218	*
219	*
220	*
221	*
222	* @subsection subsec_pgmPhys_AllocPage Allocating a page.
223	*
224	* Initially we map all guest memory to the (per VM) zero page, which
225	* means that none of the read functions will cause pages to be allocated.
226	*
227	* Exception, access bit in page tables that have been shared. This must
228	* be handled, but we must also make sure PGMGst*Modify doesn't make
229	* unnecessary modifications.
230	*
231	* Allocation points:
232	* - PGMPhysSimpleWriteGCPhys and PGMPhysWrite.
233	* - Replacing a zero page mapping at \#PF.
234	* - Replacing a shared page mapping at \#PF.
235	* - ROM registration (currently MMR3RomRegister).
236	* - VM restore (pgmR3Load).
237	*
238	* For the first three it would make sense to keep a few pages handy
239	* until we've reached the max memory commitment for the VM.
240	*
241	* For the ROM registration, we know exactly how many pages we need
242	* and will request these from ring-0. For restore, we will save
243	* the number of non-zero pages in the saved state and allocate
244	* them up front. This would allow the ring-0 component to refuse
245	* the request if the isn't sufficient memory available for VM use.
246	*
247	* Btw. for both ROM and restore allocations we won't be requiring
248	* zeroed pages as they are going to be filled instantly.
249	*
250	*
251	* @subsection subsec_pgmPhys_FreePage Freeing a page
252	*
253	* There are a few points where a page can be freed:
254	* - After being replaced by the zero page.
255	* - After being replaced by a shared page.
256	* - After being ballooned by the guest additions.
257	* - At reset.
258	* - At restore.
259	*
260	* When freeing one or more pages they will be returned to the ring-0
261	* component and replaced by the zero page.
262	*
263	* The reasoning for clearing out all the pages on reset is that it will
264	* return us to the exact same state as on power on, and may thereby help
265	* us reduce the memory load on the system. Further it might have a
266	* (temporary) positive influence on memory fragmentation (@see subsec_pgmPhys_Fragmentation).
267	*
268	* On restore, as mention under the allocation topic, pages should be
269	* freed / allocated depending on how many is actually required by the
270	* new VM state. The simplest approach is to do like on reset, and free
271	* all non-ROM pages and then allocate what we need.
272	*
273	* A measure to prevent some fragmentation, would be to let each allocation
274	* chunk have some affinity towards the VM having allocated the most pages
275	* from it. Also, try make sure to allocate from allocation chunks that
276	* are almost full. Admittedly, both these measures might work counter to
277	* our intentions and its probably not worth putting a lot of effort,
278	* cpu time or memory into this.
279	*
280	*
281	* @subsection subsec_pgmPhys_SharePage Sharing a page
282	*
283	* The basic idea is that there there will be a idle priority kernel
284	* thread walking the non-shared VM pages hashing them and looking for
285	* pages with the same checksum. If such pages are found, it will compare
286	* them byte-by-byte to see if they actually are identical. If found to be
287	* identical it will allocate a shared page, copy the content, check that
288	* the page didn't change while doing this, and finally request both the
289	* VMs to use the shared page instead. If the page is all zeros (special
290	* checksum and byte-by-byte check) it will request the VM that owns it
291	* to replace it with the zero page.
292	*
293	* To make this efficient, we will have to make sure not to try share a page
294	* that will change its contents soon. This part requires the most work.
295	* A simple idea would be to request the VM to write monitor the page for
296	* a while to make sure it isn't modified any time soon. Also, it may
297	* make sense to skip pages that are being write monitored since this
298	* information is readily available to the thread if it works on the
299	* per-VM guest memory structures (presently called PGMRAMRANGE).
300	*
301	*
302	* @subsection subsec_pgmPhys_Fragmentation Fragmentation Concerns and Counter Measures
303	*
304	* The pages are organized in allocation chunks in ring-0, this is a necessity
305	* if we wish to have an OS agnostic approach to this whole thing. (On Linux we
306	* could easily work on a page-by-page basis if we liked. Whether this is possible
307	* or efficient on NT I don't quite know.) Fragmentation within these chunks may
308	* become a problem as part of the idea here is that we wish to return memory to
309	* the host system.
310	*
311	* For instance, starting two VMs at the same time, they will both allocate the
312	* guest memory on-demand and if permitted their page allocations will be
313	* intermixed. Shut down one of the two VMs and it will be difficult to return
314	* any memory to the host system because the page allocation for the two VMs are
315	* mixed up in the same allocation chunks.
316	*
317	* To further complicate matters, when pages are freed because they have been
318	* ballooned or become shared/zero the whole idea is that the page is supposed
319	* to be reused by another VM or returned to the host system. This will cause
320	* allocation chunks to contain pages belonging to different VMs and prevent
321	* returning memory to the host when one of those VM shuts down.
322	*
323	* The only way to really deal with this problem is to move pages. This can
324	* either be done at VM shutdown and or by the idle priority worker thread
325	* that will be responsible for finding sharable/zero pages. The mechanisms
326	* involved for coercing a VM to move a page (or to do it for it) will be
327	* the same as when telling it to share/zero a page.
328	*
329	*
330	* @subsection subsec_pgmPhys_Tracking Tracking Structures And Their Cost
331	*
332	* There's a difficult balance between keeping the per-page tracking structures
333	* (global and guest page) easy to use and keeping them from eating too much
334	* memory. We have limited virtual memory resources available when operating in
335	* 32-bit kernel space (on 64-bit there'll it's quite a different story). The
336	* tracking structures will be attemted designed such that we can deal with up
337	* to 32GB of memory on a 32-bit system and essentially unlimited on 64-bit ones.
338	*
339	*
340	* @subsubsection subsubsec_pgmPhys_Tracking_Kernel Kernel Space
341	*
342	* @see pg_GMM
343	*
344	* @subsubsection subsubsec_pgmPhys_Tracking_PerVM Per-VM
345	*
346	* Fixed info is the physical address of the page (HCPhys) and the page id
347	* (described above). Theoretically we'll need 48(-12) bits for the HCPhys part.
348	* Today we've restricting ourselves to 40(-12) bits because this is the current
349	* restrictions of all AMD64 implementations (I think Barcelona will up this
350	* to 48(-12) bits, not that it really matters) and I needed the bits for
351	* tracking mappings of a page. 48-12 = 36. That leaves 28 bits, which means a
352	* decent range for the page id: 2^(28+12) = 1024TB.
353	*
354	* In additions to these, we'll have to keep maintaining the page flags as we
355	* currently do. Although it wouldn't harm to optimize these quite a bit, like
356	* for instance the ROM shouldn't depend on having a write handler installed
357	* in order for it to become read-only. A RO/RW bit should be considered so
358	* that the page syncing code doesn't have to mess about checking multiple
359	* flag combinations (ROM \|\| RW handler \|\| write monitored) in order to
360	* figure out how to setup a shadow PTE. But this of course, is second
361	* priority at present. Current this requires 12 bits, but could probably
362	* be optimized to ~8.
363	*
364	* Then there's the 24 bits used to track which shadow page tables are
365	* currently mapping a page for the purpose of speeding up physical
366	* access handlers, and thereby the page pool cache. More bit for this
367	* purpose wouldn't hurt IIRC.
368	*
369	* Then there is a new bit in which we need to record what kind of page
370	* this is, shared, zero, normal or write-monitored-normal. This'll
371	* require 2 bits. One bit might be needed for indicating whether a
372	* write monitored page has been written to. And yet another one or
373	* two for tracking migration status. 3-4 bits total then.
374	*
375	* Whatever is left will can be used to record the sharabilitiy of a
376	* page. The page checksum will not be stored in the per-VM table as
377	* the idle thread will not be permitted to do modifications to it.
378	* It will instead have to keep its own working set of potentially
379	* shareable pages and their check sums and stuff.
380	*
381	* For the present we'll keep the current packing of the
382	* PGMRAMRANGE::aHCPhys to keep the changes simple, only of course,
383	* we'll have to change it to a struct with a total of 128-bits at
384	* our disposal.
385	*
386	* The initial layout will be like this:
387	* @verbatim
388	RTHCPHYS HCPhys; The current stuff.
389	63:40 Current shadow PT tracking stuff.
390	39:12 The physical page frame number.
391	11:0 The current flags.
392	uint32_t u28PageId : 28; The page id.
393	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
394	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
395	uint32_t u1Reserved : 1; Reserved for later.
396	uint32_t u32Reserved; Reserved for later, mostly sharing stats.
397	@endverbatim
398	*
399	* The final layout will be something like this:
400	* @verbatim
401	RTHCPHYS HCPhys; The current stuff.
402	63:48 High page id (12+).
403	47:12 The physical page frame number.
404	11:0 Low page id.
405	uint32_t fReadOnly : 1; Whether it's readonly page (rom or monitored in some way).
406	uint32_t u3Type : 3; The page type {RESERVED, MMIO, MMIO2, ROM, shadowed ROM, RAM}.
407	uint32_t u2PhysMon : 2; Physical access handler type {none, read, write, all}.
408	uint32_t u2VirtMon : 2; Virtual access handler type {none, read, write, all}..
409	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
410	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
411	uint32_t u20Reserved : 20; Reserved for later, mostly sharing stats.
412	uint32_t u32Tracking; The shadow PT tracking stuff, roughly.
413	@endverbatim
414	*
415	* Cost wise, this means we'll double the cost for guest memory. There isn't anyway
416	* around that I'm afraid. It means that the cost of dealing out 32GB of memory
417	* to one or more VMs is: (32GB >> PAGE_SHIFT) * 16 bytes, or 128MBs. Or another
418	* example, the VM heap cost when assigning 1GB to a VM will be: 4MB.
419	*
420	* A couple of cost examples for the total cost per-VM + kernel.
421	* 32-bit Windows and 32-bit linux:
422	* 1GB guest ram, 256K pages: 4MB + 2MB(+) = 6MB
423	* 4GB guest ram, 1M pages: 16MB + 8MB(+) = 24MB
424	* 32GB guest ram, 8M pages: 128MB + 64MB(+) = 192MB
425	* 64-bit Windows and 64-bit linux:
426	* 1GB guest ram, 256K pages: 4MB + 3MB(+) = 7MB
427	* 4GB guest ram, 1M pages: 16MB + 12MB(+) = 28MB
428	* 32GB guest ram, 8M pages: 128MB + 96MB(+) = 224MB
429	*
430	* UPDATE - 2007-09-27:
431	* Will need a ballooned flag/state too because we cannot
432	* trust the guest 100% and reporting the same page as ballooned more
433	* than once will put the GMM off balance.
434	*
435	*
436	* @subsection subsec_pgmPhys_Serializing Serializing Access
437	*
438	* Initially, we'll try a simple scheme:
439	*
440	* - The per-VM RAM tracking structures (PGMRAMRANGE) is only modified
441	* by the EMT thread of that VM while in the pgm critsect.
442	* - Other threads in the VM process that needs to make reliable use of
443	* the per-VM RAM tracking structures will enter the critsect.
444	* - No process external thread or kernel thread will ever try enter
445	* the pgm critical section, as that just won't work.
446	* - The idle thread (and similar threads) doesn't not need 100% reliable
447	* data when performing it tasks as the EMT thread will be the one to
448	* do the actual changes later anyway. So, as long as it only accesses
449	* the main ram range, it can do so by somehow preventing the VM from
450	* being destroyed while it works on it...
451	*
452	* - The over-commitment management, including the allocating/freeing
453	* chunks, is serialized by a ring-0 mutex lock (a fast one since the
454	* more mundane mutex implementation is broken on Linux).
455	* - A separeate mutex is protecting the set of allocation chunks so
456	* that pages can be shared or/and freed up while some other VM is
457	* allocating more chunks. This mutex can be take from under the other
458	* one, but not the otherway around.
459	*
460	*
461	* @subsection subsec_pgmPhys_Request VM Request interface
462	*
463	* When in ring-0 it will become necessary to send requests to a VM so it can
464	* for instance move a page while defragmenting during VM destroy. The idle
465	* thread will make use of this interface to request VMs to setup shared
466	* pages and to perform write monitoring of pages.
467	*
468	* I would propose an interface similar to the current VMReq interface, similar
469	* in that it doesn't require locking and that the one sending the request may
470	* wait for completion if it wishes to. This shouldn't be very difficult to
471	* realize.
472	*
473	* The requests themselves are also pretty simple. They are basically:
474	* -# Check that some precondition is still true.
475	* -# Do the update.
476	* -# Update all shadow page tables involved with the page.
477	*
478	* The 3rd step is identical to what we're already doing when updating a
479	* physical handler, see pgmHandlerPhysicalSetRamFlagsAndFlushShadowPTs.
480	*
481	*
482	*
483	* @section sec_pgmPhys_MappingCaches Mapping Caches
484	*
485	* In order to be able to map in and out memory and to be able to support
486	* guest with more RAM than we've got virtual address space, we'll employing
487	* a mapping cache. There is already a tiny one for GC (see PGMGCDynMapGCPageEx)
488	* and we'll create a similar one for ring-0 unless we decide to setup a dedicate
489	* memory context for the HWACCM execution.
490	*
491	*
492	* @subsection subsec_pgmPhys_MappingCaches_R3 Ring-3
493	*
494	* We've considered implementing the ring-3 mapping cache page based but found
495	* that this was bother some when one had to take into account TLBs+SMP and
496	* portability (missing the necessary APIs on several platforms). There were
497	* also some performance concerns with this approach which hadn't quite been
498	* worked out.
499	*
500	* Instead, we'll be mapping allocation chunks into the VM process. This simplifies
501	* matters greatly quite a bit since we don't need to invent any new ring-0 stuff,
502	* only some minor RTR0MEMOBJ mapping stuff. The main concern here is that mapping
503	* compared to the previous idea is that mapping or unmapping a 1MB chunk is more
504	* costly than a single page, although how much more costly is uncertain. We'll
505	* try address this by using a very big cache, preferably bigger than the actual
506	* VM RAM size if possible. The current VM RAM sizes should give some idea for
507	* 32-bit boxes, while on 64-bit we can probably get away with employing an
508	* unlimited cache.
509	*
510	* The cache have to parts, as already indicated, the ring-3 side and the
511	* ring-0 side.
512	*
513	* The ring-0 will be tied to the page allocator since it will operate on the
514	* memory objects it contains. It will therefore require the first ring-0 mutex
515	* discussed in @ref subsec_pgmPhys_Serializing. We
516	* some double house keeping wrt to who has mapped what I think, since both
517	* VMMR0.r0 and RTR0MemObj will keep track of mapping relataions
518	*
519	* The ring-3 part will be protected by the pgm critsect. For simplicity, we'll
520	* require anyone that desires to do changes to the mapping cache to do that
521	* from within this critsect. Alternatively, we could employ a separate critsect
522	* for serializing changes to the mapping cache as this would reduce potential
523	* contention with other threads accessing mappings unrelated to the changes
524	* that are in process. We can see about this later, contention will show
525	* up in the statistics anyway, so it'll be simple to tell.
526	*
527	* The organization of the ring-3 part will be very much like how the allocation
528	* chunks are organized in ring-0, that is in an AVL tree by chunk id. To avoid
529	* having to walk the tree all the time, we'll have a couple of lookaside entries
530	* like in we do for I/O ports and MMIO in IOM.
531	*
532	* The simplified flow of a PGMPhysRead/Write function:
533	* -# Enter the PGM critsect.
534	* -# Lookup GCPhys in the ram ranges and get the Page ID.
535	* -# Calc the Allocation Chunk ID from the Page ID.
536	* -# Check the lookaside entries and then the AVL tree for the Chunk ID.
537	* If not found in cache:
538	* -# Call ring-0 and request it to be mapped and supply
539	* a chunk to be unmapped if the cache is maxed out already.
540	* -# Insert the new mapping into the AVL tree (id + R3 address).
541	* -# Update the relevant lookaside entry and return the mapping address.
542	* -# Do the read/write according to monitoring flags and everything.
543	* -# Leave the critsect.
544	*
545	*
546	* @section sec_pgmPhys_Fallback Fallback
547	*
548	* Current all the "second tier" hosts will not support the RTR0MemObjAllocPhysNC
549	* API and thus require a fallback.
550	*
551	* So, when RTR0MemObjAllocPhysNC returns VERR_NOT_SUPPORTED the page allocator
552	* will return to the ring-3 caller (and later ring-0) and asking it to seed
553	* the page allocator with some fresh pages (VERR_GMM_SEED_ME). Ring-3 will
554	* then perform an SUPPageAlloc(cbChunk >> PAGE_SHIFT) call and make a
555	* "SeededAllocPages" call to ring-0.
556	*
557	* The first time ring-0 sees the VERR_NOT_SUPPORTED failure it will disable
558	* all page sharing (zero page detection will continue). It will also force
559	* all allocations to come from the VM which seeded the page. Both these
560	* measures are taken to make sure that there will never be any need for
561	* mapping anything into ring-3 - everything will be mapped already.
562	*
563	* Whether we'll continue to use the current MM locked memory management
564	* for this I don't quite know (I'd prefer not to and just ditch that all
565	* togther), we'll see what's simplest to do.
566	*
567	*
568	*
569	* @section sec_pgmPhys_Changes Changes
570	*
571	* Breakdown of the changes involved?
572	*/
573
574	/*******************************************************************************
575	* Header Files *
576	*******************************************************************************/
577	#define LOG_GROUP LOG_GROUP_PGM
578	#include <VBox/dbgf.h>
579	#include <VBox/pgm.h>
580	#include <VBox/cpum.h>
581	#include <VBox/iom.h>
582	#include <VBox/sup.h>
583	#include <VBox/mm.h>
584	#include <VBox/em.h>
585	#include <VBox/stam.h>
586	#include <VBox/rem.h>
587	#include <VBox/dbgf.h>
588	#include <VBox/rem.h>
589	#include <VBox/selm.h>
590	#include <VBox/ssm.h>
591	#include "PGMInternal.h"
592	#include <VBox/vm.h>
593	#include <VBox/dbg.h>
594	#include <VBox/hwaccm.h>
595
596	#include <iprt/assert.h>
597	#include <iprt/alloc.h>
598	#include <iprt/asm.h>
599	#include <iprt/thread.h>
600	#include <iprt/string.h>
601	#ifdef DEBUG_bird
602	# include <iprt/env.h>
603	#endif
604	#include <VBox/param.h>
605	#include <VBox/err.h>
606
607
608	/*******************************************************************************
609	* Defined Constants And Macros *
610	*******************************************************************************/
611	/** Saved state data unit version for 2.5.x and later. */
612	#define PGM_SAVED_STATE_VERSION 9
613	/** Saved state data unit version for 2.2.0. */
614	#define PGM_SAVED_STATE_VERSION_RR_DESC 7
615	/** Saved state data unit version. */
616	#define PGM_SAVED_STATE_VERSION_OLD_PHYS_CODE 6
617
618
619	/*******************************************************************************
620	* Internal Functions *
621	*******************************************************************************/
622	static int pgmR3InitPaging(PVM pVM);
623	static void pgmR3InitStats(PVM pVM);
624	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
625	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
626	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
627	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser);
628	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
629	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
630	#ifdef VBOX_STRICT
631	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser);
632	#endif
633	static DECLCALLBACK(int) pgmR3Save(PVM pVM, PSSMHANDLE pSSM);
634	static DECLCALLBACK(int) pgmR3Load(PVM pVM, PSSMHANDLE pSSM, uint32_t u32Version);
635	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0);
636	static void pgmR3ModeDataSwitch(PVM pVM, PVMCPU pVCpu, PGMMODE enmShw, PGMMODE enmGst);
637	static PGMMODE pgmR3CalcShadowMode(PVM pVM, PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher);
638
639	#ifdef VBOX_WITH_DEBUGGER
640	/** @todo Convert the first two commands to 'info' items. */
641	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
642	static DECLCALLBACK(int) pgmR3CmdMap(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
643	static DECLCALLBACK(int) pgmR3CmdError(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
644	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
645	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
646	# ifdef VBOX_STRICT
647	static DECLCALLBACK(int) pgmR3CmdAssertCR3(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
648	# endif
649	#endif
650
651
652	/*******************************************************************************
653	* Global Variables *
654	*******************************************************************************/
655	#ifdef VBOX_WITH_DEBUGGER
656	/** Argument descriptors for '.pgmerror' and '.pgmerroroff'. */
657	static const DBGCVARDESC g_aPgmErrorArgs[] =
658	{
659	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
660	{ 0, 1, DBGCVAR_CAT_STRING, 0, "where", "Error injection location." },
661	};
662
663	/** Command descriptors. */
664	static const DBGCCMD g_aCmds[] =
665	{
666	/* pszCmd, cArgsMin, cArgsMax, paArgDesc, cArgDescs, pResultDesc, fFlags, pfnHandler pszSyntax, ....pszDescription */
667	{ "pgmram", 0, 0, NULL, 0, NULL, 0, pgmR3CmdRam, "", "Display the ram ranges." },
668	{ "pgmmap", 0, 0, NULL, 0, NULL, 0, pgmR3CmdMap, "", "Display the mapping ranges." },
669	{ "pgmsync", 0, 0, NULL, 0, NULL, 0, pgmR3CmdSync, "", "Sync the CR3 page." },
670	{ "pgmerror", 0, 1, &g_aPgmErrorArgs[0],1, NULL, 0, pgmR3CmdError, "", "Enables inject runtime of errors into parts of PGM." },
671	{ "pgmerroroff", 0, 1, &g_aPgmErrorArgs[0],1, NULL, 0, pgmR3CmdError, "", "Disables inject runtime errors into parts of PGM." },
672	#ifdef VBOX_STRICT
673	{ "pgmassertcr3", 0, 0, NULL, 0, NULL, 0, pgmR3CmdAssertCR3, "", "Check the shadow CR3 mapping." },
674	#endif
675	{ "pgmsyncalways", 0, 0, NULL, 0, NULL, 0, pgmR3CmdSyncAlways, "", "Toggle permanent CR3 syncing." },
676	};
677	#endif
678
679
680
681
682	/*
683	* Shadow - 32-bit mode
684	*/
685	#define PGM_SHW_TYPE PGM_TYPE_32BIT
686	#define PGM_SHW_NAME(name) PGM_SHW_NAME_32BIT(name)
687	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_32BIT_STR(name)
688	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_32BIT_STR(name)
689	#include "PGMShw.h"
690
691	/* Guest - real mode */
692	#define PGM_GST_TYPE PGM_TYPE_REAL
693	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
694	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
695	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
696	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_REAL(name)
697	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_REAL_STR(name)
698	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_REAL_STR(name)
699	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
700	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD_PHYS
701	#include "PGMBth.h"
702	#include "PGMGstDefs.h"
703	#include "PGMGst.h"
704	#undef BTH_PGMPOOLKIND_PT_FOR_PT
705	#undef BTH_PGMPOOLKIND_ROOT
706	#undef PGM_BTH_NAME
707	#undef PGM_BTH_NAME_RC_STR
708	#undef PGM_BTH_NAME_R0_STR
709	#undef PGM_GST_TYPE
710	#undef PGM_GST_NAME
711	#undef PGM_GST_NAME_RC_STR
712	#undef PGM_GST_NAME_R0_STR
713
714	/* Guest - protected mode */
715	#define PGM_GST_TYPE PGM_TYPE_PROT
716	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
717	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
718	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
719	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_PROT(name)
720	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_PROT_STR(name)
721	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_PROT_STR(name)
722	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
723	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD_PHYS
724	#include "PGMBth.h"
725	#include "PGMGstDefs.h"
726	#include "PGMGst.h"
727	#undef BTH_PGMPOOLKIND_PT_FOR_PT
728	#undef BTH_PGMPOOLKIND_ROOT
729	#undef PGM_BTH_NAME
730	#undef PGM_BTH_NAME_RC_STR
731	#undef PGM_BTH_NAME_R0_STR
732	#undef PGM_GST_TYPE
733	#undef PGM_GST_NAME
734	#undef PGM_GST_NAME_RC_STR
735	#undef PGM_GST_NAME_R0_STR
736
737	/* Guest - 32-bit mode */
738	#define PGM_GST_TYPE PGM_TYPE_32BIT
739	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
740	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
741	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
742	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_32BIT(name)
743	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_32BIT_STR(name)
744	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_32BIT_STR(name)
745	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_32BIT_PT
746	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_32BIT_PT_FOR_32BIT_4MB
747	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD
748	#include "PGMBth.h"
749	#include "PGMGstDefs.h"
750	#include "PGMGst.h"
751	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
752	#undef BTH_PGMPOOLKIND_PT_FOR_PT
753	#undef BTH_PGMPOOLKIND_ROOT
754	#undef PGM_BTH_NAME
755	#undef PGM_BTH_NAME_RC_STR
756	#undef PGM_BTH_NAME_R0_STR
757	#undef PGM_GST_TYPE
758	#undef PGM_GST_NAME
759	#undef PGM_GST_NAME_RC_STR
760	#undef PGM_GST_NAME_R0_STR
761
762	#undef PGM_SHW_TYPE
763	#undef PGM_SHW_NAME
764	#undef PGM_SHW_NAME_RC_STR
765	#undef PGM_SHW_NAME_R0_STR
766
767
768	/*
769	* Shadow - PAE mode
770	*/
771	#define PGM_SHW_TYPE PGM_TYPE_PAE
772	#define PGM_SHW_NAME(name) PGM_SHW_NAME_PAE(name)
773	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_PAE_STR(name)
774	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_PAE_STR(name)
775	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
776	#include "PGMShw.h"
777
778	/* Guest - real mode */
779	#define PGM_GST_TYPE PGM_TYPE_REAL
780	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
781	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
782	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
783	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
784	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_REAL_STR(name)
785	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_REAL_STR(name)
786	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
787	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_PHYS
788	#include "PGMGstDefs.h"
789	#include "PGMBth.h"
790	#undef BTH_PGMPOOLKIND_PT_FOR_PT
791	#undef BTH_PGMPOOLKIND_ROOT
792	#undef PGM_BTH_NAME
793	#undef PGM_BTH_NAME_RC_STR
794	#undef PGM_BTH_NAME_R0_STR
795	#undef PGM_GST_TYPE
796	#undef PGM_GST_NAME
797	#undef PGM_GST_NAME_RC_STR
798	#undef PGM_GST_NAME_R0_STR
799
800	/* Guest - protected mode */
801	#define PGM_GST_TYPE PGM_TYPE_PROT
802	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
803	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
804	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
805	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PROT(name)
806	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_PROT_STR(name)
807	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PROT_STR(name)
808	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
809	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_PHYS
810	#include "PGMGstDefs.h"
811	#include "PGMBth.h"
812	#undef BTH_PGMPOOLKIND_PT_FOR_PT
813	#undef BTH_PGMPOOLKIND_ROOT
814	#undef PGM_BTH_NAME
815	#undef PGM_BTH_NAME_RC_STR
816	#undef PGM_BTH_NAME_R0_STR
817	#undef PGM_GST_TYPE
818	#undef PGM_GST_NAME
819	#undef PGM_GST_NAME_RC_STR
820	#undef PGM_GST_NAME_R0_STR
821
822	/* Guest - 32-bit mode */
823	#define PGM_GST_TYPE PGM_TYPE_32BIT
824	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
825	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
826	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
827	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_32BIT(name)
828	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_32BIT_STR(name)
829	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_32BIT_STR(name)
830	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
831	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
832	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_FOR_32BIT
833	#include "PGMGstDefs.h"
834	#include "PGMBth.h"
835	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
836	#undef BTH_PGMPOOLKIND_PT_FOR_PT
837	#undef BTH_PGMPOOLKIND_ROOT
838	#undef PGM_BTH_NAME
839	#undef PGM_BTH_NAME_RC_STR
840	#undef PGM_BTH_NAME_R0_STR
841	#undef PGM_GST_TYPE
842	#undef PGM_GST_NAME
843	#undef PGM_GST_NAME_RC_STR
844	#undef PGM_GST_NAME_R0_STR
845
846	/* Guest - PAE mode */
847	#define PGM_GST_TYPE PGM_TYPE_PAE
848	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
849	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
850	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
851	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PAE(name)
852	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_PAE_STR(name)
853	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PAE_STR(name)
854	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
855	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
856	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT
857	#include "PGMBth.h"
858	#include "PGMGstDefs.h"
859	#include "PGMGst.h"
860	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
861	#undef BTH_PGMPOOLKIND_PT_FOR_PT
862	#undef BTH_PGMPOOLKIND_ROOT
863	#undef PGM_BTH_NAME
864	#undef PGM_BTH_NAME_RC_STR
865	#undef PGM_BTH_NAME_R0_STR
866	#undef PGM_GST_TYPE
867	#undef PGM_GST_NAME
868	#undef PGM_GST_NAME_RC_STR
869	#undef PGM_GST_NAME_R0_STR
870
871	#undef PGM_SHW_TYPE
872	#undef PGM_SHW_NAME
873	#undef PGM_SHW_NAME_RC_STR
874	#undef PGM_SHW_NAME_R0_STR
875
876
877	/*
878	* Shadow - AMD64 mode
879	*/
880	#define PGM_SHW_TYPE PGM_TYPE_AMD64
881	#define PGM_SHW_NAME(name) PGM_SHW_NAME_AMD64(name)
882	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_AMD64_STR(name)
883	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_AMD64_STR(name)
884	#include "PGMShw.h"
885
886	#ifdef VBOX_WITH_64_BITS_GUESTS
887	/* Guest - AMD64 mode */
888	# define PGM_GST_TYPE PGM_TYPE_AMD64
889	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
890	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
891	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
892	# define PGM_BTH_NAME(name) PGM_BTH_NAME_AMD64_AMD64(name)
893	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_AMD64_AMD64_STR(name)
894	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_AMD64_AMD64_STR(name)
895	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
896	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
897	# define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_64BIT_PML4
898	# include "PGMBth.h"
899	# include "PGMGstDefs.h"
900	# include "PGMGst.h"
901	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
902	# undef BTH_PGMPOOLKIND_PT_FOR_PT
903	# undef BTH_PGMPOOLKIND_ROOT
904	# undef PGM_BTH_NAME
905	# undef PGM_BTH_NAME_RC_STR
906	# undef PGM_BTH_NAME_R0_STR
907	# undef PGM_GST_TYPE
908	# undef PGM_GST_NAME
909	# undef PGM_GST_NAME_RC_STR
910	# undef PGM_GST_NAME_R0_STR
911	#endif /* VBOX_WITH_64_BITS_GUESTS */
912
913	#undef PGM_SHW_TYPE
914	#undef PGM_SHW_NAME
915	#undef PGM_SHW_NAME_RC_STR
916	#undef PGM_SHW_NAME_R0_STR
917
918
919	/*
920	* Shadow - Nested paging mode
921	*/
922	#define PGM_SHW_TYPE PGM_TYPE_NESTED
923	#define PGM_SHW_NAME(name) PGM_SHW_NAME_NESTED(name)
924	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_NESTED_STR(name)
925	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_NESTED_STR(name)
926	#include "PGMShw.h"
927
928	/* Guest - real mode */
929	#define PGM_GST_TYPE PGM_TYPE_REAL
930	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
931	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
932	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
933	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_REAL(name)
934	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_REAL_STR(name)
935	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_REAL_STR(name)
936	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
937	#include "PGMGstDefs.h"
938	#include "PGMBth.h"
939	#undef BTH_PGMPOOLKIND_PT_FOR_PT
940	#undef PGM_BTH_NAME
941	#undef PGM_BTH_NAME_RC_STR
942	#undef PGM_BTH_NAME_R0_STR
943	#undef PGM_GST_TYPE
944	#undef PGM_GST_NAME
945	#undef PGM_GST_NAME_RC_STR
946	#undef PGM_GST_NAME_R0_STR
947
948	/* Guest - protected mode */
949	#define PGM_GST_TYPE PGM_TYPE_PROT
950	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
951	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
952	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
953	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PROT(name)
954	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_PROT_STR(name)
955	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PROT_STR(name)
956	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
957	#include "PGMGstDefs.h"
958	#include "PGMBth.h"
959	#undef BTH_PGMPOOLKIND_PT_FOR_PT
960	#undef PGM_BTH_NAME
961	#undef PGM_BTH_NAME_RC_STR
962	#undef PGM_BTH_NAME_R0_STR
963	#undef PGM_GST_TYPE
964	#undef PGM_GST_NAME
965	#undef PGM_GST_NAME_RC_STR
966	#undef PGM_GST_NAME_R0_STR
967
968	/* Guest - 32-bit mode */
969	#define PGM_GST_TYPE PGM_TYPE_32BIT
970	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
971	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
972	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
973	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_32BIT(name)
974	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_32BIT_STR(name)
975	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_32BIT_STR(name)
976	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
977	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
978	#include "PGMGstDefs.h"
979	#include "PGMBth.h"
980	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
981	#undef BTH_PGMPOOLKIND_PT_FOR_PT
982	#undef PGM_BTH_NAME
983	#undef PGM_BTH_NAME_RC_STR
984	#undef PGM_BTH_NAME_R0_STR
985	#undef PGM_GST_TYPE
986	#undef PGM_GST_NAME
987	#undef PGM_GST_NAME_RC_STR
988	#undef PGM_GST_NAME_R0_STR
989
990	/* Guest - PAE mode */
991	#define PGM_GST_TYPE PGM_TYPE_PAE
992	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
993	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
994	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
995	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PAE(name)
996	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_PAE_STR(name)
997	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PAE_STR(name)
998	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
999	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1000	#include "PGMGstDefs.h"
1001	#include "PGMBth.h"
1002	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1003	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1004	#undef PGM_BTH_NAME
1005	#undef PGM_BTH_NAME_RC_STR
1006	#undef PGM_BTH_NAME_R0_STR
1007	#undef PGM_GST_TYPE
1008	#undef PGM_GST_NAME
1009	#undef PGM_GST_NAME_RC_STR
1010	#undef PGM_GST_NAME_R0_STR
1011
1012	#ifdef VBOX_WITH_64_BITS_GUESTS
1013	/* Guest - AMD64 mode */
1014	# define PGM_GST_TYPE PGM_TYPE_AMD64
1015	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
1016	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
1017	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
1018	# define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_AMD64(name)
1019	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_AMD64_STR(name)
1020	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_AMD64_STR(name)
1021	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1022	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1023	# include "PGMGstDefs.h"
1024	# include "PGMBth.h"
1025	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
1026	# undef BTH_PGMPOOLKIND_PT_FOR_PT
1027	# undef PGM_BTH_NAME
1028	# undef PGM_BTH_NAME_RC_STR
1029	# undef PGM_BTH_NAME_R0_STR
1030	# undef PGM_GST_TYPE
1031	# undef PGM_GST_NAME
1032	# undef PGM_GST_NAME_RC_STR
1033	# undef PGM_GST_NAME_R0_STR
1034	#endif /* VBOX_WITH_64_BITS_GUESTS */
1035
1036	#undef PGM_SHW_TYPE
1037	#undef PGM_SHW_NAME
1038	#undef PGM_SHW_NAME_RC_STR
1039	#undef PGM_SHW_NAME_R0_STR
1040
1041
1042	/*
1043	* Shadow - EPT
1044	*/
1045	#define PGM_SHW_TYPE PGM_TYPE_EPT
1046	#define PGM_SHW_NAME(name) PGM_SHW_NAME_EPT(name)
1047	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_EPT_STR(name)
1048	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_EPT_STR(name)
1049	#include "PGMShw.h"
1050
1051	/* Guest - real mode */
1052	#define PGM_GST_TYPE PGM_TYPE_REAL
1053	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
1054	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
1055	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
1056	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_REAL(name)
1057	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_REAL_STR(name)
1058	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_REAL_STR(name)
1059	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
1060	#include "PGMGstDefs.h"
1061	#include "PGMBth.h"
1062	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1063	#undef PGM_BTH_NAME
1064	#undef PGM_BTH_NAME_RC_STR
1065	#undef PGM_BTH_NAME_R0_STR
1066	#undef PGM_GST_TYPE
1067	#undef PGM_GST_NAME
1068	#undef PGM_GST_NAME_RC_STR
1069	#undef PGM_GST_NAME_R0_STR
1070
1071	/* Guest - protected mode */
1072	#define PGM_GST_TYPE PGM_TYPE_PROT
1073	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
1074	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
1075	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
1076	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_PROT(name)
1077	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_PROT_STR(name)
1078	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_PROT_STR(name)
1079	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
1080	#include "PGMGstDefs.h"
1081	#include "PGMBth.h"
1082	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1083	#undef PGM_BTH_NAME
1084	#undef PGM_BTH_NAME_RC_STR
1085	#undef PGM_BTH_NAME_R0_STR
1086	#undef PGM_GST_TYPE
1087	#undef PGM_GST_NAME
1088	#undef PGM_GST_NAME_RC_STR
1089	#undef PGM_GST_NAME_R0_STR
1090
1091	/* Guest - 32-bit mode */
1092	#define PGM_GST_TYPE PGM_TYPE_32BIT
1093	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
1094	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
1095	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
1096	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_32BIT(name)
1097	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_32BIT_STR(name)
1098	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_32BIT_STR(name)
1099	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
1100	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
1101	#include "PGMGstDefs.h"
1102	#include "PGMBth.h"
1103	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1104	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1105	#undef PGM_BTH_NAME
1106	#undef PGM_BTH_NAME_RC_STR
1107	#undef PGM_BTH_NAME_R0_STR
1108	#undef PGM_GST_TYPE
1109	#undef PGM_GST_NAME
1110	#undef PGM_GST_NAME_RC_STR
1111	#undef PGM_GST_NAME_R0_STR
1112
1113	/* Guest - PAE mode */
1114	#define PGM_GST_TYPE PGM_TYPE_PAE
1115	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
1116	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
1117	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
1118	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_PAE(name)
1119	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_PAE_STR(name)
1120	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_PAE_STR(name)
1121	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1122	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1123	#include "PGMGstDefs.h"
1124	#include "PGMBth.h"
1125	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1126	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1127	#undef PGM_BTH_NAME
1128	#undef PGM_BTH_NAME_RC_STR
1129	#undef PGM_BTH_NAME_R0_STR
1130	#undef PGM_GST_TYPE
1131	#undef PGM_GST_NAME
1132	#undef PGM_GST_NAME_RC_STR
1133	#undef PGM_GST_NAME_R0_STR
1134
1135	#ifdef VBOX_WITH_64_BITS_GUESTS
1136	/* Guest - AMD64 mode */
1137	# define PGM_GST_TYPE PGM_TYPE_AMD64
1138	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
1139	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
1140	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
1141	# define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_AMD64(name)
1142	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_AMD64_STR(name)
1143	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_AMD64_STR(name)
1144	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1145	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1146	# include "PGMGstDefs.h"
1147	# include "PGMBth.h"
1148	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
1149	# undef BTH_PGMPOOLKIND_PT_FOR_PT
1150	# undef PGM_BTH_NAME
1151	# undef PGM_BTH_NAME_RC_STR
1152	# undef PGM_BTH_NAME_R0_STR
1153	# undef PGM_GST_TYPE
1154	# undef PGM_GST_NAME
1155	# undef PGM_GST_NAME_RC_STR
1156	# undef PGM_GST_NAME_R0_STR
1157	#endif /* VBOX_WITH_64_BITS_GUESTS */
1158
1159	#undef PGM_SHW_TYPE
1160	#undef PGM_SHW_NAME
1161	#undef PGM_SHW_NAME_RC_STR
1162	#undef PGM_SHW_NAME_R0_STR
1163
1164
1165
1166	/**
1167	* Initiates the paging of VM.
1168	*
1169	* @returns VBox status code.
1170	* @param pVM Pointer to VM structure.
1171	*/
1172	VMMR3DECL(int) PGMR3Init(PVM pVM)
1173	{
1174	LogFlow(("PGMR3Init:\n"));
1175	PCFGMNODE pCfgPGM = CFGMR3GetChild(CFGMR3GetRoot(pVM), "/PGM");
1176	int rc;
1177
1178	/*
1179	* Assert alignment and sizes.
1180	*/
1181	AssertCompile(sizeof(pVM->pgm.s) <= sizeof(pVM->pgm.padding));
1182
1183	/*
1184	* Init the structure.
1185	*/
1186	pVM->pgm.s.offVM = RT_OFFSETOF(VM, pgm.s);
1187	pVM->pgm.s.offVCpu = RT_OFFSETOF(VMCPU, pgm.s);
1188
1189	/* Init the per-CPU part. */
1190	for (unsigned i=0;i<pVM->cCPUs;i++)
1191	{
1192	PVMCPU pVCpu = &pVM->aCpus[i];
1193	PPGMCPU pPGM = &pVCpu->pgm.s;
1194
1195	pPGM->offVM = (uintptr_t)&pVCpu->pgm.s - (uintptr_t)pVM;
1196	pPGM->offVCpu = RT_OFFSETOF(VMCPU, pgm.s);
1197	pPGM->offPGM = (uintptr_t)&pVCpu->pgm.s - (uintptr_t)&pVM->pgm.s;
1198
1199	pPGM->enmShadowMode = PGMMODE_INVALID;
1200	pPGM->enmGuestMode = PGMMODE_INVALID;
1201
1202	pPGM->GCPhysCR3 = NIL_RTGCPHYS;
1203
1204	pPGM->pGstPaePdptR3 = NULL;
1205	#ifndef VBOX_WITH_2X_4GB_ADDR_SPACE
1206	pPGM->pGstPaePdptR0 = NIL_RTR0PTR;
1207	#endif
1208	pPGM->pGstPaePdptRC = NIL_RTRCPTR;
1209	for (unsigned i = 0; i < RT_ELEMENTS(pVCpu->pgm.s.apGstPaePDsR3); i++)
1210	{
1211	pPGM->apGstPaePDsR3[i] = NULL;
1212	#ifndef VBOX_WITH_2X_4GB_ADDR_SPACE
1213	pPGM->apGstPaePDsR0[i] = NIL_RTR0PTR;
1214	#endif
1215	pPGM->apGstPaePDsRC[i] = NIL_RTRCPTR;
1216	pPGM->aGCPhysGstPaePDs[i] = NIL_RTGCPHYS;
1217	pPGM->aGCPhysGstPaePDsMonitored[i] = NIL_RTGCPHYS;
1218	}
1219
1220	pPGM->fA20Enabled = true;
1221	}
1222
1223	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1224	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(32) - 1; /* default; checked later */
1225	pVM->pgm.s.GCPtrPrevRamRangeMapping = MM_HYPER_AREA_ADDRESS;
1226
1227	rc = CFGMR3QueryBoolDef(CFGMR3GetRoot(pVM), "RamPreAlloc", &pVM->pgm.s.fRamPreAlloc,
1228	#ifdef VBOX_WITH_PREALLOC_RAM_BY_DEFAULT
1229	true
1230	#else
1231	false
1232	#endif
1233	);
1234	AssertLogRelRCReturn(rc, rc);
1235
1236	#if HC_ARCH_BITS == 64 \|\| 1 /** @todo 4GB/32-bit: remove \|\| 1 later and adjust the limit. */
1237	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, UINT32_MAX);
1238	#else
1239	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, _1G / GMM_CHUNK_SIZE);
1240	#endif
1241	AssertLogRelRCReturn(rc, rc);
1242	for (uint32_t i = 0; i < RT_ELEMENTS(pVM->pgm.s.ChunkR3Map.Tlb.aEntries); i++)
1243	pVM->pgm.s.ChunkR3Map.Tlb.aEntries[i].idChunk = NIL_GMM_CHUNKID;
1244
1245	/*
1246	* Get the configured RAM size - to estimate saved state size.
1247	*/
1248	uint64_t cbRam;
1249	rc = CFGMR3QueryU64(CFGMR3GetRoot(pVM), "RamSize", &cbRam);
1250	if (rc == VERR_CFGM_VALUE_NOT_FOUND)
1251	cbRam = 0;
1252	else if (RT_SUCCESS(rc))
1253	{
1254	if (cbRam < PAGE_SIZE)
1255	cbRam = 0;
1256	cbRam = RT_ALIGN_64(cbRam, PAGE_SIZE);
1257	}
1258	else
1259	{
1260	AssertMsgFailed(("Configuration error: Failed to query integer \"RamSize\", rc=%Rrc.\n", rc));
1261	return rc;
1262	}
1263
1264	/*
1265	* Register callbacks, string formatters and the saved state data unit.
1266	*/
1267	#ifdef VBOX_STRICT
1268	VMR3AtStateRegister(pVM, pgmR3ResetNoMorePhysWritesFlag, NULL);
1269	#endif
1270	PGMRegisterStringFormatTypes();
1271
1272	rc = SSMR3RegisterInternal(pVM, "pgm", 1, PGM_SAVED_STATE_VERSION, (size_t)cbRam + sizeof(PGM),
1273	NULL, pgmR3Save, NULL,
1274	NULL, pgmR3Load, NULL);
1275	if (RT_FAILURE(rc))
1276	return rc;
1277
1278	/*
1279	* Initialize the PGM critical section and flush the phys TLBs
1280	*/
1281	rc = PDMR3CritSectInit(pVM, &pVM->pgm.s.CritSect, "PGM");
1282	AssertRCReturn(rc, rc);
1283
1284	PGMR3PhysChunkInvalidateTLB(pVM);
1285	PGMPhysInvalidatePageR3MapTLB(pVM);
1286	PGMPhysInvalidatePageR0MapTLB(pVM);
1287	PGMPhysInvalidatePageGCMapTLB(pVM);
1288
1289	/*
1290	* For the time being we sport a full set of handy pages in addition to the base
1291	* memory to simplify things.
1292	*/
1293	rc = MMR3ReserveHandyPages(pVM, RT_ELEMENTS(pVM->pgm.s.aHandyPages)); /** @todo this should be changed to PGM_HANDY_PAGES_MIN but this needs proper testing... */
1294	AssertRCReturn(rc, rc);
1295
1296	/*
1297	* Trees
1298	*/
1299	rc = MMHyperAlloc(pVM, sizeof(PGMTREES), 0, MM_TAG_PGM, (void **)&pVM->pgm.s.pTreesR3);
1300	if (RT_SUCCESS(rc))
1301	{
1302	pVM->pgm.s.pTreesR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pTreesR3);
1303	pVM->pgm.s.pTreesRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pTreesR3);
1304
1305	/*
1306	* Alocate the zero page.
1307	*/
1308	rc = MMHyperAlloc(pVM, PAGE_SIZE, PAGE_SIZE, MM_TAG_PGM, &pVM->pgm.s.pvZeroPgR3);
1309	}
1310	if (RT_SUCCESS(rc))
1311	{
1312	pVM->pgm.s.pvZeroPgRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pvZeroPgR3);
1313	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
1314	pVM->pgm.s.HCPhysZeroPg = MMR3HyperHCVirt2HCPhys(pVM, pVM->pgm.s.pvZeroPgR3);
1315	AssertRelease(pVM->pgm.s.HCPhysZeroPg != NIL_RTHCPHYS);
1316
1317	/*
1318	* Init the paging.
1319	*/
1320	rc = pgmR3InitPaging(pVM);
1321	}
1322	if (RT_SUCCESS(rc))
1323	{
1324	/*
1325	* Init the page pool.
1326	*/
1327	rc = pgmR3PoolInit(pVM);
1328	}
1329	if (RT_SUCCESS(rc))
1330	{
1331	for (unsigned i=0;i<pVM->cCPUs;i++)
1332	{
1333	PVMCPU pVCpu = &pVM->aCpus[i];
1334
1335	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
1336	if (RT_FAILURE(rc))
1337	break;
1338	}
1339	}
1340
1341	if (RT_SUCCESS(rc))
1342	{
1343	/*
1344	* Info & statistics
1345	*/
1346	DBGFR3InfoRegisterInternal(pVM, "mode",
1347	"Shows the current paging mode. "
1348	"Recognizes 'all', 'guest', 'shadow' and 'host' as arguments, defaulting to 'all' if nothing's given.",
1349	pgmR3InfoMode);
1350	DBGFR3InfoRegisterInternal(pVM, "pgmcr3",
1351	"Dumps all the entries in the top level paging table. No arguments.",
1352	pgmR3InfoCr3);
1353	DBGFR3InfoRegisterInternal(pVM, "phys",
1354	"Dumps all the physical address ranges. No arguments.",
1355	pgmR3PhysInfo);
1356	DBGFR3InfoRegisterInternal(pVM, "handlers",
1357	"Dumps physical, virtual and hyper virtual handlers. "
1358	"Pass 'phys', 'virt', 'hyper' as argument if only one kind is wanted."
1359	"Add 'nost' if the statistics are unwanted, use together with 'all' or explicit selection.",
1360	pgmR3InfoHandlers);
1361	DBGFR3InfoRegisterInternal(pVM, "mappings",
1362	"Dumps guest mappings.",
1363	pgmR3MapInfo);
1364
1365	pgmR3InitStats(pVM);
1366
1367	#ifdef VBOX_WITH_DEBUGGER
1368	/*
1369	* Debugger commands.
1370	*/
1371	static bool s_fRegisteredCmds = false;
1372	if (!s_fRegisteredCmds)
1373	{
1374	int rc = DBGCRegisterCommands(&g_aCmds[0], RT_ELEMENTS(g_aCmds));
1375	if (RT_SUCCESS(rc))
1376	s_fRegisteredCmds = true;
1377	}
1378	#endif
1379	return VINF_SUCCESS;
1380	}
1381
1382	/* Almost no cleanup necessary, MM frees all memory. */
1383	PDMR3CritSectDelete(&pVM->pgm.s.CritSect);
1384
1385	return rc;
1386	}
1387
1388
1389	/**
1390	* Initializes the per-VCPU PGM.
1391	*
1392	* @returns VBox status code.
1393	* @param pVM The VM to operate on.
1394	*/
1395	VMMR3DECL(int) PGMR3InitCPU(PVM pVM)
1396	{
1397	LogFlow(("PGMR3InitCPU\n"));
1398	return VINF_SUCCESS;
1399	}
1400
1401
1402	/**
1403	* Init paging.
1404	*
1405	* Since we need to check what mode the host is operating in before we can choose
1406	* the right paging functions for the host we have to delay this until R0 has
1407	* been initialized.
1408	*
1409	* @returns VBox status code.
1410	* @param pVM VM handle.
1411	*/
1412	static int pgmR3InitPaging(PVM pVM)
1413	{
1414	/*
1415	* Force a recalculation of modes and switcher so everyone gets notified.
1416	*/
1417	for (unsigned i=0;i<pVM->cCPUs;i++)
1418	{
1419	PVMCPU pVCpu = &pVM->aCpus[i];
1420
1421	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
1422	pVCpu->pgm.s.enmGuestMode = PGMMODE_INVALID;
1423	}
1424
1425	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1426
1427	/*
1428	* Allocate static mapping space for whatever the cr3 register
1429	* points to and in the case of PAE mode to the 4 PDs.
1430	*/
1431	int rc = MMR3HyperReserve(pVM, PAGE_SIZE * 5, "CR3 mapping", &pVM->pgm.s.GCPtrCR3Mapping);
1432	if (RT_FAILURE(rc))
1433	{
1434	AssertMsgFailed(("Failed to reserve two pages for cr mapping in HMA, rc=%Rrc\n", rc));
1435	return rc;
1436	}
1437	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
1438
1439	/*
1440	* Allocate pages for the three possible intermediate contexts
1441	* (AMD64, PAE and plain 32-Bit). We maintain all three contexts
1442	* for the sake of simplicity. The AMD64 uses the PAE for the
1443	* lower levels, making the total number of pages 11 (3 + 7 + 1).
1444	*
1445	* We assume that two page tables will be enought for the core code
1446	* mappings (HC virtual and identity).
1447	*/
1448	pVM->pgm.s.pInterPD = (PX86PD)MMR3PageAllocLow(pVM);
1449	pVM->pgm.s.apInterPTs[0] = (PX86PT)MMR3PageAllocLow(pVM);
1450	pVM->pgm.s.apInterPTs[1] = (PX86PT)MMR3PageAllocLow(pVM);
1451	pVM->pgm.s.apInterPaePTs[0] = (PX86PTPAE)MMR3PageAlloc(pVM);
1452	pVM->pgm.s.apInterPaePTs[1] = (PX86PTPAE)MMR3PageAlloc(pVM);
1453	pVM->pgm.s.apInterPaePDs[0] = (PX86PDPAE)MMR3PageAlloc(pVM);
1454	pVM->pgm.s.apInterPaePDs[1] = (PX86PDPAE)MMR3PageAlloc(pVM);
1455	pVM->pgm.s.apInterPaePDs[2] = (PX86PDPAE)MMR3PageAlloc(pVM);
1456	pVM->pgm.s.apInterPaePDs[3] = (PX86PDPAE)MMR3PageAlloc(pVM);
1457	pVM->pgm.s.pInterPaePDPT = (PX86PDPT)MMR3PageAllocLow(pVM);
1458	pVM->pgm.s.pInterPaePDPT64 = (PX86PDPT)MMR3PageAllocLow(pVM);
1459	pVM->pgm.s.pInterPaePML4 = (PX86PML4)MMR3PageAllocLow(pVM);
1460	if ( !pVM->pgm.s.pInterPD
1461	\|\| !pVM->pgm.s.apInterPTs[0]
1462	\|\| !pVM->pgm.s.apInterPTs[1]
1463	\|\| !pVM->pgm.s.apInterPaePTs[0]
1464	\|\| !pVM->pgm.s.apInterPaePTs[1]
1465	\|\| !pVM->pgm.s.apInterPaePDs[0]
1466	\|\| !pVM->pgm.s.apInterPaePDs[1]
1467	\|\| !pVM->pgm.s.apInterPaePDs[2]
1468	\|\| !pVM->pgm.s.apInterPaePDs[3]
1469	\|\| !pVM->pgm.s.pInterPaePDPT
1470	\|\| !pVM->pgm.s.pInterPaePDPT64
1471	\|\| !pVM->pgm.s.pInterPaePML4)
1472	{
1473	AssertMsgFailed(("Failed to allocate pages for the intermediate context!\n"));
1474	return VERR_NO_PAGE_MEMORY;
1475	}
1476
1477	pVM->pgm.s.HCPhysInterPD = MMPage2Phys(pVM, pVM->pgm.s.pInterPD);
1478	AssertRelease(pVM->pgm.s.HCPhysInterPD != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPD & PAGE_OFFSET_MASK));
1479	pVM->pgm.s.HCPhysInterPaePDPT = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT);
1480	AssertRelease(pVM->pgm.s.HCPhysInterPaePDPT != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePDPT & PAGE_OFFSET_MASK));
1481	pVM->pgm.s.HCPhysInterPaePML4 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePML4);
1482	AssertRelease(pVM->pgm.s.HCPhysInterPaePML4 != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePML4 & PAGE_OFFSET_MASK) && pVM->pgm.s.HCPhysInterPaePML4 < 0xffffffff);
1483
1484	/*
1485	* Initialize the pages, setting up the PML4 and PDPT for repetitive 4GB action.
1486	*/
1487	ASMMemZeroPage(pVM->pgm.s.pInterPD);
1488	ASMMemZeroPage(pVM->pgm.s.apInterPTs[0]);
1489	ASMMemZeroPage(pVM->pgm.s.apInterPTs[1]);
1490
1491	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[0]);
1492	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[1]);
1493
1494	ASMMemZeroPage(pVM->pgm.s.pInterPaePDPT);
1495	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.apInterPaePDs); i++)
1496	{
1497	ASMMemZeroPage(pVM->pgm.s.apInterPaePDs[i]);
1498	pVM->pgm.s.pInterPaePDPT->a[i].u = X86_PDPE_P \| PGM_PLXFLAGS_PERMANENT
1499	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[i]);
1500	}
1501
1502	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.pInterPaePDPT64->a); i++)
1503	{
1504	const unsigned iPD = i % RT_ELEMENTS(pVM->pgm.s.apInterPaePDs);
1505	pVM->pgm.s.pInterPaePDPT64->a[i].u = X86_PDPE_P \| X86_PDPE_RW \| X86_PDPE_US \| X86_PDPE_A \| PGM_PLXFLAGS_PERMANENT
1506	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[iPD]);
1507	}
1508
1509	RTHCPHYS HCPhysInterPaePDPT64 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64);
1510	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.pInterPaePML4->a); i++)
1511	pVM->pgm.s.pInterPaePML4->a[i].u = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_US \| X86_PML4E_A \| PGM_PLXFLAGS_PERMANENT
1512	\| HCPhysInterPaePDPT64;
1513
1514	/*
1515	* Initialize paging workers and mode from current host mode
1516	* and the guest running in real mode.
1517	*/
1518	pVM->pgm.s.enmHostMode = SUPGetPagingMode();
1519	switch (pVM->pgm.s.enmHostMode)
1520	{
1521	case SUPPAGINGMODE_32_BIT:
1522	case SUPPAGINGMODE_32_BIT_GLOBAL:
1523	case SUPPAGINGMODE_PAE:
1524	case SUPPAGINGMODE_PAE_GLOBAL:
1525	case SUPPAGINGMODE_PAE_NX:
1526	case SUPPAGINGMODE_PAE_GLOBAL_NX:
1527	break;
1528
1529	case SUPPAGINGMODE_AMD64:
1530	case SUPPAGINGMODE_AMD64_GLOBAL:
1531	case SUPPAGINGMODE_AMD64_NX:
1532	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
1533	#ifndef VBOX_WITH_HYBRID_32BIT_KERNEL
1534	if (ARCH_BITS != 64)
1535	{
1536	AssertMsgFailed(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1537	LogRel(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1538	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1539	}
1540	#endif
1541	break;
1542	default:
1543	AssertMsgFailed(("Host mode %d is not supported\n", pVM->pgm.s.enmHostMode));
1544	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1545	}
1546	rc = pgmR3ModeDataInit(pVM, false /* don't resolve GC and R0 syms yet */);
1547	if (RT_SUCCESS(rc))
1548	{
1549	LogFlow(("pgmR3InitPaging: returns successfully\n"));
1550	#if HC_ARCH_BITS == 64
1551	LogRel(("Debug: HCPhysInterPD=%RHp HCPhysInterPaePDPT=%RHp HCPhysInterPaePML4=%RHp\n",
1552	pVM->pgm.s.HCPhysInterPD, pVM->pgm.s.HCPhysInterPaePDPT, pVM->pgm.s.HCPhysInterPaePML4));
1553	LogRel(("Debug: apInterPTs={%RHp,%RHp} apInterPaePTs={%RHp,%RHp} apInterPaePDs={%RHp,%RHp,%RHp,%RHp} pInterPaePDPT64=%RHp\n",
1554	MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[1]),
1555	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[1]),
1556	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[1]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[2]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[3]),
1557	MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64)));
1558	#endif
1559
1560	return VINF_SUCCESS;
1561	}
1562
1563	LogFlow(("pgmR3InitPaging: returns %Rrc\n", rc));
1564	return rc;
1565	}
1566
1567
1568	/**
1569	* Init statistics
1570	*/
1571	static void pgmR3InitStats(PVM pVM)
1572	{
1573	PPGM pPGM = &pVM->pgm.s;
1574	int rc;
1575
1576	/* Common - misc variables */
1577	STAM_REL_REG(pVM, &pPGM->cAllPages, STAMTYPE_U32, "/PGM/Page/cAllPages", STAMUNIT_OCCURENCES, "The total number of pages.");
1578	STAM_REL_REG(pVM, &pPGM->cPrivatePages, STAMTYPE_U32, "/PGM/Page/cPrivatePages", STAMUNIT_OCCURENCES, "The number of private pages.");
1579	STAM_REL_REG(pVM, &pPGM->cSharedPages, STAMTYPE_U32, "/PGM/Page/cSharedPages", STAMUNIT_OCCURENCES, "The number of shared pages.");
1580	STAM_REL_REG(pVM, &pPGM->cZeroPages, STAMTYPE_U32, "/PGM/Page/cZeroPages", STAMUNIT_OCCURENCES, "The number of zero backed pages.");
1581	STAM_REL_REG(pVM, &pPGM->cHandyPages, STAMTYPE_U32, "/PGM/Page/cHandyPages", STAMUNIT_OCCURENCES, "The number of handy pages (not included in cAllPages).");
1582	STAM_REL_REG(pVM, &pPGM->cRelocations, STAMTYPE_COUNTER, "/PGM/cRelocations", STAMUNIT_OCCURENCES, "Number of hypervisor relocations.");
1583	STAM_REL_REG(pVM, &pPGM->ChunkR3Map.c, STAMTYPE_U32, "/PGM/ChunkR3Map/c", STAMUNIT_OCCURENCES, "Number of mapped chunks.");
1584	STAM_REL_REG(pVM, &pPGM->ChunkR3Map.cMax, STAMTYPE_U32, "/PGM/ChunkR3Map/cMax", STAMUNIT_OCCURENCES, "Maximum number of mapped chunks.");
1585
1586	#ifdef VBOX_WITH_STATISTICS
1587
1588	# define PGM_REG_COUNTER(a, b, c) \
1589	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_OCCURENCES, c, b); \
1590	AssertRC(rc);
1591
1592	# define PGM_REG_PROFILE(a, b, c) \
1593	rc = STAMR3RegisterF(pVM, a, STAMTYPE_PROFILE, STAMVISIBILITY_ALWAYS, STAMUNIT_TICKS_PER_CALL, c, b); \
1594	AssertRC(rc);
1595
1596	PGM_REG_COUNTER(&pPGM->StatR3DetectedConflicts, "/PGM/R3/DetectedConflicts", "The number of times PGMR3CheckMappingConflicts() detected a conflict.");
1597	PGM_REG_PROFILE(&pPGM->StatR3ResolveConflict, "/PGM/R3/ResolveConflict", "pgmR3SyncPTResolveConflict() profiling (includes the entire relocation).");
1598
1599	PGM_REG_COUNTER(&pPGM->StatRZChunkR3MapTlbHits, "/PGM/ChunkR3Map/TlbHitsRZ", "TLB hits.");
1600	PGM_REG_COUNTER(&pPGM->StatRZChunkR3MapTlbMisses, "/PGM/ChunkR3Map/TlbMissesRZ", "TLB misses.");
1601	PGM_REG_COUNTER(&pPGM->StatRZPageMapTlbHits, "/PGM/RZ/Page/MapTlbHits", "TLB hits.");
1602	PGM_REG_COUNTER(&pPGM->StatRZPageMapTlbMisses, "/PGM/RZ/Page/MapTlbMisses", "TLB misses.");
1603	PGM_REG_COUNTER(&pPGM->StatR3ChunkR3MapTlbHits, "/PGM/ChunkR3Map/TlbHitsR3", "TLB hits.");
1604	PGM_REG_COUNTER(&pPGM->StatR3ChunkR3MapTlbMisses, "/PGM/ChunkR3Map/TlbMissesR3", "TLB misses.");
1605	PGM_REG_COUNTER(&pPGM->StatR3PageMapTlbHits, "/PGM/R3/Page/MapTlbHits", "TLB hits.");
1606	PGM_REG_COUNTER(&pPGM->StatR3PageMapTlbMisses, "/PGM/R3/Page/MapTlbMisses", "TLB misses.");
1607
1608	PGM_REG_PROFILE(&pPGM->StatRZSyncCR3HandlerVirtualUpdate, "/PGM/RZ/SyncCR3/Handlers/VirtualUpdate", "Profiling of the virtual handler updates.");
1609	PGM_REG_PROFILE(&pPGM->StatRZSyncCR3HandlerVirtualReset, "/PGM/RZ/SyncCR3/Handlers/VirtualReset", "Profiling of the virtual handler resets.");
1610	PGM_REG_PROFILE(&pPGM->StatR3SyncCR3HandlerVirtualUpdate, "/PGM/R3/SyncCR3/Handlers/VirtualUpdate", "Profiling of the virtual handler updates.");
1611	PGM_REG_PROFILE(&pPGM->StatR3SyncCR3HandlerVirtualReset, "/PGM/R3/SyncCR3/Handlers/VirtualReset", "Profiling of the virtual handler resets.");
1612
1613	PGM_REG_COUNTER(&pPGM->StatRZPhysHandlerReset, "/PGM/RZ/PhysHandlerReset", "The number of times PGMHandlerPhysicalReset is called.");
1614	PGM_REG_COUNTER(&pPGM->StatR3PhysHandlerReset, "/PGM/R3/PhysHandlerReset", "The number of times PGMHandlerPhysicalReset is called.");
1615	PGM_REG_PROFILE(&pPGM->StatRZVirtHandlerSearchByPhys, "/PGM/RZ/VirtHandlerSearchByPhys", "Profiling of pgmHandlerVirtualFindByPhysAddr.");
1616	PGM_REG_PROFILE(&pPGM->StatR3VirtHandlerSearchByPhys, "/PGM/R3/VirtHandlerSearchByPhys", "Profiling of pgmHandlerVirtualFindByPhysAddr.");
1617
1618	PGM_REG_COUNTER(&pPGM->StatRZPageReplaceShared, "/PGM/RZ/Page/ReplacedShared", "Times a shared page was replaced.");
1619	PGM_REG_COUNTER(&pPGM->StatRZPageReplaceZero, "/PGM/RZ/Page/ReplacedZero", "Times the zero page was replaced.");
1620	/// @todo PGM_REG_COUNTER(&pPGM->StatRZPageHandyAllocs, "/PGM/RZ/Page/HandyAllocs", "Number of times we've allocated more handy pages.");
1621	PGM_REG_COUNTER(&pPGM->StatR3PageReplaceShared, "/PGM/R3/Page/ReplacedShared", "Times a shared page was replaced.");
1622	PGM_REG_COUNTER(&pPGM->StatR3PageReplaceZero, "/PGM/R3/Page/ReplacedZero", "Times the zero page was replaced.");
1623	/// @todo PGM_REG_COUNTER(&pPGM->StatR3PageHandyAllocs, "/PGM/R3/Page/HandyAllocs", "Number of times we've allocated more handy pages.");
1624
1625	/* GC only: */
1626	PGM_REG_COUNTER(&pPGM->StatRCDynMapCacheHits, "/PGM/RC/DynMapCache/Hits" , "Number of dynamic page mapping cache hits.");
1627	PGM_REG_COUNTER(&pPGM->StatRCDynMapCacheMisses, "/PGM/RC/DynMapCache/Misses" , "Number of dynamic page mapping cache misses.");
1628	PGM_REG_COUNTER(&pPGM->StatRCInvlPgConflict, "/PGM/RC/InvlPgConflict", "Number of times PGMInvalidatePage() detected a mapping conflict.");
1629	PGM_REG_COUNTER(&pPGM->StatRCInvlPgSyncMonCR3, "/PGM/RC/InvlPgSyncMonitorCR3", "Number of times PGMInvalidatePage() ran into PGM_SYNC_MONITOR_CR3.");
1630
1631	# ifdef PGMPOOL_WITH_GCPHYS_TRACKING
1632	PGM_REG_COUNTER(&pPGM->StatTrackVirgin, "/PGM/Track/Virgin", "The number of first time shadowings");
1633	PGM_REG_COUNTER(&pPGM->StatTrackAliased, "/PGM/Track/Aliased", "The number of times switching to cRef2, i.e. the page is being shadowed by two PTs.");
1634	PGM_REG_COUNTER(&pPGM->StatTrackAliasedMany, "/PGM/Track/AliasedMany", "The number of times we're tracking using cRef2.");
1635	PGM_REG_COUNTER(&pPGM->StatTrackAliasedLots, "/PGM/Track/AliasedLots", "The number of times we're hitting pages which has overflowed cRef2");
1636	PGM_REG_COUNTER(&pPGM->StatTrackOverflows, "/PGM/Track/Overflows", "The number of times the extent list grows too long.");
1637	PGM_REG_PROFILE(&pPGM->StatTrackDeref, "/PGM/Track/Deref", "Profiling of SyncPageWorkerTrackDeref (expensive).");
1638	# endif
1639
1640	# undef PGM_REG_COUNTER
1641	# undef PGM_REG_PROFILE
1642	#endif
1643
1644	/*
1645	* Note! The layout below matches the member layout exactly!
1646	*/
1647
1648	/*
1649	* Common - stats
1650	*/
1651	for (unsigned i=0;i<pVM->cCPUs;i++)
1652	{
1653	PVMCPU pVCpu = &pVM->aCpus[i];
1654	PPGMCPU pPGM = &pVCpu->pgm.s;
1655
1656	#define PGM_REG_COUNTER(a, b, c) \
1657	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_OCCURENCES, c, b, i); \
1658	AssertRC(rc);
1659	#define PGM_REG_PROFILE(a, b, c) \
1660	rc = STAMR3RegisterF(pVM, a, STAMTYPE_PROFILE, STAMVISIBILITY_ALWAYS, STAMUNIT_TICKS_PER_CALL, c, b, i); \
1661	AssertRC(rc);
1662
1663	PGM_REG_COUNTER(&pPGM->cGuestModeChanges, "/PGM/CPU%d/cGuestModeChanges", "Number of guest mode changes.");
1664
1665	#ifdef VBOX_WITH_STATISTICS
1666	for (unsigned j = 0; j < RT_ELEMENTS(pPGM->StatSyncPtPD); j++)
1667	STAMR3RegisterF(pVM, &pPGM->StatSyncPtPD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1668	"The number of SyncPT per PD n.", "/PGM/CPU%d/PDSyncPT/%04X", i, j);
1669	for (unsigned j = 0; j < RT_ELEMENTS(pPGM->StatSyncPagePD); j++)
1670	STAMR3RegisterF(pVM, &pPGM->StatSyncPagePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1671	"The number of SyncPage per PD n.", "/PGM/CPU%d/PDSyncPage/%04X", i, j);
1672
1673	/* R0 only: */
1674	PGM_REG_COUNTER(&pPGM->StatR0DynMapMigrateInvlPg, "/PGM/CPU%d/R0/DynMapMigrateInvlPg", "invlpg count in PGMDynMapMigrateAutoSet.");
1675	PGM_REG_PROFILE(&pPGM->StatR0DynMapGCPageInl, "/PGM/CPU%d/R0/DynMapPageGCPageInl", "Calls to pgmR0DynMapGCPageInlined.");
1676	PGM_REG_COUNTER(&pPGM->StatR0DynMapGCPageInlHits, "/PGM/CPU%d/R0/DynMapPageGCPageInl/Hits", "Hash table lookup hits.");
1677	PGM_REG_COUNTER(&pPGM->StatR0DynMapGCPageInlMisses, "/PGM/CPU%d/R0/DynMapPageGCPageInl/Misses", "Misses that falls back to code common with PGMDynMapHCPage.");
1678	PGM_REG_COUNTER(&pPGM->StatR0DynMapGCPageInlRamHits, "/PGM/CPU%d/R0/DynMapPageGCPageInl/RamHits", "1st ram range hits.");
1679	PGM_REG_COUNTER(&pPGM->StatR0DynMapGCPageInlRamMisses, "/PGM/CPU%d/R0/DynMapPageGCPageInl/RamMisses", "1st ram range misses, takes slow path.");
1680	PGM_REG_PROFILE(&pPGM->StatR0DynMapHCPageInl, "/PGM/CPU%d/R0/DynMapPageHCPageInl", "Calls to pgmR0DynMapHCPageInlined.");
1681	PGM_REG_COUNTER(&pPGM->StatR0DynMapHCPageInlHits, "/PGM/CPU%d/R0/DynMapPageHCPageInl/Hits", "Hash table lookup hits.");
1682	PGM_REG_COUNTER(&pPGM->StatR0DynMapHCPageInlMisses, "/PGM/CPU%d/R0/DynMapPageHCPageInl/Misses", "Misses that falls back to code common with PGMDynMapHCPage.");
1683	PGM_REG_COUNTER(&pPGM->StatR0DynMapPage, "/PGM/CPU%d/R0/DynMapPage", "Calls to pgmR0DynMapPage");
1684	PGM_REG_COUNTER(&pPGM->StatR0DynMapSetOptimize, "/PGM/CPU%d/R0/DynMapPage/SetOptimize", "Calls to pgmDynMapOptimizeAutoSet.");
1685	PGM_REG_COUNTER(&pPGM->StatR0DynMapSetSearchFlushes, "/PGM/CPU%d/R0/DynMapPage/SetSearchFlushes","Set search restorting to subset flushes.");
1686	PGM_REG_COUNTER(&pPGM->StatR0DynMapSetSearchHits, "/PGM/CPU%d/R0/DynMapPage/SetSearchHits", "Set search hits.");
1687	PGM_REG_COUNTER(&pPGM->StatR0DynMapSetSearchMisses, "/PGM/CPU%d/R0/DynMapPage/SetSearchMisses", "Set search misses.");
1688	PGM_REG_PROFILE(&pPGM->StatR0DynMapHCPage, "/PGM/CPU%d/R0/DynMapPage/HCPage", "Calls to PGMDynMapHCPage (ring-0).");
1689	PGM_REG_COUNTER(&pPGM->StatR0DynMapPageHits0, "/PGM/CPU%d/R0/DynMapPage/Hits0", "Hits at iPage+0");
1690	PGM_REG_COUNTER(&pPGM->StatR0DynMapPageHits1, "/PGM/CPU%d/R0/DynMapPage/Hits1", "Hits at iPage+1");
1691	PGM_REG_COUNTER(&pPGM->StatR0DynMapPageHits2, "/PGM/CPU%d/R0/DynMapPage/Hits2", "Hits at iPage+2");
1692	PGM_REG_COUNTER(&pPGM->StatR0DynMapPageInvlPg, "/PGM/CPU%d/R0/DynMapPage/InvlPg", "invlpg count in pgmR0DynMapPageSlow.");
1693	PGM_REG_COUNTER(&pPGM->StatR0DynMapPageSlow, "/PGM/CPU%d/R0/DynMapPage/Slow", "Calls to pgmR0DynMapPageSlow - subtract this from pgmR0DynMapPage to get 1st level hits.");
1694	PGM_REG_COUNTER(&pPGM->StatR0DynMapPageSlowLoopHits, "/PGM/CPU%d/R0/DynMapPage/SlowLoopHits" , "Hits in the loop path.");
1695	PGM_REG_COUNTER(&pPGM->StatR0DynMapPageSlowLoopMisses, "/PGM/CPU%d/R0/DynMapPage/SlowLoopMisses", "Misses in the loop path. NonLoopMisses = Slow - SlowLoopHit - SlowLoopMisses");
1696	//PGM_REG_COUNTER(&pPGM->StatR0DynMapPageSlowLostHits, "/PGM/CPU%d/R0/DynMapPage/SlowLostHits", "Lost hits.");
1697	PGM_REG_COUNTER(&pPGM->StatR0DynMapSubsets, "/PGM/CPU%d/R0/Subsets", "Times PGMDynMapPushAutoSubset was called.");
1698	PGM_REG_COUNTER(&pPGM->StatR0DynMapPopFlushes, "/PGM/CPU%d/R0/SubsetPopFlushes", "Times PGMDynMapPopAutoSubset flushes the subset.");
1699	PGM_REG_COUNTER(&pPGM->aStatR0DynMapSetSize[0], "/PGM/CPU%d/R0/SetSize000..09", "00-09% filled");
1700	PGM_REG_COUNTER(&pPGM->aStatR0DynMapSetSize[1], "/PGM/CPU%d/R0/SetSize010..19", "10-19% filled");
1701	PGM_REG_COUNTER(&pPGM->aStatR0DynMapSetSize[2], "/PGM/CPU%d/R0/SetSize020..29", "20-29% filled");
1702	PGM_REG_COUNTER(&pPGM->aStatR0DynMapSetSize[3], "/PGM/CPU%d/R0/SetSize030..39", "30-39% filled");
1703	PGM_REG_COUNTER(&pPGM->aStatR0DynMapSetSize[4], "/PGM/CPU%d/R0/SetSize040..49", "40-49% filled");
1704	PGM_REG_COUNTER(&pPGM->aStatR0DynMapSetSize[5], "/PGM/CPU%d/R0/SetSize050..59", "50-59% filled");
1705	PGM_REG_COUNTER(&pPGM->aStatR0DynMapSetSize[6], "/PGM/CPU%d/R0/SetSize060..69", "60-69% filled");
1706	PGM_REG_COUNTER(&pPGM->aStatR0DynMapSetSize[7], "/PGM/CPU%d/R0/SetSize070..79", "70-79% filled");
1707	PGM_REG_COUNTER(&pPGM->aStatR0DynMapSetSize[8], "/PGM/CPU%d/R0/SetSize080..89", "80-89% filled");
1708	PGM_REG_COUNTER(&pPGM->aStatR0DynMapSetSize[9], "/PGM/CPU%d/R0/SetSize090..99", "90-99% filled");
1709	PGM_REG_COUNTER(&pPGM->aStatR0DynMapSetSize[10], "/PGM/CPU%d/R0/SetSize100", "100% filled");
1710
1711	/* RZ only: */
1712	PGM_REG_PROFILE(&pPGM->StatRZTrap0e, "/PGM/CPU%d/RZ/Trap0e", "Profiling of the PGMTrap0eHandler() body.");
1713	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTimeCheckPageFault, "/PGM/CPU%d/RZ/Trap0e/Time/CheckPageFault", "Profiling of checking for dirty/access emulation faults.");
1714	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTimeSyncPT, "/PGM/CPU%d/RZ/Trap0e/Time/SyncPT", "Profiling of lazy page table syncing.");
1715	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTimeMapping, "/PGM/CPU%d/RZ/Trap0e/Time/Mapping", "Profiling of checking virtual mappings.");
1716	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTimeOutOfSync, "/PGM/CPU%d/RZ/Trap0e/Time/OutOfSync", "Profiling of out of sync page handling.");
1717	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTimeHandlers, "/PGM/CPU%d/RZ/Trap0e/Time/Handlers", "Profiling of checking handlers.");
1718	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTime2CSAM, "/PGM/CPU%d/RZ/Trap0e/Time2/CSAM", "Profiling of the Trap0eHandler body when the cause is CSAM.");
1719	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTime2DirtyAndAccessed, "/PGM/CPU%d/RZ/Trap0e/Time2/DirtyAndAccessedBits", "Profiling of the Trap0eHandler body when the cause is dirty and/or accessed bit emulation.");
1720	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTime2GuestTrap, "/PGM/CPU%d/RZ/Trap0e/Time2/GuestTrap", "Profiling of the Trap0eHandler body when the cause is a guest trap.");
1721	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTime2HndPhys, "/PGM/CPU%d/RZ/Trap0e/Time2/HandlerPhysical", "Profiling of the Trap0eHandler body when the cause is a physical handler.");
1722	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTime2HndVirt, "/PGM/CPU%d/RZ/Trap0e/Time2/HandlerVirtual", "Profiling of the Trap0eHandler body when the cause is a virtual handler.");
1723	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTime2HndUnhandled, "/PGM/CPU%d/RZ/Trap0e/Time2/HandlerUnhandled", "Profiling of the Trap0eHandler body when the cause is access outside the monitored areas of a monitored page.");
1724	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTime2Misc, "/PGM/CPU%d/RZ/Trap0e/Time2/Misc", "Profiling of the Trap0eHandler body when the cause is not known.");
1725	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTime2OutOfSync, "/PGM/CPU%d/RZ/Trap0e/Time2/OutOfSync", "Profiling of the Trap0eHandler body when the cause is an out-of-sync page.");
1726	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTime2OutOfSyncHndPhys, "/PGM/CPU%d/RZ/Trap0e/Time2/OutOfSyncHndPhys", "Profiling of the Trap0eHandler body when the cause is an out-of-sync physical handler page.");
1727	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTime2OutOfSyncHndVirt, "/PGM/CPU%d/RZ/Trap0e/Time2/OutOfSyncHndVirt", "Profiling of the Trap0eHandler body when the cause is an out-of-sync virtual handler page.");
1728	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTime2OutOfSyncHndObs, "/PGM/CPU%d/RZ/Trap0e/Time2/OutOfSyncObsHnd", "Profiling of the Trap0eHandler body when the cause is an obsolete handler page.");
1729	PGM_REG_PROFILE(&pPGM->StatRZTrap0eTime2SyncPT, "/PGM/CPU%d/RZ/Trap0e/Time2/SyncPT", "Profiling of the Trap0eHandler body when the cause is lazy syncing of a PT.");
1730	PGM_REG_COUNTER(&pPGM->StatRZTrap0eConflicts, "/PGM/CPU%d/RZ/Trap0e/Conflicts", "The number of times #PF was caused by an undetected conflict.");
1731	PGM_REG_COUNTER(&pPGM->StatRZTrap0eHandlersMapping, "/PGM/CPU%d/RZ/Trap0e/Handlers/Mapping", "Number of traps due to access handlers in mappings.");
1732	PGM_REG_COUNTER(&pPGM->StatRZTrap0eHandlersOutOfSync, "/PGM/CPU%d/RZ/Trap0e/Handlers/OutOfSync", "Number of traps due to out-of-sync handled pages.");
1733	PGM_REG_COUNTER(&pPGM->StatRZTrap0eHandlersPhysical, "/PGM/CPU%d/RZ/Trap0e/Handlers/Physical", "Number of traps due to physical access handlers.");
1734	PGM_REG_COUNTER(&pPGM->StatRZTrap0eHandlersVirtual, "/PGM/CPU%d/RZ/Trap0e/Handlers/Virtual", "Number of traps due to virtual access handlers.");
1735	PGM_REG_COUNTER(&pPGM->StatRZTrap0eHandlersVirtualByPhys, "/PGM/CPU%d/RZ/Trap0e/Handlers/VirtualByPhys", "Number of traps due to virtual access handlers by physical address.");
1736	PGM_REG_COUNTER(&pPGM->StatRZTrap0eHandlersVirtualUnmarked,"/PGM/CPU%d/RZ/Trap0e/Handlers/VirtualUnmarked","Number of traps due to virtual access handlers by virtual address (without proper physical flags).");
1737	PGM_REG_COUNTER(&pPGM->StatRZTrap0eHandlersUnhandled, "/PGM/CPU%d/RZ/Trap0e/Handlers/Unhandled", "Number of traps due to access outside range of monitored page(s).");
1738	PGM_REG_COUNTER(&pPGM->StatRZTrap0eHandlersInvalid, "/PGM/CPU%d/RZ/Trap0e/Handlers/Invalid", "Number of traps due to access to invalid physical memory.");
1739	PGM_REG_COUNTER(&pPGM->StatRZTrap0eUSNotPresentRead, "/PGM/CPU%d/RZ/Trap0e/Err/User/NPRead", "Number of user mode not present read page faults.");
1740	PGM_REG_COUNTER(&pPGM->StatRZTrap0eUSNotPresentWrite, "/PGM/CPU%d/RZ/Trap0e/Err/User/NPWrite", "Number of user mode not present write page faults.");
1741	PGM_REG_COUNTER(&pPGM->StatRZTrap0eUSWrite, "/PGM/CPU%d/RZ/Trap0e/Err/User/Write", "Number of user mode write page faults.");
1742	PGM_REG_COUNTER(&pPGM->StatRZTrap0eUSReserved, "/PGM/CPU%d/RZ/Trap0e/Err/User/Reserved", "Number of user mode reserved bit page faults.");
1743	PGM_REG_COUNTER(&pPGM->StatRZTrap0eUSNXE, "/PGM/CPU%d/RZ/Trap0e/Err/User/NXE", "Number of user mode NXE page faults.");
1744	PGM_REG_COUNTER(&pPGM->StatRZTrap0eUSRead, "/PGM/CPU%d/RZ/Trap0e/Err/User/Read", "Number of user mode read page faults.");
1745	PGM_REG_COUNTER(&pPGM->StatRZTrap0eSVNotPresentRead, "/PGM/CPU%d/RZ/Trap0e/Err/Supervisor/NPRead", "Number of supervisor mode not present read page faults.");
1746	PGM_REG_COUNTER(&pPGM->StatRZTrap0eSVNotPresentWrite, "/PGM/CPU%d/RZ/Trap0e/Err/Supervisor/NPWrite", "Number of supervisor mode not present write page faults.");
1747	PGM_REG_COUNTER(&pPGM->StatRZTrap0eSVWrite, "/PGM/CPU%d/RZ/Trap0e/Err/Supervisor/Write", "Number of supervisor mode write page faults.");
1748	PGM_REG_COUNTER(&pPGM->StatRZTrap0eSVReserved, "/PGM/CPU%d/RZ/Trap0e/Err/Supervisor/Reserved", "Number of supervisor mode reserved bit page faults.");
1749	PGM_REG_COUNTER(&pPGM->StatRZTrap0eSNXE, "/PGM/CPU%d/RZ/Trap0e/Err/Supervisor/NXE", "Number of supervisor mode NXE page faults.");
1750	PGM_REG_COUNTER(&pPGM->StatRZTrap0eGuestPF, "/PGM/CPU%d/RZ/Trap0e/GuestPF", "Number of real guest page faults.");
1751	PGM_REG_COUNTER(&pPGM->StatRZTrap0eGuestPFUnh, "/PGM/CPU%d/RZ/Trap0e/GuestPF/Unhandled", "Number of real guest page faults from the 'unhandled' case.");
1752	PGM_REG_COUNTER(&pPGM->StatRZTrap0eGuestPFMapping, "/PGM/CPU%d/RZ/Trap0e/GuestPF/InMapping", "Number of real guest page faults in a mapping.");
1753	PGM_REG_COUNTER(&pPGM->StatRZTrap0eWPEmulInRZ, "/PGM/CPU%d/RZ/Trap0e/WP/InRZ", "Number of guest page faults due to X86_CR0_WP emulation.");
1754	PGM_REG_COUNTER(&pPGM->StatRZTrap0eWPEmulToR3, "/PGM/CPU%d/RZ/Trap0e/WP/ToR3", "Number of guest page faults due to X86_CR0_WP emulation (forward to R3 for emulation).");
1755	for (unsigned j = 0; j < RT_ELEMENTS(pPGM->StatRZTrap0ePD); j++)
1756	STAMR3RegisterF(pVM, &pPGM->StatRZTrap0ePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1757	"The number of traps in page directory n.", "/PGM/CPU%d/RZ/Trap0e/PD/%04X", i, j);
1758
1759	PGM_REG_COUNTER(&pPGM->StatRZGuestCR3WriteHandled, "/PGM/CPU%d/RZ/CR3WriteHandled", "The number of times the Guest CR3 change was successfully handled.");
1760	PGM_REG_COUNTER(&pPGM->StatRZGuestCR3WriteUnhandled, "/PGM/CPU%d/RZ/CR3WriteUnhandled", "The number of times the Guest CR3 change was passed back to the recompiler.");
1761	PGM_REG_COUNTER(&pPGM->StatRZGuestCR3WriteConflict, "/PGM/CPU%d/RZ/CR3WriteConflict", "The number of times the Guest CR3 monitoring detected a conflict.");
1762	PGM_REG_COUNTER(&pPGM->StatRZGuestROMWriteHandled, "/PGM/CPU%d/RZ/ROMWriteHandled", "The number of times the Guest ROM change was successfully handled.");
1763	PGM_REG_COUNTER(&pPGM->StatRZGuestROMWriteUnhandled, "/PGM/CPU%d/RZ/ROMWriteUnhandled", "The number of times the Guest ROM change was passed back to the recompiler.");
1764
1765	/* HC only: */
1766
1767	/* RZ & R3: */
1768	PGM_REG_PROFILE(&pPGM->StatRZSyncCR3, "/PGM/CPU%d/RZ/SyncCR3", "Profiling of the PGMSyncCR3() body.");
1769	PGM_REG_PROFILE(&pPGM->StatRZSyncCR3Handlers, "/PGM/CPU%d/RZ/SyncCR3/Handlers", "Profiling of the PGMSyncCR3() update handler section.");
1770	PGM_REG_COUNTER(&pPGM->StatRZSyncCR3Global, "/PGM/CPU%d/RZ/SyncCR3/Global", "The number of global CR3 syncs.");
1771	PGM_REG_COUNTER(&pPGM->StatRZSyncCR3NotGlobal, "/PGM/CPU%d/RZ/SyncCR3/NotGlobal", "The number of non-global CR3 syncs.");
1772	PGM_REG_COUNTER(&pPGM->StatRZSyncCR3DstCacheHit, "/PGM/CPU%d/RZ/SyncCR3/DstChacheHit", "The number of times we got some kind of a cache hit.");
1773	PGM_REG_COUNTER(&pPGM->StatRZSyncCR3DstFreed, "/PGM/CPU%d/RZ/SyncCR3/DstFreed", "The number of times we've had to free a shadow entry.");
1774	PGM_REG_COUNTER(&pPGM->StatRZSyncCR3DstFreedSrcNP, "/PGM/CPU%d/RZ/SyncCR3/DstFreedSrcNP", "The number of times we've had to free a shadow entry for which the source entry was not present.");
1775	PGM_REG_COUNTER(&pPGM->StatRZSyncCR3DstNotPresent, "/PGM/CPU%d/RZ/SyncCR3/DstNotPresent", "The number of times we've encountered a not present shadow entry for a present guest entry.");
1776	PGM_REG_COUNTER(&pPGM->StatRZSyncCR3DstSkippedGlobalPD, "/PGM/CPU%d/RZ/SyncCR3/DstSkippedGlobalPD", "The number of times a global page directory wasn't flushed.");
1777	PGM_REG_COUNTER(&pPGM->StatRZSyncCR3DstSkippedGlobalPT, "/PGM/CPU%d/RZ/SyncCR3/DstSkippedGlobalPT", "The number of times a page table with only global entries wasn't flushed.");
1778	PGM_REG_PROFILE(&pPGM->StatRZSyncPT, "/PGM/CPU%d/RZ/SyncPT", "Profiling of the pfnSyncPT() body.");
1779	PGM_REG_COUNTER(&pPGM->StatRZSyncPTFailed, "/PGM/CPU%d/RZ/SyncPT/Failed", "The number of times pfnSyncPT() failed.");
1780	PGM_REG_COUNTER(&pPGM->StatRZSyncPT4K, "/PGM/CPU%d/RZ/SyncPT/4K", "Nr of 4K PT syncs");
1781	PGM_REG_COUNTER(&pPGM->StatRZSyncPT4M, "/PGM/CPU%d/RZ/SyncPT/4M", "Nr of 4M PT syncs");
1782	PGM_REG_COUNTER(&pPGM->StatRZSyncPagePDNAs, "/PGM/CPU%d/RZ/SyncPagePDNAs", "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1783	PGM_REG_COUNTER(&pPGM->StatRZSyncPagePDOutOfSync, "/PGM/CPU%d/RZ/SyncPagePDOutOfSync", "The number of time we've encountered an out-of-sync PD in SyncPage.");
1784	PGM_REG_COUNTER(&pPGM->StatRZAccessedPage, "/PGM/CPU%d/RZ/AccessedPage", "The number of pages marked not present for accessed bit emulation.");
1785	PGM_REG_PROFILE(&pPGM->StatRZDirtyBitTracking, "/PGM/CPU%d/RZ/DirtyPage", "Profiling the dirty bit tracking in CheckPageFault().");
1786	PGM_REG_COUNTER(&pPGM->StatRZDirtyPage, "/PGM/CPU%d/RZ/DirtyPage/Mark", "The number of pages marked read-only for dirty bit tracking.");
1787	PGM_REG_COUNTER(&pPGM->StatRZDirtyPageBig, "/PGM/CPU%d/RZ/DirtyPage/MarkBig", "The number of 4MB pages marked read-only for dirty bit tracking.");
1788	PGM_REG_COUNTER(&pPGM->StatRZDirtyPageSkipped, "/PGM/CPU%d/RZ/DirtyPage/Skipped", "The number of pages already dirty or readonly.");
1789	PGM_REG_COUNTER(&pPGM->StatRZDirtyPageTrap, "/PGM/CPU%d/RZ/DirtyPage/Trap", "The number of traps generated for dirty bit tracking.");
1790	PGM_REG_COUNTER(&pPGM->StatRZDirtiedPage, "/PGM/CPU%d/RZ/DirtyPage/SetDirty", "The number of pages marked dirty because of write accesses.");
1791	PGM_REG_COUNTER(&pPGM->StatRZDirtyTrackRealPF, "/PGM/CPU%d/RZ/DirtyPage/RealPF", "The number of real pages faults during dirty bit tracking.");
1792	PGM_REG_COUNTER(&pPGM->StatRZPageAlreadyDirty, "/PGM/CPU%d/RZ/DirtyPage/AlreadySet", "The number of pages already marked dirty because of write accesses.");
1793	PGM_REG_PROFILE(&pPGM->StatRZInvalidatePage, "/PGM/CPU%d/RZ/InvalidatePage", "PGMInvalidatePage() profiling.");
1794	PGM_REG_COUNTER(&pPGM->StatRZInvalidatePage4KBPages, "/PGM/CPU%d/RZ/InvalidatePage/4KBPages", "The number of times PGMInvalidatePage() was called for a 4KB page.");
1795	PGM_REG_COUNTER(&pPGM->StatRZInvalidatePage4MBPages, "/PGM/CPU%d/RZ/InvalidatePage/4MBPages", "The number of times PGMInvalidatePage() was called for a 4MB page.");
1796	PGM_REG_COUNTER(&pPGM->StatRZInvalidatePage4MBPagesSkip, "/PGM/CPU%d/RZ/InvalidatePage/4MBPagesSkip","The number of times PGMInvalidatePage() skipped a 4MB page.");
1797	PGM_REG_COUNTER(&pPGM->StatRZInvalidatePagePDMappings, "/PGM/CPU%d/RZ/InvalidatePage/PDMappings", "The number of times PGMInvalidatePage() was called for a page directory containing mappings (no conflict).");
1798	PGM_REG_COUNTER(&pPGM->StatRZInvalidatePagePDNAs, "/PGM/CPU%d/RZ/InvalidatePage/PDNAs", "The number of times PGMInvalidatePage() was called for a not accessed page directory.");
1799	PGM_REG_COUNTER(&pPGM->StatRZInvalidatePagePDNPs, "/PGM/CPU%d/RZ/InvalidatePage/PDNPs", "The number of times PGMInvalidatePage() was called for a not present page directory.");
1800	PGM_REG_COUNTER(&pPGM->StatRZInvalidatePagePDOutOfSync, "/PGM/CPU%d/RZ/InvalidatePage/PDOutOfSync", "The number of times PGMInvalidatePage() was called for an out of sync page directory.");
1801	PGM_REG_COUNTER(&pPGM->StatRZInvalidatePageSkipped, "/PGM/CPU%d/RZ/InvalidatePage/Skipped", "The number of times PGMInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
1802	PGM_REG_COUNTER(&pPGM->StatRZPageOutOfSyncSupervisor, "/PGM/CPU%d/RZ/OutOfSync/SuperVisor", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
1803	PGM_REG_COUNTER(&pPGM->StatRZPageOutOfSyncUser, "/PGM/CPU%d/RZ/OutOfSync/User", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
1804	PGM_REG_PROFILE(&pPGM->StatRZPrefetch, "/PGM/CPU%d/RZ/Prefetch", "PGMPrefetchPage profiling.");
1805	PGM_REG_PROFILE(&pPGM->StatRZFlushTLB, "/PGM/CPU%d/RZ/FlushTLB", "Profiling of the PGMFlushTLB() body.");
1806	PGM_REG_COUNTER(&pPGM->StatRZFlushTLBNewCR3, "/PGM/CPU%d/RZ/FlushTLB/NewCR3", "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
1807	PGM_REG_COUNTER(&pPGM->StatRZFlushTLBNewCR3Global, "/PGM/CPU%d/RZ/FlushTLB/NewCR3Global", "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
1808	PGM_REG_COUNTER(&pPGM->StatRZFlushTLBSameCR3, "/PGM/CPU%d/RZ/FlushTLB/SameCR3", "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
1809	PGM_REG_COUNTER(&pPGM->StatRZFlushTLBSameCR3Global, "/PGM/CPU%d/RZ/FlushTLB/SameCR3Global", "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
1810	PGM_REG_PROFILE(&pPGM->StatRZGstModifyPage, "/PGM/CPU%d/RZ/GstModifyPage", "Profiling of the PGMGstModifyPage() body.");
1811
1812	PGM_REG_PROFILE(&pPGM->StatR3SyncCR3, "/PGM/CPU%d/R3/SyncCR3", "Profiling of the PGMSyncCR3() body.");
1813	PGM_REG_PROFILE(&pPGM->StatR3SyncCR3Handlers, "/PGM/CPU%d/R3/SyncCR3/Handlers", "Profiling of the PGMSyncCR3() update handler section.");
1814	PGM_REG_COUNTER(&pPGM->StatR3SyncCR3Global, "/PGM/CPU%d/R3/SyncCR3/Global", "The number of global CR3 syncs.");
1815	PGM_REG_COUNTER(&pPGM->StatR3SyncCR3NotGlobal, "/PGM/CPU%d/R3/SyncCR3/NotGlobal", "The number of non-global CR3 syncs.");
1816	PGM_REG_COUNTER(&pPGM->StatR3SyncCR3DstCacheHit, "/PGM/CPU%d/R3/SyncCR3/DstChacheHit", "The number of times we got some kind of a cache hit.");
1817	PGM_REG_COUNTER(&pPGM->StatR3SyncCR3DstFreed, "/PGM/CPU%d/R3/SyncCR3/DstFreed", "The number of times we've had to free a shadow entry.");
1818	PGM_REG_COUNTER(&pPGM->StatR3SyncCR3DstFreedSrcNP, "/PGM/CPU%d/R3/SyncCR3/DstFreedSrcNP", "The number of times we've had to free a shadow entry for which the source entry was not present.");
1819	PGM_REG_COUNTER(&pPGM->StatR3SyncCR3DstNotPresent, "/PGM/CPU%d/R3/SyncCR3/DstNotPresent", "The number of times we've encountered a not present shadow entry for a present guest entry.");
1820	PGM_REG_COUNTER(&pPGM->StatR3SyncCR3DstSkippedGlobalPD, "/PGM/CPU%d/R3/SyncCR3/DstSkippedGlobalPD", "The number of times a global page directory wasn't flushed.");
1821	PGM_REG_COUNTER(&pPGM->StatR3SyncCR3DstSkippedGlobalPT, "/PGM/CPU%d/R3/SyncCR3/DstSkippedGlobalPT", "The number of times a page table with only global entries wasn't flushed.");
1822	PGM_REG_PROFILE(&pPGM->StatR3SyncPT, "/PGM/CPU%d/R3/SyncPT", "Profiling of the pfnSyncPT() body.");
1823	PGM_REG_COUNTER(&pPGM->StatR3SyncPTFailed, "/PGM/CPU%d/R3/SyncPT/Failed", "The number of times pfnSyncPT() failed.");
1824	PGM_REG_COUNTER(&pPGM->StatR3SyncPT4K, "/PGM/CPU%d/R3/SyncPT/4K", "Nr of 4K PT syncs");
1825	PGM_REG_COUNTER(&pPGM->StatR3SyncPT4M, "/PGM/CPU%d/R3/SyncPT/4M", "Nr of 4M PT syncs");
1826	PGM_REG_COUNTER(&pPGM->StatR3SyncPagePDNAs, "/PGM/CPU%d/R3/SyncPagePDNAs", "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1827	PGM_REG_COUNTER(&pPGM->StatR3SyncPagePDOutOfSync, "/PGM/CPU%d/R3/SyncPagePDOutOfSync", "The number of time we've encountered an out-of-sync PD in SyncPage.");
1828	PGM_REG_COUNTER(&pPGM->StatR3AccessedPage, "/PGM/CPU%d/R3/AccessedPage", "The number of pages marked not present for accessed bit emulation.");
1829	PGM_REG_PROFILE(&pPGM->StatR3DirtyBitTracking, "/PGM/CPU%d/R3/DirtyPage", "Profiling the dirty bit tracking in CheckPageFault().");
1830	PGM_REG_COUNTER(&pPGM->StatR3DirtyPage, "/PGM/CPU%d/R3/DirtyPage/Mark", "The number of pages marked read-only for dirty bit tracking.");
1831	PGM_REG_COUNTER(&pPGM->StatR3DirtyPageBig, "/PGM/CPU%d/R3/DirtyPage/MarkBig", "The number of 4MB pages marked read-only for dirty bit tracking.");
1832	PGM_REG_COUNTER(&pPGM->StatR3DirtyPageSkipped, "/PGM/CPU%d/R3/DirtyPage/Skipped", "The number of pages already dirty or readonly.");
1833	PGM_REG_COUNTER(&pPGM->StatR3DirtyPageTrap, "/PGM/CPU%d/R3/DirtyPage/Trap", "The number of traps generated for dirty bit tracking.");
1834	PGM_REG_COUNTER(&pPGM->StatR3DirtiedPage, "/PGM/CPU%d/R3/DirtyPage/SetDirty", "The number of pages marked dirty because of write accesses.");
1835	PGM_REG_COUNTER(&pPGM->StatR3DirtyTrackRealPF, "/PGM/CPU%d/R3/DirtyPage/RealPF", "The number of real pages faults during dirty bit tracking.");
1836	PGM_REG_COUNTER(&pPGM->StatR3PageAlreadyDirty, "/PGM/CPU%d/R3/DirtyPage/AlreadySet", "The number of pages already marked dirty because of write accesses.");
1837	PGM_REG_PROFILE(&pPGM->StatR3InvalidatePage, "/PGM/CPU%d/R3/InvalidatePage", "PGMInvalidatePage() profiling.");
1838	PGM_REG_COUNTER(&pPGM->StatR3InvalidatePage4KBPages, "/PGM/CPU%d/R3/InvalidatePage/4KBPages", "The number of times PGMInvalidatePage() was called for a 4KB page.");
1839	PGM_REG_COUNTER(&pPGM->StatR3InvalidatePage4MBPages, "/PGM/CPU%d/R3/InvalidatePage/4MBPages", "The number of times PGMInvalidatePage() was called for a 4MB page.");
1840	PGM_REG_COUNTER(&pPGM->StatR3InvalidatePage4MBPagesSkip, "/PGM/CPU%d/R3/InvalidatePage/4MBPagesSkip","The number of times PGMInvalidatePage() skipped a 4MB page.");
1841	PGM_REG_COUNTER(&pPGM->StatR3InvalidatePagePDMappings, "/PGM/CPU%d/R3/InvalidatePage/PDMappings", "The number of times PGMInvalidatePage() was called for a page directory containing mappings (no conflict).");
1842	PGM_REG_COUNTER(&pPGM->StatR3InvalidatePagePDNAs, "/PGM/CPU%d/R3/InvalidatePage/PDNAs", "The number of times PGMInvalidatePage() was called for a not accessed page directory.");
1843	PGM_REG_COUNTER(&pPGM->StatR3InvalidatePagePDNPs, "/PGM/CPU%d/R3/InvalidatePage/PDNPs", "The number of times PGMInvalidatePage() was called for a not present page directory.");
1844	PGM_REG_COUNTER(&pPGM->StatR3InvalidatePagePDOutOfSync, "/PGM/CPU%d/R3/InvalidatePage/PDOutOfSync", "The number of times PGMInvalidatePage() was called for an out of sync page directory.");
1845	PGM_REG_COUNTER(&pPGM->StatR3InvalidatePageSkipped, "/PGM/CPU%d/R3/InvalidatePage/Skipped", "The number of times PGMInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
1846	PGM_REG_COUNTER(&pPGM->StatR3PageOutOfSyncSupervisor, "/PGM/CPU%d/R3/OutOfSync/SuperVisor", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
1847	PGM_REG_COUNTER(&pPGM->StatR3PageOutOfSyncUser, "/PGM/CPU%d/R3/OutOfSync/User", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
1848	PGM_REG_PROFILE(&pPGM->StatR3Prefetch, "/PGM/CPU%d/R3/Prefetch", "PGMPrefetchPage profiling.");
1849	PGM_REG_PROFILE(&pPGM->StatR3FlushTLB, "/PGM/CPU%d/R3/FlushTLB", "Profiling of the PGMFlushTLB() body.");
1850	PGM_REG_COUNTER(&pPGM->StatR3FlushTLBNewCR3, "/PGM/CPU%d/R3/FlushTLB/NewCR3", "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
1851	PGM_REG_COUNTER(&pPGM->StatR3FlushTLBNewCR3Global, "/PGM/CPU%d/R3/FlushTLB/NewCR3Global", "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
1852	PGM_REG_COUNTER(&pPGM->StatR3FlushTLBSameCR3, "/PGM/CPU%d/R3/FlushTLB/SameCR3", "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
1853	PGM_REG_COUNTER(&pPGM->StatR3FlushTLBSameCR3Global, "/PGM/CPU%d/R3/FlushTLB/SameCR3Global", "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
1854	PGM_REG_PROFILE(&pPGM->StatR3GstModifyPage, "/PGM/CPU%d/R3/GstModifyPage", "Profiling of the PGMGstModifyPage() body.");
1855	#endif /* VBOX_WITH_STATISTICS */
1856
1857	#undef PGM_REG_PROFILE
1858	#undef PGM_REG_COUNTER
1859
1860	}
1861	}
1862
1863
1864	/**
1865	* Init the PGM bits that rely on VMMR0 and MM to be fully initialized.
1866	*
1867	* The dynamic mapping area will also be allocated and initialized at this
1868	* time. We could allocate it during PGMR3Init of course, but the mapping
1869	* wouldn't be allocated at that time preventing us from setting up the
1870	* page table entries with the dummy page.
1871	*
1872	* @returns VBox status code.
1873	* @param pVM VM handle.
1874	*/
1875	VMMR3DECL(int) PGMR3InitDynMap(PVM pVM)
1876	{
1877	RTGCPTR GCPtr;
1878	int rc;
1879
1880	/*
1881	* Reserve space for the dynamic mappings.
1882	*/
1883	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping", &GCPtr);
1884	if (RT_SUCCESS(rc))
1885	pVM->pgm.s.pbDynPageMapBaseGC = GCPtr;
1886
1887	if ( RT_SUCCESS(rc)
1888	&& (pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_PAE_SHIFT) != ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_PAE_SHIFT))
1889	{
1890	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping not crossing", &GCPtr);
1891	if (RT_SUCCESS(rc))
1892	pVM->pgm.s.pbDynPageMapBaseGC = GCPtr;
1893	}
1894	if (RT_SUCCESS(rc))
1895	{
1896	AssertRelease((pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_PAE_SHIFT) == ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_PAE_SHIFT));
1897	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
1898	}
1899	return rc;
1900	}
1901
1902
1903	/**
1904	* Ring-3 init finalizing.
1905	*
1906	* @returns VBox status code.
1907	* @param pVM The VM handle.
1908	*/
1909	VMMR3DECL(int) PGMR3InitFinalize(PVM pVM)
1910	{
1911	int rc;
1912
1913	/*
1914	* Reserve space for the dynamic mappings.
1915	* Initialize the dynamic mapping pages with dummy pages to simply the cache.
1916	*/
1917	/* get the pointer to the page table entries. */
1918	PPGMMAPPING pMapping = pgmGetMapping(pVM, pVM->pgm.s.pbDynPageMapBaseGC);
1919	AssertRelease(pMapping);
1920	const uintptr_t off = pVM->pgm.s.pbDynPageMapBaseGC - pMapping->GCPtr;
1921	const unsigned iPT = off >> X86_PD_SHIFT;
1922	const unsigned iPG = (off >> X86_PT_SHIFT) & X86_PT_MASK;
1923	pVM->pgm.s.paDynPageMap32BitPTEsGC = pMapping->aPTs[iPT].pPTRC + iPG * sizeof(pMapping->aPTs[0].pPTR3->a[0]);
1924	pVM->pgm.s.paDynPageMapPaePTEsGC = pMapping->aPTs[iPT].paPaePTsRC + iPG * sizeof(pMapping->aPTs[0].paPaePTsR3->a[0]);
1925
1926	/* init cache */
1927	RTHCPHYS HCPhysDummy = MMR3PageDummyHCPhys(pVM);
1928	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.aHCPhysDynPageMapCache); i++)
1929	pVM->pgm.s.aHCPhysDynPageMapCache[i] = HCPhysDummy;
1930
1931	for (unsigned i = 0; i < MM_HYPER_DYNAMIC_SIZE; i += PAGE_SIZE)
1932	{
1933	rc = PGMMap(pVM, pVM->pgm.s.pbDynPageMapBaseGC + i, HCPhysDummy, PAGE_SIZE, 0);
1934	AssertRCReturn(rc, rc);
1935	}
1936
1937	/*
1938	* Note that AMD uses all the 8 reserved bits for the address (so 40 bits in total);
1939	* Intel only goes up to 36 bits, so we stick to 36 as well.
1940	*/
1941	/** @todo How to test for the 40 bits support? Long mode seems to be the test criterium. */
1942	uint32_t u32Dummy, u32Features;
1943	CPUMGetGuestCpuId(pVM, 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
1944
1945	if (u32Features & X86_CPUID_FEATURE_EDX_PSE36)
1946	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(36) - 1;
1947	else
1948	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(32) - 1;
1949
1950	/*
1951	* Allocate memory if we're supposed to do that.
1952	*/
1953	if (pVM->pgm.s.fRamPreAlloc)
1954	rc = pgmR3PhysRamPreAllocate(pVM);
1955
1956	LogRel(("PGMR3InitFinalize: 4 MB PSE mask %RGp\n", pVM->pgm.s.GCPhys4MBPSEMask));
1957	return rc;
1958	}
1959
1960
1961	/**
1962	* Applies relocations to data and code managed by this component.
1963	*
1964	* This function will be called at init and whenever the VMM need to relocate it
1965	* self inside the GC.
1966	*
1967	* @param pVM The VM.
1968	* @param offDelta Relocation delta relative to old location.
1969	*/
1970	VMMR3DECL(void) PGMR3Relocate(PVM pVM, RTGCINTPTR offDelta)
1971	{
1972	LogFlow(("PGMR3Relocate %RGv to %RGv\n", pVM->pgm.s.GCPtrCR3Mapping, pVM->pgm.s.GCPtrCR3Mapping + offDelta));
1973
1974	/*
1975	* Paging stuff.
1976	*/
1977	pVM->pgm.s.GCPtrCR3Mapping += offDelta;
1978
1979	pgmR3ModeDataInit(pVM, true /* resolve GC/R0 symbols */);
1980
1981	/* Shadow, guest and both mode switch & relocation for each VCPU. */
1982	for (unsigned i=0;i<pVM->cCPUs;i++)
1983	{
1984	PVMCPU pVCpu = &pVM->aCpus[i];
1985
1986	pgmR3ModeDataSwitch(pVM, pVCpu, pVCpu->pgm.s.enmShadowMode, pVCpu->pgm.s.enmGuestMode);
1987
1988	PGM_SHW_PFN(Relocate, pVM)(pVM, pVCpu, offDelta);
1989	PGM_GST_PFN(Relocate, pVM)(pVM, pVCpu, offDelta);
1990	PGM_BTH_PFN(Relocate, pVM)(pVM, pVCpu, offDelta);
1991	}
1992
1993	/*
1994	* Trees.
1995	*/
1996	pVM->pgm.s.pTreesRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pTreesR3);
1997
1998	/*
1999	* Ram ranges.
2000	*/
2001	if (pVM->pgm.s.pRamRangesR3)
2002	{
2003	/* Update the pSelfRC pointers and relink them. */
2004	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesR3; pCur; pCur = pCur->pNextR3)
2005	if (!(pCur->fFlags & PGM_RAM_RANGE_FLAGS_FLOATING))
2006	pCur->pSelfRC = MMHyperCCToRC(pVM, pCur);
2007	pgmR3PhysRelinkRamRanges(pVM);
2008	}
2009
2010	/*
2011	* Update the two page directories with all page table mappings.
2012	* (One or more of them have changed, that's why we're here.)
2013	*/
2014	pVM->pgm.s.pMappingsRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pMappingsR3);
2015	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur->pNextR3; pCur = pCur->pNextR3)
2016	pCur->pNextRC = MMHyperR3ToRC(pVM, pCur->pNextR3);
2017
2018	/* Relocate GC addresses of Page Tables. */
2019	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur; pCur = pCur->pNextR3)
2020	{
2021	for (RTHCUINT i = 0; i < pCur->cPTs; i++)
2022	{
2023	pCur->aPTs[i].pPTRC = MMHyperR3ToRC(pVM, pCur->aPTs[i].pPTR3);
2024	pCur->aPTs[i].paPaePTsRC = MMHyperR3ToRC(pVM, pCur->aPTs[i].paPaePTsR3);
2025	}
2026	}
2027
2028	/*
2029	* Dynamic page mapping area.
2030	*/
2031	pVM->pgm.s.paDynPageMap32BitPTEsGC += offDelta;
2032	pVM->pgm.s.paDynPageMapPaePTEsGC += offDelta;
2033	pVM->pgm.s.pbDynPageMapBaseGC += offDelta;
2034
2035	/*
2036	* The Zero page.
2037	*/
2038	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
2039	#ifdef VBOX_WITH_2X_4GB_ADDR_SPACE
2040	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTR0PTR \|\| !VMMIsHwVirtExtForced(pVM));
2041	#else
2042	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTR0PTR);
2043	#endif
2044
2045	/*
2046	* Physical and virtual handlers.
2047	*/
2048	RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, true, pgmR3RelocatePhysHandler, &offDelta);
2049	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesR3->VirtHandlers, true, pgmR3RelocateVirtHandler, &offDelta);
2050	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesR3->HyperVirtHandlers, true, pgmR3RelocateHyperVirtHandler, &offDelta);
2051
2052	/*
2053	* The page pool.
2054	*/
2055	pgmR3PoolRelocate(pVM);
2056	}
2057
2058
2059	/**
2060	* Callback function for relocating a physical access handler.
2061	*
2062	* @returns 0 (continue enum)
2063	* @param pNode Pointer to a PGMPHYSHANDLER node.
2064	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2065	* not certain the delta will fit in a void pointer for all possible configs.
2066	*/
2067	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser)
2068	{
2069	PPGMPHYSHANDLER pHandler = (PPGMPHYSHANDLER)pNode;
2070	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2071	if (pHandler->pfnHandlerRC)
2072	pHandler->pfnHandlerRC += offDelta;
2073	if (pHandler->pvUserRC >= 0x10000)
2074	pHandler->pvUserRC += offDelta;
2075	return 0;
2076	}
2077
2078
2079	/**
2080	* Callback function for relocating a virtual access handler.
2081	*
2082	* @returns 0 (continue enum)
2083	* @param pNode Pointer to a PGMVIRTHANDLER node.
2084	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2085	* not certain the delta will fit in a void pointer for all possible configs.
2086	*/
2087	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
2088	{
2089	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
2090	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2091	Assert( pHandler->enmType == PGMVIRTHANDLERTYPE_ALL
2092	\|\| pHandler->enmType == PGMVIRTHANDLERTYPE_WRITE);
2093	Assert(pHandler->pfnHandlerRC);
2094	pHandler->pfnHandlerRC += offDelta;
2095	return 0;
2096	}
2097
2098
2099	/**
2100	* Callback function for relocating a virtual access handler for the hypervisor mapping.
2101	*
2102	* @returns 0 (continue enum)
2103	* @param pNode Pointer to a PGMVIRTHANDLER node.
2104	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2105	* not certain the delta will fit in a void pointer for all possible configs.
2106	*/
2107	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
2108	{
2109	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
2110	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2111	Assert(pHandler->enmType == PGMVIRTHANDLERTYPE_HYPERVISOR);
2112	Assert(pHandler->pfnHandlerRC);
2113	pHandler->pfnHandlerRC += offDelta;
2114	return 0;
2115	}
2116
2117
2118	/**
2119	* The VM is being reset.
2120	*
2121	* For the PGM component this means that any PD write monitors
2122	* needs to be removed.
2123	*
2124	* @param pVM VM handle.
2125	*/
2126	VMMR3DECL(void) PGMR3Reset(PVM pVM)
2127	{
2128	int rc;
2129
2130	LogFlow(("PGMR3Reset:\n"));
2131	VM_ASSERT_EMT(pVM);
2132
2133	pgmLock(pVM);
2134
2135	/*
2136	* Unfix any fixed mappings and disable CR3 monitoring.
2137	*/
2138	pVM->pgm.s.fMappingsFixed = false;
2139	pVM->pgm.s.GCPtrMappingFixed = 0;
2140	pVM->pgm.s.cbMappingFixed = 0;
2141
2142	/* Exit the guest paging mode before the pgm pool gets reset.
2143	* Important to clean up the amd64 case.
2144	*/
2145	for (unsigned i=0;i<pVM->cCPUs;i++)
2146	{
2147	PVMCPU pVCpu = &pVM->aCpus[i];
2148
2149	rc = PGM_GST_PFN(Exit, pVM)(pVM, pVCpu);
2150	AssertRC(rc);
2151	}
2152
2153	#ifdef DEBUG
2154	DBGFR3InfoLog(pVM, "mappings", NULL);
2155	DBGFR3InfoLog(pVM, "handlers", "all nostat");
2156	#endif
2157
2158	/*
2159	* Reset the shadow page pool.
2160	*/
2161	pgmR3PoolReset(pVM);
2162
2163	for (unsigned i=0;i<pVM->cCPUs;i++)
2164	{
2165	PVMCPU pVCpu = &pVM->aCpus[i];
2166
2167	/*
2168	* Re-init other members.
2169	*/
2170	pVCpu->pgm.s.fA20Enabled = true;
2171
2172	/*
2173	* Clear the FFs PGM owns.
2174	*/
2175	VM_FF_CLEAR(pVM, VM_FF_PGM_SYNC_CR3);
2176	VM_FF_CLEAR(pVM, VM_FF_PGM_SYNC_CR3_NON_GLOBAL);
2177	}
2178
2179	/*
2180	* Reset (zero) RAM pages.
2181	*/
2182	rc = pgmR3PhysRamReset(pVM);
2183	if (RT_SUCCESS(rc))
2184	{
2185	/*
2186	* Reset (zero) shadow ROM pages.
2187	*/
2188	rc = pgmR3PhysRomReset(pVM);
2189	if (RT_SUCCESS(rc))
2190	{
2191	/*
2192	* Switch mode back to real mode.
2193	*/
2194	for (unsigned i=0;i<pVM->cCPUs;i++)
2195	{
2196	PVMCPU pVCpu = &pVM->aCpus[i];
2197
2198	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
2199	AssertRC(rc);
2200
2201	STAM_REL_COUNTER_RESET(&pVCpu->pgm.s.cGuestModeChanges);
2202	}
2203	}
2204	}
2205
2206	pgmUnlock(pVM);
2207	//return rc;
2208	AssertReleaseRC(rc);
2209	}
2210
2211
2212	#ifdef VBOX_STRICT
2213	/**
2214	* VM state change callback for clearing fNoMorePhysWrites after
2215	* a snapshot has been created.
2216	*/
2217	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser)
2218	{
2219	if (enmState == VMSTATE_RUNNING)
2220	pVM->pgm.s.fNoMorePhysWrites = false;
2221	}
2222	#endif
2223
2224
2225	/**
2226	* Terminates the PGM.
2227	*
2228	* @returns VBox status code.
2229	* @param pVM Pointer to VM structure.
2230	*/
2231	VMMR3DECL(int) PGMR3Term(PVM pVM)
2232	{
2233	PGMDeregisterStringFormatTypes();
2234	return PDMR3CritSectDelete(&pVM->pgm.s.CritSect);
2235	}
2236
2237
2238	/**
2239	* Terminates the per-VCPU PGM.
2240	*
2241	* Termination means cleaning up and freeing all resources,
2242	* the VM it self is at this point powered off or suspended.
2243	*
2244	* @returns VBox status code.
2245	* @param pVM The VM to operate on.
2246	*/
2247	VMMR3DECL(int) PGMR3TermCPU(PVM pVM)
2248	{
2249	return 0;
2250	}
2251
2252
2253	/**
2254	* Find the ROM tracking structure for the given page.
2255	*
2256	* @returns Pointer to the ROM page structure. NULL if the caller didn't check
2257	* that it's a ROM page.
2258	* @param pVM The VM handle.
2259	* @param GCPhys The address of the ROM page.
2260	*/
2261	static PPGMROMPAGE pgmR3GetRomPage(PVM pVM, RTGCPHYS GCPhys)
2262	{
2263	for (PPGMROMRANGE pRomRange = pVM->pgm.s.CTX_SUFF(pRomRanges);
2264	pRomRange;
2265	pRomRange = pRomRange->CTX_SUFF(pNext))
2266	{
2267	RTGCPHYS off = GCPhys - pRomRange->GCPhys;
2268	if (GCPhys - pRomRange->GCPhys < pRomRange->cb)
2269	return &pRomRange->aPages[off >> PAGE_SHIFT];
2270	}
2271	return NULL;
2272	}
2273
2274
2275	/**
2276	* Save zero indicator + bits for the specified page.
2277	*
2278	* @returns VBox status code, errors are logged/asserted before returning.
2279	* @param pVM The VM handle.
2280	* @param pSSH The saved state handle.
2281	* @param pPage The page to save.
2282	* @param GCPhys The address of the page.
2283	* @param pRam The ram range (for error logging).
2284	*/
2285	static int pgmR3SavePage(PVM pVM, PSSMHANDLE pSSM, PPGMPAGE pPage, RTGCPHYS GCPhys, PPGMRAMRANGE pRam)
2286	{
2287	int rc;
2288	if (PGM_PAGE_IS_ZERO(pPage))
2289	rc = SSMR3PutU8(pSSM, 0);
2290	else
2291	{
2292	void const *pvPage;
2293	rc = pgmPhysGCPhys2CCPtrInternalReadOnly(pVM, pPage, GCPhys, &pvPage);
2294	AssertLogRelMsgRCReturn(rc, ("pPage=%R[pgmpage] GCPhys=%#x %s\n", pPage, GCPhys, pRam->pszDesc), rc);
2295
2296	SSMR3PutU8(pSSM, 1);
2297	rc = SSMR3PutMem(pSSM, pvPage, PAGE_SIZE);
2298	}
2299	return rc;
2300	}
2301
2302
2303	/**
2304	* Save a shadowed ROM page.
2305	*
2306	* Format: Type, protection, and two pages with zero indicators.
2307	*
2308	* @returns VBox status code, errors are logged/asserted before returning.
2309	* @param pVM The VM handle.
2310	* @param pSSH The saved state handle.
2311	* @param pPage The page to save.
2312	* @param GCPhys The address of the page.
2313	* @param pRam The ram range (for error logging).
2314	*/
2315	static int pgmR3SaveShadowedRomPage(PVM pVM, PSSMHANDLE pSSM, PPGMPAGE pPage, RTGCPHYS GCPhys, PPGMRAMRANGE pRam)
2316	{
2317	/* Need to save both pages and the current state. */
2318	PPGMROMPAGE pRomPage = pgmR3GetRomPage(pVM, GCPhys);
2319	AssertLogRelMsgReturn(pRomPage, ("GCPhys=%RGp %s\n", GCPhys, pRam->pszDesc), VERR_INTERNAL_ERROR);
2320
2321	SSMR3PutU8(pSSM, PGMPAGETYPE_ROM_SHADOW);
2322	SSMR3PutU8(pSSM, pRomPage->enmProt);
2323
2324	int rc = pgmR3SavePage(pVM, pSSM, pPage, GCPhys, pRam);
2325	if (RT_SUCCESS(rc))
2326	{
2327	PPGMPAGE pPagePassive = PGMROMPROT_IS_ROM(pRomPage->enmProt) ? &pRomPage->Shadow : &pRomPage->Virgin;
2328	rc = pgmR3SavePage(pVM, pSSM, pPagePassive, GCPhys, pRam);
2329	}
2330	return rc;
2331	}
2332
2333	/** PGM fields to save/load. */
2334	static SSMFIELD s_aPGMFields[] =
2335	{
2336	SSMFIELD_ENTRY( PGM, fMappingsFixed),
2337	SSMFIELD_ENTRY_GCPTR( PGM, GCPtrMappingFixed),
2338	SSMFIELD_ENTRY( PGM, cbMappingFixed),
2339	SSMFIELD_ENTRY_TERM()
2340	};
2341
2342	static SSMFIELD s_aPGMCpuFields[] =
2343	{
2344	SSMFIELD_ENTRY( PGMCPU, fA20Enabled),
2345	SSMFIELD_ENTRY_GCPHYS( PGMCPU, GCPhysA20Mask),
2346	SSMFIELD_ENTRY( PGMCPU, enmGuestMode),
2347	SSMFIELD_ENTRY_TERM()
2348	};
2349
2350	/* For loading old saved states. (pre-smp) */
2351	typedef struct
2352	{
2353	/** If set no conflict checks are required. (boolean) */
2354	bool fMappingsFixed;
2355	/** Size of fixed mapping */
2356	uint32_t cbMappingFixed;
2357	/** Base address (GC) of fixed mapping */
2358	RTGCPTR GCPtrMappingFixed;
2359	/** A20 gate mask.
2360	* Our current approach to A20 emulation is to let REM do it and don't bother
2361	* anywhere else. The interesting Guests will be operating with it enabled anyway.
2362	* But whould need arrise, we'll subject physical addresses to this mask. */
2363	RTGCPHYS GCPhysA20Mask;
2364	/** A20 gate state - boolean! */
2365	bool fA20Enabled;
2366	/** The guest paging mode. */
2367	PGMMODE enmGuestMode;
2368	} PGMOLD;
2369
2370	static SSMFIELD s_aPGMFields_Old[] =
2371	{
2372	SSMFIELD_ENTRY( PGMOLD, fMappingsFixed),
2373	SSMFIELD_ENTRY_GCPTR( PGMOLD, GCPtrMappingFixed),
2374	SSMFIELD_ENTRY( PGMOLD, cbMappingFixed),
2375	SSMFIELD_ENTRY( PGMOLD, fA20Enabled),
2376	SSMFIELD_ENTRY_GCPHYS( PGMOLD, GCPhysA20Mask),
2377	SSMFIELD_ENTRY( PGMOLD, enmGuestMode),
2378	SSMFIELD_ENTRY_TERM()
2379	};
2380
2381
2382	/**
2383	* Execute state save operation.
2384	*
2385	* @returns VBox status code.
2386	* @param pVM VM Handle.
2387	* @param pSSM SSM operation handle.
2388	*/
2389	static DECLCALLBACK(int) pgmR3Save(PVM pVM, PSSMHANDLE pSSM)
2390	{
2391	int rc;
2392	unsigned i;
2393	PPGM pPGM = &pVM->pgm.s;
2394
2395	/*
2396	* Lock PGM and set the no-more-writes indicator.
2397	*/
2398	pgmLock(pVM);
2399	pVM->pgm.s.fNoMorePhysWrites = true;
2400
2401	/*
2402	* Save basic data (required / unaffected by relocation).
2403	*/
2404	SSMR3PutStruct(pSSM, pPGM, &s_aPGMFields[0]);
2405
2406	for (i=0;i<pVM->cCPUs;i++)
2407	{
2408	PVMCPU pVCpu = &pVM->aCpus[i];
2409
2410	SSMR3PutStruct(pSSM, &pVCpu->pgm.s, &s_aPGMCpuFields[0]);
2411	}
2412
2413	/*
2414	* The guest mappings.
2415	*/
2416	i = 0;
2417	for (PPGMMAPPING pMapping = pPGM->pMappingsR3; pMapping; pMapping = pMapping->pNextR3, i++)
2418	{
2419	SSMR3PutU32( pSSM, i);
2420	SSMR3PutStrZ( pSSM, pMapping->pszDesc); /* This is the best unique id we have... */
2421	SSMR3PutGCPtr( pSSM, pMapping->GCPtr);
2422	SSMR3PutGCUIntPtr(pSSM, pMapping->cPTs);
2423	}
2424	rc = SSMR3PutU32(pSSM, ~0); /* terminator. */
2425
2426	/*
2427	* Ram ranges and the memory they describe.
2428	*/
2429	i = 0;
2430	for (PPGMRAMRANGE pRam = pPGM->pRamRangesR3; pRam; pRam = pRam->pNextR3, i++)
2431	{
2432	/*
2433	* Save the ram range details.
2434	*/
2435	SSMR3PutU32(pSSM, i);
2436	SSMR3PutGCPhys(pSSM, pRam->GCPhys);
2437	SSMR3PutGCPhys(pSSM, pRam->GCPhysLast);
2438	SSMR3PutGCPhys(pSSM, pRam->cb);
2439	SSMR3PutU8(pSSM, !!pRam->pvR3); /* Boolean indicating memory or not. */
2440	SSMR3PutStrZ(pSSM, pRam->pszDesc); /* This is the best unique id we have... */
2441
2442	/*
2443	* Iterate the pages, only two special case.
2444	*/
2445	uint32_t const cPages = pRam->cb >> PAGE_SHIFT;
2446	for (uint32_t iPage = 0; iPage < cPages; iPage++)
2447	{
2448	RTGCPHYS GCPhysPage = pRam->GCPhys + ((RTGCPHYS)iPage << PAGE_SHIFT);
2449	PPGMPAGE pPage = &pRam->aPages[iPage];
2450	uint8_t uType = PGM_PAGE_GET_TYPE(pPage);
2451
2452	if (uType == PGMPAGETYPE_ROM_SHADOW)
2453	rc = pgmR3SaveShadowedRomPage(pVM, pSSM, pPage, GCPhysPage, pRam);
2454	else if (uType == PGMPAGETYPE_MMIO2_ALIAS_MMIO)
2455	{
2456	/* MMIO2 alias -> MMIO; the device will just have to deal with this. */
2457	SSMR3PutU8(pSSM, PGMPAGETYPE_MMIO);
2458	rc = SSMR3PutU8(pSSM, 0 /* ZERO */);
2459	}
2460	else
2461	{
2462	SSMR3PutU8(pSSM, uType);
2463	rc = pgmR3SavePage(pVM, pSSM, pPage, GCPhysPage, pRam);
2464	}
2465	if (RT_FAILURE(rc))
2466	break;
2467	}
2468	if (RT_FAILURE(rc))
2469	break;
2470	}
2471
2472	pgmUnlock(pVM);
2473	return SSMR3PutU32(pSSM, ~0); /* terminator. */
2474	}
2475
2476
2477	/**
2478	* Load an ignored page.
2479	*
2480	* @returns VBox status code.
2481	* @param pSSM The saved state handle.
2482	*/
2483	static int pgmR3LoadPageToDevNull(PSSMHANDLE pSSM)
2484	{
2485	uint8_t abPage[PAGE_SIZE];
2486	return SSMR3GetMem(pSSM, &abPage[0], sizeof(abPage));
2487	}
2488
2489
2490	/**
2491	* Loads a page without any bits in the saved state, i.e. making sure it's
2492	* really zero.
2493	*
2494	* @returns VBox status code.
2495	* @param pVM The VM handle.
2496	* @param uType The page type or PGMPAGETYPE_INVALID (old saved
2497	* state).
2498	* @param pPage The guest page tracking structure.
2499	* @param GCPhys The page address.
2500	* @param pRam The ram range (logging).
2501	*/
2502	static int pgmR3LoadPageZero(PVM pVM, uint8_t uType, PPGMPAGE pPage, RTGCPHYS GCPhys, PPGMRAMRANGE pRam)
2503	{
2504	if ( PGM_PAGE_GET_TYPE(pPage) != uType
2505	&& uType != PGMPAGETYPE_INVALID)
2506	return VERR_SSM_UNEXPECTED_DATA;
2507
2508	/* I think this should be sufficient. */
2509	if (!PGM_PAGE_IS_ZERO(pPage))
2510	return VERR_SSM_UNEXPECTED_DATA;
2511
2512	NOREF(pVM);
2513	NOREF(GCPhys);
2514	NOREF(pRam);
2515	return VINF_SUCCESS;
2516	}
2517
2518
2519	/**
2520	* Loads a page from the saved state.
2521	*
2522	* @returns VBox status code.
2523	* @param pVM The VM handle.
2524	* @param pSSM The SSM handle.
2525	* @param uType The page type or PGMPAGETYEP_INVALID (old saved
2526	* state).
2527	* @param pPage The guest page tracking structure.
2528	* @param GCPhys The page address.
2529	* @param pRam The ram range (logging).
2530	*/
2531	static int pgmR3LoadPageBits(PVM pVM, PSSMHANDLE pSSM, uint8_t uType, PPGMPAGE pPage, RTGCPHYS GCPhys, PPGMRAMRANGE pRam)
2532	{
2533	int rc;
2534
2535	/*
2536	* Match up the type, dealing with MMIO2 aliases (dropped).
2537	*/
2538	AssertLogRelMsgReturn( PGM_PAGE_GET_TYPE(pPage) == uType
2539	\|\| uType == PGMPAGETYPE_INVALID,
2540	("pPage=%R[pgmpage] GCPhys=%#x %s\n", pPage, GCPhys, pRam->pszDesc),
2541	VERR_SSM_UNEXPECTED_DATA);
2542
2543	/*
2544	* Load the page.
2545	*/
2546	void *pvPage;
2547	rc = pgmPhysGCPhys2CCPtrInternal(pVM, pPage, GCPhys, &pvPage);
2548	if (RT_SUCCESS(rc))
2549	rc = SSMR3GetMem(pSSM, pvPage, PAGE_SIZE);
2550
2551	return rc;
2552	}
2553
2554
2555	/**
2556	* Loads a page (counter part to pgmR3SavePage).
2557	*
2558	* @returns VBox status code, fully bitched errors.
2559	* @param pVM The VM handle.
2560	* @param pSSM The SSM handle.
2561	* @param uType The page type.
2562	* @param pPage The page.
2563	* @param GCPhys The page address.
2564	* @param pRam The RAM range (for error messages).
2565	*/
2566	static int pgmR3LoadPage(PVM pVM, PSSMHANDLE pSSM, uint8_t uType, PPGMPAGE pPage, RTGCPHYS GCPhys, PPGMRAMRANGE pRam)
2567	{
2568	uint8_t uState;
2569	int rc = SSMR3GetU8(pSSM, &uState);
2570	AssertLogRelMsgRCReturn(rc, ("pPage=%R[pgmpage] GCPhys=%#x %s rc=%Rrc\n", pPage, GCPhys, pRam->pszDesc, rc), rc);
2571	if (uState == 0 /* zero */)
2572	rc = pgmR3LoadPageZero(pVM, uType, pPage, GCPhys, pRam);
2573	else if (uState == 1)
2574	rc = pgmR3LoadPageBits(pVM, pSSM, uType, pPage, GCPhys, pRam);
2575	else
2576	rc = VERR_INTERNAL_ERROR;
2577	AssertLogRelMsgRCReturn(rc, ("pPage=%R[pgmpage] uState=%d uType=%d GCPhys=%RGp %s rc=%Rrc\n",
2578	pPage, uState, uType, GCPhys, pRam->pszDesc, rc),
2579	rc);
2580	return VINF_SUCCESS;
2581	}
2582
2583
2584	/**
2585	* Loads a shadowed ROM page.
2586	*
2587	* @returns VBox status code, errors are fully bitched.
2588	* @param pVM The VM handle.
2589	* @param pSSM The saved state handle.
2590	* @param pPage The page.
2591	* @param GCPhys The page address.
2592	* @param pRam The RAM range (for error messages).
2593	*/
2594	static int pgmR3LoadShadowedRomPage(PVM pVM, PSSMHANDLE pSSM, PPGMPAGE pPage, RTGCPHYS GCPhys, PPGMRAMRANGE pRam)
2595	{
2596	/*
2597	* Load and set the protection first, then load the two pages, the first
2598	* one is the active the other is the passive.
2599	*/
2600	PPGMROMPAGE pRomPage = pgmR3GetRomPage(pVM, GCPhys);
2601	AssertLogRelMsgReturn(pRomPage, ("GCPhys=%RGp %s\n", GCPhys, pRam->pszDesc), VERR_INTERNAL_ERROR);
2602
2603	uint8_t uProt;
2604	int rc = SSMR3GetU8(pSSM, &uProt);
2605	AssertLogRelMsgRCReturn(rc, ("pPage=%R[pgmpage] GCPhys=%#x %s\n", pPage, GCPhys, pRam->pszDesc), rc);
2606	PGMROMPROT enmProt = (PGMROMPROT)uProt;
2607	AssertLogRelMsgReturn( enmProt >= PGMROMPROT_INVALID
2608	&& enmProt < PGMROMPROT_END,
2609	("enmProt=%d pPage=%R[pgmpage] GCPhys=%#x %s\n", enmProt, pPage, GCPhys, pRam->pszDesc),
2610	VERR_SSM_UNEXPECTED_DATA);
2611
2612	if (pRomPage->enmProt != enmProt)
2613	{
2614	rc = PGMR3PhysRomProtect(pVM, GCPhys, PAGE_SIZE, enmProt);
2615	AssertLogRelRCReturn(rc, rc);
2616	AssertLogRelReturn(pRomPage->enmProt == enmProt, VERR_INTERNAL_ERROR);
2617	}
2618
2619	PPGMPAGE pPageActive = PGMROMPROT_IS_ROM(enmProt) ? &pRomPage->Virgin : &pRomPage->Shadow;
2620	PPGMPAGE pPagePassive = PGMROMPROT_IS_ROM(enmProt) ? &pRomPage->Shadow : &pRomPage->Virgin;
2621	uint8_t u8ActiveType = PGMROMPROT_IS_ROM(enmProt) ? PGMPAGETYPE_ROM : PGMPAGETYPE_ROM_SHADOW;
2622	uint8_t u8PassiveType= PGMROMPROT_IS_ROM(enmProt) ? PGMPAGETYPE_ROM_SHADOW : PGMPAGETYPE_ROM;
2623
2624	rc = pgmR3LoadPage(pVM, pSSM, u8ActiveType, pPage, GCPhys, pRam);
2625	if (RT_SUCCESS(rc))
2626	{
2627	pPageActive = pPage;
2628	rc = pgmR3LoadPage(pVM, pSSM, u8PassiveType, pPagePassive, GCPhys, pRam);
2629	}
2630	return rc;
2631	}
2632
2633
2634	/**
2635	* Worker for pgmR3Load.
2636	*
2637	* @returns VBox status code.
2638	*
2639	* @param pVM The VM handle.
2640	* @param pSSM The SSM handle.
2641	* @param u32Version The saved state version.
2642	*/
2643	static int pgmR3LoadLocked(PVM pVM, PSSMHANDLE pSSM, uint32_t u32Version)
2644	{
2645	int rc;
2646	PPGM pPGM = &pVM->pgm.s;
2647	uint32_t u32Sep;
2648
2649	/*
2650	* Load basic data (required / unaffected by relocation).
2651	*/
2652	if (u32Version >= PGM_SAVED_STATE_VERSION)
2653	{
2654	rc = SSMR3GetStruct(pSSM, pPGM, &s_aPGMFields[0]);
2655	AssertLogRelRCReturn(rc, rc);
2656
2657	for (unsigned i=0;i<pVM->cCPUs;i++)
2658	{
2659	PVMCPU pVCpu = &pVM->aCpus[i];
2660
2661	rc = SSMR3GetStruct(pSSM, &pVCpu->pgm.s, &s_aPGMCpuFields[0]);
2662	AssertLogRelRCReturn(rc, rc);
2663	}
2664	}
2665	else
2666	if (u32Version >= PGM_SAVED_STATE_VERSION_RR_DESC)
2667	{
2668	PGMOLD pgmOld;
2669
2670	AssertRelease(pVM->cCPUs == 1);
2671
2672	rc = SSMR3GetStruct(pSSM, &pgmOld, &s_aPGMFields_Old[0]);
2673	AssertLogRelRCReturn(rc, rc);
2674
2675	pPGM->fMappingsFixed = pgmOld.fMappingsFixed;
2676	pPGM->GCPtrMappingFixed = pgmOld.GCPtrMappingFixed;
2677	pPGM->cbMappingFixed = pgmOld.cbMappingFixed;
2678
2679	pVM->aCpus[0].pgm.s.fA20Enabled = pgmOld.fA20Enabled;
2680	pVM->aCpus[0].pgm.s.GCPhysA20Mask = pgmOld.GCPhysA20Mask;
2681	pVM->aCpus[0].pgm.s.enmGuestMode = pgmOld.enmGuestMode;
2682	}
2683	else
2684	{
2685	AssertRelease(pVM->cCPUs == 1);
2686
2687	SSMR3GetBool(pSSM, &pPGM->fMappingsFixed);
2688	SSMR3GetGCPtr(pSSM, &pPGM->GCPtrMappingFixed);
2689	SSMR3GetU32(pSSM, &pPGM->cbMappingFixed);
2690
2691	uint32_t cbRamSizeIgnored;
2692	rc = SSMR3GetU32(pSSM, &cbRamSizeIgnored);
2693	if (RT_FAILURE(rc))
2694	return rc;
2695	SSMR3GetGCPhys(pSSM, &pVM->aCpus[0].pgm.s.GCPhysA20Mask);
2696
2697	uint32_t u32 = 0;
2698	SSMR3GetUInt(pSSM, &u32);
2699	pVM->aCpus[0].pgm.s.fA20Enabled = !!u32;
2700	SSMR3GetUInt(pSSM, &pVM->aCpus[0].pgm.s.fSyncFlags);
2701	RTUINT uGuestMode;
2702	SSMR3GetUInt(pSSM, &uGuestMode);
2703	pVM->aCpus[0].pgm.s.enmGuestMode = (PGMMODE)uGuestMode;
2704
2705	/* check separator. */
2706	SSMR3GetU32(pSSM, &u32Sep);
2707	if (RT_FAILURE(rc))
2708	return rc;
2709	if (u32Sep != (uint32_t)~0)
2710	{
2711	AssertMsgFailed(("u32Sep=%#x (first)\n", u32Sep));
2712	return VERR_SSM_DATA_UNIT_FORMAT_CHANGED;
2713	}
2714	}
2715
2716	/*
2717	* The guest mappings.
2718	*/
2719	uint32_t i = 0;
2720	for (;; i++)
2721	{
2722	/* Check the seqence number / separator. */
2723	rc = SSMR3GetU32(pSSM, &u32Sep);
2724	if (RT_FAILURE(rc))
2725	return rc;
2726	if (u32Sep == ~0U)
2727	break;
2728	if (u32Sep != i)
2729	{
2730	AssertMsgFailed(("u32Sep=%#x (last)\n", u32Sep));
2731	return VERR_SSM_DATA_UNIT_FORMAT_CHANGED;
2732	}
2733
2734	/* get the mapping details. */
2735	char szDesc[256];
2736	szDesc[0] = '\0';
2737	rc = SSMR3GetStrZ(pSSM, szDesc, sizeof(szDesc));
2738	if (RT_FAILURE(rc))
2739	return rc;
2740	RTGCPTR GCPtr;
2741	SSMR3GetGCPtr(pSSM, &GCPtr);
2742	RTGCPTR cPTs;
2743	rc = SSMR3GetGCUIntPtr(pSSM, &cPTs);
2744	if (RT_FAILURE(rc))
2745	return rc;
2746
2747	/* find matching range. */
2748	PPGMMAPPING pMapping;
2749	for (pMapping = pPGM->pMappingsR3; pMapping; pMapping = pMapping->pNextR3)
2750	if ( pMapping->cPTs == cPTs
2751	&& !strcmp(pMapping->pszDesc, szDesc))
2752	break;
2753	AssertLogRelMsgReturn(pMapping, ("Couldn't find mapping: cPTs=%#x szDesc=%s (GCPtr=%RGv)\n",
2754	cPTs, szDesc, GCPtr),
2755	VERR_SSM_LOAD_CONFIG_MISMATCH);
2756
2757	/* relocate it. */
2758	if (pMapping->GCPtr != GCPtr)
2759	{
2760	AssertMsg((GCPtr >> X86_PD_SHIFT << X86_PD_SHIFT) == GCPtr, ("GCPtr=%RGv\n", GCPtr));
2761	pgmR3MapRelocate(pVM, pMapping, pMapping->GCPtr, GCPtr);
2762	}
2763	else
2764	Log(("pgmR3Load: '%s' needed no relocation (%RGv)\n", szDesc, GCPtr));
2765	}
2766
2767	/*
2768	* Ram range flags and bits.
2769	*/
2770	i = 0;
2771	for (PPGMRAMRANGE pRam = pPGM->pRamRangesR3; pRam; pRam = pRam->pNextR3, i++)
2772	{
2773	/** @todo MMIO ranges may move (PCI reconfig), we currently assume they don't. */
2774
2775	/* Check the seqence number / separator. */
2776	rc = SSMR3GetU32(pSSM, &u32Sep);
2777	if (RT_FAILURE(rc))
2778	return rc;
2779	if (u32Sep == ~0U)
2780	break;
2781	if (u32Sep != i)
2782	{
2783	AssertMsgFailed(("u32Sep=%#x (last)\n", u32Sep));
2784	return VERR_SSM_DATA_UNIT_FORMAT_CHANGED;
2785	}
2786
2787	/* Get the range details. */
2788	RTGCPHYS GCPhys;
2789	SSMR3GetGCPhys(pSSM, &GCPhys);
2790	RTGCPHYS GCPhysLast;
2791	SSMR3GetGCPhys(pSSM, &GCPhysLast);
2792	RTGCPHYS cb;
2793	SSMR3GetGCPhys(pSSM, &cb);
2794	uint8_t fHaveBits;
2795	rc = SSMR3GetU8(pSSM, &fHaveBits);
2796	if (RT_FAILURE(rc))
2797	return rc;
2798	if (fHaveBits & ~1)
2799	{
2800	AssertMsgFailed(("u32Sep=%#x (last)\n", u32Sep));
2801	return VERR_SSM_DATA_UNIT_FORMAT_CHANGED;
2802	}
2803	size_t cchDesc = 0;
2804	char szDesc[256];
2805	szDesc[0] = '\0';
2806	if (u32Version >= PGM_SAVED_STATE_VERSION_RR_DESC)
2807	{
2808	rc = SSMR3GetStrZ(pSSM, szDesc, sizeof(szDesc));
2809	if (RT_FAILURE(rc))
2810	return rc;
2811	/* Since we've modified the description strings in r45878, only compare
2812	them if the saved state is more recent. */
2813	if (u32Version != PGM_SAVED_STATE_VERSION_RR_DESC)
2814	cchDesc = strlen(szDesc);
2815	}
2816
2817	/*
2818	* Match it up with the current range.
2819	*
2820	* Note there is a hack for dealing with the high BIOS mapping
2821	* in the old saved state format, this means we might not have
2822	* a 1:1 match on success.
2823	*/
2824	if ( ( GCPhys != pRam->GCPhys
2825	\|\| GCPhysLast != pRam->GCPhysLast
2826	\|\| cb != pRam->cb
2827	\|\| ( cchDesc
2828	&& strcmp(szDesc, pRam->pszDesc)) )
2829	/* Hack for PDMDevHlpPhysReserve(pDevIns, 0xfff80000, 0x80000, "High ROM Region"); */
2830	&& ( u32Version != PGM_SAVED_STATE_VERSION_OLD_PHYS_CODE
2831	\|\| GCPhys != UINT32_C(0xfff80000)
2832	\|\| GCPhysLast != UINT32_C(0xffffffff)
2833	\|\| pRam->GCPhysLast != GCPhysLast
2834	\|\| pRam->GCPhys < GCPhys
2835	\|\| !fHaveBits)
2836	)
2837	{
2838	LogRel(("Ram range: %RGp-%RGp %RGp bytes %s %s\n"
2839	"State : %RGp-%RGp %RGp bytes %s %s\n",
2840	pRam->GCPhys, pRam->GCPhysLast, pRam->cb, pRam->pvR3 ? "bits" : "nobits", pRam->pszDesc,
2841	GCPhys, GCPhysLast, cb, fHaveBits ? "bits" : "nobits", szDesc));
2842	/*
2843	* If we're loading a state for debugging purpose, don't make a fuss if
2844	* the MMIO and ROM stuff isn't 100% right, just skip the mismatches.
2845	*/
2846	if ( SSMR3HandleGetAfter(pSSM) != SSMAFTER_DEBUG_IT
2847	\|\| GCPhys < 8 * _1M)
2848	AssertFailedReturn(VERR_SSM_LOAD_CONFIG_MISMATCH);
2849
2850	AssertMsgFailed(("debug skipping not implemented, sorry\n"));
2851	continue;
2852	}
2853
2854	uint32_t cPages = (GCPhysLast - GCPhys + 1) >> PAGE_SHIFT;
2855	if (u32Version >= PGM_SAVED_STATE_VERSION_RR_DESC)
2856	{
2857	/*
2858	* Load the pages one by one.
2859	*/
2860	for (uint32_t iPage = 0; iPage < cPages; iPage++)
2861	{
2862	RTGCPHYS const GCPhysPage = ((RTGCPHYS)iPage << PAGE_SHIFT) + pRam->GCPhys;
2863	PPGMPAGE pPage = &pRam->aPages[iPage];
2864	uint8_t uType;
2865	rc = SSMR3GetU8(pSSM, &uType);
2866	AssertLogRelMsgRCReturn(rc, ("pPage=%R[pgmpage] iPage=%#x GCPhysPage=%#x %s\n", pPage, iPage, GCPhysPage, pRam->pszDesc), rc);
2867	if (uType == PGMPAGETYPE_ROM_SHADOW)
2868	rc = pgmR3LoadShadowedRomPage(pVM, pSSM, pPage, GCPhysPage, pRam);
2869	else
2870	rc = pgmR3LoadPage(pVM, pSSM, uType, pPage, GCPhysPage, pRam);
2871	AssertLogRelMsgRCReturn(rc, ("rc=%Rrc iPage=%#x GCPhysPage=%#x %s\n", rc, iPage, GCPhysPage, pRam->pszDesc), rc);
2872	}
2873	}
2874	else
2875	{
2876	/*
2877	* Old format.
2878	*/
2879	AssertLogRelReturn(!pVM->pgm.s.fRamPreAlloc, VERR_NOT_SUPPORTED); /* can't be detected. */
2880
2881	/* Of the page flags, pick up MMIO2 and ROM/RESERVED for the !fHaveBits case.
2882	The rest is generally irrelevant and wrong since the stuff have to match registrations. */
2883	uint32_t fFlags = 0;
2884	for (uint32_t iPage = 0; iPage < cPages; iPage++)
2885	{
2886	uint16_t u16Flags;
2887	rc = SSMR3GetU16(pSSM, &u16Flags);
2888	AssertLogRelMsgRCReturn(rc, ("rc=%Rrc iPage=%#x GCPhys=%#x %s\n", rc, iPage, pRam->GCPhys, pRam->pszDesc), rc);
2889	fFlags \|= u16Flags;
2890	}
2891
2892	/* Load the bits */
2893	if ( !fHaveBits
2894	&& GCPhysLast < UINT32_C(0xe0000000))
2895	{
2896	/*
2897	* Dynamic chunks.
2898	*/
2899	const uint32_t cPagesInChunk = (110241024) >> PAGE_SHIFT;
2900	AssertLogRelMsgReturn(cPages % cPagesInChunk == 0,
2901	("cPages=%#x cPagesInChunk=%#x\n", cPages, cPagesInChunk, pRam->GCPhys, pRam->pszDesc),
2902	VERR_SSM_DATA_UNIT_FORMAT_CHANGED);
2903
2904	for (uint32_t iPage = 0; iPage < cPages; /* incremented by inner loop */ )
2905	{
2906	uint8_t fPresent;
2907	rc = SSMR3GetU8(pSSM, &fPresent);
2908	AssertLogRelMsgRCReturn(rc, ("rc=%Rrc iPage=%#x GCPhys=%#x %s\n", rc, iPage, pRam->GCPhys, pRam->pszDesc), rc);
2909	AssertLogRelMsgReturn(fPresent == (uint8_t)true \|\| fPresent == (uint8_t)false,
2910	("fPresent=%#x iPage=%#x GCPhys=%#x %s\n", fPresent, iPage, pRam->GCPhys, pRam->pszDesc),
2911	VERR_SSM_DATA_UNIT_FORMAT_CHANGED);
2912
2913	for (uint32_t iChunkPage = 0; iChunkPage < cPagesInChunk; iChunkPage++, iPage++)
2914	{
2915	RTGCPHYS const GCPhysPage = ((RTGCPHYS)iPage << PAGE_SHIFT) + pRam->GCPhys;
2916	PPGMPAGE pPage = &pRam->aPages[iPage];
2917	if (fPresent)
2918	{
2919	if (PGM_PAGE_GET_TYPE(pPage) == PGMPAGETYPE_MMIO)
2920	rc = pgmR3LoadPageToDevNull(pSSM);
2921	else
2922	rc = pgmR3LoadPageBits(pVM, pSSM, PGMPAGETYPE_INVALID, pPage, GCPhysPage, pRam);
2923	}
2924	else
2925	rc = pgmR3LoadPageZero(pVM, PGMPAGETYPE_INVALID, pPage, GCPhysPage, pRam);
2926	AssertLogRelMsgRCReturn(rc, ("rc=%Rrc iPage=%#x GCPhysPage=%#x %s\n", rc, iPage, GCPhysPage, pRam->pszDesc), rc);
2927	}
2928	}
2929	}
2930	else if (pRam->pvR3)
2931	{
2932	/*
2933	* MMIO2.
2934	*/
2935	AssertLogRelMsgReturn((fFlags & 0x0f) == RT_BIT(3) /MM_RAM_FLAGS_MMIO2/,
2936	("fFlags=%#x GCPhys=%#x %s\n", fFlags, pRam->GCPhys, pRam->pszDesc),
2937	VERR_SSM_DATA_UNIT_FORMAT_CHANGED);
2938	AssertLogRelMsgReturn(pRam->pvR3,
2939	("GCPhys=%#x %s\n", pRam->GCPhys, pRam->pszDesc),
2940	VERR_SSM_DATA_UNIT_FORMAT_CHANGED);
2941
2942	rc = SSMR3GetMem(pSSM, pRam->pvR3, pRam->cb);
2943	AssertLogRelMsgRCReturn(rc, ("GCPhys=%#x %s\n", pRam->GCPhys, pRam->pszDesc), rc);
2944	}
2945	else if (GCPhysLast < UINT32_C(0xfff80000))
2946	{
2947	/*
2948	* PCI MMIO, no pages saved.
2949	*/
2950	}
2951	else
2952	{
2953	/*
2954	* Load the 0xfff80000..0xffffffff BIOS range.
2955	* It starts with X reserved pages that we have to skip over since
2956	* the RAMRANGE create by the new code won't include those.
2957	*/
2958	AssertLogRelMsgReturn( !(fFlags & RT_BIT(3) /MM_RAM_FLAGS_MMIO2/)
2959	&& (fFlags & RT_BIT(0) /MM_RAM_FLAGS_RESERVED/),
2960	("fFlags=%#x GCPhys=%#x %s\n", fFlags, pRam->GCPhys, pRam->pszDesc),
2961	VERR_SSM_DATA_UNIT_FORMAT_CHANGED);
2962	AssertLogRelMsgReturn(GCPhys == UINT32_C(0xfff80000),
2963	("GCPhys=%RGp pRamRange{GCPhys=%#x %s}\n", GCPhys, pRam->GCPhys, pRam->pszDesc),
2964	VERR_SSM_DATA_UNIT_FORMAT_CHANGED);
2965
2966	/* Skip wasted reserved pages before the ROM. */
2967	while (GCPhys < pRam->GCPhys)
2968	{
2969	rc = pgmR3LoadPageToDevNull(pSSM);
2970	GCPhys += PAGE_SIZE;
2971	}
2972
2973	/* Load the bios pages. */
2974	cPages = pRam->cb >> PAGE_SHIFT;
2975	for (uint32_t iPage = 0; iPage < cPages; iPage++)
2976	{
2977	RTGCPHYS const GCPhysPage = ((RTGCPHYS)iPage << PAGE_SHIFT) + pRam->GCPhys;
2978	PPGMPAGE pPage = &pRam->aPages[iPage];
2979
2980	AssertLogRelMsgReturn(PGM_PAGE_GET_TYPE(pPage) == PGMPAGETYPE_ROM,
2981	("GCPhys=%RGp pPage=%R[pgmpage]\n", GCPhys, GCPhys),
2982	VERR_SSM_DATA_UNIT_FORMAT_CHANGED);
2983	rc = pgmR3LoadPageBits(pVM, pSSM, PGMPAGETYPE_ROM, pPage, GCPhysPage, pRam);
2984	AssertLogRelMsgRCReturn(rc, ("rc=%Rrc iPage=%#x GCPhys=%#x %s\n", rc, iPage, pRam->GCPhys, pRam->pszDesc), rc);
2985	}
2986	}
2987	}
2988	}
2989
2990	return rc;
2991	}
2992
2993
2994	/**
2995	* Execute state load operation.
2996	*
2997	* @returns VBox status code.
2998	* @param pVM VM Handle.
2999	* @param pSSM SSM operation handle.
3000	* @param u32Version Data layout version.
3001	*/
3002	static DECLCALLBACK(int) pgmR3Load(PVM pVM, PSSMHANDLE pSSM, uint32_t u32Version)
3003	{
3004	int rc;
3005	PPGM pPGM = &pVM->pgm.s;
3006
3007	/*
3008	* Validate version.
3009	*/
3010	if ( u32Version != PGM_SAVED_STATE_VERSION
3011	&& u32Version != PGM_SAVED_STATE_VERSION_RR_DESC
3012	&& u32Version != PGM_SAVED_STATE_VERSION_OLD_PHYS_CODE)
3013	{
3014	AssertMsgFailed(("pgmR3Load: Invalid version u32Version=%d (current %d)!\n", u32Version, PGM_SAVED_STATE_VERSION));
3015	return VERR_SSM_UNSUPPORTED_DATA_UNIT_VERSION;
3016	}
3017
3018	/*
3019	* Call the reset function to make sure all the memory is cleared.
3020	*/
3021	PGMR3Reset(pVM);
3022
3023	/*
3024	* Do the loading while owning the lock because a bunch of the functions
3025	* we're using requires this.
3026	*/
3027	pgmLock(pVM);
3028	rc = pgmR3LoadLocked(pVM, pSSM, u32Version);
3029	pgmUnlock(pVM);
3030	if (RT_SUCCESS(rc))
3031	{
3032	/*
3033	* We require a full resync now.
3034	*/
3035	for (unsigned i=0;i<pVM->cCPUs;i++)
3036	{
3037	PVMCPU pVCpu = &pVM->aCpus[i];
3038	VM_FF_SET(pVM, VM_FF_PGM_SYNC_CR3_NON_GLOBAL);
3039	VM_FF_SET(pVM, VM_FF_PGM_SYNC_CR3);
3040
3041	pVCpu->pgm.s.fSyncFlags \|= PGM_SYNC_UPDATE_PAGE_BIT_VIRTUAL;
3042	}
3043
3044	pPGM->fPhysCacheFlushPending = true;
3045	pgmR3HandlerPhysicalUpdateAll(pVM);
3046
3047	for (unsigned i=0;i<pVM->cCPUs;i++)
3048	{
3049	PVMCPU pVCpu = &pVM->aCpus[i];
3050
3051	/*
3052	* Change the paging mode.
3053	*/
3054	rc = PGMR3ChangeMode(pVM, pVCpu, pVCpu->pgm.s.enmGuestMode);
3055
3056	/* Restore pVM->pgm.s.GCPhysCR3. */
3057	Assert(pVCpu->pgm.s.GCPhysCR3 == NIL_RTGCPHYS);
3058	RTGCPHYS GCPhysCR3 = CPUMGetGuestCR3(pVCpu);
3059	if ( pVCpu->pgm.s.enmGuestMode == PGMMODE_PAE
3060	\|\| pVCpu->pgm.s.enmGuestMode == PGMMODE_PAE_NX
3061	\|\| pVCpu->pgm.s.enmGuestMode == PGMMODE_AMD64
3062	\|\| pVCpu->pgm.s.enmGuestMode == PGMMODE_AMD64_NX)
3063	GCPhysCR3 = (GCPhysCR3 & X86_CR3_PAE_PAGE_MASK);
3064	else
3065	GCPhysCR3 = (GCPhysCR3 & X86_CR3_PAGE_MASK);
3066	pVCpu->pgm.s.GCPhysCR3 = GCPhysCR3;
3067	}
3068	}
3069
3070	return rc;
3071	}
3072
3073
3074	/**
3075	* Show paging mode.
3076	*
3077	* @param pVM VM Handle.
3078	* @param pHlp The info helpers.
3079	* @param pszArgs "all" (default), "guest", "shadow" or "host".
3080	*/
3081	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
3082	{
3083	/* digest argument. */
3084	bool fGuest, fShadow, fHost;
3085	if (pszArgs)
3086	pszArgs = RTStrStripL(pszArgs);
3087	if (!pszArgs \|\| !*pszArgs \|\| strstr(pszArgs, "all"))
3088	fShadow = fHost = fGuest = true;
3089	else
3090	{
3091	fShadow = fHost = fGuest = false;
3092	if (strstr(pszArgs, "guest"))
3093	fGuest = true;
3094	if (strstr(pszArgs, "shadow"))
3095	fShadow = true;
3096	if (strstr(pszArgs, "host"))
3097	fHost = true;
3098	}
3099
3100	/** @todo SMP support! */
3101	/* print info. */
3102	if (fGuest)
3103	pHlp->pfnPrintf(pHlp, "Guest paging mode: %s, changed %RU64 times, A20 %s\n",
3104	PGMGetModeName(pVM->aCpus[0].pgm.s.enmGuestMode), pVM->aCpus[0].pgm.s.cGuestModeChanges.c,
3105	pVM->aCpus[0].pgm.s.fA20Enabled ? "enabled" : "disabled");
3106	if (fShadow)
3107	pHlp->pfnPrintf(pHlp, "Shadow paging mode: %s\n", PGMGetModeName(pVM->aCpus[0].pgm.s.enmShadowMode));
3108	if (fHost)
3109	{
3110	const char *psz;
3111	switch (pVM->pgm.s.enmHostMode)
3112	{
3113	case SUPPAGINGMODE_INVALID: psz = "invalid"; break;
3114	case SUPPAGINGMODE_32_BIT: psz = "32-bit"; break;
3115	case SUPPAGINGMODE_32_BIT_GLOBAL: psz = "32-bit+G"; break;
3116	case SUPPAGINGMODE_PAE: psz = "PAE"; break;
3117	case SUPPAGINGMODE_PAE_GLOBAL: psz = "PAE+G"; break;
3118	case SUPPAGINGMODE_PAE_NX: psz = "PAE+NX"; break;
3119	case SUPPAGINGMODE_PAE_GLOBAL_NX: psz = "PAE+G+NX"; break;
3120	case SUPPAGINGMODE_AMD64: psz = "AMD64"; break;
3121	case SUPPAGINGMODE_AMD64_GLOBAL: psz = "AMD64+G"; break;
3122	case SUPPAGINGMODE_AMD64_NX: psz = "AMD64+NX"; break;
3123	case SUPPAGINGMODE_AMD64_GLOBAL_NX: psz = "AMD64+G+NX"; break;
3124	default: psz = "unknown"; break;
3125	}
3126	pHlp->pfnPrintf(pHlp, "Host paging mode: %s\n", psz);
3127	}
3128	}
3129
3130
3131	/**
3132	* Dump registered MMIO ranges to the log.
3133	*
3134	* @param pVM VM Handle.
3135	* @param pHlp The info helpers.
3136	* @param pszArgs Arguments, ignored.
3137	*/
3138	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
3139	{
3140	NOREF(pszArgs);
3141	pHlp->pfnPrintf(pHlp,
3142	"RAM ranges (pVM=%p)\n"
3143	"%.s %.s\n",
3144	pVM,
3145	sizeof(RTGCPHYS) * 4 + 1, "GC Phys Range ",
3146	sizeof(RTHCPTR) * 2, "pvHC ");
3147
3148	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesR3; pCur; pCur = pCur->pNextR3)
3149	pHlp->pfnPrintf(pHlp,
3150	"%RGp-%RGp %RHv %s\n",
3151	pCur->GCPhys,
3152	pCur->GCPhysLast,
3153	pCur->pvR3,
3154	pCur->pszDesc);
3155	}
3156
3157	/**
3158	* Dump the page directory to the log.
3159	*
3160	* @param pVM VM Handle.
3161	* @param pHlp The info helpers.
3162	* @param pszArgs Arguments, ignored.
3163	*/
3164	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
3165	{
3166	/** @todo SMP support!! */
3167	PVMCPU pVCpu = &pVM->aCpus[0];
3168
3169	/** @todo fix this! Convert the PGMR3DumpHierarchyHC functions to do guest stuff. */
3170	/* Big pages supported? */
3171	const bool fPSE = !!(CPUMGetGuestCR4(pVCpu) & X86_CR4_PSE);
3172
3173	/* Global pages supported? */
3174	const bool fPGE = !!(CPUMGetGuestCR4(pVCpu) & X86_CR4_PGE);
3175
3176	NOREF(pszArgs);
3177
3178	/*
3179	* Get page directory addresses.
3180	*/
3181	PX86PD pPDSrc = pgmGstGet32bitPDPtr(&pVCpu->pgm.s);
3182	Assert(pPDSrc);
3183	Assert(PGMPhysGCPhys2R3PtrAssert(pVM, (RTGCPHYS)(CPUMGetGuestCR3(pVCpu) & X86_CR3_PAGE_MASK), sizeof(*pPDSrc)) == pPDSrc);
3184
3185	/*
3186	* Iterate the page directory.
3187	*/
3188	for (unsigned iPD = 0; iPD < RT_ELEMENTS(pPDSrc->a); iPD++)
3189	{
3190	X86PDE PdeSrc = pPDSrc->a[iPD];
3191	if (PdeSrc.n.u1Present)
3192	{
3193	if (PdeSrc.b.u1Size && fPSE)
3194	pHlp->pfnPrintf(pHlp,
3195	"%04X - %RGp P=%d U=%d RW=%d G=%d - BIG\n",
3196	iPD,
3197	pgmGstGet4MBPhysPage(&pVM->pgm.s, PdeSrc),
3198	PdeSrc.b.u1Present, PdeSrc.b.u1User, PdeSrc.b.u1Write, PdeSrc.b.u1Global && fPGE);
3199	else
3200	pHlp->pfnPrintf(pHlp,
3201	"%04X - %RGp P=%d U=%d RW=%d [G=%d]\n",
3202	iPD,
3203	(RTGCPHYS)(PdeSrc.u & X86_PDE_PG_MASK),
3204	PdeSrc.n.u1Present, PdeSrc.n.u1User, PdeSrc.n.u1Write, PdeSrc.b.u1Global && fPGE);
3205	}
3206	}
3207	}
3208
3209
3210	/**
3211	* Serivce a VMMCALLHOST_PGM_LOCK call.
3212	*
3213	* @returns VBox status code.
3214	* @param pVM The VM handle.
3215	*/
3216	VMMR3DECL(int) PGMR3LockCall(PVM pVM)
3217	{
3218	int rc = PDMR3CritSectEnterEx(&pVM->pgm.s.CritSect, true /* fHostCall */);
3219	AssertRC(rc);
3220	return rc;
3221	}
3222
3223
3224	/**
3225	* Converts a PGMMODE value to a PGM_TYPE_* \#define.
3226	*
3227	* @returns PGM_TYPE_*.
3228	* @param pgmMode The mode value to convert.
3229	*/
3230	DECLINLINE(unsigned) pgmModeToType(PGMMODE pgmMode)
3231	{
3232	switch (pgmMode)
3233	{
3234	case PGMMODE_REAL: return PGM_TYPE_REAL;
3235	case PGMMODE_PROTECTED: return PGM_TYPE_PROT;
3236	case PGMMODE_32_BIT: return PGM_TYPE_32BIT;
3237	case PGMMODE_PAE:
3238	case PGMMODE_PAE_NX: return PGM_TYPE_PAE;
3239	case PGMMODE_AMD64:
3240	case PGMMODE_AMD64_NX: return PGM_TYPE_AMD64;
3241	case PGMMODE_NESTED: return PGM_TYPE_NESTED;
3242	case PGMMODE_EPT: return PGM_TYPE_EPT;
3243	default:
3244	AssertFatalMsgFailed(("pgmMode=%d\n", pgmMode));
3245	}
3246	}
3247
3248
3249	/**
3250	* Gets the index into the paging mode data array of a SHW+GST mode.
3251	*
3252	* @returns PGM::paPagingData index.
3253	* @param uShwType The shadow paging mode type.
3254	* @param uGstType The guest paging mode type.
3255	*/
3256	DECLINLINE(unsigned) pgmModeDataIndex(unsigned uShwType, unsigned uGstType)
3257	{
3258	Assert(uShwType >= PGM_TYPE_32BIT && uShwType <= PGM_TYPE_MAX);
3259	Assert(uGstType >= PGM_TYPE_REAL && uGstType <= PGM_TYPE_AMD64);
3260	return (uShwType - PGM_TYPE_32BIT) * (PGM_TYPE_AMD64 - PGM_TYPE_REAL + 1)
3261	+ (uGstType - PGM_TYPE_REAL);
3262	}
3263
3264
3265	/**
3266	* Gets the index into the paging mode data array of a SHW+GST mode.
3267	*
3268	* @returns PGM::paPagingData index.
3269	* @param enmShw The shadow paging mode.
3270	* @param enmGst The guest paging mode.
3271	*/
3272	DECLINLINE(unsigned) pgmModeDataIndexByMode(PGMMODE enmShw, PGMMODE enmGst)
3273	{
3274	Assert(enmShw >= PGMMODE_32_BIT && enmShw <= PGMMODE_MAX);
3275	Assert(enmGst > PGMMODE_INVALID && enmGst < PGMMODE_MAX);
3276	return pgmModeDataIndex(pgmModeToType(enmShw), pgmModeToType(enmGst));
3277	}
3278
3279
3280	/**
3281	* Calculates the max data index.
3282	* @returns The number of entries in the paging data array.
3283	*/
3284	DECLINLINE(unsigned) pgmModeDataMaxIndex(void)
3285	{
3286	return pgmModeDataIndex(PGM_TYPE_MAX, PGM_TYPE_AMD64) + 1;
3287	}
3288
3289
3290	/**
3291	* Initializes the paging mode data kept in PGM::paModeData.
3292	*
3293	* @param pVM The VM handle.
3294	* @param fResolveGCAndR0 Indicate whether or not GC and Ring-0 symbols can be resolved now.
3295	* This is used early in the init process to avoid trouble with PDM
3296	* not being initialized yet.
3297	*/
3298	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0)
3299	{
3300	PPGMMODEDATA pModeData;
3301	int rc;
3302
3303	/*
3304	* Allocate the array on the first call.
3305	*/
3306	if (!pVM->pgm.s.paModeData)
3307	{
3308	pVM->pgm.s.paModeData = (PPGMMODEDATA)MMR3HeapAllocZ(pVM, MM_TAG_PGM, sizeof(PGMMODEDATA) * pgmModeDataMaxIndex());
3309	AssertReturn(pVM->pgm.s.paModeData, VERR_NO_MEMORY);
3310	}
3311
3312	/*
3313	* Initialize the array entries.
3314	*/
3315	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_REAL)];
3316	pModeData->uShwType = PGM_TYPE_32BIT;
3317	pModeData->uGstType = PGM_TYPE_REAL;
3318	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3319	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3320	rc = PGM_BTH_NAME_32BIT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3321
3322	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGMMODE_PROTECTED)];
3323	pModeData->uShwType = PGM_TYPE_32BIT;
3324	pModeData->uGstType = PGM_TYPE_PROT;
3325	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3326	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3327	rc = PGM_BTH_NAME_32BIT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3328
3329	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_32BIT)];
3330	pModeData->uShwType = PGM_TYPE_32BIT;
3331	pModeData->uGstType = PGM_TYPE_32BIT;
3332	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3333	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3334	rc = PGM_BTH_NAME_32BIT_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3335
3336	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_REAL)];
3337	pModeData->uShwType = PGM_TYPE_PAE;
3338	pModeData->uGstType = PGM_TYPE_REAL;
3339	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3340	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3341	rc = PGM_BTH_NAME_PAE_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3342
3343	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PROT)];
3344	pModeData->uShwType = PGM_TYPE_PAE;
3345	pModeData->uGstType = PGM_TYPE_PROT;
3346	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3347	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3348	rc = PGM_BTH_NAME_PAE_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3349
3350	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_32BIT)];
3351	pModeData->uShwType = PGM_TYPE_PAE;
3352	pModeData->uGstType = PGM_TYPE_32BIT;
3353	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3354	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3355	rc = PGM_BTH_NAME_PAE_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3356
3357	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PAE)];
3358	pModeData->uShwType = PGM_TYPE_PAE;
3359	pModeData->uGstType = PGM_TYPE_PAE;
3360	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3361	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3362	rc = PGM_BTH_NAME_PAE_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3363
3364	#ifdef VBOX_WITH_64_BITS_GUESTS
3365	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_AMD64, PGM_TYPE_AMD64)];
3366	pModeData->uShwType = PGM_TYPE_AMD64;
3367	pModeData->uGstType = PGM_TYPE_AMD64;
3368	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3369	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3370	rc = PGM_BTH_NAME_AMD64_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3371	#endif
3372
3373	/* The nested paging mode. */
3374	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_REAL)];
3375	pModeData->uShwType = PGM_TYPE_NESTED;
3376	pModeData->uGstType = PGM_TYPE_REAL;
3377	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3378	rc = PGM_BTH_NAME_NESTED_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3379
3380	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGMMODE_PROTECTED)];
3381	pModeData->uShwType = PGM_TYPE_NESTED;
3382	pModeData->uGstType = PGM_TYPE_PROT;
3383	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3384	rc = PGM_BTH_NAME_NESTED_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3385
3386	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_32BIT)];
3387	pModeData->uShwType = PGM_TYPE_NESTED;
3388	pModeData->uGstType = PGM_TYPE_32BIT;
3389	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3390	rc = PGM_BTH_NAME_NESTED_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3391
3392	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_PAE)];
3393	pModeData->uShwType = PGM_TYPE_NESTED;
3394	pModeData->uGstType = PGM_TYPE_PAE;
3395	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3396	rc = PGM_BTH_NAME_NESTED_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3397
3398	#ifdef VBOX_WITH_64_BITS_GUESTS
3399	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
3400	pModeData->uShwType = PGM_TYPE_NESTED;
3401	pModeData->uGstType = PGM_TYPE_AMD64;
3402	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3403	rc = PGM_BTH_NAME_NESTED_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3404	#endif
3405
3406	/* The shadow part of the nested callback mode depends on the host paging mode (AMD-V only). */
3407	switch (pVM->pgm.s.enmHostMode)
3408	{
3409	#if HC_ARCH_BITS == 32
3410	case SUPPAGINGMODE_32_BIT:
3411	case SUPPAGINGMODE_32_BIT_GLOBAL:
3412	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
3413	{
3414	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
3415	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3416	}
3417	# ifdef VBOX_WITH_64_BITS_GUESTS
3418	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
3419	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3420	# endif
3421	break;
3422
3423	case SUPPAGINGMODE_PAE:
3424	case SUPPAGINGMODE_PAE_NX:
3425	case SUPPAGINGMODE_PAE_GLOBAL:
3426	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3427	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
3428	{
3429	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
3430	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3431	}
3432	# ifdef VBOX_WITH_64_BITS_GUESTS
3433	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
3434	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3435	# endif
3436	break;
3437	#endif /* HC_ARCH_BITS == 32 */
3438
3439	#if HC_ARCH_BITS == 64 \|\| defined(RT_OS_DARWIN)
3440	case SUPPAGINGMODE_AMD64:
3441	case SUPPAGINGMODE_AMD64_GLOBAL:
3442	case SUPPAGINGMODE_AMD64_NX:
3443	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3444	# ifdef VBOX_WITH_64_BITS_GUESTS
3445	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_AMD64; i++)
3446	# else
3447	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
3448	# endif
3449	{
3450	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
3451	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3452	}
3453	break;
3454	#endif /* HC_ARCH_BITS == 64 \|\| RT_OS_DARWIN */
3455
3456	default:
3457	AssertFailed();
3458	break;
3459	}
3460
3461	/* Extended paging (EPT) / Intel VT-x */
3462	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_REAL)];
3463	pModeData->uShwType = PGM_TYPE_EPT;
3464	pModeData->uGstType = PGM_TYPE_REAL;
3465	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3466	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3467	rc = PGM_BTH_NAME_EPT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3468
3469	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_PROT)];
3470	pModeData->uShwType = PGM_TYPE_EPT;
3471	pModeData->uGstType = PGM_TYPE_PROT;
3472	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3473	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3474	rc = PGM_BTH_NAME_EPT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3475
3476	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_32BIT)];
3477	pModeData->uShwType = PGM_TYPE_EPT;
3478	pModeData->uGstType = PGM_TYPE_32BIT;
3479	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3480	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3481	rc = PGM_BTH_NAME_EPT_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3482
3483	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_PAE)];
3484	pModeData->uShwType = PGM_TYPE_EPT;
3485	pModeData->uGstType = PGM_TYPE_PAE;
3486	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3487	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3488	rc = PGM_BTH_NAME_EPT_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3489
3490	#ifdef VBOX_WITH_64_BITS_GUESTS
3491	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_AMD64)];
3492	pModeData->uShwType = PGM_TYPE_EPT;
3493	pModeData->uGstType = PGM_TYPE_AMD64;
3494	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3495	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3496	rc = PGM_BTH_NAME_EPT_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3497	#endif
3498	return VINF_SUCCESS;
3499	}
3500
3501
3502	/**
3503	* Switch to different (or relocated in the relocate case) mode data.
3504	*
3505	* @param pVM The VM handle.
3506	* @param pVCpu The VMCPU to operate on.
3507	* @param enmShw The the shadow paging mode.
3508	* @param enmGst The the guest paging mode.
3509	*/
3510	static void pgmR3ModeDataSwitch(PVM pVM, PVMCPU pVCpu, PGMMODE enmShw, PGMMODE enmGst)
3511	{
3512	PPGMMODEDATA pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndexByMode(enmShw, enmGst)];
3513
3514	Assert(pModeData->uGstType == pgmModeToType(enmGst));
3515	Assert(pModeData->uShwType == pgmModeToType(enmShw));
3516
3517	/* shadow */
3518	pVCpu->pgm.s.pfnR3ShwRelocate = pModeData->pfnR3ShwRelocate;
3519	pVCpu->pgm.s.pfnR3ShwExit = pModeData->pfnR3ShwExit;
3520	pVCpu->pgm.s.pfnR3ShwGetPage = pModeData->pfnR3ShwGetPage;
3521	Assert(pVCpu->pgm.s.pfnR3ShwGetPage);
3522	pVCpu->pgm.s.pfnR3ShwModifyPage = pModeData->pfnR3ShwModifyPage;
3523
3524	pVCpu->pgm.s.pfnRCShwGetPage = pModeData->pfnRCShwGetPage;
3525	pVCpu->pgm.s.pfnRCShwModifyPage = pModeData->pfnRCShwModifyPage;
3526
3527	pVCpu->pgm.s.pfnR0ShwGetPage = pModeData->pfnR0ShwGetPage;
3528	pVCpu->pgm.s.pfnR0ShwModifyPage = pModeData->pfnR0ShwModifyPage;
3529
3530
3531	/* guest */
3532	pVCpu->pgm.s.pfnR3GstRelocate = pModeData->pfnR3GstRelocate;
3533	pVCpu->pgm.s.pfnR3GstExit = pModeData->pfnR3GstExit;
3534	pVCpu->pgm.s.pfnR3GstGetPage = pModeData->pfnR3GstGetPage;
3535	Assert(pVCpu->pgm.s.pfnR3GstGetPage);
3536	pVCpu->pgm.s.pfnR3GstModifyPage = pModeData->pfnR3GstModifyPage;
3537	pVCpu->pgm.s.pfnR3GstGetPDE = pModeData->pfnR3GstGetPDE;
3538	pVCpu->pgm.s.pfnRCGstGetPage = pModeData->pfnRCGstGetPage;
3539	pVCpu->pgm.s.pfnRCGstModifyPage = pModeData->pfnRCGstModifyPage;
3540	pVCpu->pgm.s.pfnRCGstGetPDE = pModeData->pfnRCGstGetPDE;
3541	pVCpu->pgm.s.pfnR0GstGetPage = pModeData->pfnR0GstGetPage;
3542	pVCpu->pgm.s.pfnR0GstModifyPage = pModeData->pfnR0GstModifyPage;
3543	pVCpu->pgm.s.pfnR0GstGetPDE = pModeData->pfnR0GstGetPDE;
3544
3545	/* both */
3546	pVCpu->pgm.s.pfnR3BthRelocate = pModeData->pfnR3BthRelocate;
3547	pVCpu->pgm.s.pfnR3BthInvalidatePage = pModeData->pfnR3BthInvalidatePage;
3548	pVCpu->pgm.s.pfnR3BthSyncCR3 = pModeData->pfnR3BthSyncCR3;
3549	Assert(pVCpu->pgm.s.pfnR3BthSyncCR3);
3550	pVCpu->pgm.s.pfnR3BthSyncPage = pModeData->pfnR3BthSyncPage;
3551	pVCpu->pgm.s.pfnR3BthPrefetchPage = pModeData->pfnR3BthPrefetchPage;
3552	pVCpu->pgm.s.pfnR3BthVerifyAccessSyncPage = pModeData->pfnR3BthVerifyAccessSyncPage;
3553	#ifdef VBOX_STRICT
3554	pVCpu->pgm.s.pfnR3BthAssertCR3 = pModeData->pfnR3BthAssertCR3;
3555	#endif
3556	pVCpu->pgm.s.pfnR3BthMapCR3 = pModeData->pfnR3BthMapCR3;
3557	pVCpu->pgm.s.pfnR3BthUnmapCR3 = pModeData->pfnR3BthUnmapCR3;
3558
3559	pVCpu->pgm.s.pfnRCBthTrap0eHandler = pModeData->pfnRCBthTrap0eHandler;
3560	pVCpu->pgm.s.pfnRCBthInvalidatePage = pModeData->pfnRCBthInvalidatePage;
3561	pVCpu->pgm.s.pfnRCBthSyncCR3 = pModeData->pfnRCBthSyncCR3;
3562	pVCpu->pgm.s.pfnRCBthSyncPage = pModeData->pfnRCBthSyncPage;
3563	pVCpu->pgm.s.pfnRCBthPrefetchPage = pModeData->pfnRCBthPrefetchPage;
3564	pVCpu->pgm.s.pfnRCBthVerifyAccessSyncPage = pModeData->pfnRCBthVerifyAccessSyncPage;
3565	#ifdef VBOX_STRICT
3566	pVCpu->pgm.s.pfnRCBthAssertCR3 = pModeData->pfnRCBthAssertCR3;
3567	#endif
3568	pVCpu->pgm.s.pfnRCBthMapCR3 = pModeData->pfnRCBthMapCR3;
3569	pVCpu->pgm.s.pfnRCBthUnmapCR3 = pModeData->pfnRCBthUnmapCR3;
3570
3571	pVCpu->pgm.s.pfnR0BthTrap0eHandler = pModeData->pfnR0BthTrap0eHandler;
3572	pVCpu->pgm.s.pfnR0BthInvalidatePage = pModeData->pfnR0BthInvalidatePage;
3573	pVCpu->pgm.s.pfnR0BthSyncCR3 = pModeData->pfnR0BthSyncCR3;
3574	pVCpu->pgm.s.pfnR0BthSyncPage = pModeData->pfnR0BthSyncPage;
3575	pVCpu->pgm.s.pfnR0BthPrefetchPage = pModeData->pfnR0BthPrefetchPage;
3576	pVCpu->pgm.s.pfnR0BthVerifyAccessSyncPage = pModeData->pfnR0BthVerifyAccessSyncPage;
3577	#ifdef VBOX_STRICT
3578	pVCpu->pgm.s.pfnR0BthAssertCR3 = pModeData->pfnR0BthAssertCR3;
3579	#endif
3580	pVCpu->pgm.s.pfnR0BthMapCR3 = pModeData->pfnR0BthMapCR3;
3581	pVCpu->pgm.s.pfnR0BthUnmapCR3 = pModeData->pfnR0BthUnmapCR3;
3582	}
3583
3584
3585	/**
3586	* Calculates the shadow paging mode.
3587	*
3588	* @returns The shadow paging mode.
3589	* @param pVM VM handle.
3590	* @param enmGuestMode The guest mode.
3591	* @param enmHostMode The host mode.
3592	* @param enmShadowMode The current shadow mode.
3593	* @param penmSwitcher Where to store the switcher to use.
3594	* VMMSWITCHER_INVALID means no change.
3595	*/
3596	static PGMMODE pgmR3CalcShadowMode(PVM pVM, PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher)
3597	{
3598	VMMSWITCHER enmSwitcher = VMMSWITCHER_INVALID;
3599	switch (enmGuestMode)
3600	{
3601	/*
3602	* When switching to real or protected mode we don't change
3603	* anything since it's likely that we'll switch back pretty soon.
3604	*
3605	* During pgmR3InitPaging we'll end up here with PGMMODE_INVALID
3606	* and is supposed to determine which shadow paging and switcher to
3607	* use during init.
3608	*/
3609	case PGMMODE_REAL:
3610	case PGMMODE_PROTECTED:
3611	if ( enmShadowMode != PGMMODE_INVALID
3612	&& !HWACCMIsEnabled(pVM) /* always switch in hwaccm mode! */)
3613	break; /* (no change) */
3614
3615	switch (enmHostMode)
3616	{
3617	case SUPPAGINGMODE_32_BIT:
3618	case SUPPAGINGMODE_32_BIT_GLOBAL:
3619	enmShadowMode = PGMMODE_32_BIT;
3620	enmSwitcher = VMMSWITCHER_32_TO_32;
3621	break;
3622
3623	case SUPPAGINGMODE_PAE:
3624	case SUPPAGINGMODE_PAE_NX:
3625	case SUPPAGINGMODE_PAE_GLOBAL:
3626	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3627	enmShadowMode = PGMMODE_PAE;
3628	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3629	#ifdef DEBUG_bird
3630	if (RTEnvExist("VBOX_32BIT"))
3631	{
3632	enmShadowMode = PGMMODE_32_BIT;
3633	enmSwitcher = VMMSWITCHER_PAE_TO_32;
3634	}
3635	#endif
3636	break;
3637
3638	case SUPPAGINGMODE_AMD64:
3639	case SUPPAGINGMODE_AMD64_GLOBAL:
3640	case SUPPAGINGMODE_AMD64_NX:
3641	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3642	enmShadowMode = PGMMODE_PAE;
3643	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3644	#ifdef DEBUG_bird
3645	if (RTEnvExist("VBOX_32BIT"))
3646	{
3647	enmShadowMode = PGMMODE_32_BIT;
3648	enmSwitcher = VMMSWITCHER_AMD64_TO_32;
3649	}
3650	#endif
3651	break;
3652
3653	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3654	}
3655	break;
3656
3657	case PGMMODE_32_BIT:
3658	switch (enmHostMode)
3659	{
3660	case SUPPAGINGMODE_32_BIT:
3661	case SUPPAGINGMODE_32_BIT_GLOBAL:
3662	enmShadowMode = PGMMODE_32_BIT;
3663	enmSwitcher = VMMSWITCHER_32_TO_32;
3664	break;
3665
3666	case SUPPAGINGMODE_PAE:
3667	case SUPPAGINGMODE_PAE_NX:
3668	case SUPPAGINGMODE_PAE_GLOBAL:
3669	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3670	enmShadowMode = PGMMODE_PAE;
3671	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3672	#ifdef DEBUG_bird
3673	if (RTEnvExist("VBOX_32BIT"))
3674	{
3675	enmShadowMode = PGMMODE_32_BIT;
3676	enmSwitcher = VMMSWITCHER_PAE_TO_32;
3677	}
3678	#endif
3679	break;
3680
3681	case SUPPAGINGMODE_AMD64:
3682	case SUPPAGINGMODE_AMD64_GLOBAL:
3683	case SUPPAGINGMODE_AMD64_NX:
3684	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3685	enmShadowMode = PGMMODE_PAE;
3686	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3687	#ifdef DEBUG_bird
3688	if (RTEnvExist("VBOX_32BIT"))
3689	{
3690	enmShadowMode = PGMMODE_32_BIT;
3691	enmSwitcher = VMMSWITCHER_AMD64_TO_32;
3692	}
3693	#endif
3694	break;
3695
3696	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3697	}
3698	break;
3699
3700	case PGMMODE_PAE:
3701	case PGMMODE_PAE_NX: /** @todo This might require more switchers and guest+both modes. */
3702	switch (enmHostMode)
3703	{
3704	case SUPPAGINGMODE_32_BIT:
3705	case SUPPAGINGMODE_32_BIT_GLOBAL:
3706	enmShadowMode = PGMMODE_PAE;
3707	enmSwitcher = VMMSWITCHER_32_TO_PAE;
3708	break;
3709
3710	case SUPPAGINGMODE_PAE:
3711	case SUPPAGINGMODE_PAE_NX:
3712	case SUPPAGINGMODE_PAE_GLOBAL:
3713	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3714	enmShadowMode = PGMMODE_PAE;
3715	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3716	break;
3717
3718	case SUPPAGINGMODE_AMD64:
3719	case SUPPAGINGMODE_AMD64_GLOBAL:
3720	case SUPPAGINGMODE_AMD64_NX:
3721	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3722	enmShadowMode = PGMMODE_PAE;
3723	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3724	break;
3725
3726	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3727	}
3728	break;
3729
3730	case PGMMODE_AMD64:
3731	case PGMMODE_AMD64_NX:
3732	switch (enmHostMode)
3733	{
3734	case SUPPAGINGMODE_32_BIT:
3735	case SUPPAGINGMODE_32_BIT_GLOBAL:
3736	enmShadowMode = PGMMODE_AMD64;
3737	enmSwitcher = VMMSWITCHER_32_TO_AMD64;
3738	break;
3739
3740	case SUPPAGINGMODE_PAE:
3741	case SUPPAGINGMODE_PAE_NX:
3742	case SUPPAGINGMODE_PAE_GLOBAL:
3743	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3744	enmShadowMode = PGMMODE_AMD64;
3745	enmSwitcher = VMMSWITCHER_PAE_TO_AMD64;
3746	break;
3747
3748	case SUPPAGINGMODE_AMD64:
3749	case SUPPAGINGMODE_AMD64_GLOBAL:
3750	case SUPPAGINGMODE_AMD64_NX:
3751	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3752	enmShadowMode = PGMMODE_AMD64;
3753	enmSwitcher = VMMSWITCHER_AMD64_TO_AMD64;
3754	break;
3755
3756	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3757	}
3758	break;
3759
3760
3761	default:
3762	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
3763	return PGMMODE_INVALID;
3764	}
3765	/* Override the shadow mode is nested paging is active. */
3766	if (HWACCMIsNestedPagingActive(pVM))
3767	enmShadowMode = HWACCMGetShwPagingMode(pVM);
3768
3769	*penmSwitcher = enmSwitcher;
3770	return enmShadowMode;
3771	}
3772
3773
3774	/**
3775	* Performs the actual mode change.
3776	* This is called by PGMChangeMode and pgmR3InitPaging().
3777	*
3778	* @returns VBox status code. May suspend or power off the VM on error, but this
3779	* will trigger using FFs and not status codes.
3780	*
3781	* @param pVM VM handle.
3782	* @param pVCpu The VMCPU to operate on.
3783	* @param enmGuestMode The new guest mode. This is assumed to be different from
3784	* the current mode.
3785	*/
3786	VMMR3DECL(int) PGMR3ChangeMode(PVM pVM, PVMCPU pVCpu, PGMMODE enmGuestMode)
3787	{
3788	Log(("PGMR3ChangeMode: Guest mode: %s -> %s\n", PGMGetModeName(pVCpu->pgm.s.enmGuestMode), PGMGetModeName(enmGuestMode)));
3789	STAM_REL_COUNTER_INC(&pVCpu->pgm.s.cGuestModeChanges);
3790
3791	/*
3792	* Calc the shadow mode and switcher.
3793	*/
3794	VMMSWITCHER enmSwitcher;
3795	PGMMODE enmShadowMode = pgmR3CalcShadowMode(pVM, enmGuestMode, pVM->pgm.s.enmHostMode, pVCpu->pgm.s.enmShadowMode, &enmSwitcher);
3796	if (enmSwitcher != VMMSWITCHER_INVALID)
3797	{
3798	/*
3799	* Select new switcher.
3800	*/
3801	int rc = VMMR3SelectSwitcher(pVM, enmSwitcher);
3802	if (RT_FAILURE(rc))
3803	{
3804	AssertReleaseMsgFailed(("VMMR3SelectSwitcher(%d) -> %Rrc\n", enmSwitcher, rc));
3805	return rc;
3806	}
3807	}
3808
3809	/*
3810	* Exit old mode(s).
3811	*/
3812	/* shadow */
3813	if (enmShadowMode != pVCpu->pgm.s.enmShadowMode)
3814	{
3815	LogFlow(("PGMR3ChangeMode: Shadow mode: %s -> %s\n", PGMGetModeName(pVCpu->pgm.s.enmShadowMode), PGMGetModeName(enmShadowMode)));
3816	if (PGM_SHW_PFN(Exit, pVM))
3817	{
3818	int rc = PGM_SHW_PFN(Exit, pVM)(pVM, pVCpu);
3819	if (RT_FAILURE(rc))
3820	{
3821	AssertMsgFailed(("Exit failed for shadow mode %d: %Rrc\n", pVCpu->pgm.s.enmShadowMode, rc));
3822	return rc;
3823	}
3824	}
3825
3826	}
3827	else
3828	LogFlow(("PGMR3ChangeMode: Shadow mode remains: %s\n", PGMGetModeName(pVCpu->pgm.s.enmShadowMode)));
3829
3830	/* guest */
3831	if (PGM_GST_PFN(Exit, pVM))
3832	{
3833	int rc = PGM_GST_PFN(Exit, pVM)(pVM, pVCpu);
3834	if (RT_FAILURE(rc))
3835	{
3836	AssertMsgFailed(("Exit failed for guest mode %d: %Rrc\n", pVCpu->pgm.s.enmGuestMode, rc));
3837	return rc;
3838	}
3839	}
3840
3841	/*
3842	* Load new paging mode data.
3843	*/
3844	pgmR3ModeDataSwitch(pVM, pVCpu, enmShadowMode, enmGuestMode);
3845
3846	/*
3847	* Enter new shadow mode (if changed).
3848	*/
3849	if (enmShadowMode != pVCpu->pgm.s.enmShadowMode)
3850	{
3851	int rc;
3852	pVCpu->pgm.s.enmShadowMode = enmShadowMode;
3853	switch (enmShadowMode)
3854	{
3855	case PGMMODE_32_BIT:
3856	rc = PGM_SHW_NAME_32BIT(Enter)(pVM, pVCpu);
3857	break;
3858	case PGMMODE_PAE:
3859	case PGMMODE_PAE_NX:
3860	rc = PGM_SHW_NAME_PAE(Enter)(pVM, pVCpu);
3861	break;
3862	case PGMMODE_AMD64:
3863	case PGMMODE_AMD64_NX:
3864	rc = PGM_SHW_NAME_AMD64(Enter)(pVM, pVCpu);
3865	break;
3866	case PGMMODE_NESTED:
3867	rc = PGM_SHW_NAME_NESTED(Enter)(pVM, pVCpu);
3868	break;
3869	case PGMMODE_EPT:
3870	rc = PGM_SHW_NAME_EPT(Enter)(pVM, pVCpu);
3871	break;
3872	case PGMMODE_REAL:
3873	case PGMMODE_PROTECTED:
3874	default:
3875	AssertReleaseMsgFailed(("enmShadowMode=%d\n", enmShadowMode));
3876	return VERR_INTERNAL_ERROR;
3877	}
3878	if (RT_FAILURE(rc))
3879	{
3880	AssertReleaseMsgFailed(("Entering enmShadowMode=%d failed: %Rrc\n", enmShadowMode, rc));
3881	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
3882	return rc;
3883	}
3884	}
3885
3886	/*
3887	* Always flag the necessary updates
3888	*/
3889	VM_FF_SET(pVM, VM_FF_PGM_SYNC_CR3);
3890
3891	/*
3892	* Enter the new guest and shadow+guest modes.
3893	*/
3894	int rc = -1;
3895	int rc2 = -1;
3896	RTGCPHYS GCPhysCR3 = NIL_RTGCPHYS;
3897	pVCpu->pgm.s.enmGuestMode = enmGuestMode;
3898	switch (enmGuestMode)
3899	{
3900	case PGMMODE_REAL:
3901	rc = PGM_GST_NAME_REAL(Enter)(pVM, pVCpu, NIL_RTGCPHYS);
3902	switch (pVCpu->pgm.s.enmShadowMode)
3903	{
3904	case PGMMODE_32_BIT:
3905	rc2 = PGM_BTH_NAME_32BIT_REAL(Enter)(pVM, pVCpu, NIL_RTGCPHYS);
3906	break;
3907	case PGMMODE_PAE:
3908	case PGMMODE_PAE_NX:
3909	rc2 = PGM_BTH_NAME_PAE_REAL(Enter)(pVM, pVCpu, NIL_RTGCPHYS);
3910	break;
3911	case PGMMODE_NESTED:
3912	rc2 = PGM_BTH_NAME_NESTED_REAL(Enter)(pVM, pVCpu, NIL_RTGCPHYS);
3913	break;
3914	case PGMMODE_EPT:
3915	rc2 = PGM_BTH_NAME_EPT_REAL(Enter)(pVM, pVCpu, NIL_RTGCPHYS);
3916	break;
3917	case PGMMODE_AMD64:
3918	case PGMMODE_AMD64_NX:
3919	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3920	default: AssertFailed(); break;
3921	}
3922	break;
3923
3924	case PGMMODE_PROTECTED:
3925	rc = PGM_GST_NAME_PROT(Enter)(pVM, pVCpu, NIL_RTGCPHYS);
3926	switch (pVCpu->pgm.s.enmShadowMode)
3927	{
3928	case PGMMODE_32_BIT:
3929	rc2 = PGM_BTH_NAME_32BIT_PROT(Enter)(pVM, pVCpu, NIL_RTGCPHYS);
3930	break;
3931	case PGMMODE_PAE:
3932	case PGMMODE_PAE_NX:
3933	rc2 = PGM_BTH_NAME_PAE_PROT(Enter)(pVM, pVCpu, NIL_RTGCPHYS);
3934	break;
3935	case PGMMODE_NESTED:
3936	rc2 = PGM_BTH_NAME_NESTED_PROT(Enter)(pVM, pVCpu, NIL_RTGCPHYS);
3937	break;
3938	case PGMMODE_EPT:
3939	rc2 = PGM_BTH_NAME_EPT_PROT(Enter)(pVM, pVCpu, NIL_RTGCPHYS);
3940	break;
3941	case PGMMODE_AMD64:
3942	case PGMMODE_AMD64_NX:
3943	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3944	default: AssertFailed(); break;
3945	}
3946	break;
3947
3948	case PGMMODE_32_BIT:
3949	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & X86_CR3_PAGE_MASK;
3950	rc = PGM_GST_NAME_32BIT(Enter)(pVM, pVCpu, GCPhysCR3);
3951	switch (pVCpu->pgm.s.enmShadowMode)
3952	{
3953	case PGMMODE_32_BIT:
3954	rc2 = PGM_BTH_NAME_32BIT_32BIT(Enter)(pVM, pVCpu, GCPhysCR3);
3955	break;
3956	case PGMMODE_PAE:
3957	case PGMMODE_PAE_NX:
3958	rc2 = PGM_BTH_NAME_PAE_32BIT(Enter)(pVM, pVCpu, GCPhysCR3);
3959	break;
3960	case PGMMODE_NESTED:
3961	rc2 = PGM_BTH_NAME_NESTED_32BIT(Enter)(pVM, pVCpu, GCPhysCR3);
3962	break;
3963	case PGMMODE_EPT:
3964	rc2 = PGM_BTH_NAME_EPT_32BIT(Enter)(pVM, pVCpu, GCPhysCR3);
3965	break;
3966	case PGMMODE_AMD64:
3967	case PGMMODE_AMD64_NX:
3968	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3969	default: AssertFailed(); break;
3970	}
3971	break;
3972
3973	case PGMMODE_PAE_NX:
3974	case PGMMODE_PAE:
3975	{
3976	uint32_t u32Dummy, u32Features;
3977
3978	CPUMGetGuestCpuId(pVM, 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
3979	if (!(u32Features & X86_CPUID_FEATURE_EDX_PAE))
3980	return VMSetRuntimeError(pVM, VMSETRTERR_FLAGS_FATAL, "PAEmode",
3981	N_("The guest is trying to switch to the PAE mode which is currently disabled by default in VirtualBox. PAE support can be enabled using the VM settings (General/Advanced)"));
3982
3983	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & X86_CR3_PAE_PAGE_MASK;
3984	rc = PGM_GST_NAME_PAE(Enter)(pVM, pVCpu, GCPhysCR3);
3985	switch (pVCpu->pgm.s.enmShadowMode)
3986	{
3987	case PGMMODE_PAE:
3988	case PGMMODE_PAE_NX:
3989	rc2 = PGM_BTH_NAME_PAE_PAE(Enter)(pVM, pVCpu, GCPhysCR3);
3990	break;
3991	case PGMMODE_NESTED:
3992	rc2 = PGM_BTH_NAME_NESTED_PAE(Enter)(pVM, pVCpu, GCPhysCR3);
3993	break;
3994	case PGMMODE_EPT:
3995	rc2 = PGM_BTH_NAME_EPT_PAE(Enter)(pVM, pVCpu, GCPhysCR3);
3996	break;
3997	case PGMMODE_32_BIT:
3998	case PGMMODE_AMD64:
3999	case PGMMODE_AMD64_NX:
4000	AssertMsgFailed(("Should use PAE shadow mode!\n"));
4001	default: AssertFailed(); break;
4002	}
4003	break;
4004	}
4005
4006	#ifdef VBOX_WITH_64_BITS_GUESTS
4007	case PGMMODE_AMD64_NX:
4008	case PGMMODE_AMD64:
4009	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & UINT64_C(0xfffffffffffff000); /** @todo define this mask! */
4010	rc = PGM_GST_NAME_AMD64(Enter)(pVM, pVCpu, GCPhysCR3);
4011	switch (pVCpu->pgm.s.enmShadowMode)
4012	{
4013	case PGMMODE_AMD64:
4014	case PGMMODE_AMD64_NX:
4015	rc2 = PGM_BTH_NAME_AMD64_AMD64(Enter)(pVM, pVCpu, GCPhysCR3);
4016	break;
4017	case PGMMODE_NESTED:
4018	rc2 = PGM_BTH_NAME_NESTED_AMD64(Enter)(pVM, pVCpu, GCPhysCR3);
4019	break;
4020	case PGMMODE_EPT:
4021	rc2 = PGM_BTH_NAME_EPT_AMD64(Enter)(pVM, pVCpu, GCPhysCR3);
4022	break;
4023	case PGMMODE_32_BIT:
4024	case PGMMODE_PAE:
4025	case PGMMODE_PAE_NX:
4026	AssertMsgFailed(("Should use AMD64 shadow mode!\n"));
4027	default: AssertFailed(); break;
4028	}
4029	break;
4030	#endif
4031
4032	default:
4033	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
4034	rc = VERR_NOT_IMPLEMENTED;
4035	break;
4036	}
4037
4038	/* status codes. */
4039	AssertRC(rc);
4040	AssertRC(rc2);
4041	if (RT_SUCCESS(rc))
4042	{
4043	rc = rc2;
4044	if (RT_SUCCESS(rc)) /* no informational status codes. */
4045	rc = VINF_SUCCESS;
4046	}
4047
4048	/* Notify HWACCM as well. */
4049	HWACCMR3PagingModeChanged(pVM, pVCpu, pVCpu->pgm.s.enmShadowMode, pVCpu->pgm.s.enmGuestMode);
4050	return rc;
4051	}
4052
4053
4054	/**
4055	* Dumps a PAE shadow page table.
4056	*
4057	* @returns VBox status code (VINF_SUCCESS).
4058	* @param pVM The VM handle.
4059	* @param pPT Pointer to the page table.
4060	* @param u64Address The virtual address of the page table starts.
4061	* @param fLongMode Set if this a long mode table; clear if it's a legacy mode table.
4062	* @param cMaxDepth The maxium depth.
4063	* @param pHlp Pointer to the output functions.
4064	*/
4065	static int pgmR3DumpHierarchyHCPaePT(PVM pVM, PX86PTPAE pPT, uint64_t u64Address, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
4066	{
4067	for (unsigned i = 0; i < RT_ELEMENTS(pPT->a); i++)
4068	{
4069	X86PTEPAE Pte = pPT->a[i];
4070	if (Pte.n.u1Present)
4071	{
4072	pHlp->pfnPrintf(pHlp,
4073	fLongMode /P R S A D G WT CD AT NX 4M a p ? /
4074	? "%016llx 3 \| P %c %c %c %c %c %s %s %s %s 4K %c%c%c %016llx\n"
4075	: "%08llx 2 \| P %c %c %c %c %c %s %s %s %s 4K %c%c%c %016llx\n",
4076	u64Address + ((uint64_t)i << X86_PT_PAE_SHIFT),
4077	Pte.n.u1Write ? 'W' : 'R',
4078	Pte.n.u1User ? 'U' : 'S',
4079	Pte.n.u1Accessed ? 'A' : '-',
4080	Pte.n.u1Dirty ? 'D' : '-',
4081	Pte.n.u1Global ? 'G' : '-',
4082	Pte.n.u1WriteThru ? "WT" : "--",
4083	Pte.n.u1CacheDisable? "CD" : "--",
4084	Pte.n.u1PAT ? "AT" : "--",
4085	Pte.n.u1NoExecute ? "NX" : "--",
4086	Pte.u & PGM_PTFLAGS_TRACK_DIRTY ? 'd' : '-',
4087	Pte.u & RT_BIT(10) ? '1' : '0',
4088	Pte.u & PGM_PTFLAGS_CSAM_VALIDATED? 'v' : '-',
4089	Pte.u & X86_PTE_PAE_PG_MASK);
4090	}
4091	}
4092	return VINF_SUCCESS;
4093	}
4094
4095
4096	/**
4097	* Dumps a PAE shadow page directory table.
4098	*
4099	* @returns VBox status code (VINF_SUCCESS).
4100	* @param pVM The VM handle.
4101	* @param HCPhys The physical address of the page directory table.
4102	* @param u64Address The virtual address of the page table starts.
4103	* @param cr4 The CR4, PSE is currently used.
4104	* @param fLongMode Set if this a long mode table; clear if it's a legacy mode table.
4105	* @param cMaxDepth The maxium depth.
4106	* @param pHlp Pointer to the output functions.
4107	*/
4108	static int pgmR3DumpHierarchyHCPaePD(PVM pVM, RTHCPHYS HCPhys, uint64_t u64Address, uint32_t cr4, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
4109	{
4110	PX86PDPAE pPD = (PX86PDPAE)MMPagePhys2Page(pVM, HCPhys);
4111	if (!pPD)
4112	{
4113	pHlp->pfnPrintf(pHlp, "%0*llx error! Page directory at HCPhys=%RHp was not found in the page pool!\n",
4114	fLongMode ? 16 : 8, u64Address, HCPhys);
4115	return VERR_INVALID_PARAMETER;
4116	}
4117	const bool fBigPagesSupported = fLongMode \|\| !!(cr4 & X86_CR4_PSE);
4118
4119	int rc = VINF_SUCCESS;
4120	for (unsigned i = 0; i < RT_ELEMENTS(pPD->a); i++)
4121	{
4122	X86PDEPAE Pde = pPD->a[i];
4123	if (Pde.n.u1Present)
4124	{
4125	if (fBigPagesSupported && Pde.b.u1Size)
4126	pHlp->pfnPrintf(pHlp,
4127	fLongMode /P R S A D G WT CD AT NX 4M a p ? /
4128	? "%016llx 2 \| P %c %c %c %c %c %s %s %s %s 4M %c%c%c %016llx\n"
4129	: "%08llx 1 \| P %c %c %c %c %c %s %s %s %s 4M %c%c%c %016llx\n",
4130	u64Address + ((uint64_t)i << X86_PD_PAE_SHIFT),
4131	Pde.b.u1Write ? 'W' : 'R',
4132	Pde.b.u1User ? 'U' : 'S',
4133	Pde.b.u1Accessed ? 'A' : '-',
4134	Pde.b.u1Dirty ? 'D' : '-',
4135	Pde.b.u1Global ? 'G' : '-',
4136	Pde.b.u1WriteThru ? "WT" : "--",
4137	Pde.b.u1CacheDisable? "CD" : "--",
4138	Pde.b.u1PAT ? "AT" : "--",
4139	Pde.b.u1NoExecute ? "NX" : "--",
4140	Pde.u & RT_BIT_64(9) ? '1' : '0',
4141	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
4142	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
4143	Pde.u & X86_PDE_PAE_PG_MASK);
4144	else
4145	{
4146	pHlp->pfnPrintf(pHlp,
4147	fLongMode /P R S A D G WT CD AT NX 4M a p ? /
4148	? "%016llx 2 \| P %c %c %c %c %c %s %s .. %s 4K %c%c%c %016llx\n"
4149	: "%08llx 1 \| P %c %c %c %c %c %s %s .. %s 4K %c%c%c %016llx\n",
4150	u64Address + ((uint64_t)i << X86_PD_PAE_SHIFT),
4151	Pde.n.u1Write ? 'W' : 'R',
4152	Pde.n.u1User ? 'U' : 'S',
4153	Pde.n.u1Accessed ? 'A' : '-',
4154	Pde.n.u1Reserved0 ? '?' : '.', /* ignored */
4155	Pde.n.u1Reserved1 ? '?' : '.', /* ignored */
4156	Pde.n.u1WriteThru ? "WT" : "--",
4157	Pde.n.u1CacheDisable? "CD" : "--",
4158	Pde.n.u1NoExecute ? "NX" : "--",
4159	Pde.u & RT_BIT_64(9) ? '1' : '0',
4160	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
4161	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
4162	Pde.u & X86_PDE_PAE_PG_MASK);
4163	if (cMaxDepth >= 1)
4164	{
4165	/** @todo what about using the page pool for mapping PTs? */
4166	uint64_t u64AddressPT = u64Address + ((uint64_t)i << X86_PD_PAE_SHIFT);
4167	RTHCPHYS HCPhysPT = Pde.u & X86_PDE_PAE_PG_MASK;
4168	PX86PTPAE pPT = NULL;
4169	if (!(Pde.u & PGM_PDFLAGS_MAPPING))
4170	pPT = (PX86PTPAE)MMPagePhys2Page(pVM, HCPhysPT);
4171	else
4172	{
4173	for (PPGMMAPPING pMap = pVM->pgm.s.pMappingsR3; pMap; pMap = pMap->pNextR3)
4174	{
4175	uint64_t off = u64AddressPT - pMap->GCPtr;
4176	if (off < pMap->cb)
4177	{
4178	const int iPDE = (uint32_t)(off >> X86_PD_SHIFT);
4179	const int iSub = (int)((off >> X86_PD_PAE_SHIFT) & 1); /* MSC is a pain sometimes */
4180	if ((iSub ? pMap->aPTs[iPDE].HCPhysPaePT1 : pMap->aPTs[iPDE].HCPhysPaePT0) != HCPhysPT)
4181	pHlp->pfnPrintf(pHlp, "%0*llx error! Mapping error! PT %d has HCPhysPT=%RHp not %RHp is in the PD.\n",
4182	fLongMode ? 16 : 8, u64AddressPT, iPDE,
4183	iSub ? pMap->aPTs[iPDE].HCPhysPaePT1 : pMap->aPTs[iPDE].HCPhysPaePT0, HCPhysPT);
4184	pPT = &pMap->aPTs[iPDE].paPaePTsR3[iSub];
4185	}
4186	}
4187	}
4188	int rc2 = VERR_INVALID_PARAMETER;
4189	if (pPT)
4190	rc2 = pgmR3DumpHierarchyHCPaePT(pVM, pPT, u64AddressPT, fLongMode, cMaxDepth - 1, pHlp);
4191	else
4192	pHlp->pfnPrintf(pHlp, "%0*llx error! Page table at HCPhys=%RHp was not found in the page pool!\n",
4193	fLongMode ? 16 : 8, u64AddressPT, HCPhysPT);
4194	if (rc2 < rc && RT_SUCCESS(rc))
4195	rc = rc2;
4196	}
4197	}
4198	}
4199	}
4200	return rc;
4201	}
4202
4203
4204	/**
4205	* Dumps a PAE shadow page directory pointer table.
4206	*
4207	* @returns VBox status code (VINF_SUCCESS).
4208	* @param pVM The VM handle.
4209	* @param HCPhys The physical address of the page directory pointer table.
4210	* @param u64Address The virtual address of the page table starts.
4211	* @param cr4 The CR4, PSE is currently used.
4212	* @param fLongMode Set if this a long mode table; clear if it's a legacy mode table.
4213	* @param cMaxDepth The maxium depth.
4214	* @param pHlp Pointer to the output functions.
4215	*/
4216	static int pgmR3DumpHierarchyHCPaePDPT(PVM pVM, RTHCPHYS HCPhys, uint64_t u64Address, uint32_t cr4, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
4217	{
4218	PX86PDPT pPDPT = (PX86PDPT)MMPagePhys2Page(pVM, HCPhys);
4219	if (!pPDPT)
4220	{
4221	pHlp->pfnPrintf(pHlp, "%0*llx error! Page directory pointer table at HCPhys=%RHp was not found in the page pool!\n",
4222	fLongMode ? 16 : 8, u64Address, HCPhys);
4223	return VERR_INVALID_PARAMETER;
4224	}
4225
4226	int rc = VINF_SUCCESS;
4227	const unsigned c = fLongMode ? RT_ELEMENTS(pPDPT->a) : X86_PG_PAE_PDPE_ENTRIES;
4228	for (unsigned i = 0; i < c; i++)
4229	{
4230	X86PDPE Pdpe = pPDPT->a[i];
4231	if (Pdpe.n.u1Present)
4232	{
4233	if (fLongMode)
4234	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a p ? /
4235	"%016llx 1 \| P %c %c %c %c %c %s %s %s %s .. %c%c%c %016llx\n",
4236	u64Address + ((uint64_t)i << X86_PDPT_SHIFT),
4237	Pdpe.lm.u1Write ? 'W' : 'R',
4238	Pdpe.lm.u1User ? 'U' : 'S',
4239	Pdpe.lm.u1Accessed ? 'A' : '-',
4240	Pdpe.lm.u3Reserved & 1? '?' : '.', /* ignored */
4241	Pdpe.lm.u3Reserved & 4? '!' : '.', /* mbz */
4242	Pdpe.lm.u1WriteThru ? "WT" : "--",
4243	Pdpe.lm.u1CacheDisable? "CD" : "--",
4244	Pdpe.lm.u3Reserved & 2? "!" : "..",/* mbz */
4245	Pdpe.lm.u1NoExecute ? "NX" : "--",
4246	Pdpe.u & RT_BIT(9) ? '1' : '0',
4247	Pdpe.u & PGM_PLXFLAGS_PERMANENT ? 'p' : '-',
4248	Pdpe.u & RT_BIT(11) ? '1' : '0',
4249	Pdpe.u & X86_PDPE_PG_MASK);
4250	else
4251	pHlp->pfnPrintf(pHlp, /P G WT CD AT NX 4M a p ? /
4252	"%08x 0 \| P %c %s %s %s %s .. %c%c%c %016llx\n",
4253	i << X86_PDPT_SHIFT,
4254	Pdpe.n.u4Reserved & 1? '!' : '.', /* mbz */
4255	Pdpe.n.u4Reserved & 4? '!' : '.', /* mbz */
4256	Pdpe.n.u1WriteThru ? "WT" : "--",
4257	Pdpe.n.u1CacheDisable? "CD" : "--",
4258	Pdpe.n.u4Reserved & 2? "!" : "..",/* mbz */
4259	Pdpe.u & RT_BIT(9) ? '1' : '0',
4260	Pdpe.u & PGM_PLXFLAGS_PERMANENT ? 'p' : '-',
4261	Pdpe.u & RT_BIT(11) ? '1' : '0',
4262	Pdpe.u & X86_PDPE_PG_MASK);
4263	if (cMaxDepth >= 1)
4264	{
4265	int rc2 = pgmR3DumpHierarchyHCPaePD(pVM, Pdpe.u & X86_PDPE_PG_MASK, u64Address + ((uint64_t)i << X86_PDPT_SHIFT),
4266	cr4, fLongMode, cMaxDepth - 1, pHlp);
4267	if (rc2 < rc && RT_SUCCESS(rc))
4268	rc = rc2;
4269	}
4270	}
4271	}
4272	return rc;
4273	}
4274
4275
4276	/**
4277	* Dumps a 32-bit shadow page table.
4278	*
4279	* @returns VBox status code (VINF_SUCCESS).
4280	* @param pVM The VM handle.
4281	* @param HCPhys The physical address of the table.
4282	* @param cr4 The CR4, PSE is currently used.
4283	* @param cMaxDepth The maxium depth.
4284	* @param pHlp Pointer to the output functions.
4285	*/
4286	static int pgmR3DumpHierarchyHcPaePML4(PVM pVM, RTHCPHYS HCPhys, uint32_t cr4, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
4287	{
4288	PX86PML4 pPML4 = (PX86PML4)MMPagePhys2Page(pVM, HCPhys);
4289	if (!pPML4)
4290	{
4291	pHlp->pfnPrintf(pHlp, "Page map level 4 at HCPhys=%RHp was not found in the page pool!\n", HCPhys);
4292	return VERR_INVALID_PARAMETER;
4293	}
4294
4295	int rc = VINF_SUCCESS;
4296	for (unsigned i = 0; i < RT_ELEMENTS(pPML4->a); i++)
4297	{
4298	X86PML4E Pml4e = pPML4->a[i];
4299	if (Pml4e.n.u1Present)
4300	{
4301	uint64_t u64Address = ((uint64_t)i << X86_PML4_SHIFT) \| (((uint64_t)i >> (X86_PML4_SHIFT - X86_PDPT_SHIFT - 1)) * 0xffff000000000000ULL);
4302	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a p ? /
4303	"%016llx 0 \| P %c %c %c %c %c %s %s %s %s .. %c%c%c %016llx\n",
4304	u64Address,
4305	Pml4e.n.u1Write ? 'W' : 'R',
4306	Pml4e.n.u1User ? 'U' : 'S',
4307	Pml4e.n.u1Accessed ? 'A' : '-',
4308	Pml4e.n.u3Reserved & 1? '?' : '.', /* ignored */
4309	Pml4e.n.u3Reserved & 4? '!' : '.', /* mbz */
4310	Pml4e.n.u1WriteThru ? "WT" : "--",
4311	Pml4e.n.u1CacheDisable? "CD" : "--",
4312	Pml4e.n.u3Reserved & 2? "!" : "..",/* mbz */
4313	Pml4e.n.u1NoExecute ? "NX" : "--",
4314	Pml4e.u & RT_BIT(9) ? '1' : '0',
4315	Pml4e.u & PGM_PLXFLAGS_PERMANENT ? 'p' : '-',
4316	Pml4e.u & RT_BIT(11) ? '1' : '0',
4317	Pml4e.u & X86_PML4E_PG_MASK);
4318
4319	if (cMaxDepth >= 1)
4320	{
4321	int rc2 = pgmR3DumpHierarchyHCPaePDPT(pVM, Pml4e.u & X86_PML4E_PG_MASK, u64Address, cr4, true, cMaxDepth - 1, pHlp);
4322	if (rc2 < rc && RT_SUCCESS(rc))
4323	rc = rc2;
4324	}
4325	}
4326	}
4327	return rc;
4328	}
4329
4330
4331	/**
4332	* Dumps a 32-bit shadow page table.
4333	*
4334	* @returns VBox status code (VINF_SUCCESS).
4335	* @param pVM The VM handle.
4336	* @param pPT Pointer to the page table.
4337	* @param u32Address The virtual address this table starts at.
4338	* @param pHlp Pointer to the output functions.
4339	*/
4340	int pgmR3DumpHierarchyHC32BitPT(PVM pVM, PX86PT pPT, uint32_t u32Address, PCDBGFINFOHLP pHlp)
4341	{
4342	for (unsigned i = 0; i < RT_ELEMENTS(pPT->a); i++)
4343	{
4344	X86PTE Pte = pPT->a[i];
4345	if (Pte.n.u1Present)
4346	{
4347	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a m d /
4348	"%08x 1 \| P %c %c %c %c %c %s %s %s .. 4K %c%c%c %08x\n",
4349	u32Address + (i << X86_PT_SHIFT),
4350	Pte.n.u1Write ? 'W' : 'R',
4351	Pte.n.u1User ? 'U' : 'S',
4352	Pte.n.u1Accessed ? 'A' : '-',
4353	Pte.n.u1Dirty ? 'D' : '-',
4354	Pte.n.u1Global ? 'G' : '-',
4355	Pte.n.u1WriteThru ? "WT" : "--",
4356	Pte.n.u1CacheDisable? "CD" : "--",
4357	Pte.n.u1PAT ? "AT" : "--",
4358	Pte.u & PGM_PTFLAGS_TRACK_DIRTY ? 'd' : '-',
4359	Pte.u & RT_BIT(10) ? '1' : '0',
4360	Pte.u & PGM_PTFLAGS_CSAM_VALIDATED ? 'v' : '-',
4361	Pte.u & X86_PDE_PG_MASK);
4362	}
4363	}
4364	return VINF_SUCCESS;
4365	}
4366
4367
4368	/**
4369	* Dumps a 32-bit shadow page directory and page tables.
4370	*
4371	* @returns VBox status code (VINF_SUCCESS).
4372	* @param pVM The VM handle.
4373	* @param cr3 The root of the hierarchy.
4374	* @param cr4 The CR4, PSE is currently used.
4375	* @param cMaxDepth How deep into the hierarchy the dumper should go.
4376	* @param pHlp Pointer to the output functions.
4377	*/
4378	int pgmR3DumpHierarchyHC32BitPD(PVM pVM, uint32_t cr3, uint32_t cr4, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
4379	{
4380	PX86PD pPD = (PX86PD)MMPagePhys2Page(pVM, cr3 & X86_CR3_PAGE_MASK);
4381	if (!pPD)
4382	{
4383	pHlp->pfnPrintf(pHlp, "Page directory at %#x was not found in the page pool!\n", cr3 & X86_CR3_PAGE_MASK);
4384	return VERR_INVALID_PARAMETER;
4385	}
4386
4387	int rc = VINF_SUCCESS;
4388	for (unsigned i = 0; i < RT_ELEMENTS(pPD->a); i++)
4389	{
4390	X86PDE Pde = pPD->a[i];
4391	if (Pde.n.u1Present)
4392	{
4393	const uint32_t u32Address = i << X86_PD_SHIFT;
4394	if ((cr4 & X86_CR4_PSE) && Pde.b.u1Size)
4395	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a m d /
4396	"%08x 0 \| P %c %c %c %c %c %s %s %s .. 4M %c%c%c %08x\n",
4397	u32Address,
4398	Pde.b.u1Write ? 'W' : 'R',
4399	Pde.b.u1User ? 'U' : 'S',
4400	Pde.b.u1Accessed ? 'A' : '-',
4401	Pde.b.u1Dirty ? 'D' : '-',
4402	Pde.b.u1Global ? 'G' : '-',
4403	Pde.b.u1WriteThru ? "WT" : "--",
4404	Pde.b.u1CacheDisable? "CD" : "--",
4405	Pde.b.u1PAT ? "AT" : "--",
4406	Pde.u & RT_BIT_64(9) ? '1' : '0',
4407	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
4408	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
4409	Pde.u & X86_PDE4M_PG_MASK);
4410	else
4411	{
4412	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a m d /
4413	"%08x 0 \| P %c %c %c %c %c %s %s .. .. 4K %c%c%c %08x\n",
4414	u32Address,
4415	Pde.n.u1Write ? 'W' : 'R',
4416	Pde.n.u1User ? 'U' : 'S',
4417	Pde.n.u1Accessed ? 'A' : '-',
4418	Pde.n.u1Reserved0 ? '?' : '.', /* ignored */
4419	Pde.n.u1Reserved1 ? '?' : '.', /* ignored */
4420	Pde.n.u1WriteThru ? "WT" : "--",
4421	Pde.n.u1CacheDisable? "CD" : "--",
4422	Pde.u & RT_BIT_64(9) ? '1' : '0',
4423	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
4424	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
4425	Pde.u & X86_PDE_PG_MASK);
4426	if (cMaxDepth >= 1)
4427	{
4428	/** @todo what about using the page pool for mapping PTs? */
4429	RTHCPHYS HCPhys = Pde.u & X86_PDE_PG_MASK;
4430	PX86PT pPT = NULL;
4431	if (!(Pde.u & PGM_PDFLAGS_MAPPING))
4432	pPT = (PX86PT)MMPagePhys2Page(pVM, HCPhys);
4433	else
4434	{
4435	for (PPGMMAPPING pMap = pVM->pgm.s.pMappingsR3; pMap; pMap = pMap->pNextR3)
4436	if (u32Address - pMap->GCPtr < pMap->cb)
4437	{
4438	int iPDE = (u32Address - pMap->GCPtr) >> X86_PD_SHIFT;
4439	if (pMap->aPTs[iPDE].HCPhysPT != HCPhys)
4440	pHlp->pfnPrintf(pHlp, "%08x error! Mapping error! PT %d has HCPhysPT=%RHp not %RHp is in the PD.\n",
4441	u32Address, iPDE, pMap->aPTs[iPDE].HCPhysPT, HCPhys);
4442	pPT = pMap->aPTs[iPDE].pPTR3;
4443	}
4444	}
4445	int rc2 = VERR_INVALID_PARAMETER;
4446	if (pPT)
4447	rc2 = pgmR3DumpHierarchyHC32BitPT(pVM, pPT, u32Address, pHlp);
4448	else
4449	pHlp->pfnPrintf(pHlp, "%08x error! Page table at %#x was not found in the page pool!\n", u32Address, HCPhys);
4450	if (rc2 < rc && RT_SUCCESS(rc))
4451	rc = rc2;
4452	}
4453	}
4454	}
4455	}
4456
4457	return rc;
4458	}
4459
4460
4461	/**
4462	* Dumps a 32-bit shadow page table.
4463	*
4464	* @returns VBox status code (VINF_SUCCESS).
4465	* @param pVM The VM handle.
4466	* @param pPT Pointer to the page table.
4467	* @param u32Address The virtual address this table starts at.
4468	* @param PhysSearch Address to search for.
4469	*/
4470	int pgmR3DumpHierarchyGC32BitPT(PVM pVM, PX86PT pPT, uint32_t u32Address, RTGCPHYS PhysSearch)
4471	{
4472	for (unsigned i = 0; i < RT_ELEMENTS(pPT->a); i++)
4473	{
4474	X86PTE Pte = pPT->a[i];
4475	if (Pte.n.u1Present)
4476	{
4477	Log(( /P R S A D G WT CD AT NX 4M a m d /
4478	"%08x 1 \| P %c %c %c %c %c %s %s %s .. 4K %c%c%c %08x\n",
4479	u32Address + (i << X86_PT_SHIFT),
4480	Pte.n.u1Write ? 'W' : 'R',
4481	Pte.n.u1User ? 'U' : 'S',
4482	Pte.n.u1Accessed ? 'A' : '-',
4483	Pte.n.u1Dirty ? 'D' : '-',
4484	Pte.n.u1Global ? 'G' : '-',
4485	Pte.n.u1WriteThru ? "WT" : "--",
4486	Pte.n.u1CacheDisable? "CD" : "--",
4487	Pte.n.u1PAT ? "AT" : "--",
4488	Pte.u & PGM_PTFLAGS_TRACK_DIRTY ? 'd' : '-',
4489	Pte.u & RT_BIT(10) ? '1' : '0',
4490	Pte.u & PGM_PTFLAGS_CSAM_VALIDATED ? 'v' : '-',
4491	Pte.u & X86_PDE_PG_MASK));
4492
4493	if ((Pte.u & X86_PDE_PG_MASK) == PhysSearch)
4494	{
4495	uint64_t fPageShw = 0;
4496	RTHCPHYS pPhysHC = 0;
4497
4498	/** @todo SMP support!! */
4499	PGMShwGetPage(pVM, &pVM->aCpus[0], (RTGCPTR)(u32Address + (i << X86_PT_SHIFT)), &fPageShw, &pPhysHC);
4500	Log(("Found %RGp at %RGv -> flags=%llx\n", PhysSearch, (RTGCPTR)(u32Address + (i << X86_PT_SHIFT)), fPageShw));
4501	}
4502	}
4503	}
4504	return VINF_SUCCESS;
4505	}
4506
4507
4508	/**
4509	* Dumps a 32-bit guest page directory and page tables.
4510	*
4511	* @returns VBox status code (VINF_SUCCESS).
4512	* @param pVM The VM handle.
4513	* @param cr3 The root of the hierarchy.
4514	* @param cr4 The CR4, PSE is currently used.
4515	* @param PhysSearch Address to search for.
4516	*/
4517	VMMR3DECL(int) PGMR3DumpHierarchyGC(PVM pVM, uint64_t cr3, uint64_t cr4, RTGCPHYS PhysSearch)
4518	{
4519	bool fLongMode = false;
4520	const unsigned cch = fLongMode ? 16 : 8; NOREF(cch);
4521	PX86PD pPD = 0;
4522
4523	int rc = PGM_GCPHYS_2_PTR(pVM, cr3 & X86_CR3_PAGE_MASK, &pPD);
4524	if (RT_FAILURE(rc) \|\| !pPD)
4525	{
4526	Log(("Page directory at %#x was not found in the page pool!\n", cr3 & X86_CR3_PAGE_MASK));
4527	return VERR_INVALID_PARAMETER;
4528	}
4529
4530	Log(("cr3=%08x cr4=%08x%s\n"
4531	"%-*s P - Present\n"
4532	"%-*s \| R/W - Read (0) / Write (1)\n"
4533	"%-*s \| \| U/S - User (1) / Supervisor (0)\n"
4534	"%-*s \| \| \| A - Accessed\n"
4535	"%-*s \| \| \| \| D - Dirty\n"
4536	"%-*s \| \| \| \| \| G - Global\n"
4537	"%-*s \| \| \| \| \| \| WT - Write thru\n"
4538	"%-*s \| \| \| \| \| \| \| CD - Cache disable\n"
4539	"%-*s \| \| \| \| \| \| \| \| AT - Attribute table (PAT)\n"
4540	"%-*s \| \| \| \| \| \| \| \| \| NX - No execute (K8)\n"
4541	"%-*s \| \| \| \| \| \| \| \| \| \| 4K/4M/2M - Page size.\n"
4542	"%-*s \| \| \| \| \| \| \| \| \| \| \| AVL - a=allocated; m=mapping; d=track dirty;\n"
4543	"%-*s \| \| \| \| \| \| \| \| \| \| \| \| p=permanent; v=validated;\n"
4544	"%-*s Level \| \| \| \| \| \| \| \| \| \| \| \| Page\n"
4545	/* xxxx n **** P R S A D G WT CD AT NX 4M AVL xxxxxxxxxxxxx
4546	- W U - - - -- -- -- -- -- 010 */
4547	, cr3, cr4, fLongMode ? " Long Mode" : "",
4548	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "",
4549	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "Address"));
4550
4551	for (unsigned i = 0; i < RT_ELEMENTS(pPD->a); i++)
4552	{
4553	X86PDE Pde = pPD->a[i];
4554	if (Pde.n.u1Present)
4555	{
4556	const uint32_t u32Address = i << X86_PD_SHIFT;
4557
4558	if ((cr4 & X86_CR4_PSE) && Pde.b.u1Size)
4559	Log(( /P R S A D G WT CD AT NX 4M a m d /
4560	"%08x 0 \| P %c %c %c %c %c %s %s %s .. 4M %c%c%c %08x\n",
4561	u32Address,
4562	Pde.b.u1Write ? 'W' : 'R',
4563	Pde.b.u1User ? 'U' : 'S',
4564	Pde.b.u1Accessed ? 'A' : '-',
4565	Pde.b.u1Dirty ? 'D' : '-',
4566	Pde.b.u1Global ? 'G' : '-',
4567	Pde.b.u1WriteThru ? "WT" : "--",
4568	Pde.b.u1CacheDisable? "CD" : "--",
4569	Pde.b.u1PAT ? "AT" : "--",
4570	Pde.u & RT_BIT(9) ? '1' : '0',
4571	Pde.u & RT_BIT(10) ? '1' : '0',
4572	Pde.u & RT_BIT(11) ? '1' : '0',
4573	pgmGstGet4MBPhysPage(&pVM->pgm.s, Pde)));
4574	/** @todo PhysSearch */
4575	else
4576	{
4577	Log(( /P R S A D G WT CD AT NX 4M a m d /
4578	"%08x 0 \| P %c %c %c %c %c %s %s .. .. 4K %c%c%c %08x\n",
4579	u32Address,
4580	Pde.n.u1Write ? 'W' : 'R',
4581	Pde.n.u1User ? 'U' : 'S',
4582	Pde.n.u1Accessed ? 'A' : '-',
4583	Pde.n.u1Reserved0 ? '?' : '.', /* ignored */
4584	Pde.n.u1Reserved1 ? '?' : '.', /* ignored */
4585	Pde.n.u1WriteThru ? "WT" : "--",
4586	Pde.n.u1CacheDisable? "CD" : "--",
4587	Pde.u & RT_BIT(9) ? '1' : '0',
4588	Pde.u & RT_BIT(10) ? '1' : '0',
4589	Pde.u & RT_BIT(11) ? '1' : '0',
4590	Pde.u & X86_PDE_PG_MASK));
4591	////if (cMaxDepth >= 1)
4592	{
4593	/** @todo what about using the page pool for mapping PTs? */
4594	RTGCPHYS GCPhys = Pde.u & X86_PDE_PG_MASK;
4595	PX86PT pPT = NULL;
4596
4597	rc = PGM_GCPHYS_2_PTR(pVM, GCPhys, &pPT);
4598
4599	int rc2 = VERR_INVALID_PARAMETER;
4600	if (pPT)
4601	rc2 = pgmR3DumpHierarchyGC32BitPT(pVM, pPT, u32Address, PhysSearch);
4602	else
4603	Log(("%08x error! Page table at %#x was not found in the page pool!\n", u32Address, GCPhys));
4604	if (rc2 < rc && RT_SUCCESS(rc))
4605	rc = rc2;
4606	}
4607	}
4608	}
4609	}
4610
4611	return rc;
4612	}
4613
4614
4615	/**
4616	* Dumps a page table hierarchy use only physical addresses and cr4/lm flags.
4617	*
4618	* @returns VBox status code (VINF_SUCCESS).
4619	* @param pVM The VM handle.
4620	* @param cr3 The root of the hierarchy.
4621	* @param cr4 The cr4, only PAE and PSE is currently used.
4622	* @param fLongMode Set if long mode, false if not long mode.
4623	* @param cMaxDepth Number of levels to dump.
4624	* @param pHlp Pointer to the output functions.
4625	*/
4626	VMMR3DECL(int) PGMR3DumpHierarchyHC(PVM pVM, uint64_t cr3, uint64_t cr4, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
4627	{
4628	if (!pHlp)
4629	pHlp = DBGFR3InfoLogHlp();
4630	if (!cMaxDepth)
4631	return VINF_SUCCESS;
4632	const unsigned cch = fLongMode ? 16 : 8;
4633	pHlp->pfnPrintf(pHlp,
4634	"cr3=%08x cr4=%08x%s\n"
4635	"%-*s P - Present\n"
4636	"%-*s \| R/W - Read (0) / Write (1)\n"
4637	"%-*s \| \| U/S - User (1) / Supervisor (0)\n"
4638	"%-*s \| \| \| A - Accessed\n"
4639	"%-*s \| \| \| \| D - Dirty\n"
4640	"%-*s \| \| \| \| \| G - Global\n"
4641	"%-*s \| \| \| \| \| \| WT - Write thru\n"
4642	"%-*s \| \| \| \| \| \| \| CD - Cache disable\n"
4643	"%-*s \| \| \| \| \| \| \| \| AT - Attribute table (PAT)\n"
4644	"%-*s \| \| \| \| \| \| \| \| \| NX - No execute (K8)\n"
4645	"%-*s \| \| \| \| \| \| \| \| \| \| 4K/4M/2M - Page size.\n"
4646	"%-*s \| \| \| \| \| \| \| \| \| \| \| AVL - a=allocated; m=mapping; d=track dirty;\n"
4647	"%-*s \| \| \| \| \| \| \| \| \| \| \| \| p=permanent; v=validated;\n"
4648	"%-*s Level \| \| \| \| \| \| \| \| \| \| \| \| Page\n"
4649	/* xxxx n **** P R S A D G WT CD AT NX 4M AVL xxxxxxxxxxxxx
4650	- W U - - - -- -- -- -- -- 010 */
4651	, cr3, cr4, fLongMode ? " Long Mode" : "",
4652	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "",
4653	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "Address");
4654	if (cr4 & X86_CR4_PAE)
4655	{
4656	if (fLongMode)
4657	return pgmR3DumpHierarchyHcPaePML4(pVM, cr3 & X86_CR3_PAGE_MASK, cr4, cMaxDepth, pHlp);
4658	return pgmR3DumpHierarchyHCPaePDPT(pVM, cr3 & X86_CR3_PAE_PAGE_MASK, 0, cr4, false, cMaxDepth, pHlp);
4659	}
4660	return pgmR3DumpHierarchyHC32BitPD(pVM, cr3 & X86_CR3_PAGE_MASK, cr4, cMaxDepth, pHlp);
4661	}
4662
4663	#ifdef VBOX_WITH_DEBUGGER
4664
4665	/**
4666	* The '.pgmram' command.
4667	*
4668	* @returns VBox status.
4669	* @param pCmd Pointer to the command descriptor (as registered).
4670	* @param pCmdHlp Pointer to command helper functions.
4671	* @param pVM Pointer to the current VM (if any).
4672	* @param paArgs Pointer to (readonly) array of arguments.
4673	* @param cArgs Number of arguments in the array.
4674	*/
4675	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
4676	{
4677	/*
4678	* Validate input.
4679	*/
4680	if (!pVM)
4681	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
4682	if (!pVM->pgm.s.pRamRangesRC)
4683	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Sorry, no Ram is registered.\n");
4684
4685	/*
4686	* Dump the ranges.
4687	*/
4688	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "From - To (incl) pvHC\n");
4689	PPGMRAMRANGE pRam;
4690	for (pRam = pVM->pgm.s.pRamRangesR3; pRam; pRam = pRam->pNextR3)
4691	{
4692	rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL,
4693	"%RGp - %RGp %p\n",
4694	pRam->GCPhys, pRam->GCPhysLast, pRam->pvR3);
4695	if (RT_FAILURE(rc))
4696	return rc;
4697	}
4698
4699	return VINF_SUCCESS;
4700	}
4701
4702
4703	/**
4704	* The '.pgmmap' command.
4705	*
4706	* @returns VBox status.
4707	* @param pCmd Pointer to the command descriptor (as registered).
4708	* @param pCmdHlp Pointer to command helper functions.
4709	* @param pVM Pointer to the current VM (if any).
4710	* @param paArgs Pointer to (readonly) array of arguments.
4711	* @param cArgs Number of arguments in the array.
4712	*/
4713	static DECLCALLBACK(int) pgmR3CmdMap(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
4714	{
4715	/*
4716	* Validate input.
4717	*/
4718	if (!pVM)
4719	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
4720	if (!pVM->pgm.s.pMappingsR3)
4721	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Sorry, no mappings are registered.\n");
4722
4723	/*
4724	* Print message about the fixedness of the mappings.
4725	*/
4726	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, pVM->pgm.s.fMappingsFixed ? "The mappings are FIXED.\n" : "The mappings are FLOATING.\n");
4727	if (RT_FAILURE(rc))
4728	return rc;
4729
4730	/*
4731	* Dump the ranges.
4732	*/
4733	PPGMMAPPING pCur;
4734	for (pCur = pVM->pgm.s.pMappingsR3; pCur; pCur = pCur->pNextR3)
4735	{
4736	rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL,
4737	"%08x - %08x %s\n",
4738	pCur->GCPtr, pCur->GCPtrLast, pCur->pszDesc);
4739	if (RT_FAILURE(rc))
4740	return rc;
4741	}
4742
4743	return VINF_SUCCESS;
4744	}
4745
4746
4747	/**
4748	* The '.pgmerror' and '.pgmerroroff' commands.
4749	*
4750	* @returns VBox status.
4751	* @param pCmd Pointer to the command descriptor (as registered).
4752	* @param pCmdHlp Pointer to command helper functions.
4753	* @param pVM Pointer to the current VM (if any).
4754	* @param paArgs Pointer to (readonly) array of arguments.
4755	* @param cArgs Number of arguments in the array.
4756	*/
4757	static DECLCALLBACK(int) pgmR3CmdError(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
4758	{
4759	/*
4760	* Validate input.
4761	*/
4762	if (!pVM)
4763	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
4764	AssertReturn(cArgs == 0 \|\| (cArgs == 1 && paArgs[0].enmType == DBGCVAR_TYPE_STRING),
4765	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Hit bug in the parser.\n"));
4766
4767	if (!cArgs)
4768	{
4769	/*
4770	* Print the list of error injection locations with status.
4771	*/
4772	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "PGM error inject locations:\n");
4773	pCmdHlp->pfnPrintf(pCmdHlp, NULL, " handy - %RTbool\n", pVM->pgm.s.fErrInjHandyPages);
4774	}
4775	else
4776	{
4777
4778	/*
4779	* String switch on where to inject the error.
4780	*/
4781	bool const fNewState = !strcmp(pCmd->pszCmd, "pgmerror");
4782	const char *pszWhere = paArgs[0].u.pszString;
4783	if (!strcmp(pszWhere, "handy"))
4784	ASMAtomicWriteBool(&pVM->pgm.s.fErrInjHandyPages, fNewState);
4785	else
4786	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Invalid 'where' value: %s.\n", pszWhere);
4787	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "done\n");
4788	}
4789	return VINF_SUCCESS;
4790	}
4791
4792
4793	/**
4794	* The '.pgmsync' command.
4795	*
4796	* @returns VBox status.
4797	* @param pCmd Pointer to the command descriptor (as registered).
4798	* @param pCmdHlp Pointer to command helper functions.
4799	* @param pVM Pointer to the current VM (if any).
4800	* @param paArgs Pointer to (readonly) array of arguments.
4801	* @param cArgs Number of arguments in the array.
4802	*/
4803	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
4804	{
4805	/*
4806	* Validate input.
4807	*/
4808	if (!pVM)
4809	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
4810
4811	/*
4812	* Force page directory sync.
4813	*/
4814	VM_FF_SET(pVM, VM_FF_PGM_SYNC_CR3);
4815
4816	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Forcing page directory sync.\n");
4817	if (RT_FAILURE(rc))
4818	return rc;
4819
4820	return VINF_SUCCESS;
4821	}
4822
4823
4824	#ifdef VBOX_STRICT
4825	/**
4826	* The '.pgmassertcr3' command.
4827	*
4828	* @returns VBox status.
4829	* @param pCmd Pointer to the command descriptor (as registered).
4830	* @param pCmdHlp Pointer to command helper functions.
4831	* @param pVM Pointer to the current VM (if any).
4832	* @param paArgs Pointer to (readonly) array of arguments.
4833	* @param cArgs Number of arguments in the array.
4834	*/
4835	static DECLCALLBACK(int) pgmR3CmdAssertCR3(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
4836	{
4837	/** @todo SMP support!! */
4838	PVMCPU pVCpu = &pVM->aCpus[0];
4839
4840	/*
4841	* Validate input.
4842	*/
4843	if (!pVM)
4844	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
4845
4846	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Checking shadow CR3 page tables for consistency.\n");
4847	if (RT_FAILURE(rc))
4848	return rc;
4849
4850	PGMAssertCR3(pVM, pVCpu, CPUMGetGuestCR3(pVCpu), CPUMGetGuestCR4(pVCpu));
4851
4852	return VINF_SUCCESS;
4853	}
4854	#endif /* VBOX_STRICT */
4855
4856
4857	/**
4858	* The '.pgmsyncalways' command.
4859	*
4860	* @returns VBox status.
4861	* @param pCmd Pointer to the command descriptor (as registered).
4862	* @param pCmdHlp Pointer to command helper functions.
4863	* @param pVM Pointer to the current VM (if any).
4864	* @param paArgs Pointer to (readonly) array of arguments.
4865	* @param cArgs Number of arguments in the array.
4866	*/
4867	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
4868	{
4869	/** @todo SMP support!! */
4870	PVMCPU pVCpu = &pVM->aCpus[0];
4871
4872	/*
4873	* Validate input.
4874	*/
4875	if (!pVM)
4876	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
4877
4878	/*
4879	* Force page directory sync.
4880	*/
4881	if (pVCpu->pgm.s.fSyncFlags & PGM_SYNC_ALWAYS)
4882	{
4883	ASMAtomicAndU32(&pVCpu->pgm.s.fSyncFlags, ~PGM_SYNC_ALWAYS);
4884	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Disabled permanent forced page directory syncing.\n");
4885	}
4886	else
4887	{
4888	ASMAtomicOrU32(&pVCpu->pgm.s.fSyncFlags, PGM_SYNC_ALWAYS);
4889	VM_FF_SET(pVM, VM_FF_PGM_SYNC_CR3);
4890	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Enabled permanent forced page directory syncing.\n");
4891	}
4892	}
4893
4894	#endif /* VBOX_WITH_DEBUGGER */
4895
4896	/**
4897	* pvUser argument of the pgmR3CheckIntegrity*Node callbacks.
4898	*/
4899	typedef struct PGMCHECKINTARGS
4900	{
4901	bool fLeftToRight; /*< true: left-to-right; false: right-to-left. /
4902	PPGMPHYSHANDLER pPrevPhys;
4903	PPGMVIRTHANDLER pPrevVirt;
4904	PPGMPHYS2VIRTHANDLER pPrevPhys2Virt;
4905	PVM pVM;
4906	} PGMCHECKINTARGS, *PPGMCHECKINTARGS;
4907
4908	/**
4909	* Validate a node in the physical handler tree.
4910	*
4911	* @returns 0 on if ok, other wise 1.
4912	* @param pNode The handler node.
4913	* @param pvUser pVM.
4914	*/
4915	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
4916	{
4917	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
4918	PPGMPHYSHANDLER pCur = (PPGMPHYSHANDLER)pNode;
4919	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
4920	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGp-%RGp %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
4921	AssertReleaseMsg( !pArgs->pPrevPhys
4922	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys->Core.KeyLast > pCur->Core.Key),
4923	("pPrevPhys=%p %RGp-%RGp %s\n"
4924	" pCur=%p %RGp-%RGp %s\n",
4925	pArgs->pPrevPhys, pArgs->pPrevPhys->Core.Key, pArgs->pPrevPhys->Core.KeyLast, pArgs->pPrevPhys->pszDesc,
4926	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
4927	pArgs->pPrevPhys = pCur;
4928	return 0;
4929	}
4930
4931
4932	/**
4933	* Validate a node in the virtual handler tree.
4934	*
4935	* @returns 0 on if ok, other wise 1.
4936	* @param pNode The handler node.
4937	* @param pvUser pVM.
4938	*/
4939	static DECLCALLBACK(int) pgmR3CheckIntegrityVirtHandlerNode(PAVLROGCPTRNODECORE pNode, void *pvUser)
4940	{
4941	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
4942	PPGMVIRTHANDLER pCur = (PPGMVIRTHANDLER)pNode;
4943	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
4944	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGv-%RGv %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
4945	AssertReleaseMsg( !pArgs->pPrevVirt
4946	\|\| (pArgs->fLeftToRight ? pArgs->pPrevVirt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevVirt->Core.KeyLast > pCur->Core.Key),
4947	("pPrevVirt=%p %RGv-%RGv %s\n"
4948	" pCur=%p %RGv-%RGv %s\n",
4949	pArgs->pPrevVirt, pArgs->pPrevVirt->Core.Key, pArgs->pPrevVirt->Core.KeyLast, pArgs->pPrevVirt->pszDesc,
4950	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
4951	for (unsigned iPage = 0; iPage < pCur->cPages; iPage++)
4952	{
4953	AssertReleaseMsg(pCur->aPhysToVirt[iPage].offVirtHandler == -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage]),
4954	("pCur=%p %RGv-%RGv %s\n"
4955	"iPage=%d offVirtHandle=%#x expected %#x\n",
4956	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc,
4957	iPage, pCur->aPhysToVirt[iPage].offVirtHandler, -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage])));
4958	}
4959	pArgs->pPrevVirt = pCur;
4960	return 0;
4961	}
4962
4963
4964	/**
4965	* Validate a node in the virtual handler tree.
4966	*
4967	* @returns 0 on if ok, other wise 1.
4968	* @param pNode The handler node.
4969	* @param pvUser pVM.
4970	*/
4971	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysToVirtHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
4972	{
4973	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
4974	PPGMPHYS2VIRTHANDLER pCur = (PPGMPHYS2VIRTHANDLER)pNode;
4975	AssertReleaseMsgReturn(!((uintptr_t)pCur & 3), ("\n"), 1);
4976	AssertReleaseMsgReturn(!(pCur->offVirtHandler & 3), ("\n"), 1);
4977	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGp-%RGp\n", pCur, pCur->Core.Key, pCur->Core.KeyLast));
4978	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
4979	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
4980	("pPrevPhys2Virt=%p %RGp-%RGp\n"
4981	" pCur=%p %RGp-%RGp\n",
4982	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
4983	pCur, pCur->Core.Key, pCur->Core.KeyLast));
4984	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
4985	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
4986	("pPrevPhys2Virt=%p %RGp-%RGp\n"
4987	" pCur=%p %RGp-%RGp\n",
4988	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
4989	pCur, pCur->Core.Key, pCur->Core.KeyLast));
4990	AssertReleaseMsg((pCur->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD),
4991	("pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4992	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
4993	if (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK)
4994	{
4995	PPGMPHYS2VIRTHANDLER pCur2 = pCur;
4996	for (;;)
4997	{
4998	pCur2 = (PPGMPHYS2VIRTHANDLER)((intptr_t)pCur + (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK));
4999	AssertReleaseMsg(pCur2 != pCur,
5000	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
5001	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
5002	AssertReleaseMsg((pCur2->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == PGMPHYS2VIRTHANDLER_IN_TREE,
5003	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
5004	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
5005	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
5006	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
5007	AssertReleaseMsg((pCur2->Core.Key ^ pCur->Core.Key) < PAGE_SIZE,
5008	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
5009	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
5010	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
5011	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
5012	AssertReleaseMsg((pCur2->Core.KeyLast ^ pCur->Core.KeyLast) < PAGE_SIZE,
5013	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
5014	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
5015	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
5016	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
5017	if (!(pCur2->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK))
5018	break;
5019	}
5020	}
5021
5022	pArgs->pPrevPhys2Virt = pCur;
5023	return 0;
5024	}
5025
5026
5027	/**
5028	* Perform an integrity check on the PGM component.
5029	*
5030	* @returns VINF_SUCCESS if everything is fine.
5031	* @returns VBox error status after asserting on integrity breach.
5032	* @param pVM The VM handle.
5033	*/
5034	VMMR3DECL(int) PGMR3CheckIntegrity(PVM pVM)
5035	{
5036	AssertReleaseReturn(pVM->pgm.s.offVM, VERR_INTERNAL_ERROR);
5037
5038	/*
5039	* Check the trees.
5040	*/
5041	int cErrors = 0;
5042	const static PGMCHECKINTARGS s_LeftToRight = { true, NULL, NULL, NULL, pVM };
5043	const static PGMCHECKINTARGS s_RightToLeft = { false, NULL, NULL, NULL, pVM };
5044	PGMCHECKINTARGS Args = s_LeftToRight;
5045	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, true, pgmR3CheckIntegrityPhysHandlerNode, &Args);
5046	Args = s_RightToLeft;
5047	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, false, pgmR3CheckIntegrityPhysHandlerNode, &Args);
5048	Args = s_LeftToRight;
5049	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->VirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
5050	Args = s_RightToLeft;
5051	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->VirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
5052	Args = s_LeftToRight;
5053	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->HyperVirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
5054	Args = s_RightToLeft;
5055	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->HyperVirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
5056	Args = s_LeftToRight;
5057	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysToVirtHandlers, true, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
5058	Args = s_RightToLeft;
5059	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysToVirtHandlers, false, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
5060
5061	return !cErrors ? VINF_SUCCESS : VERR_INTERNAL_ERROR;
5062	}
5063
5064

Note: See TracBrowser for help on using the repository browser.

source: vbox/trunk/src/VBox/VMM/PGM.cpp@ 18927

Download in other formats: