PGM.cpp@ 9021

Last change on this file since 9021 was 9021, checked in by vboxsync, 17 years ago
Nested paging updates. Extra paging mode added to prevent illegal changes to the shadow page table.
Property svn:eol-style set to `native` Property svn:keywords set to `Id`
File size: 190.9 KB

Line
1	/* $Id: PGM.cpp 9021 2008-05-21 14:38:13Z vboxsync $ */
2	/** @file
3	* PGM - Page Manager and Monitor. (Mixing stuff here, not good?)
4	*/
5
6	/*
7	* Copyright (C) 2006-2007 Sun Microsystems, Inc.
8	*
9	* This file is part of VirtualBox Open Source Edition (OSE), as
10	* available from http://www.virtualbox.org. This file is free software;
11	* you can redistribute it and/or modify it under the terms of the GNU
12	* General Public License (GPL) as published by the Free Software
13	* Foundation, in version 2 as it comes in the "COPYING" file of the
14	* VirtualBox OSE distribution. VirtualBox OSE is distributed in the
15	* hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
16	*
17	* Please contact Sun Microsystems, Inc., 4150 Network Circle, Santa
18	* Clara, CA 95054 USA or visit http://www.sun.com if you need
19	* additional information or have any questions.
20	*/
21
22
23	/** @page pg_pgm PGM - The Page Manager and Monitor
24	*
25	*
26	*
27	* @section sec_pgm_modes Paging Modes
28	*
29	* There are three memory contexts: Host Context (HC), Guest Context (GC)
30	* and intermediate context. When talking about paging HC can also be refered to
31	* as "host paging", and GC refered to as "shadow paging".
32	*
33	* We define three basic paging modes: 32-bit, PAE and AMD64. The host paging mode
34	* is defined by the host operating system. The mode used in the shadow paging mode
35	* depends on the host paging mode and what the mode the guest is currently in. The
36	* following relation between the two is defined:
37	*
38	* @verbatim
39	Host > 32-bit \| PAE \| AMD64 \|
40	Guest \| \| \| \|
41	==v================================
42	32-bit 32-bit PAE PAE
43	-------\|--------\|--------\|--------\|
44	PAE PAE PAE PAE
45	-------\|--------\|--------\|--------\|
46	AMD64 AMD64 AMD64 AMD64
47	-------\|--------\|--------\|--------\| @endverbatim
48	*
49	* All configuration except those in the diagonal (upper left) are expected to
50	* require special effort from the switcher (i.e. a bit slower).
51	*
52	*
53	*
54	*
55	* @section sec_pgm_shw The Shadow Memory Context
56	*
57	*
58	* [..]
59	*
60	* Because of guest context mappings requires PDPT and PML4 entries to allow
61	* writing on AMD64, the two upper levels will have fixed flags whatever the
62	* guest is thinking of using there. So, when shadowing the PD level we will
63	* calculate the effective flags of PD and all the higher levels. In legacy
64	* PAE mode this only applies to the PWT and PCD bits (the rest are
65	* ignored/reserved/MBZ). We will ignore those bits for the present.
66	*
67	*
68	*
69	* @section sec_pgm_int The Intermediate Memory Context
70	*
71	* The world switch goes thru an intermediate memory context which purpose it is
72	* to provide different mappings of the switcher code. All guest mappings are also
73	* present in this context.
74	*
75	* The switcher code is mapped at the same location as on the host, at an
76	* identity mapped location (physical equals virtual address), and at the
77	* hypervisor location.
78	*
79	* PGM maintain page tables for 32-bit, PAE and AMD64 paging modes. This
80	* simplifies switching guest CPU mode and consistency at the cost of more
81	* code to do the work. All memory use for those page tables is located below
82	* 4GB (this includes page tables for guest context mappings).
83	*
84	*
85	* @subsection subsec_pgm_int_gc Guest Context Mappings
86	*
87	* During assignment and relocation of a guest context mapping the intermediate
88	* memory context is used to verify the new location.
89	*
90	* Guest context mappings are currently restricted to below 4GB, for reasons
91	* of simplicity. This may change when we implement AMD64 support.
92	*
93	*
94	*
95	*
96	* @section sec_pgm_misc Misc
97	*
98	* @subsection subsec_pgm_misc_diff Differences Between Legacy PAE and Long Mode PAE
99	*
100	* The differences between legacy PAE and long mode PAE are:
101	* -# PDPE bits 1, 2, 5 and 6 are defined differently. In leagcy mode they are
102	* all marked down as must-be-zero, while in long mode 1, 2 and 5 have the
103	* usual meanings while 6 is ignored (AMD). This means that upon switching to
104	* legacy PAE mode we'll have to clear these bits and when going to long mode
105	* they must be set. This applies to both intermediate and shadow contexts,
106	* however we don't need to do it for the intermediate one since we're
107	* executing with CR0.WP at that time.
108	* -# CR3 allows a 32-byte aligned address in legacy mode, while in long mode
109	* a page aligned one is required.
110	*
111	*
112	* @section sec_pgm_handlers Access Handlers
113	*
114	* Placeholder.
115	*
116	*
117	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
118	*
119	* Placeholder.
120	*
121	*
122	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
123	*
124	* We currently implement three types of virtual access handlers: ALL, WRITE
125	* and HYPERVISOR (WRITE). See PGMVIRTHANDLERTYPE for some more details.
126	*
127	* The HYPERVISOR access handlers is kept in a separate tree since it doesn't apply
128	* to physical pages (PGMTREES::HyperVirtHandlers) and only needs to be consulted in
129	* a special \#PF case. The ALL and WRITE are in the PGMTREES::VirtHandlers tree, the
130	* rest of this section is going to be about these handlers.
131	*
132	* We'll go thru the life cycle of a handler and try make sense of it all, don't know
133	* how successfull this is gonna be...
134	*
135	* 1. A handler is registered thru the PGMR3HandlerVirtualRegister and
136	* PGMHandlerVirtualRegisterEx APIs. We check for conflicting virtual handlers
137	* and create a new node that is inserted into the AVL tree (range key). Then
138	* a full PGM resync is flagged (clear pool, sync cr3, update virtual bit of PGMPAGE).
139	*
140	* 2. The following PGMSyncCR3/SyncCR3 operation will first make invoke HandlerVirtualUpdate.
141	*
142	* 2a. HandlerVirtualUpdate will will lookup all the pages covered by virtual handlers
143	* via the current guest CR3 and update the physical page -> virtual handler
144	* translation. Needless to say, this doesn't exactly scale very well. If any changes
145	* are detected, it will flag a virtual bit update just like we did on registration.
146	* PGMPHYS pages with changes will have their virtual handler state reset to NONE.
147	*
148	* 2b. The virtual bit update process will iterate all the pages covered by all the
149	* virtual handlers and update the PGMPAGE virtual handler state to the max of all
150	* virtual handlers on that page.
151	*
152	* 2c. Back in SyncCR3 we will now flush the entire shadow page cache to make sure
153	* we don't miss any alias mappings of the monitored pages.
154	*
155	* 2d. SyncCR3 will then proceed with syncing the CR3 table.
156	*
157	* 3. \#PF(np,read) on a page in the range. This will cause it to be synced
158	* read-only and resumed if it's a WRITE handler. If it's an ALL handler we
159	* will call the handlers like in the next step. If the physical mapping has
160	* changed we will - some time in the future - perform a handler callback
161	* (optional) and update the physical -> virtual handler cache.
162	*
163	* 4. \#PF(,write) on a page in the range. This will cause the handler to
164	* be invoked.
165	*
166	* 5. The guest invalidates the page and changes the physical backing or
167	* unmaps it. This should cause the invalidation callback to be invoked
168	* (it might not yet be 100% perfect). Exactly what happens next... is
169	* this where we mess up and end up out of sync for a while?
170	*
171	* 6. The handler is deregistered by the client via PGMHandlerVirtualDeregister.
172	* We will then set all PGMPAGEs in the physical -> virtual handler cache for
173	* this handler to NONE and trigger a full PGM resync (basically the same
174	* as int step 1). Which means 2 is executed again.
175	*
176	*
177	* @subsubsection sub_sec_pgm_handler_virt_todo TODOs
178	*
179	* There is a bunch of things that needs to be done to make the virtual handlers
180	* work 100% correctly and work more efficiently.
181	*
182	* The first bit hasn't been implemented yet because it's going to slow the
183	* whole mess down even more, and besides it seems to be working reliably for
184	* our current uses. OTOH, some of the optimizations might end up more or less
185	* implementing the missing bits, so we'll see.
186	*
187	* On the optimization side, the first thing to do is to try avoid unnecessary
188	* cache flushing. Then try team up with the shadowing code to track changes
189	* in mappings by means of access to them (shadow in), updates to shadows pages,
190	* invlpg, and shadow PT discarding (perhaps).
191	*
192	* Some idea that have popped up for optimization for current and new features:
193	* - bitmap indicating where there are virtual handlers installed.
194	* (4KB => 220 pages, page 212 => covers 32-bit address space 1:1!)
195	* - Further optimize this by min/max (needs min/max avl getters).
196	* - Shadow page table entry bit (if any left)?
197	*
198	*/
199
200
201	/** @page pg_pgmPhys PGMPhys - Physical Guest Memory Management.
202	*
203	*
204	* Objectives:
205	* - Guest RAM over-commitment using memory ballooning,
206	* zero pages and general page sharing.
207	* - Moving or mirroring a VM onto a different physical machine.
208	*
209	*
210	* @subsection subsec_pgmPhys_Definitions Definitions
211	*
212	* Allocation chunk - A RTR0MemObjAllocPhysNC object and the tracking
213	* machinery assoicated with it.
214	*
215	*
216	*
217	*
218	* @subsection subsec_pgmPhys_AllocPage Allocating a page.
219	*
220	* Initially we map all guest memory to the (per VM) zero page, which
221	* means that none of the read functions will cause pages to be allocated.
222	*
223	* Exception, access bit in page tables that have been shared. This must
224	* be handled, but we must also make sure PGMGst*Modify doesn't make
225	* unnecessary modifications.
226	*
227	* Allocation points:
228	* - PGMPhysWriteGCPhys and PGMPhysWrite.
229	* - Replacing a zero page mapping at \#PF.
230	* - Replacing a shared page mapping at \#PF.
231	* - ROM registration (currently MMR3RomRegister).
232	* - VM restore (pgmR3Load).
233	*
234	* For the first three it would make sense to keep a few pages handy
235	* until we've reached the max memory commitment for the VM.
236	*
237	* For the ROM registration, we know exactly how many pages we need
238	* and will request these from ring-0. For restore, we will save
239	* the number of non-zero pages in the saved state and allocate
240	* them up front. This would allow the ring-0 component to refuse
241	* the request if the isn't sufficient memory available for VM use.
242	*
243	* Btw. for both ROM and restore allocations we won't be requiring
244	* zeroed pages as they are going to be filled instantly.
245	*
246	*
247	* @subsection subsec_pgmPhys_FreePage Freeing a page
248	*
249	* There are a few points where a page can be freed:
250	* - After being replaced by the zero page.
251	* - After being replaced by a shared page.
252	* - After being ballooned by the guest additions.
253	* - At reset.
254	* - At restore.
255	*
256	* When freeing one or more pages they will be returned to the ring-0
257	* component and replaced by the zero page.
258	*
259	* The reasoning for clearing out all the pages on reset is that it will
260	* return us to the exact same state as on power on, and may thereby help
261	* us reduce the memory load on the system. Further it might have a
262	* (temporary) positive influence on memory fragmentation (@see subsec_pgmPhys_Fragmentation).
263	*
264	* On restore, as mention under the allocation topic, pages should be
265	* freed / allocated depending on how many is actually required by the
266	* new VM state. The simplest approach is to do like on reset, and free
267	* all non-ROM pages and then allocate what we need.
268	*
269	* A measure to prevent some fragmentation, would be to let each allocation
270	* chunk have some affinity towards the VM having allocated the most pages
271	* from it. Also, try make sure to allocate from allocation chunks that
272	* are almost full. Admittedly, both these measures might work counter to
273	* our intentions and its probably not worth putting a lot of effort,
274	* cpu time or memory into this.
275	*
276	*
277	* @subsection subsec_pgmPhys_SharePage Sharing a page
278	*
279	* The basic idea is that there there will be a idle priority kernel
280	* thread walking the non-shared VM pages hashing them and looking for
281	* pages with the same checksum. If such pages are found, it will compare
282	* them byte-by-byte to see if they actually are identical. If found to be
283	* identical it will allocate a shared page, copy the content, check that
284	* the page didn't change while doing this, and finally request both the
285	* VMs to use the shared page instead. If the page is all zeros (special
286	* checksum and byte-by-byte check) it will request the VM that owns it
287	* to replace it with the zero page.
288	*
289	* To make this efficient, we will have to make sure not to try share a page
290	* that will change its contents soon. This part requires the most work.
291	* A simple idea would be to request the VM to write monitor the page for
292	* a while to make sure it isn't modified any time soon. Also, it may
293	* make sense to skip pages that are being write monitored since this
294	* information is readily available to the thread if it works on the
295	* per-VM guest memory structures (presently called PGMRAMRANGE).
296	*
297	*
298	* @subsection subsec_pgmPhys_Fragmentation Fragmentation Concerns and Counter Measures
299	*
300	* The pages are organized in allocation chunks in ring-0, this is a necessity
301	* if we wish to have an OS agnostic approach to this whole thing. (On Linux we
302	* could easily work on a page-by-page basis if we liked. Whether this is possible
303	* or efficient on NT I don't quite know.) Fragmentation within these chunks may
304	* become a problem as part of the idea here is that we wish to return memory to
305	* the host system.
306	*
307	* For instance, starting two VMs at the same time, they will both allocate the
308	* guest memory on-demand and if permitted their page allocations will be
309	* intermixed. Shut down one of the two VMs and it will be difficult to return
310	* any memory to the host system because the page allocation for the two VMs are
311	* mixed up in the same allocation chunks.
312	*
313	* To further complicate matters, when pages are freed because they have been
314	* ballooned or become shared/zero the whole idea is that the page is supposed
315	* to be reused by another VM or returned to the host system. This will cause
316	* allocation chunks to contain pages belonging to different VMs and prevent
317	* returning memory to the host when one of those VM shuts down.
318	*
319	* The only way to really deal with this problem is to move pages. This can
320	* either be done at VM shutdown and or by the idle priority worker thread
321	* that will be responsible for finding sharable/zero pages. The mechanisms
322	* involved for coercing a VM to move a page (or to do it for it) will be
323	* the same as when telling it to share/zero a page.
324	*
325	*
326	* @subsection subsec_pgmPhys_Tracking Tracking Structures And Their Cost
327	*
328	* There's a difficult balance between keeping the per-page tracking structures
329	* (global and guest page) easy to use and keeping them from eating too much
330	* memory. We have limited virtual memory resources available when operating in
331	* 32-bit kernel space (on 64-bit there'll it's quite a different story). The
332	* tracking structures will be attemted designed such that we can deal with up
333	* to 32GB of memory on a 32-bit system and essentially unlimited on 64-bit ones.
334	*
335	*
336	* @subsubsection subsubsec_pgmPhys_Tracking_Kernel Kernel Space
337	*
338	* @see pg_GMM
339	*
340	* @subsubsection subsubsec_pgmPhys_Tracking_PerVM Per-VM
341	*
342	* Fixed info is the physical address of the page (HCPhys) and the page id
343	* (described above). Theoretically we'll need 48(-12) bits for the HCPhys part.
344	* Today we've restricting ourselves to 40(-12) bits because this is the current
345	* restrictions of all AMD64 implementations (I think Barcelona will up this
346	* to 48(-12) bits, not that it really matters) and I needed the bits for
347	* tracking mappings of a page. 48-12 = 36. That leaves 28 bits, which means a
348	* decent range for the page id: 2^(28+12) = 1024TB.
349	*
350	* In additions to these, we'll have to keep maintaining the page flags as we
351	* currently do. Although it wouldn't harm to optimize these quite a bit, like
352	* for instance the ROM shouldn't depend on having a write handler installed
353	* in order for it to become read-only. A RO/RW bit should be considered so
354	* that the page syncing code doesn't have to mess about checking multiple
355	* flag combinations (ROM \|\| RW handler \|\| write monitored) in order to
356	* figure out how to setup a shadow PTE. But this of course, is second
357	* priority at present. Current this requires 12 bits, but could probably
358	* be optimized to ~8.
359	*
360	* Then there's the 24 bits used to track which shadow page tables are
361	* currently mapping a page for the purpose of speeding up physical
362	* access handlers, and thereby the page pool cache. More bit for this
363	* purpose wouldn't hurt IIRC.
364	*
365	* Then there is a new bit in which we need to record what kind of page
366	* this is, shared, zero, normal or write-monitored-normal. This'll
367	* require 2 bits. One bit might be needed for indicating whether a
368	* write monitored page has been written to. And yet another one or
369	* two for tracking migration status. 3-4 bits total then.
370	*
371	* Whatever is left will can be used to record the sharabilitiy of a
372	* page. The page checksum will not be stored in the per-VM table as
373	* the idle thread will not be permitted to do modifications to it.
374	* It will instead have to keep its own working set of potentially
375	* shareable pages and their check sums and stuff.
376	*
377	* For the present we'll keep the current packing of the
378	* PGMRAMRANGE::aHCPhys to keep the changes simple, only of course,
379	* we'll have to change it to a struct with a total of 128-bits at
380	* our disposal.
381	*
382	* The initial layout will be like this:
383	* @verbatim
384	RTHCPHYS HCPhys; The current stuff.
385	63:40 Current shadow PT tracking stuff.
386	39:12 The physical page frame number.
387	11:0 The current flags.
388	uint32_t u28PageId : 28; The page id.
389	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
390	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
391	uint32_t u1Reserved : 1; Reserved for later.
392	uint32_t u32Reserved; Reserved for later, mostly sharing stats.
393	@endverbatim
394	*
395	* The final layout will be something like this:
396	* @verbatim
397	RTHCPHYS HCPhys; The current stuff.
398	63:48 High page id (12+).
399	47:12 The physical page frame number.
400	11:0 Low page id.
401	uint32_t fReadOnly : 1; Whether it's readonly page (rom or monitored in some way).
402	uint32_t u3Type : 3; The page type {RESERVED, MMIO, MMIO2, ROM, shadowed ROM, RAM}.
403	uint32_t u2PhysMon : 2; Physical access handler type {none, read, write, all}.
404	uint32_t u2VirtMon : 2; Virtual access handler type {none, read, write, all}..
405	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
406	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
407	uint32_t u20Reserved : 20; Reserved for later, mostly sharing stats.
408	uint32_t u32Tracking; The shadow PT tracking stuff, roughly.
409	@endverbatim
410	*
411	* Cost wise, this means we'll double the cost for guest memory. There isn't anyway
412	* around that I'm afraid. It means that the cost of dealing out 32GB of memory
413	* to one or more VMs is: (32GB >> PAGE_SHIFT) * 16 bytes, or 128MBs. Or another
414	* example, the VM heap cost when assigning 1GB to a VM will be: 4MB.
415	*
416	* A couple of cost examples for the total cost per-VM + kernel.
417	* 32-bit Windows and 32-bit linux:
418	* 1GB guest ram, 256K pages: 4MB + 2MB(+) = 6MB
419	* 4GB guest ram, 1M pages: 16MB + 8MB(+) = 24MB
420	* 32GB guest ram, 8M pages: 128MB + 64MB(+) = 192MB
421	* 64-bit Windows and 64-bit linux:
422	* 1GB guest ram, 256K pages: 4MB + 3MB(+) = 7MB
423	* 4GB guest ram, 1M pages: 16MB + 12MB(+) = 28MB
424	* 32GB guest ram, 8M pages: 128MB + 96MB(+) = 224MB
425	*
426	* UPDATE - 2007-09-27:
427	* Will need a ballooned flag/state too because we cannot
428	* trust the guest 100% and reporting the same page as ballooned more
429	* than once will put the GMM off balance.
430	*
431	*
432	* @subsection subsec_pgmPhys_Serializing Serializing Access
433	*
434	* Initially, we'll try a simple scheme:
435	*
436	* - The per-VM RAM tracking structures (PGMRAMRANGE) is only modified
437	* by the EMT thread of that VM while in the pgm critsect.
438	* - Other threads in the VM process that needs to make reliable use of
439	* the per-VM RAM tracking structures will enter the critsect.
440	* - No process external thread or kernel thread will ever try enter
441	* the pgm critical section, as that just won't work.
442	* - The idle thread (and similar threads) doesn't not need 100% reliable
443	* data when performing it tasks as the EMT thread will be the one to
444	* do the actual changes later anyway. So, as long as it only accesses
445	* the main ram range, it can do so by somehow preventing the VM from
446	* being destroyed while it works on it...
447	*
448	* - The over-commitment management, including the allocating/freeing
449	* chunks, is serialized by a ring-0 mutex lock (a fast one since the
450	* more mundane mutex implementation is broken on Linux).
451	* - A separeate mutex is protecting the set of allocation chunks so
452	* that pages can be shared or/and freed up while some other VM is
453	* allocating more chunks. This mutex can be take from under the other
454	* one, but not the otherway around.
455	*
456	*
457	* @subsection subsec_pgmPhys_Request VM Request interface
458	*
459	* When in ring-0 it will become necessary to send requests to a VM so it can
460	* for instance move a page while defragmenting during VM destroy. The idle
461	* thread will make use of this interface to request VMs to setup shared
462	* pages and to perform write monitoring of pages.
463	*
464	* I would propose an interface similar to the current VMReq interface, similar
465	* in that it doesn't require locking and that the one sending the request may
466	* wait for completion if it wishes to. This shouldn't be very difficult to
467	* realize.
468	*
469	* The requests themselves are also pretty simple. They are basically:
470	* -# Check that some precondition is still true.
471	* -# Do the update.
472	* -# Update all shadow page tables involved with the page.
473	*
474	* The 3rd step is identical to what we're already doing when updating a
475	* physical handler, see pgmHandlerPhysicalSetRamFlagsAndFlushShadowPTs.
476	*
477	*
478	*
479	* @section sec_pgmPhys_MappingCaches Mapping Caches
480	*
481	* In order to be able to map in and out memory and to be able to support
482	* guest with more RAM than we've got virtual address space, we'll employing
483	* a mapping cache. There is already a tiny one for GC (see PGMGCDynMapGCPageEx)
484	* and we'll create a similar one for ring-0 unless we decide to setup a dedicate
485	* memory context for the HWACCM execution.
486	*
487	*
488	* @subsection subsec_pgmPhys_MappingCaches_R3 Ring-3
489	*
490	* We've considered implementing the ring-3 mapping cache page based but found
491	* that this was bother some when one had to take into account TLBs+SMP and
492	* portability (missing the necessary APIs on several platforms). There were
493	* also some performance concerns with this approach which hadn't quite been
494	* worked out.
495	*
496	* Instead, we'll be mapping allocation chunks into the VM process. This simplifies
497	* matters greatly quite a bit since we don't need to invent any new ring-0 stuff,
498	* only some minor RTR0MEMOBJ mapping stuff. The main concern here is that mapping
499	* compared to the previous idea is that mapping or unmapping a 1MB chunk is more
500	* costly than a single page, although how much more costly is uncertain. We'll
501	* try address this by using a very big cache, preferably bigger than the actual
502	* VM RAM size if possible. The current VM RAM sizes should give some idea for
503	* 32-bit boxes, while on 64-bit we can probably get away with employing an
504	* unlimited cache.
505	*
506	* The cache have to parts, as already indicated, the ring-3 side and the
507	* ring-0 side.
508	*
509	* The ring-0 will be tied to the page allocator since it will operate on the
510	* memory objects it contains. It will therefore require the first ring-0 mutex
511	* discussed in @ref subsec_pgmPhys_Serializing. We
512	* some double house keeping wrt to who has mapped what I think, since both
513	* VMMR0.r0 and RTR0MemObj will keep track of mapping relataions
514	*
515	* The ring-3 part will be protected by the pgm critsect. For simplicity, we'll
516	* require anyone that desires to do changes to the mapping cache to do that
517	* from within this critsect. Alternatively, we could employ a separate critsect
518	* for serializing changes to the mapping cache as this would reduce potential
519	* contention with other threads accessing mappings unrelated to the changes
520	* that are in process. We can see about this later, contention will show
521	* up in the statistics anyway, so it'll be simple to tell.
522	*
523	* The organization of the ring-3 part will be very much like how the allocation
524	* chunks are organized in ring-0, that is in an AVL tree by chunk id. To avoid
525	* having to walk the tree all the time, we'll have a couple of lookaside entries
526	* like in we do for I/O ports and MMIO in IOM.
527	*
528	* The simplified flow of a PGMPhysRead/Write function:
529	* -# Enter the PGM critsect.
530	* -# Lookup GCPhys in the ram ranges and get the Page ID.
531	* -# Calc the Allocation Chunk ID from the Page ID.
532	* -# Check the lookaside entries and then the AVL tree for the Chunk ID.
533	* If not found in cache:
534	* -# Call ring-0 and request it to be mapped and supply
535	* a chunk to be unmapped if the cache is maxed out already.
536	* -# Insert the new mapping into the AVL tree (id + R3 address).
537	* -# Update the relevant lookaside entry and return the mapping address.
538	* -# Do the read/write according to monitoring flags and everything.
539	* -# Leave the critsect.
540	*
541	*
542	* @section sec_pgmPhys_Fallback Fallback
543	*
544	* Current all the "second tier" hosts will not support the RTR0MemObjAllocPhysNC
545	* API and thus require a fallback.
546	*
547	* So, when RTR0MemObjAllocPhysNC returns VERR_NOT_SUPPORTED the page allocator
548	* will return to the ring-3 caller (and later ring-0) and asking it to seed
549	* the page allocator with some fresh pages (VERR_GMM_SEED_ME). Ring-3 will
550	* then perform an SUPPageAlloc(cbChunk >> PAGE_SHIFT) call and make a
551	* "SeededAllocPages" call to ring-0.
552	*
553	* The first time ring-0 sees the VERR_NOT_SUPPORTED failure it will disable
554	* all page sharing (zero page detection will continue). It will also force
555	* all allocations to come from the VM which seeded the page. Both these
556	* measures are taken to make sure that there will never be any need for
557	* mapping anything into ring-3 - everything will be mapped already.
558	*
559	* Whether we'll continue to use the current MM locked memory management
560	* for this I don't quite know (I'd prefer not to and just ditch that all
561	* togther), we'll see what's simplest to do.
562	*
563	*
564	*
565	* @section sec_pgmPhys_Changes Changes
566	*
567	* Breakdown of the changes involved?
568	*/
569
570
571	/** Saved state data unit version. */
572	#define PGM_SAVED_STATE_VERSION 6
573
574	/*******************************************************************************
575	* Header Files *
576	*******************************************************************************/
577	#define LOG_GROUP LOG_GROUP_PGM
578	#include <VBox/dbgf.h>
579	#include <VBox/pgm.h>
580	#include <VBox/cpum.h>
581	#include <VBox/iom.h>
582	#include <VBox/sup.h>
583	#include <VBox/mm.h>
584	#include <VBox/em.h>
585	#include <VBox/stam.h>
586	#include <VBox/rem.h>
587	#include <VBox/dbgf.h>
588	#include <VBox/rem.h>
589	#include <VBox/selm.h>
590	#include <VBox/ssm.h>
591	#include "PGMInternal.h"
592	#include <VBox/vm.h>
593	#include <VBox/dbg.h>
594	#include <VBox/hwaccm.h>
595
596	#include <iprt/assert.h>
597	#include <iprt/alloc.h>
598	#include <iprt/asm.h>
599	#include <iprt/thread.h>
600	#include <iprt/string.h>
601	#include <VBox/param.h>
602	#include <VBox/err.h>
603
604
605
606	/*******************************************************************************
607	* Internal Functions *
608	*******************************************************************************/
609	static int pgmR3InitPaging(PVM pVM);
610	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
611	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
612	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
613	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser);
614	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
615	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
616	#ifdef VBOX_STRICT
617	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser);
618	#endif
619	static DECLCALLBACK(int) pgmR3Save(PVM pVM, PSSMHANDLE pSSM);
620	static DECLCALLBACK(int) pgmR3Load(PVM pVM, PSSMHANDLE pSSM, uint32_t u32Version);
621	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0);
622	static void pgmR3ModeDataSwitch(PVM pVM, PGMMODE enmShw, PGMMODE enmGst);
623	static PGMMODE pgmR3CalcShadowMode(PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher);
624
625	#ifdef VBOX_WITH_STATISTICS
626	static void pgmR3InitStats(PVM pVM);
627	#endif
628
629	#ifdef VBOX_WITH_DEBUGGER
630	/** @todo all but the two last commands must be converted to 'info'. */
631	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
632	static DECLCALLBACK(int) pgmR3CmdMap(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
633	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
634	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
635	#endif
636
637
638	/*******************************************************************************
639	* Global Variables *
640	*******************************************************************************/
641	#ifdef VBOX_WITH_DEBUGGER
642	/** Command descriptors. */
643	static const DBGCCMD g_aCmds[] =
644	{
645	/* pszCmd, cArgsMin, cArgsMax, paArgDesc, cArgDescs, pResultDesc, fFlags, pfnHandler pszSyntax, ....pszDescription */
646	{ "pgmram", 0, 0, NULL, 0, NULL, 0, pgmR3CmdRam, "", "Display the ram ranges." },
647	{ "pgmmap", 0, 0, NULL, 0, NULL, 0, pgmR3CmdMap, "", "Display the mapping ranges." },
648	{ "pgmsync", 0, 0, NULL, 0, NULL, 0, pgmR3CmdSync, "", "Sync the CR3 page." },
649	{ "pgmsyncalways", 0, 0, NULL, 0, NULL, 0, pgmR3CmdSyncAlways, "", "Toggle permanent CR3 syncing." },
650	};
651	#endif
652
653
654
655
656	/*
657	* Shadow - 32-bit mode
658	*/
659	#define PGM_SHW_TYPE PGM_TYPE_32BIT
660	#define PGM_SHW_NAME(name) PGM_SHW_NAME_32BIT(name)
661	#define PGM_SHW_NAME_GC_STR(name) PGM_SHW_NAME_GC_32BIT_STR(name)
662	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_32BIT_STR(name)
663	#include "PGMShw.h"
664
665	/* Guest - real mode */
666	#define PGM_GST_TYPE PGM_TYPE_REAL
667	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
668	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_REAL_STR(name)
669	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
670	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_REAL(name)
671	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_32BIT_REAL_STR(name)
672	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_REAL_STR(name)
673	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
674	#include "PGMGst.h"
675	#include "PGMBth.h"
676	#undef BTH_PGMPOOLKIND_PT_FOR_PT
677	#undef PGM_BTH_NAME
678	#undef PGM_BTH_NAME_GC_STR
679	#undef PGM_BTH_NAME_R0_STR
680	#undef PGM_GST_TYPE
681	#undef PGM_GST_NAME
682	#undef PGM_GST_NAME_GC_STR
683	#undef PGM_GST_NAME_R0_STR
684
685	/* Guest - protected mode */
686	#define PGM_GST_TYPE PGM_TYPE_PROT
687	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
688	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_PROT_STR(name)
689	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
690	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_PROT(name)
691	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_32BIT_PROT_STR(name)
692	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_PROT_STR(name)
693	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
694	#include "PGMGst.h"
695	#include "PGMBth.h"
696	#undef BTH_PGMPOOLKIND_PT_FOR_PT
697	#undef PGM_BTH_NAME
698	#undef PGM_BTH_NAME_GC_STR
699	#undef PGM_BTH_NAME_R0_STR
700	#undef PGM_GST_TYPE
701	#undef PGM_GST_NAME
702	#undef PGM_GST_NAME_GC_STR
703	#undef PGM_GST_NAME_R0_STR
704
705	/* Guest - 32-bit mode */
706	#define PGM_GST_TYPE PGM_TYPE_32BIT
707	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
708	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_32BIT_STR(name)
709	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
710	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_32BIT(name)
711	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_32BIT_32BIT_STR(name)
712	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_32BIT_STR(name)
713	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_32BIT_PT
714	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_32BIT_PT_FOR_32BIT_4MB
715	#include "PGMGst.h"
716	#include "PGMBth.h"
717	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
718	#undef BTH_PGMPOOLKIND_PT_FOR_PT
719	#undef PGM_BTH_NAME
720	#undef PGM_BTH_NAME_GC_STR
721	#undef PGM_BTH_NAME_R0_STR
722	#undef PGM_GST_TYPE
723	#undef PGM_GST_NAME
724	#undef PGM_GST_NAME_GC_STR
725	#undef PGM_GST_NAME_R0_STR
726
727	#undef PGM_SHW_TYPE
728	#undef PGM_SHW_NAME
729	#undef PGM_SHW_NAME_GC_STR
730	#undef PGM_SHW_NAME_R0_STR
731
732
733	/*
734	* Shadow - PAE mode
735	*/
736	#define PGM_SHW_TYPE PGM_TYPE_PAE
737	#define PGM_SHW_NAME(name) PGM_SHW_NAME_PAE(name)
738	#define PGM_SHW_NAME_GC_STR(name) PGM_SHW_NAME_GC_PAE_STR(name)
739	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_PAE_STR(name)
740	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
741	#include "PGMShw.h"
742
743	/* Guest - real mode */
744	#define PGM_GST_TYPE PGM_TYPE_REAL
745	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
746	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_REAL_STR(name)
747	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
748	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
749	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_PAE_REAL_STR(name)
750	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_REAL_STR(name)
751	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
752	#include "PGMBth.h"
753	#undef BTH_PGMPOOLKIND_PT_FOR_PT
754	#undef PGM_BTH_NAME
755	#undef PGM_BTH_NAME_GC_STR
756	#undef PGM_BTH_NAME_R0_STR
757	#undef PGM_GST_TYPE
758	#undef PGM_GST_NAME
759	#undef PGM_GST_NAME_GC_STR
760	#undef PGM_GST_NAME_R0_STR
761
762	/* Guest - protected mode */
763	#define PGM_GST_TYPE PGM_TYPE_PROT
764	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
765	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_PROT_STR(name)
766	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
767	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PROT(name)
768	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_PAE_PROT_STR(name)
769	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PROT_STR(name)
770	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
771	#include "PGMBth.h"
772	#undef BTH_PGMPOOLKIND_PT_FOR_PT
773	#undef PGM_BTH_NAME
774	#undef PGM_BTH_NAME_GC_STR
775	#undef PGM_BTH_NAME_R0_STR
776	#undef PGM_GST_TYPE
777	#undef PGM_GST_NAME
778	#undef PGM_GST_NAME_GC_STR
779	#undef PGM_GST_NAME_R0_STR
780
781	/* Guest - 32-bit mode */
782	#define PGM_GST_TYPE PGM_TYPE_32BIT
783	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
784	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_32BIT_STR(name)
785	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
786	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_32BIT(name)
787	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_PAE_32BIT_STR(name)
788	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_32BIT_STR(name)
789	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
790	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
791	#include "PGMBth.h"
792	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
793	#undef BTH_PGMPOOLKIND_PT_FOR_PT
794	#undef PGM_BTH_NAME
795	#undef PGM_BTH_NAME_GC_STR
796	#undef PGM_BTH_NAME_R0_STR
797	#undef PGM_GST_TYPE
798	#undef PGM_GST_NAME
799	#undef PGM_GST_NAME_GC_STR
800	#undef PGM_GST_NAME_R0_STR
801
802	/* Guest - PAE mode */
803	#define PGM_GST_TYPE PGM_TYPE_PAE
804	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
805	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_PAE_STR(name)
806	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
807	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PAE(name)
808	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_PAE_PAE_STR(name)
809	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PAE_STR(name)
810	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
811	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
812	#include "PGMGst.h"
813	#include "PGMBth.h"
814	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
815	#undef BTH_PGMPOOLKIND_PT_FOR_PT
816	#undef PGM_BTH_NAME
817	#undef PGM_BTH_NAME_GC_STR
818	#undef PGM_BTH_NAME_R0_STR
819	#undef PGM_GST_TYPE
820	#undef PGM_GST_NAME
821	#undef PGM_GST_NAME_GC_STR
822	#undef PGM_GST_NAME_R0_STR
823
824	#undef PGM_SHW_TYPE
825	#undef PGM_SHW_NAME
826	#undef PGM_SHW_NAME_GC_STR
827	#undef PGM_SHW_NAME_R0_STR
828
829
830	/*
831	* Shadow - AMD64 mode
832	*/
833	#define PGM_SHW_TYPE PGM_TYPE_AMD64
834	#define PGM_SHW_NAME(name) PGM_SHW_NAME_AMD64(name)
835	#define PGM_SHW_NAME_GC_STR(name) PGM_SHW_NAME_GC_AMD64_STR(name)
836	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_AMD64_STR(name)
837	#include "PGMShw.h"
838
839	/* Guest - AMD64 mode */
840	#define PGM_GST_TYPE PGM_TYPE_AMD64
841	#define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
842	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_AMD64_STR(name)
843	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
844	#define PGM_BTH_NAME(name) PGM_BTH_NAME_AMD64_AMD64(name)
845	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_AMD64_AMD64_STR(name)
846	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_AMD64_AMD64_STR(name)
847	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
848	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
849	#include "PGMGst.h"
850	#include "PGMBth.h"
851	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
852	#undef BTH_PGMPOOLKIND_PT_FOR_PT
853	#undef PGM_BTH_NAME
854	#undef PGM_BTH_NAME_GC_STR
855	#undef PGM_BTH_NAME_R0_STR
856	#undef PGM_GST_TYPE
857	#undef PGM_GST_NAME
858	#undef PGM_GST_NAME_GC_STR
859	#undef PGM_GST_NAME_R0_STR
860
861	#undef PGM_SHW_TYPE
862	#undef PGM_SHW_NAME
863	#undef PGM_SHW_NAME_GC_STR
864	#undef PGM_SHW_NAME_R0_STR
865
866	/*
867	* Shadow - Nested paging mode
868	*/
869	#define PGM_SHW_TYPE PGM_TYPE_NESTED
870	#define PGM_SHW_NAME(name) PGM_SHW_NAME_NESTED(name)
871	#define PGM_SHW_NAME_GC_STR(name) PGM_SHW_NAME_GC_NESTED_STR(name)
872	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_NESTED_STR(name)
873	#include "PGMShw.h"
874
875	/* Guest - real mode */
876	#define PGM_GST_TYPE PGM_TYPE_REAL
877	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
878	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_REAL_STR(name)
879	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
880	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_REAL(name)
881	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_NESTED_REAL_STR(name)
882	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_REAL_STR(name)
883	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
884	#include "PGMBth.h"
885	#undef BTH_PGMPOOLKIND_PT_FOR_PT
886	#undef PGM_BTH_NAME
887	#undef PGM_BTH_NAME_GC_STR
888	#undef PGM_BTH_NAME_R0_STR
889	#undef PGM_GST_TYPE
890	#undef PGM_GST_NAME
891	#undef PGM_GST_NAME_GC_STR
892	#undef PGM_GST_NAME_R0_STR
893
894	/* Guest - protected mode */
895	#define PGM_GST_TYPE PGM_TYPE_PROT
896	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
897	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_PROT_STR(name)
898	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
899	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PROT(name)
900	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_NESTED_PROT_STR(name)
901	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PROT_STR(name)
902	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
903	#include "PGMBth.h"
904	#undef BTH_PGMPOOLKIND_PT_FOR_PT
905	#undef PGM_BTH_NAME
906	#undef PGM_BTH_NAME_GC_STR
907	#undef PGM_BTH_NAME_R0_STR
908	#undef PGM_GST_TYPE
909	#undef PGM_GST_NAME
910	#undef PGM_GST_NAME_GC_STR
911	#undef PGM_GST_NAME_R0_STR
912
913	/* Guest - 32-bit mode */
914	#define PGM_GST_TYPE PGM_TYPE_32BIT
915	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
916	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_32BIT_STR(name)
917	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
918	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_32BIT(name)
919	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_NESTED_32BIT_STR(name)
920	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_32BIT_STR(name)
921	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
922	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
923	#include "PGMBth.h"
924	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
925	#undef BTH_PGMPOOLKIND_PT_FOR_PT
926	#undef PGM_BTH_NAME
927	#undef PGM_BTH_NAME_GC_STR
928	#undef PGM_BTH_NAME_R0_STR
929	#undef PGM_GST_TYPE
930	#undef PGM_GST_NAME
931	#undef PGM_GST_NAME_GC_STR
932	#undef PGM_GST_NAME_R0_STR
933
934	/* Guest - PAE mode */
935	#define PGM_GST_TYPE PGM_TYPE_PAE
936	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
937	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_PAE_STR(name)
938	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
939	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PAE(name)
940	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_NESTED_PAE_STR(name)
941	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PAE_STR(name)
942	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
943	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
944	#include "PGMBth.h"
945	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
946	#undef BTH_PGMPOOLKIND_PT_FOR_PT
947	#undef PGM_BTH_NAME
948	#undef PGM_BTH_NAME_GC_STR
949	#undef PGM_BTH_NAME_R0_STR
950	#undef PGM_GST_TYPE
951	#undef PGM_GST_NAME
952	#undef PGM_GST_NAME_GC_STR
953	#undef PGM_GST_NAME_R0_STR
954
955	/* Guest - AMD64 mode */
956	#define PGM_GST_TYPE PGM_TYPE_AMD64
957	#define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
958	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_AMD64_STR(name)
959	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
960	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_AMD64(name)
961	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_NESTED_AMD64_STR(name)
962	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_AMD64_STR(name)
963	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
964	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
965	#include "PGMBth.h"
966	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
967	#undef BTH_PGMPOOLKIND_PT_FOR_PT
968	#undef PGM_BTH_NAME
969	#undef PGM_BTH_NAME_GC_STR
970	#undef PGM_BTH_NAME_R0_STR
971	#undef PGM_GST_TYPE
972	#undef PGM_GST_NAME
973	#undef PGM_GST_NAME_GC_STR
974	#undef PGM_GST_NAME_R0_STR
975
976	#undef PGM_SHW_TYPE
977	#undef PGM_SHW_NAME
978	#undef PGM_SHW_NAME_GC_STR
979	#undef PGM_SHW_NAME_R0_STR
980
981
982	/**
983	* Initiates the paging of VM.
984	*
985	* @returns VBox status code.
986	* @param pVM Pointer to VM structure.
987	*/
988	PGMR3DECL(int) PGMR3Init(PVM pVM)
989	{
990	LogFlow(("PGMR3Init:\n"));
991
992	/*
993	* Assert alignment and sizes.
994	*/
995	AssertRelease(sizeof(pVM->pgm.s) <= sizeof(pVM->pgm.padding));
996
997	/*
998	* Init the structure.
999	*/
1000	pVM->pgm.s.offVM = RT_OFFSETOF(VM, pgm.s);
1001	pVM->pgm.s.enmShadowMode = PGMMODE_INVALID;
1002	pVM->pgm.s.enmGuestMode = PGMMODE_INVALID;
1003	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1004	pVM->pgm.s.GCPhysCR3 = NIL_RTGCPHYS;
1005	pVM->pgm.s.GCPhysGstCR3Monitored = NIL_RTGCPHYS;
1006	pVM->pgm.s.fA20Enabled = true;
1007	pVM->pgm.s.pGstPaePDPTHC = NULL;
1008	pVM->pgm.s.pGstPaePDPTGC = 0;
1009	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.apGstPaePDsHC); i++)
1010	{
1011	pVM->pgm.s.apGstPaePDsHC[i] = NULL;
1012	pVM->pgm.s.apGstPaePDsGC[i] = 0;
1013	pVM->pgm.s.aGCPhysGstPaePDs[i] = NIL_RTGCPHYS;
1014	pVM->pgm.s.aGCPhysGstPaePDsMonitored[i] = NIL_RTGCPHYS;
1015	}
1016
1017	#ifdef VBOX_STRICT
1018	VMR3AtStateRegister(pVM, pgmR3ResetNoMorePhysWritesFlag, NULL);
1019	#endif
1020
1021	/*
1022	* Get the configured RAM size - to estimate saved state size.
1023	*/
1024	uint64_t cbRam;
1025	int rc = CFGMR3QueryU64(CFGMR3GetRoot(pVM), "RamSize", &cbRam);
1026	if (rc == VERR_CFGM_VALUE_NOT_FOUND)
1027	cbRam = pVM->pgm.s.cbRamSize = 0;
1028	else if (VBOX_SUCCESS(rc))
1029	{
1030	if (cbRam < PAGE_SIZE)
1031	cbRam = 0;
1032	cbRam = RT_ALIGN_64(cbRam, PAGE_SIZE);
1033	pVM->pgm.s.cbRamSize = (RTUINT)cbRam;
1034	}
1035	else
1036	{
1037	AssertMsgFailed(("Configuration error: Failed to query integer \"RamSize\", rc=%Vrc.\n", rc));
1038	return rc;
1039	}
1040
1041	/*
1042	* Register saved state data unit.
1043	*/
1044	rc = SSMR3RegisterInternal(pVM, "pgm", 1, PGM_SAVED_STATE_VERSION, (size_t)cbRam + sizeof(PGM),
1045	NULL, pgmR3Save, NULL,
1046	NULL, pgmR3Load, NULL);
1047	if (VBOX_FAILURE(rc))
1048	return rc;
1049
1050	/*
1051	* Initialize the PGM critical section and flush the phys TLBs
1052	*/
1053	rc = PDMR3CritSectInit(pVM, &pVM->pgm.s.CritSect, "PGM");
1054	AssertRCReturn(rc, rc);
1055
1056	PGMR3PhysChunkInvalidateTLB(pVM);
1057	PGMPhysInvalidatePageR3MapTLB(pVM);
1058	PGMPhysInvalidatePageR0MapTLB(pVM);
1059	PGMPhysInvalidatePageGCMapTLB(pVM);
1060
1061	/*
1062	* Trees
1063	*/
1064	rc = MMHyperAlloc(pVM, sizeof(PGMTREES), 0, MM_TAG_PGM, (void **)&pVM->pgm.s.pTreesHC);
1065	if (VBOX_SUCCESS(rc))
1066	{
1067	pVM->pgm.s.pTreesGC = MMHyperHC2GC(pVM, pVM->pgm.s.pTreesHC);
1068
1069	/*
1070	* Alocate the zero page.
1071	*/
1072	rc = MMHyperAlloc(pVM, PAGE_SIZE, PAGE_SIZE, MM_TAG_PGM, &pVM->pgm.s.pvZeroPgR3);
1073	}
1074	if (VBOX_SUCCESS(rc))
1075	{
1076	pVM->pgm.s.pvZeroPgGC = MMHyperR3ToGC(pVM, pVM->pgm.s.pvZeroPgR3);
1077	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
1078	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTHCPHYS);
1079	pVM->pgm.s.HCPhysZeroPg = MMR3HyperHCVirt2HCPhys(pVM, pVM->pgm.s.pvZeroPgR3);
1080	AssertRelease(pVM->pgm.s.HCPhysZeroPg != NIL_RTHCPHYS);
1081
1082	/*
1083	* Init the paging.
1084	*/
1085	rc = pgmR3InitPaging(pVM);
1086	}
1087	if (VBOX_SUCCESS(rc))
1088	{
1089	/*
1090	* Init the page pool.
1091	*/
1092	rc = pgmR3PoolInit(pVM);
1093	}
1094	if (VBOX_SUCCESS(rc))
1095	{
1096	/*
1097	* Info & statistics
1098	*/
1099	DBGFR3InfoRegisterInternal(pVM, "mode",
1100	"Shows the current paging mode. "
1101	"Recognizes 'all', 'guest', 'shadow' and 'host' as arguments, defaulting to 'all' if nothing's given.",
1102	pgmR3InfoMode);
1103	DBGFR3InfoRegisterInternal(pVM, "pgmcr3",
1104	"Dumps all the entries in the top level paging table. No arguments.",
1105	pgmR3InfoCr3);
1106	DBGFR3InfoRegisterInternal(pVM, "phys",
1107	"Dumps all the physical address ranges. No arguments.",
1108	pgmR3PhysInfo);
1109	DBGFR3InfoRegisterInternal(pVM, "handlers",
1110	"Dumps physical, virtual and hyper virtual handlers. "
1111	"Pass 'phys', 'virt', 'hyper' as argument if only one kind is wanted."
1112	"Add 'nost' if the statistics are unwanted, use together with 'all' or explicit selection.",
1113	pgmR3InfoHandlers);
1114	DBGFR3InfoRegisterInternal(pVM, "mappings",
1115	"Dumps guest mappings.",
1116	pgmR3MapInfo);
1117
1118	STAM_REL_REG(pVM, &pVM->pgm.s.cGuestModeChanges, STAMTYPE_COUNTER, "/PGM/cGuestModeChanges", STAMUNIT_OCCURENCES, "Number of guest mode changes.");
1119	#ifdef VBOX_WITH_STATISTICS
1120	pgmR3InitStats(pVM);
1121	#endif
1122	#ifdef VBOX_WITH_DEBUGGER
1123	/*
1124	* Debugger commands.
1125	*/
1126	static bool fRegisteredCmds = false;
1127	if (!fRegisteredCmds)
1128	{
1129	int rc = DBGCRegisterCommands(&g_aCmds[0], ELEMENTS(g_aCmds));
1130	if (VBOX_SUCCESS(rc))
1131	fRegisteredCmds = true;
1132	}
1133	#endif
1134	return VINF_SUCCESS;
1135	}
1136
1137	/* Almost no cleanup necessary, MM frees all memory. */
1138	PDMR3CritSectDelete(&pVM->pgm.s.CritSect);
1139
1140	return rc;
1141	}
1142
1143
1144	/**
1145	* Init paging.
1146	*
1147	* Since we need to check what mode the host is operating in before we can choose
1148	* the right paging functions for the host we have to delay this until R0 has
1149	* been initialized.
1150	*
1151	* @returns VBox status code.
1152	* @param pVM VM handle.
1153	*/
1154	static int pgmR3InitPaging(PVM pVM)
1155	{
1156	/*
1157	* Force a recalculation of modes and switcher so everyone gets notified.
1158	*/
1159	pVM->pgm.s.enmShadowMode = PGMMODE_INVALID;
1160	pVM->pgm.s.enmGuestMode = PGMMODE_INVALID;
1161	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1162
1163	/*
1164	* Allocate static mapping space for whatever the cr3 register
1165	* points to and in the case of PAE mode to the 4 PDs.
1166	*/
1167	int rc = MMR3HyperReserve(pVM, PAGE_SIZE * 5, "CR3 mapping", &pVM->pgm.s.GCPtrCR3Mapping);
1168	if (VBOX_FAILURE(rc))
1169	{
1170	AssertMsgFailed(("Failed to reserve two pages for cr mapping in HMA, rc=%Vrc\n", rc));
1171	return rc;
1172	}
1173	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
1174
1175	/*
1176	* Allocate pages for the three possible intermediate contexts
1177	* (AMD64, PAE and plain 32-Bit). We maintain all three contexts
1178	* for the sake of simplicity. The AMD64 uses the PAE for the
1179	* lower levels, making the total number of pages 11 (3 + 7 + 1).
1180	*
1181	* We assume that two page tables will be enought for the core code
1182	* mappings (HC virtual and identity).
1183	*/
1184	pVM->pgm.s.pInterPD = (PX86PD)MMR3PageAllocLow(pVM);
1185	pVM->pgm.s.apInterPTs[0] = (PX86PT)MMR3PageAllocLow(pVM);
1186	pVM->pgm.s.apInterPTs[1] = (PX86PT)MMR3PageAllocLow(pVM);
1187	pVM->pgm.s.apInterPaePTs[0] = (PX86PTPAE)MMR3PageAlloc(pVM);
1188	pVM->pgm.s.apInterPaePTs[1] = (PX86PTPAE)MMR3PageAlloc(pVM);
1189	pVM->pgm.s.apInterPaePDs[0] = (PX86PDPAE)MMR3PageAlloc(pVM);
1190	pVM->pgm.s.apInterPaePDs[1] = (PX86PDPAE)MMR3PageAlloc(pVM);
1191	pVM->pgm.s.apInterPaePDs[2] = (PX86PDPAE)MMR3PageAlloc(pVM);
1192	pVM->pgm.s.apInterPaePDs[3] = (PX86PDPAE)MMR3PageAlloc(pVM);
1193	pVM->pgm.s.pInterPaePDPT = (PX86PDPT)MMR3PageAllocLow(pVM);
1194	pVM->pgm.s.pInterPaePDPT64 = (PX86PDPT)MMR3PageAllocLow(pVM);
1195	pVM->pgm.s.pInterPaePML4 = (PX86PML4)MMR3PageAllocLow(pVM);
1196	if ( !pVM->pgm.s.pInterPD
1197	\|\| !pVM->pgm.s.apInterPTs[0]
1198	\|\| !pVM->pgm.s.apInterPTs[1]
1199	\|\| !pVM->pgm.s.apInterPaePTs[0]
1200	\|\| !pVM->pgm.s.apInterPaePTs[1]
1201	\|\| !pVM->pgm.s.apInterPaePDs[0]
1202	\|\| !pVM->pgm.s.apInterPaePDs[1]
1203	\|\| !pVM->pgm.s.apInterPaePDs[2]
1204	\|\| !pVM->pgm.s.apInterPaePDs[3]
1205	\|\| !pVM->pgm.s.pInterPaePDPT
1206	\|\| !pVM->pgm.s.pInterPaePDPT64
1207	\|\| !pVM->pgm.s.pInterPaePML4)
1208	{
1209	AssertMsgFailed(("Failed to allocate pages for the intermediate context!\n"));
1210	return VERR_NO_PAGE_MEMORY;
1211	}
1212
1213	pVM->pgm.s.HCPhysInterPD = MMPage2Phys(pVM, pVM->pgm.s.pInterPD);
1214	AssertRelease(pVM->pgm.s.HCPhysInterPD != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPD & PAGE_OFFSET_MASK));
1215	pVM->pgm.s.HCPhysInterPaePDPT = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT);
1216	AssertRelease(pVM->pgm.s.HCPhysInterPaePDPT != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePDPT & PAGE_OFFSET_MASK));
1217	pVM->pgm.s.HCPhysInterPaePML4 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePML4);
1218	AssertRelease(pVM->pgm.s.HCPhysInterPaePML4 != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePML4 & PAGE_OFFSET_MASK));
1219
1220	/*
1221	* Initialize the pages, setting up the PML4 and PDPT for repetitive 4GB action.
1222	*/
1223	ASMMemZeroPage(pVM->pgm.s.pInterPD);
1224	ASMMemZeroPage(pVM->pgm.s.apInterPTs[0]);
1225	ASMMemZeroPage(pVM->pgm.s.apInterPTs[1]);
1226
1227	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[0]);
1228	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[1]);
1229
1230	ASMMemZeroPage(pVM->pgm.s.pInterPaePDPT);
1231	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.apInterPaePDs); i++)
1232	{
1233	ASMMemZeroPage(pVM->pgm.s.apInterPaePDs[i]);
1234	pVM->pgm.s.pInterPaePDPT->a[i].u = X86_PDPE_P \| PGM_PLXFLAGS_PERMANENT
1235	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[i]);
1236	}
1237
1238	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.pInterPaePDPT64->a); i++)
1239	{
1240	const unsigned iPD = i % ELEMENTS(pVM->pgm.s.apInterPaePDs);
1241	pVM->pgm.s.pInterPaePDPT64->a[i].u = X86_PDPE_P \| X86_PDPE_RW \| X86_PDPE_US \| X86_PDPE_A \| PGM_PLXFLAGS_PERMANENT
1242	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[iPD]);
1243	}
1244
1245	RTHCPHYS HCPhysInterPaePDPT64 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64);
1246	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.pInterPaePML4->a); i++)
1247	pVM->pgm.s.pInterPaePML4->a[i].u = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_US \| X86_PML4E_A \| PGM_PLXFLAGS_PERMANENT
1248	\| HCPhysInterPaePDPT64;
1249
1250	/*
1251	* Allocate pages for the three possible guest contexts (AMD64, PAE and plain 32-Bit).
1252	* We allocate pages for all three posibilities to in order to simplify mappings and
1253	* avoid resource failure during mode switches. So, we need to cover all levels of the
1254	* of the first 4GB down to PD level.
1255	* As with the intermediate context, AMD64 uses the PAE PDPT and PDs.
1256	*/
1257	pVM->pgm.s.pHC32BitPD = (PX86PD)MMR3PageAllocLow(pVM);
1258	pVM->pgm.s.apHCPaePDs[0] = (PX86PDPAE)MMR3PageAlloc(pVM);
1259	pVM->pgm.s.apHCPaePDs[1] = (PX86PDPAE)MMR3PageAlloc(pVM);
1260	AssertRelease((uintptr_t)pVM->pgm.s.apHCPaePDs[0] + PAGE_SIZE == (uintptr_t)pVM->pgm.s.apHCPaePDs[1]);
1261	pVM->pgm.s.apHCPaePDs[2] = (PX86PDPAE)MMR3PageAlloc(pVM);
1262	AssertRelease((uintptr_t)pVM->pgm.s.apHCPaePDs[1] + PAGE_SIZE == (uintptr_t)pVM->pgm.s.apHCPaePDs[2]);
1263	pVM->pgm.s.apHCPaePDs[3] = (PX86PDPAE)MMR3PageAlloc(pVM);
1264	AssertRelease((uintptr_t)pVM->pgm.s.apHCPaePDs[2] + PAGE_SIZE == (uintptr_t)pVM->pgm.s.apHCPaePDs[3]);
1265	pVM->pgm.s.pHCPaePDPT = (PX86PDPT)MMR3PageAllocLow(pVM);
1266	pVM->pgm.s.pHCPaePML4 = (PX86PML4)MMR3PageAllocLow(pVM);
1267	if ( !pVM->pgm.s.pHC32BitPD
1268	\|\| !pVM->pgm.s.apHCPaePDs[0]
1269	\|\| !pVM->pgm.s.apHCPaePDs[1]
1270	\|\| !pVM->pgm.s.apHCPaePDs[2]
1271	\|\| !pVM->pgm.s.apHCPaePDs[3]
1272	\|\| !pVM->pgm.s.pHCPaePDPT
1273	\|\| !pVM->pgm.s.pHCPaePML4)
1274	{
1275	AssertMsgFailed(("Failed to allocate pages for the intermediate context!\n"));
1276	return VERR_NO_PAGE_MEMORY;
1277	}
1278
1279	/* get physical addresses. */
1280	pVM->pgm.s.HCPhys32BitPD = MMPage2Phys(pVM, pVM->pgm.s.pHC32BitPD);
1281	Assert(MMPagePhys2Page(pVM, pVM->pgm.s.HCPhys32BitPD) == pVM->pgm.s.pHC32BitPD);
1282	pVM->pgm.s.aHCPhysPaePDs[0] = MMPage2Phys(pVM, pVM->pgm.s.apHCPaePDs[0]);
1283	pVM->pgm.s.aHCPhysPaePDs[1] = MMPage2Phys(pVM, pVM->pgm.s.apHCPaePDs[1]);
1284	pVM->pgm.s.aHCPhysPaePDs[2] = MMPage2Phys(pVM, pVM->pgm.s.apHCPaePDs[2]);
1285	pVM->pgm.s.aHCPhysPaePDs[3] = MMPage2Phys(pVM, pVM->pgm.s.apHCPaePDs[3]);
1286	pVM->pgm.s.HCPhysPaePDPT = MMPage2Phys(pVM, pVM->pgm.s.pHCPaePDPT);
1287	pVM->pgm.s.HCPhysPaePML4 = MMPage2Phys(pVM, pVM->pgm.s.pHCPaePML4);
1288
1289	/*
1290	* Initialize the pages, setting up the PML4 and PDPT for action below 4GB.
1291	*/
1292	ASMMemZero32(pVM->pgm.s.pHC32BitPD, PAGE_SIZE);
1293
1294	ASMMemZero32(pVM->pgm.s.pHCPaePDPT, PAGE_SIZE);
1295	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.apHCPaePDs); i++)
1296	{
1297	ASMMemZero32(pVM->pgm.s.apHCPaePDs[i], PAGE_SIZE);
1298	pVM->pgm.s.pHCPaePDPT->a[i].u = X86_PDPE_P \| PGM_PLXFLAGS_PERMANENT \| pVM->pgm.s.aHCPhysPaePDs[i];
1299	/* The flags will be corrected when entering and leaving long mode. */
1300	}
1301
1302	ASMMemZero32(pVM->pgm.s.pHCPaePML4, PAGE_SIZE);
1303	pVM->pgm.s.pHCPaePML4->a[0].u = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_A
1304	\| PGM_PLXFLAGS_PERMANENT \| pVM->pgm.s.HCPhysPaePDPT;
1305
1306	CPUMSetHyperCR3(pVM, (uint32_t)pVM->pgm.s.HCPhys32BitPD);
1307
1308	/*
1309	* Initialize paging workers and mode from current host mode
1310	* and the guest running in real mode.
1311	*/
1312	pVM->pgm.s.enmHostMode = SUPGetPagingMode();
1313	switch (pVM->pgm.s.enmHostMode)
1314	{
1315	case SUPPAGINGMODE_32_BIT:
1316	case SUPPAGINGMODE_32_BIT_GLOBAL:
1317	case SUPPAGINGMODE_PAE:
1318	case SUPPAGINGMODE_PAE_GLOBAL:
1319	case SUPPAGINGMODE_PAE_NX:
1320	case SUPPAGINGMODE_PAE_GLOBAL_NX:
1321	break;
1322
1323	case SUPPAGINGMODE_AMD64:
1324	case SUPPAGINGMODE_AMD64_GLOBAL:
1325	case SUPPAGINGMODE_AMD64_NX:
1326	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
1327	#ifndef VBOX_WITH_HYBIRD_32BIT_KERNEL
1328	if (ARCH_BITS != 64)
1329	{
1330	AssertMsgFailed(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1331	LogRel(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1332	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1333	}
1334	#endif
1335	break;
1336	default:
1337	AssertMsgFailed(("Host mode %d is not supported\n", pVM->pgm.s.enmHostMode));
1338	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1339	}
1340	rc = pgmR3ModeDataInit(pVM, false /* don't resolve GC and R0 syms yet */);
1341	if (VBOX_SUCCESS(rc))
1342	rc = pgmR3ChangeMode(pVM, PGMMODE_REAL);
1343	if (VBOX_SUCCESS(rc))
1344	{
1345	LogFlow(("pgmR3InitPaging: returns successfully\n"));
1346	#if HC_ARCH_BITS == 64
1347	LogRel(("Debug: HCPhys32BitPD=%VHp aHCPhysPaePDs={%VHp,%VHp,%VHp,%VHp} HCPhysPaePDPT=%VHp HCPhysPaePML4=%VHp\n",
1348	pVM->pgm.s.HCPhys32BitPD, pVM->pgm.s.aHCPhysPaePDs[0], pVM->pgm.s.aHCPhysPaePDs[1], pVM->pgm.s.aHCPhysPaePDs[2], pVM->pgm.s.aHCPhysPaePDs[3],
1349	pVM->pgm.s.HCPhysPaePDPT, pVM->pgm.s.HCPhysPaePML4));
1350	LogRel(("Debug: HCPhysInterPD=%VHp HCPhysInterPaePDPT=%VHp HCPhysInterPaePML4=%VHp\n",
1351	pVM->pgm.s.HCPhysInterPD, pVM->pgm.s.HCPhysInterPaePDPT, pVM->pgm.s.HCPhysInterPaePML4));
1352	LogRel(("Debug: apInterPTs={%VHp,%VHp} apInterPaePTs={%VHp,%VHp} apInterPaePDs={%VHp,%VHp,%VHp,%VHp} pInterPaePDPT64=%VHp\n",
1353	MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[1]),
1354	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[1]),
1355	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[1]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[2]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[3]),
1356	MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64)));
1357	#endif
1358
1359	return VINF_SUCCESS;
1360	}
1361
1362	LogFlow(("pgmR3InitPaging: returns %Vrc\n", rc));
1363	return rc;
1364	}
1365
1366
1367	#ifdef VBOX_WITH_STATISTICS
1368	/**
1369	* Init statistics
1370	*/
1371	static void pgmR3InitStats(PVM pVM)
1372	{
1373	PPGM pPGM = &pVM->pgm.s;
1374	STAM_REG(pVM, &pPGM->StatGCInvalidatePage, STAMTYPE_PROFILE, "/PGM/GC/InvalidatePage", STAMUNIT_TICKS_PER_CALL, "PGMGCInvalidatePage() profiling.");
1375	STAM_REG(pVM, &pPGM->StatGCInvalidatePage4KBPages, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/4KBPages", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was called for a 4KB page.");
1376	STAM_REG(pVM, &pPGM->StatGCInvalidatePage4MBPages, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/4MBPages", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was called for a 4MB page.");
1377	STAM_REG(pVM, &pPGM->StatGCInvalidatePage4MBPagesSkip, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/4MBPagesSkip",STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() skipped a 4MB page.");
1378	STAM_REG(pVM, &pPGM->StatGCInvalidatePagePDMappings, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/PDMappings", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was called for a page directory containing mappings (no conflict).");
1379	STAM_REG(pVM, &pPGM->StatGCInvalidatePagePDNAs, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/PDNAs", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was called for a not accessed page directory.");
1380	STAM_REG(pVM, &pPGM->StatGCInvalidatePagePDNPs, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/PDNPs", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was called for a not present page directory.");
1381	STAM_REG(pVM, &pPGM->StatGCInvalidatePagePDOutOfSync, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/PDOutOfSync", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was called for an out of sync page directory.");
1382	STAM_REG(pVM, &pPGM->StatGCInvalidatePageSkipped, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/Skipped", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
1383	STAM_REG(pVM, &pPGM->StatGCSyncPT, STAMTYPE_PROFILE, "/PGM/GC/SyncPT", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMGCSyncPT() body.");
1384	STAM_REG(pVM, &pPGM->StatGCAccessedPage, STAMTYPE_COUNTER, "/PGM/GC/AccessedPage", STAMUNIT_OCCURENCES, "The number of pages marked not present for accessed bit emulation.");
1385	STAM_REG(pVM, &pPGM->StatGCDirtyPage, STAMTYPE_COUNTER, "/PGM/GC/DirtyPage/Mark", STAMUNIT_OCCURENCES, "The number of pages marked read-only for dirty bit tracking.");
1386	STAM_REG(pVM, &pPGM->StatGCDirtyPageBig, STAMTYPE_COUNTER, "/PGM/GC/DirtyPage/MarkBig", STAMUNIT_OCCURENCES, "The number of 4MB pages marked read-only for dirty bit tracking.");
1387	STAM_REG(pVM, &pPGM->StatGCDirtyPageTrap, STAMTYPE_COUNTER, "/PGM/GC/DirtyPage/Trap", STAMUNIT_OCCURENCES, "The number of traps generated for dirty bit tracking.");
1388	STAM_REG(pVM, &pPGM->StatGCDirtyPageSkipped, STAMTYPE_COUNTER, "/PGM/GC/DirtyPage/Skipped", STAMUNIT_OCCURENCES, "The number of pages already dirty or readonly.");
1389	STAM_REG(pVM, &pPGM->StatGCDirtiedPage, STAMTYPE_COUNTER, "/PGM/GC/DirtyPage/SetDirty", STAMUNIT_OCCURENCES, "The number of pages marked dirty because of write accesses.");
1390	STAM_REG(pVM, &pPGM->StatGCDirtyTrackRealPF, STAMTYPE_COUNTER, "/PGM/GC/DirtyPage/RealPF", STAMUNIT_OCCURENCES, "The number of real pages faults during dirty bit tracking.");
1391	STAM_REG(pVM, &pPGM->StatGCPageAlreadyDirty, STAMTYPE_COUNTER, "/PGM/GC/DirtyPage/AlreadySet", STAMUNIT_OCCURENCES, "The number of pages already marked dirty because of write accesses.");
1392	STAM_REG(pVM, &pPGM->StatGCDirtyBitTracking, STAMTYPE_PROFILE, "/PGM/GC/DirtyPage", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMTrackDirtyBit() body.");
1393	STAM_REG(pVM, &pPGM->StatGCSyncPTAlloc, STAMTYPE_COUNTER, "/PGM/GC/SyncPT/Alloc", STAMUNIT_OCCURENCES, "The number of times PGMGCSyncPT() needed to allocate page tables.");
1394	STAM_REG(pVM, &pPGM->StatGCSyncPTConflict, STAMTYPE_COUNTER, "/PGM/GC/SyncPT/Conflicts", STAMUNIT_OCCURENCES, "The number of times PGMGCSyncPT() detected conflicts.");
1395	STAM_REG(pVM, &pPGM->StatGCSyncPTFailed, STAMTYPE_COUNTER, "/PGM/GC/SyncPT/Failed", STAMUNIT_OCCURENCES, "The number of times PGMGCSyncPT() failed.");
1396
1397	STAM_REG(pVM, &pPGM->StatGCTrap0e, STAMTYPE_PROFILE, "/PGM/GC/Trap0e", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMGCTrap0eHandler() body.");
1398	STAM_REG(pVM, &pPGM->StatCheckPageFault, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time/CheckPageFault", STAMUNIT_TICKS_PER_CALL, "Profiling of checking for dirty/access emulation faults.");
1399	STAM_REG(pVM, &pPGM->StatLazySyncPT, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time/SyncPT", STAMUNIT_TICKS_PER_CALL, "Profiling of lazy page table syncing.");
1400	STAM_REG(pVM, &pPGM->StatMapping, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time/Mapping", STAMUNIT_TICKS_PER_CALL, "Profiling of checking virtual mappings.");
1401	STAM_REG(pVM, &pPGM->StatOutOfSync, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time/OutOfSync", STAMUNIT_TICKS_PER_CALL, "Profiling of out of sync page handling.");
1402	STAM_REG(pVM, &pPGM->StatHandlers, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time/Handlers", STAMUNIT_TICKS_PER_CALL, "Profiling of checking handlers.");
1403	STAM_REG(pVM, &pPGM->StatEIPHandlers, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time/EIPHandlers", STAMUNIT_TICKS_PER_CALL, "Profiling of checking eip handlers.");
1404	STAM_REG(pVM, &pPGM->StatTrap0eCSAM, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/CSAM", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is CSAM.");
1405	STAM_REG(pVM, &pPGM->StatTrap0eDirtyAndAccessedBits, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/DirtyAndAccessedBits", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is dirty and/or accessed bit emulation.");
1406	STAM_REG(pVM, &pPGM->StatTrap0eGuestTrap, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/GuestTrap", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is a guest trap.");
1407	STAM_REG(pVM, &pPGM->StatTrap0eHndPhys, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/HandlerPhysical", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is a physical handler.");
1408	STAM_REG(pVM, &pPGM->StatTrap0eHndVirt, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/HandlerVirtual",STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is a virtual handler.");
1409	STAM_REG(pVM, &pPGM->StatTrap0eHndUnhandled, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/HandlerUnhandled", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is access outside the monitored areas of a monitored page.");
1410	STAM_REG(pVM, &pPGM->StatTrap0eMisc, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/Misc", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is not known.");
1411	STAM_REG(pVM, &pPGM->StatTrap0eOutOfSync, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/OutOfSync", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is an out-of-sync page.");
1412	STAM_REG(pVM, &pPGM->StatTrap0eOutOfSyncHndPhys, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/OutOfSyncHndPhys", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is an out-of-sync physical handler page.");
1413	STAM_REG(pVM, &pPGM->StatTrap0eOutOfSyncHndVirt, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/OutOfSyncHndVirt", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is an out-of-sync virtual handler page.");
1414	STAM_REG(pVM, &pPGM->StatTrap0eOutOfSyncObsHnd, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/OutOfSyncObsHnd", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is an obsolete handler page.");
1415	STAM_REG(pVM, &pPGM->StatTrap0eSyncPT, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/SyncPT", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is lazy syncing of a PT.");
1416
1417	STAM_REG(pVM, &pPGM->StatTrap0eMapHandler, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/Mapping", STAMUNIT_OCCURENCES, "Number of traps due to access handlers in mappings.");
1418	STAM_REG(pVM, &pPGM->StatHandlersOutOfSync, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/OutOfSync", STAMUNIT_OCCURENCES, "Number of traps due to out-of-sync handled pages.");
1419	STAM_REG(pVM, &pPGM->StatHandlersPhysical, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/Physical", STAMUNIT_OCCURENCES, "Number of traps due to physical access handlers.");
1420	STAM_REG(pVM, &pPGM->StatHandlersVirtual, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/Virtual", STAMUNIT_OCCURENCES, "Number of traps due to virtual access handlers.");
1421	STAM_REG(pVM, &pPGM->StatHandlersVirtualByPhys, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/VirtualByPhys", STAMUNIT_OCCURENCES, "Number of traps due to virtual access handlers by physical address.");
1422	STAM_REG(pVM, &pPGM->StatHandlersVirtualUnmarked, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/VirtualUnmarked", STAMUNIT_OCCURENCES,"Number of traps due to virtual access handlers by virtual address (without proper physical flags).");
1423	STAM_REG(pVM, &pPGM->StatHandlersUnhandled, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/Unhandled", STAMUNIT_OCCURENCES, "Number of traps due to access outside range of monitored page(s).");
1424	STAM_REG(pVM, &pPGM->StatHandlersInvalid, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/Invalid", STAMUNIT_OCCURENCES, "Number of traps due to access to invalid physical memory.");
1425
1426	STAM_REG(pVM, &pPGM->StatGCTrap0eConflicts, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Conflicts", STAMUNIT_OCCURENCES, "The number of times #PF was caused by an undetected conflict.");
1427	STAM_REG(pVM, &pPGM->StatGCTrap0eUSNotPresentRead, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/User/NPRead", STAMUNIT_OCCURENCES, "Number of user mode not present read page faults.");
1428	STAM_REG(pVM, &pPGM->StatGCTrap0eUSNotPresentWrite, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/User/NPWrite", STAMUNIT_OCCURENCES, "Number of user mode not present write page faults.");
1429	STAM_REG(pVM, &pPGM->StatGCTrap0eUSWrite, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/User/Write", STAMUNIT_OCCURENCES, "Number of user mode write page faults.");
1430	STAM_REG(pVM, &pPGM->StatGCTrap0eUSReserved, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/User/Reserved", STAMUNIT_OCCURENCES, "Number of user mode reserved bit page faults.");
1431	STAM_REG(pVM, &pPGM->StatGCTrap0eUSNXE, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/User/NXE", STAMUNIT_OCCURENCES, "Number of user mode NXE page faults.");
1432	STAM_REG(pVM, &pPGM->StatGCTrap0eUSRead, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/User/Read", STAMUNIT_OCCURENCES, "Number of user mode read page faults.");
1433
1434	STAM_REG(pVM, &pPGM->StatGCTrap0eSVNotPresentRead, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Supervisor/NPRead", STAMUNIT_OCCURENCES, "Number of supervisor mode not present read page faults.");
1435	STAM_REG(pVM, &pPGM->StatGCTrap0eSVNotPresentWrite, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Supervisor/NPWrite", STAMUNIT_OCCURENCES, "Number of supervisor mode not present write page faults.");
1436	STAM_REG(pVM, &pPGM->StatGCTrap0eSVWrite, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Supervisor/Write", STAMUNIT_OCCURENCES, "Number of supervisor mode write page faults.");
1437	STAM_REG(pVM, &pPGM->StatGCTrap0eSVReserved, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Supervisor/Reserved", STAMUNIT_OCCURENCES, "Number of supervisor mode reserved bit page faults.");
1438	STAM_REG(pVM, &pPGM->StatGCTrap0eSNXE, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Supervisor/NXE", STAMUNIT_OCCURENCES, "Number of supervisor mode NXE page faults.");
1439	STAM_REG(pVM, &pPGM->StatGCTrap0eUnhandled, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/GuestPF/Unhandled", STAMUNIT_OCCURENCES, "Number of guest real page faults.");
1440	STAM_REG(pVM, &pPGM->StatGCTrap0eMap, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/GuestPF/Map", STAMUNIT_OCCURENCES, "Number of guest page faults due to map accesses.");
1441
1442	STAM_REG(pVM, &pPGM->StatTrap0eWPEmulGC, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/WP/InGC", STAMUNIT_OCCURENCES, "Number of guest page faults due to X86_CR0_WP emulation.");
1443	STAM_REG(pVM, &pPGM->StatTrap0eWPEmulR3, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/WP/ToR3", STAMUNIT_OCCURENCES, "Number of guest page faults due to X86_CR0_WP emulation (forward to R3 for emulation).");
1444
1445	STAM_REG(pVM, &pPGM->StatGCGuestCR3WriteHandled, STAMTYPE_COUNTER, "/PGM/GC/CR3WriteInt", STAMUNIT_OCCURENCES, "The number of times the Guest CR3 change was successfully handled.");
1446	STAM_REG(pVM, &pPGM->StatGCGuestCR3WriteUnhandled, STAMTYPE_COUNTER, "/PGM/GC/CR3WriteEmu", STAMUNIT_OCCURENCES, "The number of times the Guest CR3 change was passed back to the recompiler.");
1447	STAM_REG(pVM, &pPGM->StatGCGuestCR3WriteConflict, STAMTYPE_COUNTER, "/PGM/GC/CR3WriteConflict", STAMUNIT_OCCURENCES, "The number of times the Guest CR3 monitoring detected a conflict.");
1448
1449	STAM_REG(pVM, &pPGM->StatGCPageOutOfSyncSupervisor, STAMTYPE_COUNTER, "/PGM/GC/OutOfSync/SuperVisor", STAMUNIT_OCCURENCES, "Number of traps due to pages out of sync.");
1450	STAM_REG(pVM, &pPGM->StatGCPageOutOfSyncUser, STAMTYPE_COUNTER, "/PGM/GC/OutOfSync/User", STAMUNIT_OCCURENCES, "Number of traps due to pages out of sync.");
1451
1452	STAM_REG(pVM, &pPGM->StatGCGuestROMWriteHandled, STAMTYPE_COUNTER, "/PGM/GC/ROMWriteInt", STAMUNIT_OCCURENCES, "The number of times the Guest ROM change was successfully handled.");
1453	STAM_REG(pVM, &pPGM->StatGCGuestROMWriteUnhandled, STAMTYPE_COUNTER, "/PGM/GC/ROMWriteEmu", STAMUNIT_OCCURENCES, "The number of times the Guest ROM change was passed back to the recompiler.");
1454
1455	STAM_REG(pVM, &pPGM->StatDynMapCacheHits, STAMTYPE_COUNTER, "/PGM/GC/DynMapCache/Hits" , STAMUNIT_OCCURENCES, "Number of dynamic page mapping cache hits.");
1456	STAM_REG(pVM, &pPGM->StatDynMapCacheMisses, STAMTYPE_COUNTER, "/PGM/GC/DynMapCache/Misses" , STAMUNIT_OCCURENCES, "Number of dynamic page mapping cache misses.");
1457
1458	STAM_REG(pVM, &pPGM->StatHCDetectedConflicts, STAMTYPE_COUNTER, "/PGM/HC/DetectedConflicts", STAMUNIT_OCCURENCES, "The number of times PGMR3CheckMappingConflicts() detected a conflict.");
1459	STAM_REG(pVM, &pPGM->StatHCGuestPDWrite, STAMTYPE_COUNTER, "/PGM/HC/PDWrite", STAMUNIT_OCCURENCES, "The total number of times pgmHCGuestPDWriteHandler() was called.");
1460	STAM_REG(pVM, &pPGM->StatHCGuestPDWriteConflict, STAMTYPE_COUNTER, "/PGM/HC/PDWriteConflict", STAMUNIT_OCCURENCES, "The number of times pgmHCGuestPDWriteHandler() detected a conflict.");
1461
1462	STAM_REG(pVM, &pPGM->StatHCInvalidatePage, STAMTYPE_PROFILE, "/PGM/HC/InvalidatePage", STAMUNIT_TICKS_PER_CALL, "PGMHCInvalidatePage() profiling.");
1463	STAM_REG(pVM, &pPGM->StatHCInvalidatePage4KBPages, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/4KBPages", STAMUNIT_OCCURENCES, "The number of times PGMHCInvalidatePage() was called for a 4KB page.");
1464	STAM_REG(pVM, &pPGM->StatHCInvalidatePage4MBPages, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/4MBPages", STAMUNIT_OCCURENCES, "The number of times PGMHCInvalidatePage() was called for a 4MB page.");
1465	STAM_REG(pVM, &pPGM->StatHCInvalidatePage4MBPagesSkip, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/4MBPagesSkip",STAMUNIT_OCCURENCES, "The number of times PGMHCInvalidatePage() skipped a 4MB page.");
1466	STAM_REG(pVM, &pPGM->StatHCInvalidatePagePDMappings, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/PDMappings", STAMUNIT_OCCURENCES, "The number of times PGMHCInvalidatePage() was called for a page directory containing mappings (no conflict).");
1467	STAM_REG(pVM, &pPGM->StatHCInvalidatePagePDNAs, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/PDNAs", STAMUNIT_OCCURENCES, "The number of times PGMHCInvalidatePage() was called for a not accessed page directory.");
1468	STAM_REG(pVM, &pPGM->StatHCInvalidatePagePDNPs, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/PDNPs", STAMUNIT_OCCURENCES, "The number of times PGMHCInvalidatePage() was called for a not present page directory.");
1469	STAM_REG(pVM, &pPGM->StatHCInvalidatePagePDOutOfSync, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/PDOutOfSync", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was called for an out of sync page directory.");
1470	STAM_REG(pVM, &pPGM->StatHCInvalidatePageSkipped, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/Skipped", STAMUNIT_OCCURENCES, "The number of times PGMHCInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
1471	STAM_REG(pVM, &pPGM->StatHCResolveConflict, STAMTYPE_PROFILE, "/PGM/HC/ResolveConflict", STAMUNIT_TICKS_PER_CALL, "pgmR3SyncPTResolveConflict() profiling (includes the entire relocation).");
1472	STAM_REG(pVM, &pPGM->StatHCPrefetch, STAMTYPE_PROFILE, "/PGM/HC/Prefetch", STAMUNIT_TICKS_PER_CALL, "PGMR3PrefetchPage profiling.");
1473
1474	STAM_REG(pVM, &pPGM->StatHCSyncPT, STAMTYPE_PROFILE, "/PGM/HC/SyncPT", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMR3SyncPT() body.");
1475	STAM_REG(pVM, &pPGM->StatHCAccessedPage, STAMTYPE_COUNTER, "/PGM/HC/AccessedPage", STAMUNIT_OCCURENCES, "The number of pages marked not present for accessed bit emulation.");
1476	STAM_REG(pVM, &pPGM->StatHCDirtyPage, STAMTYPE_COUNTER, "/PGM/HC/DirtyPage/Mark", STAMUNIT_OCCURENCES, "The number of pages marked read-only for dirty bit tracking.");
1477	STAM_REG(pVM, &pPGM->StatHCDirtyPageBig, STAMTYPE_COUNTER, "/PGM/HC/DirtyPage/MarkBig", STAMUNIT_OCCURENCES, "The number of 4MB pages marked read-only for dirty bit tracking.");
1478	STAM_REG(pVM, &pPGM->StatHCDirtyPageTrap, STAMTYPE_COUNTER, "/PGM/HC/DirtyPage/Trap", STAMUNIT_OCCURENCES, "The number of traps generated for dirty bit tracking.");
1479	STAM_REG(pVM, &pPGM->StatHCDirtyPageSkipped, STAMTYPE_COUNTER, "/PGM/HC/DirtyPage/Skipped", STAMUNIT_OCCURENCES, "The number of pages already dirty or readonly.");
1480	STAM_REG(pVM, &pPGM->StatHCDirtyBitTracking, STAMTYPE_PROFILE, "/PGM/HC/DirtyPage", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMTrackDirtyBit() body.");
1481
1482	STAM_REG(pVM, &pPGM->StatGCSyncPagePDNAs, STAMTYPE_COUNTER, "/PGM/GC/SyncPagePDNAs", STAMUNIT_OCCURENCES, "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1483	STAM_REG(pVM, &pPGM->StatGCSyncPagePDOutOfSync, STAMTYPE_COUNTER, "/PGM/GC/SyncPagePDOutOfSync", STAMUNIT_OCCURENCES, "The number of time we've encountered an out-of-sync PD in SyncPage.");
1484	STAM_REG(pVM, &pPGM->StatHCSyncPagePDNAs, STAMTYPE_COUNTER, "/PGM/HC/SyncPagePDNAs", STAMUNIT_OCCURENCES, "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1485	STAM_REG(pVM, &pPGM->StatHCSyncPagePDOutOfSync, STAMTYPE_COUNTER, "/PGM/HC/SyncPagePDOutOfSync", STAMUNIT_OCCURENCES, "The number of time we've encountered an out-of-sync PD in SyncPage.");
1486
1487	STAM_REG(pVM, &pPGM->StatFlushTLB, STAMTYPE_PROFILE, "/PGM/FlushTLB", STAMUNIT_OCCURENCES, "Profiling of the PGMFlushTLB() body.");
1488	STAM_REG(pVM, &pPGM->StatFlushTLBNewCR3, STAMTYPE_COUNTER, "/PGM/FlushTLB/NewCR3", STAMUNIT_OCCURENCES, "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
1489	STAM_REG(pVM, &pPGM->StatFlushTLBNewCR3Global, STAMTYPE_COUNTER, "/PGM/FlushTLB/NewCR3Global", STAMUNIT_OCCURENCES, "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
1490	STAM_REG(pVM, &pPGM->StatFlushTLBSameCR3, STAMTYPE_COUNTER, "/PGM/FlushTLB/SameCR3", STAMUNIT_OCCURENCES, "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
1491	STAM_REG(pVM, &pPGM->StatFlushTLBSameCR3Global, STAMTYPE_COUNTER, "/PGM/FlushTLB/SameCR3Global", STAMUNIT_OCCURENCES, "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
1492
1493	STAM_REG(pVM, &pPGM->StatGCSyncCR3, STAMTYPE_PROFILE, "/PGM/GC/SyncCR3", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMSyncCR3() body.");
1494	STAM_REG(pVM, &pPGM->StatGCSyncCR3Handlers, STAMTYPE_PROFILE, "/PGM/GC/SyncCR3/Handlers", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMSyncCR3() update handler section.");
1495	STAM_REG(pVM, &pPGM->StatGCSyncCR3HandlerVirtualUpdate, STAMTYPE_PROFILE, "/PGM/GC/SyncCR3/Handlers/VirtualUpdate",STAMUNIT_TICKS_PER_CALL, "Profiling of the virtual handler updates.");
1496	STAM_REG(pVM, &pPGM->StatGCSyncCR3HandlerVirtualReset, STAMTYPE_PROFILE, "/PGM/GC/SyncCR3/Handlers/VirtualReset", STAMUNIT_TICKS_PER_CALL, "Profiling of the virtual handler resets.");
1497	STAM_REG(pVM, &pPGM->StatGCSyncCR3Global, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/Global", STAMUNIT_OCCURENCES, "The number of global CR3 syncs.");
1498	STAM_REG(pVM, &pPGM->StatGCSyncCR3NotGlobal, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/NotGlobal", STAMUNIT_OCCURENCES, "The number of non-global CR3 syncs.");
1499	STAM_REG(pVM, &pPGM->StatGCSyncCR3DstCacheHit, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/DstChacheHit", STAMUNIT_OCCURENCES, "The number of times we got some kind of a cache hit.");
1500	STAM_REG(pVM, &pPGM->StatGCSyncCR3DstFreed, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/DstFreed", STAMUNIT_OCCURENCES, "The number of times we've had to free a shadow entry.");
1501	STAM_REG(pVM, &pPGM->StatGCSyncCR3DstFreedSrcNP, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/DstFreedSrcNP", STAMUNIT_OCCURENCES, "The number of times we've had to free a shadow entry for which the source entry was not present.");
1502	STAM_REG(pVM, &pPGM->StatGCSyncCR3DstNotPresent, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/DstNotPresent", STAMUNIT_OCCURENCES, "The number of times we've encountered a not present shadow entry for a present guest entry.");
1503	STAM_REG(pVM, &pPGM->StatGCSyncCR3DstSkippedGlobalPD, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/DstSkippedGlobalPD", STAMUNIT_OCCURENCES, "The number of times a global page directory wasn't flushed.");
1504	STAM_REG(pVM, &pPGM->StatGCSyncCR3DstSkippedGlobalPT, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/DstSkippedGlobalPT", STAMUNIT_OCCURENCES, "The number of times a page table with only global entries wasn't flushed.");
1505
1506	STAM_REG(pVM, &pPGM->StatHCSyncCR3, STAMTYPE_PROFILE, "/PGM/HC/SyncCR3", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMSyncCR3() body.");
1507	STAM_REG(pVM, &pPGM->StatHCSyncCR3Handlers, STAMTYPE_PROFILE, "/PGM/HC/SyncCR3/Handlers", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMSyncCR3() update handler section.");
1508	STAM_REG(pVM, &pPGM->StatHCSyncCR3HandlerVirtualUpdate, STAMTYPE_PROFILE, "/PGM/HC/SyncCR3/Handlers/VirtualUpdate",STAMUNIT_TICKS_PER_CALL, "Profiling of the virtual handler updates.");
1509	STAM_REG(pVM, &pPGM->StatHCSyncCR3HandlerVirtualReset, STAMTYPE_PROFILE, "/PGM/HC/SyncCR3/Handlers/VirtualReset", STAMUNIT_TICKS_PER_CALL, "Profiling of the virtual handler resets.");
1510	STAM_REG(pVM, &pPGM->StatHCSyncCR3Global, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/Global", STAMUNIT_OCCURENCES, "The number of global CR3 syncs.");
1511	STAM_REG(pVM, &pPGM->StatHCSyncCR3NotGlobal, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/NotGlobal", STAMUNIT_OCCURENCES, "The number of non-global CR3 syncs.");
1512	STAM_REG(pVM, &pPGM->StatHCSyncCR3DstCacheHit, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/DstChacheHit", STAMUNIT_OCCURENCES, "The number of times we got some kind of a cache hit.");
1513	STAM_REG(pVM, &pPGM->StatHCSyncCR3DstFreed, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/DstFreed", STAMUNIT_OCCURENCES, "The number of times we've had to free a shadow entry.");
1514	STAM_REG(pVM, &pPGM->StatHCSyncCR3DstFreedSrcNP, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/DstFreedSrcNP", STAMUNIT_OCCURENCES, "The number of times we've had to free a shadow entry for which the source entry was not present.");
1515	STAM_REG(pVM, &pPGM->StatHCSyncCR3DstNotPresent, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/DstNotPresent", STAMUNIT_OCCURENCES, "The number of times we've encountered a not present shadow entry for a present guest entry.");
1516	STAM_REG(pVM, &pPGM->StatHCSyncCR3DstSkippedGlobalPD, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/DstSkippedGlobalPD", STAMUNIT_OCCURENCES, "The number of times a global page directory wasn't flushed.");
1517	STAM_REG(pVM, &pPGM->StatHCSyncCR3DstSkippedGlobalPT, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/DstSkippedGlobalPT", STAMUNIT_OCCURENCES, "The number of times a page table with only global entries wasn't flushed.");
1518
1519	STAM_REG(pVM, &pPGM->StatVirtHandleSearchByPhysGC, STAMTYPE_PROFILE, "/PGM/VirtHandler/SearchByPhys/GC", STAMUNIT_TICKS_PER_CALL, "Profiling of pgmHandlerVirtualFindByPhysAddr in GC.");
1520	STAM_REG(pVM, &pPGM->StatVirtHandleSearchByPhysHC, STAMTYPE_PROFILE, "/PGM/VirtHandler/SearchByPhys/HC", STAMUNIT_TICKS_PER_CALL, "Profiling of pgmHandlerVirtualFindByPhysAddr in HC.");
1521	STAM_REG(pVM, &pPGM->StatHandlePhysicalReset, STAMTYPE_COUNTER, "/PGM/HC/HandlerPhysicalReset", STAMUNIT_OCCURENCES, "The number of times PGMR3HandlerPhysicalReset is called.");
1522
1523	STAM_REG(pVM, &pPGM->StatHCGstModifyPage, STAMTYPE_PROFILE, "/PGM/HC/GstModifyPage", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMGstModifyPage() body.");
1524	STAM_REG(pVM, &pPGM->StatGCGstModifyPage, STAMTYPE_PROFILE, "/PGM/GC/GstModifyPage", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMGstModifyPage() body.");
1525
1526	STAM_REG(pVM, &pPGM->StatSynPT4kGC, STAMTYPE_COUNTER, "/PGM/GC/SyncPT/4k", STAMUNIT_OCCURENCES, "Nr of 4k PT syncs");
1527	STAM_REG(pVM, &pPGM->StatSynPT4kHC, STAMTYPE_COUNTER, "/PGM/HC/SyncPT/4k", STAMUNIT_OCCURENCES, "Nr of 4k PT syncs");
1528	STAM_REG(pVM, &pPGM->StatSynPT4MGC, STAMTYPE_COUNTER, "/PGM/GC/SyncPT/4M", STAMUNIT_OCCURENCES, "Nr of 4M PT syncs");
1529	STAM_REG(pVM, &pPGM->StatSynPT4MHC, STAMTYPE_COUNTER, "/PGM/HC/SyncPT/4M", STAMUNIT_OCCURENCES, "Nr of 4M PT syncs");
1530
1531	STAM_REG(pVM, &pPGM->StatDynRamTotal, STAMTYPE_COUNTER, "/PGM/RAM/TotalAlloc", STAMUNIT_MEGABYTES, "Allocated mbs of guest ram.");
1532	STAM_REG(pVM, &pPGM->StatDynRamGrow, STAMTYPE_COUNTER, "/PGM/RAM/Grow", STAMUNIT_OCCURENCES, "Nr of pgmr3PhysGrowRange calls.");
1533
1534	STAM_REG(pVM, &pPGM->StatPageHCMapTlbHits, STAMTYPE_COUNTER, "/PGM/PageHCMap/TlbHits", STAMUNIT_OCCURENCES, "TLB hits.");
1535	STAM_REG(pVM, &pPGM->StatPageHCMapTlbMisses, STAMTYPE_COUNTER, "/PGM/PageHCMap/TlbMisses", STAMUNIT_OCCURENCES, "TLB misses.");
1536	STAM_REG(pVM, &pPGM->ChunkR3Map.c, STAMTYPE_U32, "/PGM/ChunkR3Map/c", STAMUNIT_OCCURENCES, "Number of mapped chunks.");
1537	STAM_REG(pVM, &pPGM->ChunkR3Map.cMax, STAMTYPE_U32, "/PGM/ChunkR3Map/cMax", STAMUNIT_OCCURENCES, "Maximum number of mapped chunks.");
1538	STAM_REG(pVM, &pPGM->StatChunkR3MapTlbHits, STAMTYPE_COUNTER, "/PGM/ChunkR3Map/TlbHits", STAMUNIT_OCCURENCES, "TLB hits.");
1539	STAM_REG(pVM, &pPGM->StatChunkR3MapTlbMisses, STAMTYPE_COUNTER, "/PGM/ChunkR3Map/TlbMisses", STAMUNIT_OCCURENCES, "TLB misses.");
1540	STAM_REG(pVM, &pPGM->StatPageReplaceShared, STAMTYPE_COUNTER, "/PGM/Page/ReplacedShared", STAMUNIT_OCCURENCES, "Times a shared page was replaced.");
1541	STAM_REG(pVM, &pPGM->StatPageReplaceZero, STAMTYPE_COUNTER, "/PGM/Page/ReplacedZero", STAMUNIT_OCCURENCES, "Times the zero page was replaced.");
1542	STAM_REG(pVM, &pPGM->StatPageHandyAllocs, STAMTYPE_COUNTER, "/PGM/Page/HandyAllocs", STAMUNIT_OCCURENCES, "Number of times we've allocated more handy pages.");
1543	STAM_REG(pVM, &pPGM->cAllPages, STAMTYPE_U32, "/PGM/Page/cAllPages", STAMUNIT_OCCURENCES, "The total number of pages.");
1544	STAM_REG(pVM, &pPGM->cPrivatePages, STAMTYPE_U32, "/PGM/Page/cPrivatePages", STAMUNIT_OCCURENCES, "The number of private pages.");
1545	STAM_REG(pVM, &pPGM->cSharedPages, STAMTYPE_U32, "/PGM/Page/cSharedPages", STAMUNIT_OCCURENCES, "The number of shared pages.");
1546	STAM_REG(pVM, &pPGM->cZeroPages, STAMTYPE_U32, "/PGM/Page/cZeroPages", STAMUNIT_OCCURENCES, "The number of zero backed pages.");
1547
1548	#ifdef PGMPOOL_WITH_GCPHYS_TRACKING
1549	STAM_REG(pVM, &pPGM->StatTrackVirgin, STAMTYPE_COUNTER, "/PGM/Track/Virgin", STAMUNIT_OCCURENCES, "The number of first time shadowings");
1550	STAM_REG(pVM, &pPGM->StatTrackAliased, STAMTYPE_COUNTER, "/PGM/Track/Aliased", STAMUNIT_OCCURENCES, "The number of times switching to cRef2, i.e. the page is being shadowed by two PTs.");
1551	STAM_REG(pVM, &pPGM->StatTrackAliasedMany, STAMTYPE_COUNTER, "/PGM/Track/AliasedMany", STAMUNIT_OCCURENCES, "The number of times we're tracking using cRef2.");
1552	STAM_REG(pVM, &pPGM->StatTrackAliasedLots, STAMTYPE_COUNTER, "/PGM/Track/AliasedLots", STAMUNIT_OCCURENCES, "The number of times we're hitting pages which has overflowed cRef2");
1553	STAM_REG(pVM, &pPGM->StatTrackOverflows, STAMTYPE_COUNTER, "/PGM/Track/Overflows", STAMUNIT_OCCURENCES, "The number of times the extent list grows to long.");
1554	STAM_REG(pVM, &pPGM->StatTrackDeref, STAMTYPE_PROFILE, "/PGM/Track/Deref", STAMUNIT_OCCURENCES, "Profiling of SyncPageWorkerTrackDeref (expensive).");
1555	#endif
1556
1557	for (unsigned i = 0; i < X86_PG_ENTRIES; i++)
1558	{
1559	/** @todo r=bird: We need a STAMR3RegisterF()! */
1560	char szName[32];
1561
1562	RTStrPrintf(szName, sizeof(szName), "/PGM/GC/PD/Trap0e/%04X", i);
1563	int rc = STAMR3Register(pVM, &pPGM->StatGCTrap0ePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, szName, STAMUNIT_OCCURENCES, "The number of traps in page directory n.");
1564	AssertRC(rc);
1565
1566	RTStrPrintf(szName, sizeof(szName), "/PGM/GC/PD/SyncPt/%04X", i);
1567	rc = STAMR3Register(pVM, &pPGM->StatGCSyncPtPD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, szName, STAMUNIT_OCCURENCES, "The number of syncs per PD n.");
1568	AssertRC(rc);
1569
1570	RTStrPrintf(szName, sizeof(szName), "/PGM/GC/PD/SyncPage/%04X", i);
1571	rc = STAMR3Register(pVM, &pPGM->StatGCSyncPagePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, szName, STAMUNIT_OCCURENCES, "The number of out of sync pages per page directory n.");
1572	AssertRC(rc);
1573	}
1574	}
1575	#endif /* VBOX_WITH_STATISTICS */
1576
1577	/**
1578	* Init the PGM bits that rely on VMMR0 and MM to be fully initialized.
1579	*
1580	* The dynamic mapping area will also be allocated and initialized at this
1581	* time. We could allocate it during PGMR3Init of course, but the mapping
1582	* wouldn't be allocated at that time preventing us from setting up the
1583	* page table entries with the dummy page.
1584	*
1585	* @returns VBox status code.
1586	* @param pVM VM handle.
1587	*/
1588	PGMR3DECL(int) PGMR3InitDynMap(PVM pVM)
1589	{
1590	/*
1591	* Reserve space for mapping the paging pages into guest context.
1592	*/
1593	int rc = MMR3HyperReserve(pVM, PAGE_SIZE * (2 + ELEMENTS(pVM->pgm.s.apHCPaePDs) + 1 + 2 + 2), "Paging", &pVM->pgm.s.pGC32BitPD);
1594	AssertRCReturn(rc, rc);
1595	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
1596
1597	/*
1598	* Reserve space for the dynamic mappings.
1599	*/
1600	/** @todo r=bird: Need to verify that the checks for crossing PTs are correct here. They seems to be assuming 4MB PTs.. */
1601	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping", &pVM->pgm.s.pbDynPageMapBaseGC);
1602	if ( VBOX_SUCCESS(rc)
1603	&& (pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_SHIFT) != ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_SHIFT))
1604	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping not crossing", &pVM->pgm.s.pbDynPageMapBaseGC);
1605	if (VBOX_SUCCESS(rc))
1606	{
1607	AssertRelease((pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_SHIFT) == ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_SHIFT));
1608	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
1609	}
1610	return rc;
1611	}
1612
1613
1614	/**
1615	* Ring-3 init finalizing.
1616	*
1617	* @returns VBox status code.
1618	* @param pVM The VM handle.
1619	*/
1620	PGMR3DECL(int) PGMR3InitFinalize(PVM pVM)
1621	{
1622	/*
1623	* Map the paging pages into the guest context.
1624	*/
1625	RTGCPTR GCPtr = pVM->pgm.s.pGC32BitPD;
1626	AssertReleaseReturn(GCPtr, VERR_INTERNAL_ERROR);
1627
1628	int rc = PGMMap(pVM, GCPtr, pVM->pgm.s.HCPhys32BitPD, PAGE_SIZE, 0);
1629	AssertRCReturn(rc, rc);
1630	pVM->pgm.s.pGC32BitPD = GCPtr;
1631	GCPtr += PAGE_SIZE;
1632	GCPtr += PAGE_SIZE; /* reserved page */
1633
1634	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.apHCPaePDs); i++)
1635	{
1636	rc = PGMMap(pVM, GCPtr, pVM->pgm.s.aHCPhysPaePDs[i], PAGE_SIZE, 0);
1637	AssertRCReturn(rc, rc);
1638	pVM->pgm.s.apGCPaePDs[i] = GCPtr;
1639	GCPtr += PAGE_SIZE;
1640	}
1641	/* A bit of paranoia is justified. */
1642	AssertRelease((RTGCUINTPTR)pVM->pgm.s.apGCPaePDs[0] + PAGE_SIZE == (RTGCUINTPTR)pVM->pgm.s.apGCPaePDs[1]);
1643	AssertRelease((RTGCUINTPTR)pVM->pgm.s.apGCPaePDs[1] + PAGE_SIZE == (RTGCUINTPTR)pVM->pgm.s.apGCPaePDs[2]);
1644	AssertRelease((RTGCUINTPTR)pVM->pgm.s.apGCPaePDs[2] + PAGE_SIZE == (RTGCUINTPTR)pVM->pgm.s.apGCPaePDs[3]);
1645	GCPtr += PAGE_SIZE; /* reserved page */
1646
1647	rc = PGMMap(pVM, GCPtr, pVM->pgm.s.HCPhysPaePDPT, PAGE_SIZE, 0);
1648	AssertRCReturn(rc, rc);
1649	pVM->pgm.s.pGCPaePDPT = GCPtr;
1650	GCPtr += PAGE_SIZE;
1651	GCPtr += PAGE_SIZE; /* reserved page */
1652
1653
1654	/*
1655	* Reserve space for the dynamic mappings.
1656	* Initialize the dynamic mapping pages with dummy pages to simply the cache.
1657	*/
1658	/* get the pointer to the page table entries. */
1659	PPGMMAPPING pMapping = pgmGetMapping(pVM, pVM->pgm.s.pbDynPageMapBaseGC);
1660	AssertRelease(pMapping);
1661	const uintptr_t off = pVM->pgm.s.pbDynPageMapBaseGC - pMapping->GCPtr;
1662	const unsigned iPT = off >> X86_PD_SHIFT;
1663	const unsigned iPG = (off >> X86_PT_SHIFT) & X86_PT_MASK;
1664	pVM->pgm.s.paDynPageMap32BitPTEsGC = pMapping->aPTs[iPT].pPTGC + iPG * sizeof(pMapping->aPTs[0].pPTR3->a[0]);
1665	pVM->pgm.s.paDynPageMapPaePTEsGC = pMapping->aPTs[iPT].paPaePTsGC + iPG * sizeof(pMapping->aPTs[0].paPaePTsR3->a[0]);
1666
1667	/* init cache */
1668	RTHCPHYS HCPhysDummy = MMR3PageDummyHCPhys(pVM);
1669	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.aHCPhysDynPageMapCache); i++)
1670	pVM->pgm.s.aHCPhysDynPageMapCache[i] = HCPhysDummy;
1671
1672	for (unsigned i = 0; i < MM_HYPER_DYNAMIC_SIZE; i += PAGE_SIZE)
1673	{
1674	rc = PGMMap(pVM, pVM->pgm.s.pbDynPageMapBaseGC + i, HCPhysDummy, PAGE_SIZE, 0);
1675	AssertRCReturn(rc, rc);
1676	}
1677
1678	return rc;
1679	}
1680
1681
1682	/**
1683	* Applies relocations to data and code managed by this
1684	* component. This function will be called at init and
1685	* whenever the VMM need to relocate it self inside the GC.
1686	*
1687	* @param pVM The VM.
1688	* @param offDelta Relocation delta relative to old location.
1689	*/
1690	PGMR3DECL(void) PGMR3Relocate(PVM pVM, RTGCINTPTR offDelta)
1691	{
1692	LogFlow(("PGMR3Relocate\n"));
1693
1694	/*
1695	* Paging stuff.
1696	*/
1697	pVM->pgm.s.GCPtrCR3Mapping += offDelta;
1698	/** @todo move this into shadow and guest specific relocation functions. */
1699	AssertMsg(pVM->pgm.s.pGC32BitPD, ("Init order, no relocation before paging is initialized!\n"));
1700	pVM->pgm.s.pGC32BitPD += offDelta;
1701	pVM->pgm.s.pGuestPDGC += offDelta;
1702	AssertCompile(ELEMENTS(pVM->pgm.s.apGCPaePDs) == ELEMENTS(pVM->pgm.s.apGstPaePDsGC));
1703	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.apGCPaePDs); i++)
1704	{
1705	pVM->pgm.s.apGCPaePDs[i] += offDelta;
1706	pVM->pgm.s.apGstPaePDsGC[i] += offDelta;
1707	}
1708	pVM->pgm.s.pGstPaePDPTGC += offDelta;
1709	pVM->pgm.s.pGCPaePDPT += offDelta;
1710
1711	pgmR3ModeDataInit(pVM, true /* resolve GC/R0 symbols */);
1712	pgmR3ModeDataSwitch(pVM, pVM->pgm.s.enmShadowMode, pVM->pgm.s.enmGuestMode);
1713
1714	PGM_SHW_PFN(Relocate, pVM)(pVM, offDelta);
1715	PGM_GST_PFN(Relocate, pVM)(pVM, offDelta);
1716	PGM_BTH_PFN(Relocate, pVM)(pVM, offDelta);
1717
1718	/*
1719	* Trees.
1720	*/
1721	pVM->pgm.s.pTreesGC = MMHyperHC2GC(pVM, pVM->pgm.s.pTreesHC);
1722
1723	/*
1724	* Ram ranges.
1725	*/
1726	if (pVM->pgm.s.pRamRangesR3)
1727	{
1728	pVM->pgm.s.pRamRangesGC = MMHyperHC2GC(pVM, pVM->pgm.s.pRamRangesR3);
1729	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesR3; pCur->pNextR3; pCur = pCur->pNextR3)
1730	#ifdef VBOX_WITH_NEW_PHYS_CODE
1731	pCur->pNextGC = MMHyperR3ToGC(pVM, pCur->pNextR3);
1732	#else
1733	{
1734	pCur->pNextGC = MMHyperR3ToGC(pVM, pCur->pNextR3);
1735	if (pCur->pavHCChunkGC)
1736	pCur->pavHCChunkGC = MMHyperHC2GC(pVM, pCur->pavHCChunkHC);
1737	}
1738	#endif
1739	}
1740
1741	/*
1742	* Update the two page directories with all page table mappings.
1743	* (One or more of them have changed, that's why we're here.)
1744	*/
1745	pVM->pgm.s.pMappingsGC = MMHyperHC2GC(pVM, pVM->pgm.s.pMappingsR3);
1746	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur->pNextR3; pCur = pCur->pNextR3)
1747	pCur->pNextGC = MMHyperHC2GC(pVM, pCur->pNextR3);
1748
1749	/* Relocate GC addresses of Page Tables. */
1750	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur; pCur = pCur->pNextR3)
1751	{
1752	for (RTHCUINT i = 0; i < pCur->cPTs; i++)
1753	{
1754	pCur->aPTs[i].pPTGC = MMHyperR3ToGC(pVM, pCur->aPTs[i].pPTR3);
1755	pCur->aPTs[i].paPaePTsGC = MMHyperR3ToGC(pVM, pCur->aPTs[i].paPaePTsR3);
1756	}
1757	}
1758
1759	/*
1760	* Dynamic page mapping area.
1761	*/
1762	pVM->pgm.s.paDynPageMap32BitPTEsGC += offDelta;
1763	pVM->pgm.s.paDynPageMapPaePTEsGC += offDelta;
1764	pVM->pgm.s.pbDynPageMapBaseGC += offDelta;
1765
1766	/*
1767	* The Zero page.
1768	*/
1769	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
1770	AssertRelease(pVM->pgm.s.pvZeroPgR0);
1771
1772	/*
1773	* Physical and virtual handlers.
1774	*/
1775	RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesHC->PhysHandlers, true, pgmR3RelocatePhysHandler, &offDelta);
1776	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesHC->VirtHandlers, true, pgmR3RelocateVirtHandler, &offDelta);
1777	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesHC->HyperVirtHandlers, true, pgmR3RelocateHyperVirtHandler, &offDelta);
1778
1779	/*
1780	* The page pool.
1781	*/
1782	pgmR3PoolRelocate(pVM);
1783	}
1784
1785
1786	/**
1787	* Callback function for relocating a physical access handler.
1788	*
1789	* @returns 0 (continue enum)
1790	* @param pNode Pointer to a PGMPHYSHANDLER node.
1791	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
1792	* not certain the delta will fit in a void pointer for all possible configs.
1793	*/
1794	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser)
1795	{
1796	PPGMPHYSHANDLER pHandler = (PPGMPHYSHANDLER)pNode;
1797	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
1798	if (pHandler->pfnHandlerGC)
1799	pHandler->pfnHandlerGC += offDelta;
1800	if ((RTGCUINTPTR)pHandler->pvUserGC >= 0x10000)
1801	pHandler->pvUserGC += offDelta;
1802	return 0;
1803	}
1804
1805
1806	/**
1807	* Callback function for relocating a virtual access handler.
1808	*
1809	* @returns 0 (continue enum)
1810	* @param pNode Pointer to a PGMVIRTHANDLER node.
1811	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
1812	* not certain the delta will fit in a void pointer for all possible configs.
1813	*/
1814	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
1815	{
1816	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
1817	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
1818	Assert( pHandler->enmType == PGMVIRTHANDLERTYPE_ALL
1819	\|\| pHandler->enmType == PGMVIRTHANDLERTYPE_WRITE);
1820	Assert(pHandler->pfnHandlerGC);
1821	pHandler->pfnHandlerGC += offDelta;
1822	return 0;
1823	}
1824
1825
1826	/**
1827	* Callback function for relocating a virtual access handler for the hypervisor mapping.
1828	*
1829	* @returns 0 (continue enum)
1830	* @param pNode Pointer to a PGMVIRTHANDLER node.
1831	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
1832	* not certain the delta will fit in a void pointer for all possible configs.
1833	*/
1834	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
1835	{
1836	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
1837	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
1838	Assert(pHandler->enmType == PGMVIRTHANDLERTYPE_HYPERVISOR);
1839	Assert(pHandler->pfnHandlerGC);
1840	pHandler->pfnHandlerGC += offDelta;
1841	return 0;
1842	}
1843
1844
1845	/**
1846	* The VM is being reset.
1847	*
1848	* For the PGM component this means that any PD write monitors
1849	* needs to be removed.
1850	*
1851	* @param pVM VM handle.
1852	*/
1853	PGMR3DECL(void) PGMR3Reset(PVM pVM)
1854	{
1855	LogFlow(("PGMR3Reset:\n"));
1856	VM_ASSERT_EMT(pVM);
1857
1858	pgmLock(pVM);
1859
1860	/*
1861	* Unfix any fixed mappings and disable CR3 monitoring.
1862	*/
1863	pVM->pgm.s.fMappingsFixed = false;
1864	pVM->pgm.s.GCPtrMappingFixed = 0;
1865	pVM->pgm.s.cbMappingFixed = 0;
1866
1867	int rc = PGM_GST_PFN(UnmonitorCR3, pVM)(pVM);
1868	AssertRC(rc);
1869	#ifdef DEBUG
1870	DBGFR3InfoLog(pVM, "mappings", NULL);
1871	DBGFR3InfoLog(pVM, "handlers", "all nostat");
1872	#endif
1873
1874	/*
1875	* Reset the shadow page pool.
1876	*/
1877	pgmR3PoolReset(pVM);
1878
1879	/*
1880	* Re-init other members.
1881	*/
1882	pVM->pgm.s.fA20Enabled = true;
1883
1884	/*
1885	* Clear the FFs PGM owns.
1886	*/
1887	VM_FF_CLEAR(pVM, VM_FF_PGM_SYNC_CR3);
1888	VM_FF_CLEAR(pVM, VM_FF_PGM_SYNC_CR3_NON_GLOBAL);
1889
1890	/*
1891	* Reset (zero) RAM pages.
1892	*/
1893	rc = pgmR3PhysRamReset(pVM);
1894	if (RT_SUCCESS(rc))
1895	{
1896	#ifdef VBOX_WITH_NEW_PHYS_CODE
1897	/*
1898	* Reset (zero) shadow ROM pages.
1899	*/
1900	rc = pgmR3PhysRomReset(pVM);
1901	#endif
1902	if (RT_SUCCESS(rc))
1903	{
1904	/*
1905	* Switch mode back to real mode.
1906	*/
1907	rc = pgmR3ChangeMode(pVM, PGMMODE_REAL);
1908	STAM_REL_COUNTER_RESET(&pVM->pgm.s.cGuestModeChanges);
1909	}
1910	}
1911
1912	pgmUnlock(pVM);
1913	//return rc;
1914	AssertReleaseRC(rc);
1915	}
1916
1917
1918	#ifdef VBOX_STRICT
1919	/**
1920	* VM state change callback for clearing fNoMorePhysWrites after
1921	* a snapshot has been created.
1922	*/
1923	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser)
1924	{
1925	if (enmState == VMSTATE_RUNNING)
1926	pVM->pgm.s.fNoMorePhysWrites = false;
1927	}
1928	#endif
1929
1930
1931	/**
1932	* Terminates the PGM.
1933	*
1934	* @returns VBox status code.
1935	* @param pVM Pointer to VM structure.
1936	*/
1937	PGMR3DECL(int) PGMR3Term(PVM pVM)
1938	{
1939	return PDMR3CritSectDelete(&pVM->pgm.s.CritSect);
1940	}
1941
1942
1943	/**
1944	* Execute state save operation.
1945	*
1946	* @returns VBox status code.
1947	* @param pVM VM Handle.
1948	* @param pSSM SSM operation handle.
1949	*/
1950	static DECLCALLBACK(int) pgmR3Save(PVM pVM, PSSMHANDLE pSSM)
1951	{
1952	PPGM pPGM = &pVM->pgm.s;
1953
1954	/* No more writes to physical memory after this point! */
1955	pVM->pgm.s.fNoMorePhysWrites = true;
1956
1957	/*
1958	* Save basic data (required / unaffected by relocation).
1959	*/
1960	#if 1
1961	SSMR3PutBool(pSSM, pPGM->fMappingsFixed);
1962	#else
1963	SSMR3PutUInt(pSSM, pPGM->fMappingsFixed);
1964	#endif
1965	SSMR3PutGCPtr(pSSM, pPGM->GCPtrMappingFixed);
1966	SSMR3PutU32(pSSM, pPGM->cbMappingFixed);
1967	SSMR3PutUInt(pSSM, pPGM->cbRamSize);
1968	SSMR3PutGCPhys(pSSM, pPGM->GCPhysA20Mask);
1969	SSMR3PutUInt(pSSM, pPGM->fA20Enabled);
1970	SSMR3PutUInt(pSSM, pPGM->fSyncFlags);
1971	SSMR3PutUInt(pSSM, pPGM->enmGuestMode);
1972	SSMR3PutU32(pSSM, ~0); /* Separator. */
1973
1974	/*
1975	* The guest mappings.
1976	*/
1977	uint32_t i = 0;
1978	for (PPGMMAPPING pMapping = pPGM->pMappingsR3; pMapping; pMapping = pMapping->pNextR3, i++)
1979	{
1980	SSMR3PutU32(pSSM, i);
1981	SSMR3PutStrZ(pSSM, pMapping->pszDesc); /* This is the best unique id we have... */
1982	SSMR3PutGCPtr(pSSM, pMapping->GCPtr);
1983	SSMR3PutGCUIntPtr(pSSM, pMapping->cPTs);
1984	/* flags are done by the mapping owners! */
1985	}
1986	SSMR3PutU32(pSSM, ~0); /* terminator. */
1987
1988	/*
1989	* Ram range flags and bits.
1990	*/
1991	i = 0;
1992	for (PPGMRAMRANGE pRam = pPGM->pRamRangesR3; pRam; pRam = pRam->pNextR3, i++)
1993	{
1994	/** @todo MMIO ranges may move (PCI reconfig), we currently assume they don't. */
1995
1996	SSMR3PutU32(pSSM, i);
1997	SSMR3PutGCPhys(pSSM, pRam->GCPhys);
1998	SSMR3PutGCPhys(pSSM, pRam->GCPhysLast);
1999	SSMR3PutGCPhys(pSSM, pRam->cb);
2000	SSMR3PutU8(pSSM, !!pRam->pvHC); /* boolean indicating memory or not. */
2001
2002	/* Flags. */
2003	const unsigned cPages = pRam->cb >> PAGE_SHIFT;
2004	for (unsigned iPage = 0; iPage < cPages; iPage++)
2005	SSMR3PutU16(pSSM, (uint16_t)(pRam->aPages[iPage].HCPhys & ~X86_PTE_PAE_PG_MASK)); /** @todo PAGE FLAGS */
2006
2007	/* any memory associated with the range. */
2008	if (pRam->fFlags & MM_RAM_FLAGS_DYNAMIC_ALLOC)
2009	{
2010	for (unsigned iChunk = 0; iChunk < (pRam->cb >> PGM_DYNAMIC_CHUNK_SHIFT); iChunk++)
2011	{
2012	if (pRam->pavHCChunkHC[iChunk])
2013	{
2014	SSMR3PutU8(pSSM, 1); /* chunk present */
2015	SSMR3PutMem(pSSM, pRam->pavHCChunkHC[iChunk], PGM_DYNAMIC_CHUNK_SIZE);
2016	}
2017	else
2018	SSMR3PutU8(pSSM, 0); /* no chunk present */
2019	}
2020	}
2021	else if (pRam->pvHC)
2022	{
2023	int rc = SSMR3PutMem(pSSM, pRam->pvHC, pRam->cb);
2024	if (VBOX_FAILURE(rc))
2025	{
2026	Log(("pgmR3Save: SSMR3PutMem(, %p, %#x) -> %Vrc\n", pRam->pvHC, pRam->cb, rc));
2027	return rc;
2028	}
2029	}
2030	}
2031	return SSMR3PutU32(pSSM, ~0); /* terminator. */
2032	}
2033
2034
2035	/**
2036	* Execute state load operation.
2037	*
2038	* @returns VBox status code.
2039	* @param pVM VM Handle.
2040	* @param pSSM SSM operation handle.
2041	* @param u32Version Data layout version.
2042	*/
2043	static DECLCALLBACK(int) pgmR3Load(PVM pVM, PSSMHANDLE pSSM, uint32_t u32Version)
2044	{
2045	/*
2046	* Validate version.
2047	*/
2048	if (u32Version != PGM_SAVED_STATE_VERSION)
2049	{
2050	Log(("pgmR3Load: Invalid version u32Version=%d (current %d)!\n", u32Version, PGM_SAVED_STATE_VERSION));
2051	return VERR_SSM_UNSUPPORTED_DATA_UNIT_VERSION;
2052	}
2053
2054	/*
2055	* Call the reset function to make sure all the memory is cleared.
2056	*/
2057	PGMR3Reset(pVM);
2058
2059	/*
2060	* Load basic data (required / unaffected by relocation).
2061	*/
2062	PPGM pPGM = &pVM->pgm.s;
2063	#if 1
2064	SSMR3GetBool(pSSM, &pPGM->fMappingsFixed);
2065	#else
2066	uint32_t u;
2067	SSMR3GetU32(pSSM, &u);
2068	pPGM->fMappingsFixed = u;
2069	#endif
2070	SSMR3GetGCPtr(pSSM, &pPGM->GCPtrMappingFixed);
2071	SSMR3GetU32(pSSM, &pPGM->cbMappingFixed);
2072
2073	RTUINT cbRamSize;
2074	int rc = SSMR3GetU32(pSSM, &cbRamSize);
2075	if (VBOX_FAILURE(rc))
2076	return rc;
2077	if (cbRamSize != pPGM->cbRamSize)
2078	return VERR_SSM_LOAD_MEMORY_SIZE_MISMATCH;
2079	SSMR3GetGCPhys(pSSM, &pPGM->GCPhysA20Mask);
2080	SSMR3GetUInt(pSSM, &pPGM->fA20Enabled);
2081	SSMR3GetUInt(pSSM, &pPGM->fSyncFlags);
2082	RTUINT uGuestMode;
2083	SSMR3GetUInt(pSSM, &uGuestMode);
2084	pPGM->enmGuestMode = (PGMMODE)uGuestMode;
2085
2086	/* check separator. */
2087	uint32_t u32Sep;
2088	SSMR3GetU32(pSSM, &u32Sep);
2089	if (VBOX_FAILURE(rc))
2090	return rc;
2091	if (u32Sep != (uint32_t)~0)
2092	{
2093	AssertMsgFailed(("u32Sep=%#x (first)\n", u32Sep));
2094	return VERR_SSM_DATA_UNIT_FORMAT_CHANGED;
2095	}
2096
2097	/*
2098	* The guest mappings.
2099	*/
2100	uint32_t i = 0;
2101	for (;; i++)
2102	{
2103	/* Check the seqence number / separator. */
2104	rc = SSMR3GetU32(pSSM, &u32Sep);
2105	if (VBOX_FAILURE(rc))
2106	return rc;
2107	if (u32Sep == ~0U)
2108	break;
2109	if (u32Sep != i)
2110	{
2111	AssertMsgFailed(("u32Sep=%#x (last)\n", u32Sep));
2112	return VERR_SSM_DATA_UNIT_FORMAT_CHANGED;
2113	}
2114
2115	/* get the mapping details. */
2116	char szDesc[256];
2117	szDesc[0] = '\0';
2118	rc = SSMR3GetStrZ(pSSM, szDesc, sizeof(szDesc));
2119	if (VBOX_FAILURE(rc))
2120	return rc;
2121	RTGCPTR GCPtr;
2122	SSMR3GetGCPtr(pSSM, &GCPtr);
2123	RTGCUINTPTR cPTs;
2124	rc = SSMR3GetU32(pSSM, &cPTs);
2125	if (VBOX_FAILURE(rc))
2126	return rc;
2127
2128	/* find matching range. */
2129	PPGMMAPPING pMapping;
2130	for (pMapping = pPGM->pMappingsR3; pMapping; pMapping = pMapping->pNextR3)
2131	if ( pMapping->cPTs == cPTs
2132	&& !strcmp(pMapping->pszDesc, szDesc))
2133	break;
2134	if (!pMapping)
2135	{
2136	LogRel(("Couldn't find mapping: cPTs=%#x szDesc=%s (GCPtr=%VGv)\n",
2137	cPTs, szDesc, GCPtr));
2138	AssertFailed();
2139	return VERR_SSM_LOAD_CONFIG_MISMATCH;
2140	}
2141
2142	/* relocate it. */
2143	if (pMapping->GCPtr != GCPtr)
2144	{
2145	AssertMsg((GCPtr >> X86_PD_SHIFT << X86_PD_SHIFT) == GCPtr, ("GCPtr=%VGv\n", GCPtr));
2146	#if HC_ARCH_BITS == 64
2147	LogRel(("Mapping: %VGv -> %VGv %s\n", pMapping->GCPtr, GCPtr, pMapping->pszDesc));
2148	#endif
2149	pgmR3MapRelocate(pVM, pMapping, pMapping->GCPtr, GCPtr);
2150	}
2151	else
2152	Log(("pgmR3Load: '%s' needed no relocation (%VGv)\n", szDesc, GCPtr));
2153	}
2154
2155	/*
2156	* Ram range flags and bits.
2157	*/
2158	i = 0;
2159	for (PPGMRAMRANGE pRam = pPGM->pRamRangesR3; pRam; pRam = pRam->pNextR3, i++)
2160	{
2161	/** @todo MMIO ranges may move (PCI reconfig), we currently assume they don't. */
2162	/* Check the seqence number / separator. */
2163	rc = SSMR3GetU32(pSSM, &u32Sep);
2164	if (VBOX_FAILURE(rc))
2165	return rc;
2166	if (u32Sep == ~0U)
2167	break;
2168	if (u32Sep != i)
2169	{
2170	AssertMsgFailed(("u32Sep=%#x (last)\n", u32Sep));
2171	return VERR_SSM_DATA_UNIT_FORMAT_CHANGED;
2172	}
2173
2174	/* Get the range details. */
2175	RTGCPHYS GCPhys;
2176	SSMR3GetGCPhys(pSSM, &GCPhys);
2177	RTGCPHYS GCPhysLast;
2178	SSMR3GetGCPhys(pSSM, &GCPhysLast);
2179	RTGCPHYS cb;
2180	SSMR3GetGCPhys(pSSM, &cb);
2181	uint8_t fHaveBits;
2182	rc = SSMR3GetU8(pSSM, &fHaveBits);
2183	if (VBOX_FAILURE(rc))
2184	return rc;
2185	if (fHaveBits & ~1)
2186	{
2187	AssertMsgFailed(("u32Sep=%#x (last)\n", u32Sep));
2188	return VERR_SSM_DATA_UNIT_FORMAT_CHANGED;
2189	}
2190
2191	/* Match it up with the current range. */
2192	if ( GCPhys != pRam->GCPhys
2193	\|\| GCPhysLast != pRam->GCPhysLast
2194	\|\| cb != pRam->cb
2195	\|\| fHaveBits != !!pRam->pvHC)
2196	{
2197	LogRel(("Ram range: %VGp-%VGp %VGp bytes %s\n"
2198	"State : %VGp-%VGp %VGp bytes %s\n",
2199	pRam->GCPhys, pRam->GCPhysLast, pRam->cb, pRam->pvHC ? "bits" : "nobits",
2200	GCPhys, GCPhysLast, cb, fHaveBits ? "bits" : "nobits"));
2201	/*
2202	* If we're loading a state for debugging purpose, don't make a fuss if
2203	* the MMIO[2] and ROM stuff isn't 100% right, just skip the mismatches.
2204	*/
2205	if ( SSMR3HandleGetAfter(pSSM) != SSMAFTER_DEBUG_IT
2206	\|\| GCPhys < 8 * _1M)
2207	AssertFailedReturn(VERR_SSM_LOAD_CONFIG_MISMATCH);
2208
2209	RTGCPHYS cPages = ((GCPhysLast - GCPhys) + 1) >> PAGE_SHIFT;
2210	while (cPages-- > 0)
2211	{
2212	uint16_t u16Ignore;
2213	SSMR3GetU16(pSSM, &u16Ignore);
2214	}
2215	continue;
2216	}
2217
2218	/* Flags. */
2219	const unsigned cPages = pRam->cb >> PAGE_SHIFT;
2220	for (unsigned iPage = 0; iPage < cPages; iPage++)
2221	{
2222	uint16_t u16 = 0;
2223	SSMR3GetU16(pSSM, &u16);
2224	u16 &= PAGE_OFFSET_MASK & ~( RT_BIT(4) \| RT_BIT(5) \| RT_BIT(6)
2225	\| RT_BIT(7) \| RT_BIT(8) \| RT_BIT(9) \| RT_BIT(10) );
2226	// &= MM_RAM_FLAGS_DYNAMIC_ALLOC \| MM_RAM_FLAGS_RESERVED \| MM_RAM_FLAGS_ROM \| MM_RAM_FLAGS_MMIO \| MM_RAM_FLAGS_MMIO2
2227	pRam->aPages[iPage].HCPhys = PGM_PAGE_GET_HCPHYS(&pRam->aPages[iPage]) \| (RTHCPHYS)u16; /** @todo PAGE FLAGS */
2228	}
2229
2230	/* any memory associated with the range. */
2231	if (pRam->fFlags & MM_RAM_FLAGS_DYNAMIC_ALLOC)
2232	{
2233	for (unsigned iChunk = 0; iChunk < (pRam->cb >> PGM_DYNAMIC_CHUNK_SHIFT); iChunk++)
2234	{
2235	uint8_t fValidChunk;
2236
2237	rc = SSMR3GetU8(pSSM, &fValidChunk);
2238	if (VBOX_FAILURE(rc))
2239	return rc;
2240	if (fValidChunk > 1)
2241	return VERR_SSM_DATA_UNIT_FORMAT_CHANGED;
2242
2243	if (fValidChunk)
2244	{
2245	if (!pRam->pavHCChunkHC[iChunk])
2246	{
2247	rc = pgmr3PhysGrowRange(pVM, pRam->GCPhys + iChunk * PGM_DYNAMIC_CHUNK_SIZE);
2248	if (VBOX_FAILURE(rc))
2249	return rc;
2250	}
2251	Assert(pRam->pavHCChunkHC[iChunk]);
2252
2253	SSMR3GetMem(pSSM, pRam->pavHCChunkHC[iChunk], PGM_DYNAMIC_CHUNK_SIZE);
2254	}
2255	/* else nothing to do */
2256	}
2257	}
2258	else if (pRam->pvHC)
2259	{
2260	int rc = SSMR3GetMem(pSSM, pRam->pvHC, pRam->cb);
2261	if (VBOX_FAILURE(rc))
2262	{
2263	Log(("pgmR3Save: SSMR3GetMem(, %p, %#x) -> %Vrc\n", pRam->pvHC, pRam->cb, rc));
2264	return rc;
2265	}
2266	}
2267	}
2268
2269	/*
2270	* We require a full resync now.
2271	*/
2272	VM_FF_SET(pVM, VM_FF_PGM_SYNC_CR3_NON_GLOBAL);
2273	VM_FF_SET(pVM, VM_FF_PGM_SYNC_CR3);
2274	pPGM->fSyncFlags \|= PGM_SYNC_UPDATE_PAGE_BIT_VIRTUAL;
2275	pPGM->fPhysCacheFlushPending = true;
2276	pgmR3HandlerPhysicalUpdateAll(pVM);
2277
2278	/*
2279	* Change the paging mode.
2280	*/
2281	return pgmR3ChangeMode(pVM, pPGM->enmGuestMode);
2282	}
2283
2284
2285	/**
2286	* Show paging mode.
2287	*
2288	* @param pVM VM Handle.
2289	* @param pHlp The info helpers.
2290	* @param pszArgs "all" (default), "guest", "shadow" or "host".
2291	*/
2292	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2293	{
2294	/* digest argument. */
2295	bool fGuest, fShadow, fHost;
2296	if (pszArgs)
2297	pszArgs = RTStrStripL(pszArgs);
2298	if (!pszArgs \|\| !*pszArgs \|\| strstr(pszArgs, "all"))
2299	fShadow = fHost = fGuest = true;
2300	else
2301	{
2302	fShadow = fHost = fGuest = false;
2303	if (strstr(pszArgs, "guest"))
2304	fGuest = true;
2305	if (strstr(pszArgs, "shadow"))
2306	fShadow = true;
2307	if (strstr(pszArgs, "host"))
2308	fHost = true;
2309	}
2310
2311	/* print info. */
2312	if (fGuest)
2313	pHlp->pfnPrintf(pHlp, "Guest paging mode: %s, changed %RU64 times, A20 %s\n",
2314	PGMGetModeName(pVM->pgm.s.enmGuestMode), pVM->pgm.s.cGuestModeChanges.c,
2315	pVM->pgm.s.fA20Enabled ? "enabled" : "disabled");
2316	if (fShadow)
2317	pHlp->pfnPrintf(pHlp, "Shadow paging mode: %s\n", PGMGetModeName(pVM->pgm.s.enmShadowMode));
2318	if (fHost)
2319	{
2320	const char *psz;
2321	switch (pVM->pgm.s.enmHostMode)
2322	{
2323	case SUPPAGINGMODE_INVALID: psz = "invalid"; break;
2324	case SUPPAGINGMODE_32_BIT: psz = "32-bit"; break;
2325	case SUPPAGINGMODE_32_BIT_GLOBAL: psz = "32-bit+G"; break;
2326	case SUPPAGINGMODE_PAE: psz = "PAE"; break;
2327	case SUPPAGINGMODE_PAE_GLOBAL: psz = "PAE+G"; break;
2328	case SUPPAGINGMODE_PAE_NX: psz = "PAE+NX"; break;
2329	case SUPPAGINGMODE_PAE_GLOBAL_NX: psz = "PAE+G+NX"; break;
2330	case SUPPAGINGMODE_AMD64: psz = "AMD64"; break;
2331	case SUPPAGINGMODE_AMD64_GLOBAL: psz = "AMD64+G"; break;
2332	case SUPPAGINGMODE_AMD64_NX: psz = "AMD64+NX"; break;
2333	case SUPPAGINGMODE_AMD64_GLOBAL_NX: psz = "AMD64+G+NX"; break;
2334	default: psz = "unknown"; break;
2335	}
2336	pHlp->pfnPrintf(pHlp, "Host paging mode: %s\n", psz);
2337	}
2338	}
2339
2340
2341	/**
2342	* Dump registered MMIO ranges to the log.
2343	*
2344	* @param pVM VM Handle.
2345	* @param pHlp The info helpers.
2346	* @param pszArgs Arguments, ignored.
2347	*/
2348	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2349	{
2350	NOREF(pszArgs);
2351	pHlp->pfnPrintf(pHlp,
2352	"RAM ranges (pVM=%p)\n"
2353	"%.s %.s\n",
2354	pVM,
2355	sizeof(RTGCPHYS) * 4 + 1, "GC Phys Range ",
2356	sizeof(RTHCPTR) * 2, "pvHC ");
2357
2358	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesR3; pCur; pCur = pCur->pNextR3)
2359	pHlp->pfnPrintf(pHlp,
2360	"%RGp-%RGp %RHv %s\n",
2361	pCur->GCPhys,
2362	pCur->GCPhysLast,
2363	pCur->pvHC,
2364	pCur->pszDesc);
2365	}
2366
2367	/**
2368	* Dump the page directory to the log.
2369	*
2370	* @param pVM VM Handle.
2371	* @param pHlp The info helpers.
2372	* @param pszArgs Arguments, ignored.
2373	*/
2374	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2375	{
2376	/** @todo fix this! Convert the PGMR3DumpHierarchyHC functions to do guest stuff. */
2377	/* Big pages supported? */
2378	const bool fPSE = !!(CPUMGetGuestCR4(pVM) & X86_CR4_PSE);
2379	/* Global pages supported? */
2380	const bool fPGE = !!(CPUMGetGuestCR4(pVM) & X86_CR4_PGE);
2381
2382	NOREF(pszArgs);
2383
2384	/*
2385	* Get page directory addresses.
2386	*/
2387	PX86PD pPDSrc = pVM->pgm.s.pGuestPDHC;
2388	Assert(pPDSrc);
2389	Assert(MMPhysGCPhys2HCVirt(pVM, (RTGCPHYS)(CPUMGetGuestCR3(pVM) & X86_CR3_PAGE_MASK), sizeof(*pPDSrc)) == pPDSrc);
2390
2391	/*
2392	* Iterate the page directory.
2393	*/
2394	for (unsigned iPD = 0; iPD < ELEMENTS(pPDSrc->a); iPD++)
2395	{
2396	X86PDE PdeSrc = pPDSrc->a[iPD];
2397	if (PdeSrc.n.u1Present)
2398	{
2399	if (PdeSrc.b.u1Size && fPSE)
2400	{
2401	pHlp->pfnPrintf(pHlp,
2402	"%04X - %VGp P=%d U=%d RW=%d G=%d - BIG\n",
2403	iPD,
2404	PdeSrc.u & X86_PDE_PG_MASK,
2405	PdeSrc.b.u1Present, PdeSrc.b.u1User, PdeSrc.b.u1Write, PdeSrc.b.u1Global && fPGE);
2406	}
2407	else
2408	{
2409	pHlp->pfnPrintf(pHlp,
2410	"%04X - %VGp P=%d U=%d RW=%d [G=%d]\n",
2411	iPD,
2412	PdeSrc.u & X86_PDE4M_PG_MASK,
2413	PdeSrc.n.u1Present, PdeSrc.n.u1User, PdeSrc.n.u1Write, PdeSrc.b.u1Global && fPGE);
2414	}
2415	}
2416	}
2417	}
2418
2419
2420	/**
2421	* Serivce a VMMCALLHOST_PGM_LOCK call.
2422	*
2423	* @returns VBox status code.
2424	* @param pVM The VM handle.
2425	*/
2426	PDMR3DECL(int) PGMR3LockCall(PVM pVM)
2427	{
2428	int rc = PDMR3CritSectEnterEx(&pVM->pgm.s.CritSect, true /* fHostCall */);
2429	AssertRC(rc);
2430	return rc;
2431	}
2432
2433
2434	/**
2435	* Converts a PGMMODE value to a PGM_TYPE_* \#define.
2436	*
2437	* @returns PGM_TYPE_*.
2438	* @param pgmMode The mode value to convert.
2439	*/
2440	DECLINLINE(unsigned) pgmModeToType(PGMMODE pgmMode)
2441	{
2442	switch (pgmMode)
2443	{
2444	case PGMMODE_REAL: return PGM_TYPE_REAL;
2445	case PGMMODE_PROTECTED: return PGM_TYPE_PROT;
2446	case PGMMODE_32_BIT: return PGM_TYPE_32BIT;
2447	case PGMMODE_PAE:
2448	case PGMMODE_PAE_NX: return PGM_TYPE_PAE;
2449	case PGMMODE_AMD64:
2450	case PGMMODE_AMD64_NX: return PGM_TYPE_AMD64;
2451	case PGMMODE_NESTED: return PGM_TYPE_NESTED;
2452	default:
2453	AssertFatalMsgFailed(("pgmMode=%d\n", pgmMode));
2454	}
2455	}
2456
2457
2458	/**
2459	* Gets the index into the paging mode data array of a SHW+GST mode.
2460	*
2461	* @returns PGM::paPagingData index.
2462	* @param uShwType The shadow paging mode type.
2463	* @param uGstType The guest paging mode type.
2464	*/
2465	DECLINLINE(unsigned) pgmModeDataIndex(unsigned uShwType, unsigned uGstType)
2466	{
2467	Assert(uShwType >= PGM_TYPE_32BIT && uShwType <= PGM_TYPE_NESTED);
2468	Assert(uGstType >= PGM_TYPE_REAL && uGstType <= PGM_TYPE_AMD64);
2469	return (uShwType - PGM_TYPE_32BIT) * (PGM_TYPE_AMD64 - PGM_TYPE_REAL + 1)
2470	+ (uGstType - PGM_TYPE_REAL);
2471	}
2472
2473
2474	/**
2475	* Gets the index into the paging mode data array of a SHW+GST mode.
2476	*
2477	* @returns PGM::paPagingData index.
2478	* @param enmShw The shadow paging mode.
2479	* @param enmGst The guest paging mode.
2480	*/
2481	DECLINLINE(unsigned) pgmModeDataIndexByMode(PGMMODE enmShw, PGMMODE enmGst)
2482	{
2483	Assert(enmShw >= PGMMODE_32_BIT && enmShw <= PGMMODE_MAX);
2484	Assert(enmGst > PGMMODE_INVALID && enmGst < PGMMODE_MAX);
2485	return pgmModeDataIndex(pgmModeToType(enmShw), pgmModeToType(enmGst));
2486	}
2487
2488
2489	/**
2490	* Calculates the max data index.
2491	* @returns The number of entries in the paging data array.
2492	*/
2493	DECLINLINE(unsigned) pgmModeDataMaxIndex(void)
2494	{
2495	return pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64) + 1;
2496	}
2497
2498
2499	/**
2500	* Initializes the paging mode data kept in PGM::paModeData.
2501	*
2502	* @param pVM The VM handle.
2503	* @param fResolveGCAndR0 Indicate whether or not GC and Ring-0 symbols can be resolved now.
2504	* This is used early in the init process to avoid trouble with PDM
2505	* not being initialized yet.
2506	*/
2507	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0)
2508	{
2509	PPGMMODEDATA pModeData;
2510	int rc;
2511
2512	/*
2513	* Allocate the array on the first call.
2514	*/
2515	if (!pVM->pgm.s.paModeData)
2516	{
2517	pVM->pgm.s.paModeData = (PPGMMODEDATA)MMR3HeapAllocZ(pVM, MM_TAG_PGM, sizeof(PGMMODEDATA) * pgmModeDataMaxIndex());
2518	AssertReturn(pVM->pgm.s.paModeData, VERR_NO_MEMORY);
2519	}
2520
2521	/*
2522	* Initialize the array entries.
2523	*/
2524	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_REAL)];
2525	pModeData->uShwType = PGM_TYPE_32BIT;
2526	pModeData->uGstType = PGM_TYPE_REAL;
2527	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2528	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2529	rc = PGM_BTH_NAME_32BIT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2530
2531	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGMMODE_PROTECTED)];
2532	pModeData->uShwType = PGM_TYPE_32BIT;
2533	pModeData->uGstType = PGM_TYPE_PROT;
2534	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2535	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2536	rc = PGM_BTH_NAME_32BIT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2537
2538	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_32BIT)];
2539	pModeData->uShwType = PGM_TYPE_32BIT;
2540	pModeData->uGstType = PGM_TYPE_32BIT;
2541	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2542	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2543	rc = PGM_BTH_NAME_32BIT_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2544
2545	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_REAL)];
2546	pModeData->uShwType = PGM_TYPE_PAE;
2547	pModeData->uGstType = PGM_TYPE_REAL;
2548	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2549	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2550	rc = PGM_BTH_NAME_PAE_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2551
2552	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PROT)];
2553	pModeData->uShwType = PGM_TYPE_PAE;
2554	pModeData->uGstType = PGM_TYPE_PROT;
2555	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2556	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2557	rc = PGM_BTH_NAME_PAE_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2558
2559	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_32BIT)];
2560	pModeData->uShwType = PGM_TYPE_PAE;
2561	pModeData->uGstType = PGM_TYPE_32BIT;
2562	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2563	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2564	rc = PGM_BTH_NAME_PAE_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2565
2566	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PAE)];
2567	pModeData->uShwType = PGM_TYPE_PAE;
2568	pModeData->uGstType = PGM_TYPE_PAE;
2569	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2570	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2571	rc = PGM_BTH_NAME_PAE_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2572
2573	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_AMD64, PGM_TYPE_AMD64)];
2574	pModeData->uShwType = PGM_TYPE_AMD64;
2575	pModeData->uGstType = PGM_TYPE_AMD64;
2576	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2577	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2578	rc = PGM_BTH_NAME_AMD64_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2579
2580	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_REAL)];
2581	pModeData->uShwType = PGM_TYPE_NESTED;
2582	pModeData->uGstType = PGM_TYPE_REAL;
2583	rc = PGM_SHW_NAME_NESTED(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2584	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2585	rc = PGM_BTH_NAME_NESTED_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2586
2587	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGMMODE_PROTECTED)];
2588	pModeData->uShwType = PGM_TYPE_NESTED;
2589	pModeData->uGstType = PGM_TYPE_PROT;
2590	rc = PGM_SHW_NAME_NESTED(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2591	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2592	rc = PGM_BTH_NAME_NESTED_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2593
2594	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_32BIT)];
2595	pModeData->uShwType = PGM_TYPE_NESTED;
2596	pModeData->uGstType = PGM_TYPE_32BIT;
2597	rc = PGM_SHW_NAME_NESTED(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2598	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2599	rc = PGM_BTH_NAME_NESTED_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2600
2601	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_PAE)];
2602	pModeData->uShwType = PGM_TYPE_NESTED;
2603	pModeData->uGstType = PGM_TYPE_PAE;
2604	rc = PGM_SHW_NAME_NESTED(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2605	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2606	rc = PGM_BTH_NAME_NESTED_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2607
2608	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2609	pModeData->uShwType = PGM_TYPE_NESTED;
2610	pModeData->uGstType = PGM_TYPE_AMD64;
2611	rc = PGM_SHW_NAME_NESTED(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2612	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2613	rc = PGM_BTH_NAME_NESTED_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2614
2615	return VINF_SUCCESS;
2616	}
2617
2618
2619	/**
2620	* Switch to different (or relocated in the relocate case) mode data.
2621	*
2622	* @param pVM The VM handle.
2623	* @param enmShw The the shadow paging mode.
2624	* @param enmGst The the guest paging mode.
2625	*/
2626	static void pgmR3ModeDataSwitch(PVM pVM, PGMMODE enmShw, PGMMODE enmGst)
2627	{
2628	PPGMMODEDATA pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(enmShw, enmGst)];
2629
2630	Assert(pModeData->uGstType == pgmModeToType(enmGst));
2631	Assert(pModeData->uShwType == pgmModeToType(enmShw));
2632
2633	/* shadow */
2634	pVM->pgm.s.pfnR3ShwRelocate = pModeData->pfnR3ShwRelocate;
2635	pVM->pgm.s.pfnR3ShwExit = pModeData->pfnR3ShwExit;
2636	pVM->pgm.s.pfnR3ShwGetPage = pModeData->pfnR3ShwGetPage;
2637	Assert(pVM->pgm.s.pfnR3ShwGetPage);
2638	pVM->pgm.s.pfnR3ShwModifyPage = pModeData->pfnR3ShwModifyPage;
2639
2640	pVM->pgm.s.pfnGCShwGetPage = pModeData->pfnGCShwGetPage;
2641	pVM->pgm.s.pfnGCShwModifyPage = pModeData->pfnGCShwModifyPage;
2642
2643	pVM->pgm.s.pfnR0ShwGetPage = pModeData->pfnR0ShwGetPage;
2644	pVM->pgm.s.pfnR0ShwModifyPage = pModeData->pfnR0ShwModifyPage;
2645
2646
2647	/* guest */
2648	pVM->pgm.s.pfnR3GstRelocate = pModeData->pfnR3GstRelocate;
2649	pVM->pgm.s.pfnR3GstExit = pModeData->pfnR3GstExit;
2650	pVM->pgm.s.pfnR3GstGetPage = pModeData->pfnR3GstGetPage;
2651	Assert(pVM->pgm.s.pfnR3GstGetPage);
2652	pVM->pgm.s.pfnR3GstModifyPage = pModeData->pfnR3GstModifyPage;
2653	pVM->pgm.s.pfnR3GstGetPDE = pModeData->pfnR3GstGetPDE;
2654	pVM->pgm.s.pfnR3GstMonitorCR3 = pModeData->pfnR3GstMonitorCR3;
2655	pVM->pgm.s.pfnR3GstUnmonitorCR3 = pModeData->pfnR3GstUnmonitorCR3;
2656	pVM->pgm.s.pfnR3GstMapCR3 = pModeData->pfnR3GstMapCR3;
2657	pVM->pgm.s.pfnR3GstUnmapCR3 = pModeData->pfnR3GstUnmapCR3;
2658	pVM->pgm.s.pfnR3GstWriteHandlerCR3 = pModeData->pfnR3GstWriteHandlerCR3;
2659	pVM->pgm.s.pszR3GstWriteHandlerCR3 = pModeData->pszR3GstWriteHandlerCR3;
2660	pVM->pgm.s.pfnR3GstPAEWriteHandlerCR3 = pModeData->pfnR3GstPAEWriteHandlerCR3;
2661	pVM->pgm.s.pszR3GstPAEWriteHandlerCR3 = pModeData->pszR3GstPAEWriteHandlerCR3;
2662
2663	pVM->pgm.s.pfnGCGstGetPage = pModeData->pfnGCGstGetPage;
2664	pVM->pgm.s.pfnGCGstModifyPage = pModeData->pfnGCGstModifyPage;
2665	pVM->pgm.s.pfnGCGstGetPDE = pModeData->pfnGCGstGetPDE;
2666	pVM->pgm.s.pfnGCGstMonitorCR3 = pModeData->pfnGCGstMonitorCR3;
2667	pVM->pgm.s.pfnGCGstUnmonitorCR3 = pModeData->pfnGCGstUnmonitorCR3;
2668	pVM->pgm.s.pfnGCGstMapCR3 = pModeData->pfnGCGstMapCR3;
2669	pVM->pgm.s.pfnGCGstUnmapCR3 = pModeData->pfnGCGstUnmapCR3;
2670	pVM->pgm.s.pfnGCGstWriteHandlerCR3 = pModeData->pfnGCGstWriteHandlerCR3;
2671	pVM->pgm.s.pfnGCGstPAEWriteHandlerCR3 = pModeData->pfnGCGstPAEWriteHandlerCR3;
2672
2673	pVM->pgm.s.pfnR0GstGetPage = pModeData->pfnR0GstGetPage;
2674	pVM->pgm.s.pfnR0GstModifyPage = pModeData->pfnR0GstModifyPage;
2675	pVM->pgm.s.pfnR0GstGetPDE = pModeData->pfnR0GstGetPDE;
2676	pVM->pgm.s.pfnR0GstMonitorCR3 = pModeData->pfnR0GstMonitorCR3;
2677	pVM->pgm.s.pfnR0GstUnmonitorCR3 = pModeData->pfnR0GstUnmonitorCR3;
2678	pVM->pgm.s.pfnR0GstMapCR3 = pModeData->pfnR0GstMapCR3;
2679	pVM->pgm.s.pfnR0GstUnmapCR3 = pModeData->pfnR0GstUnmapCR3;
2680	pVM->pgm.s.pfnR0GstWriteHandlerCR3 = pModeData->pfnR0GstWriteHandlerCR3;
2681	pVM->pgm.s.pfnR0GstPAEWriteHandlerCR3 = pModeData->pfnR0GstPAEWriteHandlerCR3;
2682
2683
2684	/* both */
2685	pVM->pgm.s.pfnR3BthRelocate = pModeData->pfnR3BthRelocate;
2686	pVM->pgm.s.pfnR3BthTrap0eHandler = pModeData->pfnR3BthTrap0eHandler;
2687	pVM->pgm.s.pfnR3BthInvalidatePage = pModeData->pfnR3BthInvalidatePage;
2688	pVM->pgm.s.pfnR3BthSyncCR3 = pModeData->pfnR3BthSyncCR3;
2689	Assert(pVM->pgm.s.pfnR3BthSyncCR3);
2690	pVM->pgm.s.pfnR3BthSyncPage = pModeData->pfnR3BthSyncPage;
2691	pVM->pgm.s.pfnR3BthPrefetchPage = pModeData->pfnR3BthPrefetchPage;
2692	pVM->pgm.s.pfnR3BthVerifyAccessSyncPage = pModeData->pfnR3BthVerifyAccessSyncPage;
2693	#ifdef VBOX_STRICT
2694	pVM->pgm.s.pfnR3BthAssertCR3 = pModeData->pfnR3BthAssertCR3;
2695	#endif
2696
2697	pVM->pgm.s.pfnGCBthTrap0eHandler = pModeData->pfnGCBthTrap0eHandler;
2698	pVM->pgm.s.pfnGCBthInvalidatePage = pModeData->pfnGCBthInvalidatePage;
2699	pVM->pgm.s.pfnGCBthSyncCR3 = pModeData->pfnGCBthSyncCR3;
2700	pVM->pgm.s.pfnGCBthSyncPage = pModeData->pfnGCBthSyncPage;
2701	pVM->pgm.s.pfnGCBthPrefetchPage = pModeData->pfnGCBthPrefetchPage;
2702	pVM->pgm.s.pfnGCBthVerifyAccessSyncPage = pModeData->pfnGCBthVerifyAccessSyncPage;
2703	#ifdef VBOX_STRICT
2704	pVM->pgm.s.pfnGCBthAssertCR3 = pModeData->pfnGCBthAssertCR3;
2705	#endif
2706
2707	pVM->pgm.s.pfnR0BthTrap0eHandler = pModeData->pfnR0BthTrap0eHandler;
2708	pVM->pgm.s.pfnR0BthInvalidatePage = pModeData->pfnR0BthInvalidatePage;
2709	pVM->pgm.s.pfnR0BthSyncCR3 = pModeData->pfnR0BthSyncCR3;
2710	pVM->pgm.s.pfnR0BthSyncPage = pModeData->pfnR0BthSyncPage;
2711	pVM->pgm.s.pfnR0BthPrefetchPage = pModeData->pfnR0BthPrefetchPage;
2712	pVM->pgm.s.pfnR0BthVerifyAccessSyncPage = pModeData->pfnR0BthVerifyAccessSyncPage;
2713	#ifdef VBOX_STRICT
2714	pVM->pgm.s.pfnR0BthAssertCR3 = pModeData->pfnR0BthAssertCR3;
2715	#endif
2716	}
2717
2718
2719	#ifdef DEBUG_bird
2720	#include <stdlib.h> /* getenv() remove me! */
2721	#endif
2722
2723	/**
2724	* Calculates the shadow paging mode.
2725	*
2726	* @returns The shadow paging mode.
2727	* @param enmGuestMode The guest mode.
2728	* @param enmHostMode The host mode.
2729	* @param enmShadowMode The current shadow mode.
2730	* @param penmSwitcher Where to store the switcher to use.
2731	* VMMSWITCHER_INVALID means no change.
2732	*/
2733	static PGMMODE pgmR3CalcShadowMode(PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher)
2734	{
2735	VMMSWITCHER enmSwitcher = VMMSWITCHER_INVALID;
2736	switch (enmGuestMode)
2737	{
2738	/*
2739	* When switching to real or protected mode we don't change
2740	* anything since it's likely that we'll switch back pretty soon.
2741	*
2742	* During pgmR3InitPaging we'll end up here with PGMMODE_INVALID
2743	* and is supposed to determin which shadow paging and switcher to
2744	* use during init.
2745	*/
2746	case PGMMODE_REAL:
2747	case PGMMODE_PROTECTED:
2748	if (enmShadowMode != PGMMODE_INVALID)
2749	break; /* (no change) */
2750	switch (enmHostMode)
2751	{
2752	case SUPPAGINGMODE_32_BIT:
2753	case SUPPAGINGMODE_32_BIT_GLOBAL:
2754	enmShadowMode = PGMMODE_32_BIT;
2755	enmSwitcher = VMMSWITCHER_32_TO_32;
2756	break;
2757
2758	case SUPPAGINGMODE_PAE:
2759	case SUPPAGINGMODE_PAE_NX:
2760	case SUPPAGINGMODE_PAE_GLOBAL:
2761	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2762	enmShadowMode = PGMMODE_PAE;
2763	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
2764	#ifdef DEBUG_bird
2765	if (getenv("VBOX_32BIT"))
2766	{
2767	enmShadowMode = PGMMODE_32_BIT;
2768	enmSwitcher = VMMSWITCHER_PAE_TO_32;
2769	}
2770	#endif
2771	break;
2772
2773	case SUPPAGINGMODE_AMD64:
2774	case SUPPAGINGMODE_AMD64_GLOBAL:
2775	case SUPPAGINGMODE_AMD64_NX:
2776	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2777	enmShadowMode = PGMMODE_PAE;
2778	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
2779	break;
2780
2781	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
2782	}
2783	break;
2784
2785	case PGMMODE_32_BIT:
2786	switch (enmHostMode)
2787	{
2788	case SUPPAGINGMODE_32_BIT:
2789	case SUPPAGINGMODE_32_BIT_GLOBAL:
2790	enmShadowMode = PGMMODE_32_BIT;
2791	enmSwitcher = VMMSWITCHER_32_TO_32;
2792	break;
2793
2794	case SUPPAGINGMODE_PAE:
2795	case SUPPAGINGMODE_PAE_NX:
2796	case SUPPAGINGMODE_PAE_GLOBAL:
2797	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2798	enmShadowMode = PGMMODE_PAE;
2799	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
2800	#ifdef DEBUG_bird
2801	if (getenv("VBOX_32BIT"))
2802	{
2803	enmShadowMode = PGMMODE_32_BIT;
2804	enmSwitcher = VMMSWITCHER_PAE_TO_32;
2805	}
2806	#endif
2807	break;
2808
2809	case SUPPAGINGMODE_AMD64:
2810	case SUPPAGINGMODE_AMD64_GLOBAL:
2811	case SUPPAGINGMODE_AMD64_NX:
2812	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2813	enmShadowMode = PGMMODE_PAE;
2814	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
2815	break;
2816
2817	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
2818	}
2819	break;
2820
2821	case PGMMODE_PAE:
2822	case PGMMODE_PAE_NX: /** @todo This might require more switchers and guest+both modes. */
2823	switch (enmHostMode)
2824	{
2825	case SUPPAGINGMODE_32_BIT:
2826	case SUPPAGINGMODE_32_BIT_GLOBAL:
2827	enmShadowMode = PGMMODE_PAE;
2828	enmSwitcher = VMMSWITCHER_32_TO_PAE;
2829	break;
2830
2831	case SUPPAGINGMODE_PAE:
2832	case SUPPAGINGMODE_PAE_NX:
2833	case SUPPAGINGMODE_PAE_GLOBAL:
2834	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2835	enmShadowMode = PGMMODE_PAE;
2836	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
2837	break;
2838
2839	case SUPPAGINGMODE_AMD64:
2840	case SUPPAGINGMODE_AMD64_GLOBAL:
2841	case SUPPAGINGMODE_AMD64_NX:
2842	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2843	enmShadowMode = PGMMODE_PAE;
2844	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
2845	break;
2846
2847	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
2848	}
2849	break;
2850
2851	case PGMMODE_AMD64:
2852	case PGMMODE_AMD64_NX:
2853	switch (enmHostMode)
2854	{
2855	case SUPPAGINGMODE_32_BIT:
2856	case SUPPAGINGMODE_32_BIT_GLOBAL:
2857	enmShadowMode = PGMMODE_PAE;
2858	enmSwitcher = VMMSWITCHER_32_TO_AMD64;
2859	break;
2860
2861	case SUPPAGINGMODE_PAE:
2862	case SUPPAGINGMODE_PAE_NX:
2863	case SUPPAGINGMODE_PAE_GLOBAL:
2864	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2865	enmShadowMode = PGMMODE_PAE;
2866	enmSwitcher = VMMSWITCHER_PAE_TO_AMD64;
2867	break;
2868
2869	case SUPPAGINGMODE_AMD64:
2870	case SUPPAGINGMODE_AMD64_GLOBAL:
2871	case SUPPAGINGMODE_AMD64_NX:
2872	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2873	enmShadowMode = PGMMODE_AMD64;
2874	enmSwitcher = VMMSWITCHER_AMD64_TO_AMD64;
2875	break;
2876
2877	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
2878	}
2879	break;
2880
2881
2882	default:
2883	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
2884	return PGMMODE_INVALID;
2885	}
2886
2887	*penmSwitcher = enmSwitcher;
2888	return enmShadowMode;
2889	}
2890
2891
2892	/**
2893	* Performs the actual mode change.
2894	* This is called by PGMChangeMode and pgmR3InitPaging().
2895	*
2896	* @returns VBox status code.
2897	* @param pVM VM handle.
2898	* @param enmGuestMode The new guest mode. This is assumed to be different from
2899	* the current mode.
2900	*/
2901	int pgmR3ChangeMode(PVM pVM, PGMMODE enmGuestMode)
2902	{
2903	LogFlow(("pgmR3ChangeMode: Guest mode: %d -> %d\n", pVM->pgm.s.enmGuestMode, enmGuestMode));
2904	STAM_REL_COUNTER_INC(&pVM->pgm.s.cGuestModeChanges);
2905
2906	/*
2907	* Calc the shadow mode and switcher.
2908	*/
2909	VMMSWITCHER enmSwitcher;
2910	PGMMODE enmShadowMode = pgmR3CalcShadowMode(enmGuestMode, pVM->pgm.s.enmHostMode, pVM->pgm.s.enmShadowMode, &enmSwitcher);
2911	if (enmSwitcher != VMMSWITCHER_INVALID)
2912	{
2913	/*
2914	* Select new switcher.
2915	*/
2916	int rc = VMMR3SelectSwitcher(pVM, enmSwitcher);
2917	if (VBOX_FAILURE(rc))
2918	{
2919	AssertReleaseMsgFailed(("VMMR3SelectSwitcher(%d) -> %Vrc\n", enmSwitcher, rc));
2920	return rc;
2921	}
2922	}
2923
2924	/*
2925	* Exit old mode(s).
2926	*/
2927	/* shadow */
2928	if (enmShadowMode != pVM->pgm.s.enmShadowMode)
2929	{
2930	LogFlow(("pgmR3ChangeMode: Shadow mode: %d -> %d\n", pVM->pgm.s.enmShadowMode, enmShadowMode));
2931	if (PGM_SHW_PFN(Exit, pVM))
2932	{
2933	int rc = PGM_SHW_PFN(Exit, pVM)(pVM);
2934	if (VBOX_FAILURE(rc))
2935	{
2936	AssertMsgFailed(("Exit failed for shadow mode %d: %Vrc\n", pVM->pgm.s.enmShadowMode, rc));
2937	return rc;
2938	}
2939	}
2940
2941	}
2942
2943	/* guest */
2944	if (PGM_GST_PFN(Exit, pVM))
2945	{
2946	int rc = PGM_GST_PFN(Exit, pVM)(pVM);
2947	if (VBOX_FAILURE(rc))
2948	{
2949	AssertMsgFailed(("Exit failed for guest mode %d: %Vrc\n", pVM->pgm.s.enmGuestMode, rc));
2950	return rc;
2951	}
2952	}
2953
2954	/*
2955	* Load new paging mode data.
2956	*/
2957	pgmR3ModeDataSwitch(pVM, enmShadowMode, enmGuestMode);
2958
2959	/*
2960	* Enter new shadow mode (if changed).
2961	*/
2962	if (enmShadowMode != pVM->pgm.s.enmShadowMode)
2963	{
2964	int rc;
2965	pVM->pgm.s.enmShadowMode = enmShadowMode;
2966	switch (enmShadowMode)
2967	{
2968	case PGMMODE_32_BIT:
2969	rc = PGM_SHW_NAME_32BIT(Enter)(pVM);
2970	break;
2971	case PGMMODE_PAE:
2972	case PGMMODE_PAE_NX:
2973	rc = PGM_SHW_NAME_PAE(Enter)(pVM);
2974	break;
2975	case PGMMODE_AMD64:
2976	case PGMMODE_AMD64_NX:
2977	rc = PGM_SHW_NAME_AMD64(Enter)(pVM);
2978	break;
2979	case PGMMODE_REAL:
2980	case PGMMODE_PROTECTED:
2981	default:
2982	AssertReleaseMsgFailed(("enmShadowMode=%d\n", enmShadowMode));
2983	return VERR_INTERNAL_ERROR;
2984	}
2985	if (VBOX_FAILURE(rc))
2986	{
2987	AssertReleaseMsgFailed(("Entering enmShadowMode=%d failed: %Vrc\n", enmShadowMode, rc));
2988	pVM->pgm.s.enmShadowMode = PGMMODE_INVALID;
2989	return rc;
2990	}
2991	}
2992
2993	/*
2994	* Enter the new guest and shadow+guest modes.
2995	*/
2996	int rc = -1;
2997	int rc2 = -1;
2998	RTGCPHYS GCPhysCR3 = NIL_RTGCPHYS;
2999	pVM->pgm.s.enmGuestMode = enmGuestMode;
3000	switch (enmGuestMode)
3001	{
3002	case PGMMODE_REAL:
3003	rc = PGM_GST_NAME_REAL(Enter)(pVM, NIL_RTGCPHYS);
3004	switch (pVM->pgm.s.enmShadowMode)
3005	{
3006	case PGMMODE_32_BIT:
3007	rc2 = PGM_BTH_NAME_32BIT_REAL(Enter)(pVM, NIL_RTGCPHYS);
3008	break;
3009	case PGMMODE_PAE:
3010	case PGMMODE_PAE_NX:
3011	rc2 = PGM_BTH_NAME_PAE_REAL(Enter)(pVM, NIL_RTGCPHYS);
3012	break;
3013	case PGMMODE_AMD64:
3014	case PGMMODE_AMD64_NX:
3015	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3016	default: AssertFailed(); break;
3017	}
3018	break;
3019
3020	case PGMMODE_PROTECTED:
3021	rc = PGM_GST_NAME_PROT(Enter)(pVM, NIL_RTGCPHYS);
3022	switch (pVM->pgm.s.enmShadowMode)
3023	{
3024	case PGMMODE_32_BIT:
3025	rc2 = PGM_BTH_NAME_32BIT_PROT(Enter)(pVM, NIL_RTGCPHYS);
3026	break;
3027	case PGMMODE_PAE:
3028	case PGMMODE_PAE_NX:
3029	rc2 = PGM_BTH_NAME_PAE_PROT(Enter)(pVM, NIL_RTGCPHYS);
3030	break;
3031	case PGMMODE_AMD64:
3032	case PGMMODE_AMD64_NX:
3033	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3034	default: AssertFailed(); break;
3035	}
3036	break;
3037
3038	case PGMMODE_32_BIT:
3039	GCPhysCR3 = CPUMGetGuestCR3(pVM) & X86_CR3_PAGE_MASK;
3040	rc = PGM_GST_NAME_32BIT(Enter)(pVM, GCPhysCR3);
3041	switch (pVM->pgm.s.enmShadowMode)
3042	{
3043	case PGMMODE_32_BIT:
3044	rc2 = PGM_BTH_NAME_32BIT_32BIT(Enter)(pVM, GCPhysCR3);
3045	break;
3046	case PGMMODE_PAE:
3047	case PGMMODE_PAE_NX:
3048	rc2 = PGM_BTH_NAME_PAE_32BIT(Enter)(pVM, GCPhysCR3);
3049	break;
3050	case PGMMODE_AMD64:
3051	case PGMMODE_AMD64_NX:
3052	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3053	default: AssertFailed(); break;
3054	}
3055	break;
3056
3057	//case PGMMODE_PAE_NX:
3058	case PGMMODE_PAE:
3059	{
3060	uint32_t u32Dummy, u32Features;
3061
3062	CPUMGetGuestCpuId(pVM, 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
3063	if (!(u32Features & X86_CPUID_FEATURE_EDX_PAE))
3064	{
3065	/* Pause first, then inform Main. */
3066	rc = VMR3SuspendNoSave(pVM);
3067	AssertRC(rc);
3068
3069	VMSetRuntimeError(pVM, true, "PAEmode",
3070	N_("The guest is trying to switch to the PAE mode which is currently disabled by default in VirtualBox. Experimental PAE support can be enabled using the -pae option with VBoxManage."));
3071	/* we must return TRUE here otherwise the recompiler will assert */
3072	return VINF_SUCCESS;
3073	}
3074	GCPhysCR3 = CPUMGetGuestCR3(pVM) & X86_CR3_PAE_PAGE_MASK;
3075	rc = PGM_GST_NAME_PAE(Enter)(pVM, GCPhysCR3);
3076	switch (pVM->pgm.s.enmShadowMode)
3077	{
3078	case PGMMODE_PAE:
3079	case PGMMODE_PAE_NX:
3080	rc2 = PGM_BTH_NAME_PAE_PAE(Enter)(pVM, GCPhysCR3);
3081	break;
3082	case PGMMODE_32_BIT:
3083	case PGMMODE_AMD64:
3084	case PGMMODE_AMD64_NX:
3085	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3086	default: AssertFailed(); break;
3087	}
3088	break;
3089	}
3090
3091	//case PGMMODE_AMD64_NX:
3092	case PGMMODE_AMD64:
3093	GCPhysCR3 = CPUMGetGuestCR3(pVM) & 0xfffffffffffff000ULL; /** @todo define this mask and make CR3 64-bit in this case! */
3094	rc = PGM_GST_NAME_AMD64(Enter)(pVM, GCPhysCR3);
3095	switch (pVM->pgm.s.enmShadowMode)
3096	{
3097	case PGMMODE_AMD64:
3098	case PGMMODE_AMD64_NX:
3099	rc2 = PGM_BTH_NAME_AMD64_AMD64(Enter)(pVM, GCPhysCR3);
3100	break;
3101	case PGMMODE_32_BIT:
3102	case PGMMODE_PAE:
3103	case PGMMODE_PAE_NX:
3104	AssertMsgFailed(("Should use AMD64 shadow mode!\n"));
3105	default: AssertFailed(); break;
3106	}
3107	break;
3108
3109	default:
3110	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
3111	rc = VERR_NOT_IMPLEMENTED;
3112	break;
3113	}
3114
3115	/* status codes. */
3116	AssertRC(rc);
3117	AssertRC(rc2);
3118	if (VBOX_SUCCESS(rc))
3119	{
3120	rc = rc2;
3121	if (VBOX_SUCCESS(rc)) /* no informational status codes. */
3122	rc = VINF_SUCCESS;
3123	}
3124
3125	/*
3126	* Notify SELM so it can update the TSSes with correct CR3s.
3127	*/
3128	SELMR3PagingModeChanged(pVM);
3129
3130	/* Notify HWACCM as well. */
3131	HWACCMR3PagingModeChanged(pVM, pVM->pgm.s.enmShadowMode);
3132	return rc;
3133	}
3134
3135
3136	/**
3137	* Dumps a PAE shadow page table.
3138	*
3139	* @returns VBox status code (VINF_SUCCESS).
3140	* @param pVM The VM handle.
3141	* @param pPT Pointer to the page table.
3142	* @param u64Address The virtual address of the page table starts.
3143	* @param fLongMode Set if this a long mode table; clear if it's a legacy mode table.
3144	* @param cMaxDepth The maxium depth.
3145	* @param pHlp Pointer to the output functions.
3146	*/
3147	static int pgmR3DumpHierarchyHCPaePT(PVM pVM, PX86PTPAE pPT, uint64_t u64Address, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3148	{
3149	for (unsigned i = 0; i < ELEMENTS(pPT->a); i++)
3150	{
3151	X86PTEPAE Pte = pPT->a[i];
3152	if (Pte.n.u1Present)
3153	{
3154	pHlp->pfnPrintf(pHlp,
3155	fLongMode /P R S A D G WT CD AT NX 4M a p ? /
3156	? "%016llx 3 \| P %c %c %c %c %c %s %s %s %s 4K %c%c%c %016llx\n"
3157	: "%08llx 2 \| P %c %c %c %c %c %s %s %s %s 4K %c%c%c %016llx\n",
3158	u64Address + ((uint64_t)i << X86_PT_PAE_SHIFT),
3159	Pte.n.u1Write ? 'W' : 'R',
3160	Pte.n.u1User ? 'U' : 'S',
3161	Pte.n.u1Accessed ? 'A' : '-',
3162	Pte.n.u1Dirty ? 'D' : '-',
3163	Pte.n.u1Global ? 'G' : '-',
3164	Pte.n.u1WriteThru ? "WT" : "--",
3165	Pte.n.u1CacheDisable? "CD" : "--",
3166	Pte.n.u1PAT ? "AT" : "--",
3167	Pte.n.u1NoExecute ? "NX" : "--",
3168	Pte.u & PGM_PTFLAGS_TRACK_DIRTY ? 'd' : '-',
3169	Pte.u & RT_BIT(10) ? '1' : '0',
3170	Pte.u & PGM_PTFLAGS_CSAM_VALIDATED? 'v' : '-',
3171	Pte.u & X86_PTE_PAE_PG_MASK);
3172	}
3173	}
3174	return VINF_SUCCESS;
3175	}
3176
3177
3178	/**
3179	* Dumps a PAE shadow page directory table.
3180	*
3181	* @returns VBox status code (VINF_SUCCESS).
3182	* @param pVM The VM handle.
3183	* @param HCPhys The physical address of the page directory table.
3184	* @param u64Address The virtual address of the page table starts.
3185	* @param cr4 The CR4, PSE is currently used.
3186	* @param fLongMode Set if this a long mode table; clear if it's a legacy mode table.
3187	* @param cMaxDepth The maxium depth.
3188	* @param pHlp Pointer to the output functions.
3189	*/
3190	static int pgmR3DumpHierarchyHCPaePD(PVM pVM, RTHCPHYS HCPhys, uint64_t u64Address, uint32_t cr4, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3191	{
3192	PX86PDPAE pPD = (PX86PDPAE)MMPagePhys2Page(pVM, HCPhys);
3193	if (!pPD)
3194	{
3195	pHlp->pfnPrintf(pHlp, "%0*llx error! Page directory at HCPhys=%#VHp was not found in the page pool!\n",
3196	fLongMode ? 16 : 8, u64Address, HCPhys);
3197	return VERR_INVALID_PARAMETER;
3198	}
3199	int rc = VINF_SUCCESS;
3200	for (unsigned i = 0; i < ELEMENTS(pPD->a); i++)
3201	{
3202	X86PDEPAE Pde = pPD->a[i];
3203	if (Pde.n.u1Present)
3204	{
3205	if ((cr4 & X86_CR4_PSE) && Pde.b.u1Size)
3206	pHlp->pfnPrintf(pHlp,
3207	fLongMode /P R S A D G WT CD AT NX 4M a p ? /
3208	? "%016llx 2 \| P %c %c %c %c %c %s %s %s %s 4M %c%c%c %016llx\n"
3209	: "%08llx 1 \| P %c %c %c %c %c %s %s %s %s 4M %c%c%c %016llx\n",
3210	u64Address + ((uint64_t)i << X86_PD_PAE_SHIFT),
3211	Pde.b.u1Write ? 'W' : 'R',
3212	Pde.b.u1User ? 'U' : 'S',
3213	Pde.b.u1Accessed ? 'A' : '-',
3214	Pde.b.u1Dirty ? 'D' : '-',
3215	Pde.b.u1Global ? 'G' : '-',
3216	Pde.b.u1WriteThru ? "WT" : "--",
3217	Pde.b.u1CacheDisable? "CD" : "--",
3218	Pde.b.u1PAT ? "AT" : "--",
3219	Pde.b.u1NoExecute ? "NX" : "--",
3220	Pde.u & RT_BIT_64(9) ? '1' : '0',
3221	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
3222	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
3223	Pde.u & X86_PDE_PAE_PG_MASK);
3224	else
3225	{
3226	pHlp->pfnPrintf(pHlp,
3227	fLongMode /P R S A D G WT CD AT NX 4M a p ? /
3228	? "%016llx 2 \| P %c %c %c %c %c %s %s .. %s 4K %c%c%c %016llx\n"
3229	: "%08llx 1 \| P %c %c %c %c %c %s %s .. %s 4K %c%c%c %016llx\n",
3230	u64Address + ((uint64_t)i << X86_PD_PAE_SHIFT),
3231	Pde.n.u1Write ? 'W' : 'R',
3232	Pde.n.u1User ? 'U' : 'S',
3233	Pde.n.u1Accessed ? 'A' : '-',
3234	Pde.n.u1Reserved0 ? '?' : '.', /* ignored */
3235	Pde.n.u1Reserved1 ? '?' : '.', /* ignored */
3236	Pde.n.u1WriteThru ? "WT" : "--",
3237	Pde.n.u1CacheDisable? "CD" : "--",
3238	Pde.n.u1NoExecute ? "NX" : "--",
3239	Pde.u & RT_BIT_64(9) ? '1' : '0',
3240	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
3241	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
3242	Pde.u & X86_PDE_PAE_PG_MASK);
3243	if (cMaxDepth >= 1)
3244	{
3245	/** @todo what about using the page pool for mapping PTs? */
3246	uint64_t u64AddressPT = u64Address + ((uint64_t)i << X86_PD_PAE_SHIFT);
3247	RTHCPHYS HCPhysPT = Pde.u & X86_PDE_PAE_PG_MASK;
3248	PX86PTPAE pPT = NULL;
3249	if (!(Pde.u & PGM_PDFLAGS_MAPPING))
3250	pPT = (PX86PTPAE)MMPagePhys2Page(pVM, HCPhysPT);
3251	else
3252	{
3253	for (PPGMMAPPING pMap = pVM->pgm.s.pMappingsR3; pMap; pMap = pMap->pNextR3)
3254	{
3255	uint64_t off = u64AddressPT - pMap->GCPtr;
3256	if (off < pMap->cb)
3257	{
3258	const int iPDE = (uint32_t)(off >> X86_PD_SHIFT);
3259	const int iSub = (int)((off >> X86_PD_PAE_SHIFT) & 1); /* MSC is a pain sometimes */
3260	if ((iSub ? pMap->aPTs[iPDE].HCPhysPaePT1 : pMap->aPTs[iPDE].HCPhysPaePT0) != HCPhysPT)
3261	pHlp->pfnPrintf(pHlp, "%0*llx error! Mapping error! PT %d has HCPhysPT=%VHp not %VHp is in the PD.\n",
3262	fLongMode ? 16 : 8, u64AddressPT, iPDE,
3263	iSub ? pMap->aPTs[iPDE].HCPhysPaePT1 : pMap->aPTs[iPDE].HCPhysPaePT0, HCPhysPT);
3264	pPT = &pMap->aPTs[iPDE].paPaePTsR3[iSub];
3265	}
3266	}
3267	}
3268	int rc2 = VERR_INVALID_PARAMETER;
3269	if (pPT)
3270	rc2 = pgmR3DumpHierarchyHCPaePT(pVM, pPT, u64AddressPT, fLongMode, cMaxDepth - 1, pHlp);
3271	else
3272	pHlp->pfnPrintf(pHlp, "%0*llx error! Page table at HCPhys=%#VHp was not found in the page pool!\n",
3273	fLongMode ? 16 : 8, u64AddressPT, HCPhysPT);
3274	if (rc2 < rc && VBOX_SUCCESS(rc))
3275	rc = rc2;
3276	}
3277	}
3278	}
3279	}
3280	return rc;
3281	}
3282
3283
3284	/**
3285	* Dumps a PAE shadow page directory pointer table.
3286	*
3287	* @returns VBox status code (VINF_SUCCESS).
3288	* @param pVM The VM handle.
3289	* @param HCPhys The physical address of the page directory pointer table.
3290	* @param u64Address The virtual address of the page table starts.
3291	* @param cr4 The CR4, PSE is currently used.
3292	* @param fLongMode Set if this a long mode table; clear if it's a legacy mode table.
3293	* @param cMaxDepth The maxium depth.
3294	* @param pHlp Pointer to the output functions.
3295	*/
3296	static int pgmR3DumpHierarchyHCPaePDPT(PVM pVM, RTHCPHYS HCPhys, uint64_t u64Address, uint32_t cr4, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3297	{
3298	PX86PDPT pPDPT = (PX86PDPT)MMPagePhys2Page(pVM, HCPhys);
3299	if (!pPDPT)
3300	{
3301	pHlp->pfnPrintf(pHlp, "%0*llx error! Page directory pointer table at HCPhys=%#VHp was not found in the page pool!\n",
3302	fLongMode ? 16 : 8, u64Address, HCPhys);
3303	return VERR_INVALID_PARAMETER;
3304	}
3305
3306	int rc = VINF_SUCCESS;
3307	const unsigned c = fLongMode ? ELEMENTS(pPDPT->a) : X86_PG_PAE_PDPE_ENTRIES;
3308	for (unsigned i = 0; i < c; i++)
3309	{
3310	X86PDPE Pdpe = pPDPT->a[i];
3311	if (Pdpe.n.u1Present)
3312	{
3313	if (fLongMode)
3314	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a p ? /
3315	"%016llx 1 \| P %c %c %c %c %c %s %s %s %s .. %c%c%c %016llx\n",
3316	u64Address + ((uint64_t)i << X86_PDPT_SHIFT),
3317	Pdpe.lm.u1Write ? 'W' : 'R',
3318	Pdpe.lm.u1User ? 'U' : 'S',
3319	Pdpe.lm.u1Accessed ? 'A' : '-',
3320	Pdpe.lm.u3Reserved & 1? '?' : '.', /* ignored */
3321	Pdpe.lm.u3Reserved & 4? '!' : '.', /* mbz */
3322	Pdpe.lm.u1WriteThru ? "WT" : "--",
3323	Pdpe.lm.u1CacheDisable? "CD" : "--",
3324	Pdpe.lm.u3Reserved & 2? "!" : "..",/* mbz */
3325	Pdpe.lm.u1NoExecute ? "NX" : "--",
3326	Pdpe.u & RT_BIT(9) ? '1' : '0',
3327	Pdpe.u & PGM_PLXFLAGS_PERMANENT ? 'p' : '-',
3328	Pdpe.u & RT_BIT(11) ? '1' : '0',
3329	Pdpe.u & X86_PDPE_PG_MASK);
3330	else
3331	pHlp->pfnPrintf(pHlp, /P G WT CD AT NX 4M a p ? /
3332	"%08x 0 \| P %c %s %s %s %s .. %c%c%c %016llx\n",
3333	i << X86_PDPT_SHIFT,
3334	Pdpe.n.u4Reserved & 1? '!' : '.', /* mbz */
3335	Pdpe.n.u4Reserved & 4? '!' : '.', /* mbz */
3336	Pdpe.n.u1WriteThru ? "WT" : "--",
3337	Pdpe.n.u1CacheDisable? "CD" : "--",
3338	Pdpe.n.u4Reserved & 2? "!" : "..",/* mbz */
3339	Pdpe.u & RT_BIT(9) ? '1' : '0',
3340	Pdpe.u & PGM_PLXFLAGS_PERMANENT ? 'p' : '-',
3341	Pdpe.u & RT_BIT(11) ? '1' : '0',
3342	Pdpe.u & X86_PDPE_PG_MASK);
3343	if (cMaxDepth >= 1)
3344	{
3345	int rc2 = pgmR3DumpHierarchyHCPaePD(pVM, Pdpe.u & X86_PDPE_PG_MASK, u64Address + ((uint64_t)i << X86_PDPT_SHIFT),
3346	cr4, fLongMode, cMaxDepth - 1, pHlp);
3347	if (rc2 < rc && VBOX_SUCCESS(rc))
3348	rc = rc2;
3349	}
3350	}
3351	}
3352	return rc;
3353	}
3354
3355
3356	/**
3357	* Dumps a 32-bit shadow page table.
3358	*
3359	* @returns VBox status code (VINF_SUCCESS).
3360	* @param pVM The VM handle.
3361	* @param HCPhys The physical address of the table.
3362	* @param cr4 The CR4, PSE is currently used.
3363	* @param cMaxDepth The maxium depth.
3364	* @param pHlp Pointer to the output functions.
3365	*/
3366	static int pgmR3DumpHierarchyHcPaePML4(PVM pVM, RTHCPHYS HCPhys, uint32_t cr4, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3367	{
3368	PX86PML4 pPML4 = (PX86PML4)MMPagePhys2Page(pVM, HCPhys);
3369	if (!pPML4)
3370	{
3371	pHlp->pfnPrintf(pHlp, "Page map level 4 at HCPhys=%#VHp was not found in the page pool!\n", HCPhys);
3372	return VERR_INVALID_PARAMETER;
3373	}
3374
3375	int rc = VINF_SUCCESS;
3376	for (unsigned i = 0; i < ELEMENTS(pPML4->a); i++)
3377	{
3378	X86PML4E Pml4e = pPML4->a[i];
3379	if (Pml4e.n.u1Present)
3380	{
3381	uint64_t u64Address = ((uint64_t)i << X86_PML4_SHIFT) \| (((uint64_t)i >> (X86_PML4_SHIFT - X86_PDPT_SHIFT - 1)) * 0xffff000000000000ULL);
3382	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a p ? /
3383	"%016llx 0 \| P %c %c %c %c %c %s %s %s %s .. %c%c%c %016llx\n",
3384	u64Address,
3385	Pml4e.n.u1Write ? 'W' : 'R',
3386	Pml4e.n.u1User ? 'U' : 'S',
3387	Pml4e.n.u1Accessed ? 'A' : '-',
3388	Pml4e.n.u3Reserved & 1? '?' : '.', /* ignored */
3389	Pml4e.n.u3Reserved & 4? '!' : '.', /* mbz */
3390	Pml4e.n.u1WriteThru ? "WT" : "--",
3391	Pml4e.n.u1CacheDisable? "CD" : "--",
3392	Pml4e.n.u3Reserved & 2? "!" : "..",/* mbz */
3393	Pml4e.n.u1NoExecute ? "NX" : "--",
3394	Pml4e.u & RT_BIT(9) ? '1' : '0',
3395	Pml4e.u & PGM_PLXFLAGS_PERMANENT ? 'p' : '-',
3396	Pml4e.u & RT_BIT(11) ? '1' : '0',
3397	Pml4e.u & X86_PML4E_PG_MASK);
3398
3399	if (cMaxDepth >= 1)
3400	{
3401	int rc2 = pgmR3DumpHierarchyHCPaePDPT(pVM, Pml4e.u & X86_PML4E_PG_MASK, u64Address, cr4, true, cMaxDepth - 1, pHlp);
3402	if (rc2 < rc && VBOX_SUCCESS(rc))
3403	rc = rc2;
3404	}
3405	}
3406	}
3407	return rc;
3408	}
3409
3410
3411	/**
3412	* Dumps a 32-bit shadow page table.
3413	*
3414	* @returns VBox status code (VINF_SUCCESS).
3415	* @param pVM The VM handle.
3416	* @param pPT Pointer to the page table.
3417	* @param u32Address The virtual address this table starts at.
3418	* @param pHlp Pointer to the output functions.
3419	*/
3420	int pgmR3DumpHierarchyHC32BitPT(PVM pVM, PX86PT pPT, uint32_t u32Address, PCDBGFINFOHLP pHlp)
3421	{
3422	for (unsigned i = 0; i < ELEMENTS(pPT->a); i++)
3423	{
3424	X86PTE Pte = pPT->a[i];
3425	if (Pte.n.u1Present)
3426	{
3427	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a m d /
3428	"%08x 1 \| P %c %c %c %c %c %s %s %s .. 4K %c%c%c %08x\n",
3429	u32Address + (i << X86_PT_SHIFT),
3430	Pte.n.u1Write ? 'W' : 'R',
3431	Pte.n.u1User ? 'U' : 'S',
3432	Pte.n.u1Accessed ? 'A' : '-',
3433	Pte.n.u1Dirty ? 'D' : '-',
3434	Pte.n.u1Global ? 'G' : '-',
3435	Pte.n.u1WriteThru ? "WT" : "--",
3436	Pte.n.u1CacheDisable? "CD" : "--",
3437	Pte.n.u1PAT ? "AT" : "--",
3438	Pte.u & PGM_PTFLAGS_TRACK_DIRTY ? 'd' : '-',
3439	Pte.u & RT_BIT(10) ? '1' : '0',
3440	Pte.u & PGM_PTFLAGS_CSAM_VALIDATED ? 'v' : '-',
3441	Pte.u & X86_PDE_PG_MASK);
3442	}
3443	}
3444	return VINF_SUCCESS;
3445	}
3446
3447
3448	/**
3449	* Dumps a 32-bit shadow page directory and page tables.
3450	*
3451	* @returns VBox status code (VINF_SUCCESS).
3452	* @param pVM The VM handle.
3453	* @param cr3 The root of the hierarchy.
3454	* @param cr4 The CR4, PSE is currently used.
3455	* @param cMaxDepth How deep into the hierarchy the dumper should go.
3456	* @param pHlp Pointer to the output functions.
3457	*/
3458	int pgmR3DumpHierarchyHC32BitPD(PVM pVM, uint32_t cr3, uint32_t cr4, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3459	{
3460	PX86PD pPD = (PX86PD)MMPagePhys2Page(pVM, cr3 & X86_CR3_PAGE_MASK);
3461	if (!pPD)
3462	{
3463	pHlp->pfnPrintf(pHlp, "Page directory at %#x was not found in the page pool!\n", cr3 & X86_CR3_PAGE_MASK);
3464	return VERR_INVALID_PARAMETER;
3465	}
3466
3467	int rc = VINF_SUCCESS;
3468	for (unsigned i = 0; i < ELEMENTS(pPD->a); i++)
3469	{
3470	X86PDE Pde = pPD->a[i];
3471	if (Pde.n.u1Present)
3472	{
3473	const uint32_t u32Address = i << X86_PD_SHIFT;
3474	if ((cr4 & X86_CR4_PSE) && Pde.b.u1Size)
3475	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a m d /
3476	"%08x 0 \| P %c %c %c %c %c %s %s %s .. 4M %c%c%c %08x\n",
3477	u32Address,
3478	Pde.b.u1Write ? 'W' : 'R',
3479	Pde.b.u1User ? 'U' : 'S',
3480	Pde.b.u1Accessed ? 'A' : '-',
3481	Pde.b.u1Dirty ? 'D' : '-',
3482	Pde.b.u1Global ? 'G' : '-',
3483	Pde.b.u1WriteThru ? "WT" : "--",
3484	Pde.b.u1CacheDisable? "CD" : "--",
3485	Pde.b.u1PAT ? "AT" : "--",
3486	Pde.u & RT_BIT_64(9) ? '1' : '0',
3487	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
3488	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
3489	Pde.u & X86_PDE4M_PG_MASK);
3490	else
3491	{
3492	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a m d /
3493	"%08x 0 \| P %c %c %c %c %c %s %s .. .. 4K %c%c%c %08x\n",
3494	u32Address,
3495	Pde.n.u1Write ? 'W' : 'R',
3496	Pde.n.u1User ? 'U' : 'S',
3497	Pde.n.u1Accessed ? 'A' : '-',
3498	Pde.n.u1Reserved0 ? '?' : '.', /* ignored */
3499	Pde.n.u1Reserved1 ? '?' : '.', /* ignored */
3500	Pde.n.u1WriteThru ? "WT" : "--",
3501	Pde.n.u1CacheDisable? "CD" : "--",
3502	Pde.u & RT_BIT_64(9) ? '1' : '0',
3503	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
3504	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
3505	Pde.u & X86_PDE_PG_MASK);
3506	if (cMaxDepth >= 1)
3507	{
3508	/** @todo what about using the page pool for mapping PTs? */
3509	RTHCPHYS HCPhys = Pde.u & X86_PDE_PG_MASK;
3510	PX86PT pPT = NULL;
3511	if (!(Pde.u & PGM_PDFLAGS_MAPPING))
3512	pPT = (PX86PT)MMPagePhys2Page(pVM, HCPhys);
3513	else
3514	{
3515	for (PPGMMAPPING pMap = pVM->pgm.s.pMappingsR3; pMap; pMap = pMap->pNextR3)
3516	if (u32Address - pMap->GCPtr < pMap->cb)
3517	{
3518	int iPDE = (u32Address - pMap->GCPtr) >> X86_PD_SHIFT;
3519	if (pMap->aPTs[iPDE].HCPhysPT != HCPhys)
3520	pHlp->pfnPrintf(pHlp, "%08x error! Mapping error! PT %d has HCPhysPT=%VHp not %VHp is in the PD.\n",
3521	u32Address, iPDE, pMap->aPTs[iPDE].HCPhysPT, HCPhys);
3522	pPT = pMap->aPTs[iPDE].pPTR3;
3523	}
3524	}
3525	int rc2 = VERR_INVALID_PARAMETER;
3526	if (pPT)
3527	rc2 = pgmR3DumpHierarchyHC32BitPT(pVM, pPT, u32Address, pHlp);
3528	else
3529	pHlp->pfnPrintf(pHlp, "%08x error! Page table at %#x was not found in the page pool!\n", u32Address, HCPhys);
3530	if (rc2 < rc && VBOX_SUCCESS(rc))
3531	rc = rc2;
3532	}
3533	}
3534	}
3535	}
3536
3537	return rc;
3538	}
3539
3540
3541	/**
3542	* Dumps a 32-bit shadow page table.
3543	*
3544	* @returns VBox status code (VINF_SUCCESS).
3545	* @param pVM The VM handle.
3546	* @param pPT Pointer to the page table.
3547	* @param u32Address The virtual address this table starts at.
3548	* @param PhysSearch Address to search for.
3549	*/
3550	int pgmR3DumpHierarchyGC32BitPT(PVM pVM, PX86PT pPT, uint32_t u32Address, RTGCPHYS PhysSearch)
3551	{
3552	for (unsigned i = 0; i < ELEMENTS(pPT->a); i++)
3553	{
3554	X86PTE Pte = pPT->a[i];
3555	if (Pte.n.u1Present)
3556	{
3557	Log(( /P R S A D G WT CD AT NX 4M a m d /
3558	"%08x 1 \| P %c %c %c %c %c %s %s %s .. 4K %c%c%c %08x\n",
3559	u32Address + (i << X86_PT_SHIFT),
3560	Pte.n.u1Write ? 'W' : 'R',
3561	Pte.n.u1User ? 'U' : 'S',
3562	Pte.n.u1Accessed ? 'A' : '-',
3563	Pte.n.u1Dirty ? 'D' : '-',
3564	Pte.n.u1Global ? 'G' : '-',
3565	Pte.n.u1WriteThru ? "WT" : "--",
3566	Pte.n.u1CacheDisable? "CD" : "--",
3567	Pte.n.u1PAT ? "AT" : "--",
3568	Pte.u & PGM_PTFLAGS_TRACK_DIRTY ? 'd' : '-',
3569	Pte.u & RT_BIT(10) ? '1' : '0',
3570	Pte.u & PGM_PTFLAGS_CSAM_VALIDATED ? 'v' : '-',
3571	Pte.u & X86_PDE_PG_MASK));
3572
3573	if ((Pte.u & X86_PDE_PG_MASK) == PhysSearch)
3574	{
3575	uint64_t fPageShw = 0;
3576	RTHCPHYS pPhysHC = 0;
3577
3578	PGMShwGetPage(pVM, (RTGCPTR)(u32Address + (i << X86_PT_SHIFT)), &fPageShw, &pPhysHC);
3579	Log(("Found %VGp at %VGv -> flags=%llx\n", PhysSearch, (RTGCPTR)(u32Address + (i << X86_PT_SHIFT)), fPageShw));
3580	}
3581	}
3582	}
3583	return VINF_SUCCESS;
3584	}
3585
3586
3587	/**
3588	* Dumps a 32-bit guest page directory and page tables.
3589	*
3590	* @returns VBox status code (VINF_SUCCESS).
3591	* @param pVM The VM handle.
3592	* @param cr3 The root of the hierarchy.
3593	* @param cr4 The CR4, PSE is currently used.
3594	* @param PhysSearch Address to search for.
3595	*/
3596	PGMR3DECL(int) PGMR3DumpHierarchyGC(PVM pVM, uint64_t cr3, uint64_t cr4, RTGCPHYS PhysSearch)
3597	{
3598	bool fLongMode = false;
3599	const unsigned cch = fLongMode ? 16 : 8; NOREF(cch);
3600	PX86PD pPD = 0;
3601
3602	int rc = PGM_GCPHYS_2_PTR(pVM, cr3 & X86_CR3_PAGE_MASK, &pPD);
3603	if (VBOX_FAILURE(rc) \|\| !pPD)
3604	{
3605	Log(("Page directory at %#x was not found in the page pool!\n", cr3 & X86_CR3_PAGE_MASK));
3606	return VERR_INVALID_PARAMETER;
3607	}
3608
3609	Log(("cr3=%08x cr4=%08x%s\n"
3610	"%-*s P - Present\n"
3611	"%-*s \| R/W - Read (0) / Write (1)\n"
3612	"%-*s \| \| U/S - User (1) / Supervisor (0)\n"
3613	"%-*s \| \| \| A - Accessed\n"
3614	"%-*s \| \| \| \| D - Dirty\n"
3615	"%-*s \| \| \| \| \| G - Global\n"
3616	"%-*s \| \| \| \| \| \| WT - Write thru\n"
3617	"%-*s \| \| \| \| \| \| \| CD - Cache disable\n"
3618	"%-*s \| \| \| \| \| \| \| \| AT - Attribute table (PAT)\n"
3619	"%-*s \| \| \| \| \| \| \| \| \| NX - No execute (K8)\n"
3620	"%-*s \| \| \| \| \| \| \| \| \| \| 4K/4M/2M - Page size.\n"
3621	"%-*s \| \| \| \| \| \| \| \| \| \| \| AVL - a=allocated; m=mapping; d=track dirty;\n"
3622	"%-*s \| \| \| \| \| \| \| \| \| \| \| \| p=permanent; v=validated;\n"
3623	"%-*s Level \| \| \| \| \| \| \| \| \| \| \| \| Page\n"
3624	/* xxxx n **** P R S A D G WT CD AT NX 4M AVL xxxxxxxxxxxxx
3625	- W U - - - -- -- -- -- -- 010 */
3626	, cr3, cr4, fLongMode ? " Long Mode" : "",
3627	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "",
3628	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "Address"));
3629
3630	for (unsigned i = 0; i < ELEMENTS(pPD->a); i++)
3631	{
3632	X86PDE Pde = pPD->a[i];
3633	if (Pde.n.u1Present)
3634	{
3635	const uint32_t u32Address = i << X86_PD_SHIFT;
3636
3637	if ((cr4 & X86_CR4_PSE) && Pde.b.u1Size)
3638	Log(( /P R S A D G WT CD AT NX 4M a m d /
3639	"%08x 0 \| P %c %c %c %c %c %s %s %s .. 4M %c%c%c %08x\n",
3640	u32Address,
3641	Pde.b.u1Write ? 'W' : 'R',
3642	Pde.b.u1User ? 'U' : 'S',
3643	Pde.b.u1Accessed ? 'A' : '-',
3644	Pde.b.u1Dirty ? 'D' : '-',
3645	Pde.b.u1Global ? 'G' : '-',
3646	Pde.b.u1WriteThru ? "WT" : "--",
3647	Pde.b.u1CacheDisable? "CD" : "--",
3648	Pde.b.u1PAT ? "AT" : "--",
3649	Pde.u & RT_BIT(9) ? '1' : '0',
3650	Pde.u & RT_BIT(10) ? '1' : '0',
3651	Pde.u & RT_BIT(11) ? '1' : '0',
3652	Pde.u & X86_PDE4M_PG_MASK));
3653	/** @todo PhysSearch */
3654	else
3655	{
3656	Log(( /P R S A D G WT CD AT NX 4M a m d /
3657	"%08x 0 \| P %c %c %c %c %c %s %s .. .. 4K %c%c%c %08x\n",
3658	u32Address,
3659	Pde.n.u1Write ? 'W' : 'R',
3660	Pde.n.u1User ? 'U' : 'S',
3661	Pde.n.u1Accessed ? 'A' : '-',
3662	Pde.n.u1Reserved0 ? '?' : '.', /* ignored */
3663	Pde.n.u1Reserved1 ? '?' : '.', /* ignored */
3664	Pde.n.u1WriteThru ? "WT" : "--",
3665	Pde.n.u1CacheDisable? "CD" : "--",
3666	Pde.u & RT_BIT(9) ? '1' : '0',
3667	Pde.u & RT_BIT(10) ? '1' : '0',
3668	Pde.u & RT_BIT(11) ? '1' : '0',
3669	Pde.u & X86_PDE_PG_MASK));
3670	////if (cMaxDepth >= 1)
3671	{
3672	/** @todo what about using the page pool for mapping PTs? */
3673	RTGCPHYS GCPhys = Pde.u & X86_PDE_PG_MASK;
3674	PX86PT pPT = NULL;
3675
3676	rc = PGM_GCPHYS_2_PTR(pVM, GCPhys, &pPT);
3677
3678	int rc2 = VERR_INVALID_PARAMETER;
3679	if (pPT)
3680	rc2 = pgmR3DumpHierarchyGC32BitPT(pVM, pPT, u32Address, PhysSearch);
3681	else
3682	Log(("%08x error! Page table at %#x was not found in the page pool!\n", u32Address, GCPhys));
3683	if (rc2 < rc && VBOX_SUCCESS(rc))
3684	rc = rc2;
3685	}
3686	}
3687	}
3688	}
3689
3690	return rc;
3691	}
3692
3693
3694	/**
3695	* Dumps a page table hierarchy use only physical addresses and cr4/lm flags.
3696	*
3697	* @returns VBox status code (VINF_SUCCESS).
3698	* @param pVM The VM handle.
3699	* @param cr3 The root of the hierarchy.
3700	* @param cr4 The cr4, only PAE and PSE is currently used.
3701	* @param fLongMode Set if long mode, false if not long mode.
3702	* @param cMaxDepth Number of levels to dump.
3703	* @param pHlp Pointer to the output functions.
3704	*/
3705	PGMR3DECL(int) PGMR3DumpHierarchyHC(PVM pVM, uint64_t cr3, uint64_t cr4, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3706	{
3707	if (!pHlp)
3708	pHlp = DBGFR3InfoLogHlp();
3709	if (!cMaxDepth)
3710	return VINF_SUCCESS;
3711	const unsigned cch = fLongMode ? 16 : 8;
3712	pHlp->pfnPrintf(pHlp,
3713	"cr3=%08x cr4=%08x%s\n"
3714	"%-*s P - Present\n"
3715	"%-*s \| R/W - Read (0) / Write (1)\n"
3716	"%-*s \| \| U/S - User (1) / Supervisor (0)\n"
3717	"%-*s \| \| \| A - Accessed\n"
3718	"%-*s \| \| \| \| D - Dirty\n"
3719	"%-*s \| \| \| \| \| G - Global\n"
3720	"%-*s \| \| \| \| \| \| WT - Write thru\n"
3721	"%-*s \| \| \| \| \| \| \| CD - Cache disable\n"
3722	"%-*s \| \| \| \| \| \| \| \| AT - Attribute table (PAT)\n"
3723	"%-*s \| \| \| \| \| \| \| \| \| NX - No execute (K8)\n"
3724	"%-*s \| \| \| \| \| \| \| \| \| \| 4K/4M/2M - Page size.\n"
3725	"%-*s \| \| \| \| \| \| \| \| \| \| \| AVL - a=allocated; m=mapping; d=track dirty;\n"
3726	"%-*s \| \| \| \| \| \| \| \| \| \| \| \| p=permanent; v=validated;\n"
3727	"%-*s Level \| \| \| \| \| \| \| \| \| \| \| \| Page\n"
3728	/* xxxx n **** P R S A D G WT CD AT NX 4M AVL xxxxxxxxxxxxx
3729	- W U - - - -- -- -- -- -- 010 */
3730	, cr3, cr4, fLongMode ? " Long Mode" : "",
3731	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "",
3732	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "Address");
3733	if (cr4 & X86_CR4_PAE)
3734	{
3735	if (fLongMode)
3736	return pgmR3DumpHierarchyHcPaePML4(pVM, cr3 & X86_CR3_PAGE_MASK, cr4, cMaxDepth, pHlp);
3737	return pgmR3DumpHierarchyHCPaePDPT(pVM, cr3 & X86_CR3_PAE_PAGE_MASK, 0, cr4, false, cMaxDepth, pHlp);
3738	}
3739	return pgmR3DumpHierarchyHC32BitPD(pVM, cr3 & X86_CR3_PAGE_MASK, cr4, cMaxDepth, pHlp);
3740	}
3741
3742
3743
3744	#ifdef VBOX_WITH_DEBUGGER
3745	/**
3746	* The '.pgmram' command.
3747	*
3748	* @returns VBox status.
3749	* @param pCmd Pointer to the command descriptor (as registered).
3750	* @param pCmdHlp Pointer to command helper functions.
3751	* @param pVM Pointer to the current VM (if any).
3752	* @param paArgs Pointer to (readonly) array of arguments.
3753	* @param cArgs Number of arguments in the array.
3754	*/
3755	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
3756	{
3757	/*
3758	* Validate input.
3759	*/
3760	if (!pVM)
3761	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires VM to be selected.\n");
3762	if (!pVM->pgm.s.pRamRangesGC)
3763	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Sorry, no Ram is registered.\n");
3764
3765	/*
3766	* Dump the ranges.
3767	*/
3768	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "From - To (incl) pvHC\n");
3769	PPGMRAMRANGE pRam;
3770	for (pRam = pVM->pgm.s.pRamRangesR3; pRam; pRam = pRam->pNextR3)
3771	{
3772	rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL,
3773	"%VGp - %VGp %p\n",
3774	pRam->GCPhys, pRam->GCPhysLast, pRam->pvHC);
3775	if (VBOX_FAILURE(rc))
3776	return rc;
3777	}
3778
3779	return VINF_SUCCESS;
3780	}
3781
3782
3783	/**
3784	* The '.pgmmap' command.
3785	*
3786	* @returns VBox status.
3787	* @param pCmd Pointer to the command descriptor (as registered).
3788	* @param pCmdHlp Pointer to command helper functions.
3789	* @param pVM Pointer to the current VM (if any).
3790	* @param paArgs Pointer to (readonly) array of arguments.
3791	* @param cArgs Number of arguments in the array.
3792	*/
3793	static DECLCALLBACK(int) pgmR3CmdMap(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
3794	{
3795	/*
3796	* Validate input.
3797	*/
3798	if (!pVM)
3799	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires VM to be selected.\n");
3800	if (!pVM->pgm.s.pMappingsR3)
3801	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Sorry, no mappings are registered.\n");
3802
3803	/*
3804	* Print message about the fixedness of the mappings.
3805	*/
3806	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, pVM->pgm.s.fMappingsFixed ? "The mappings are FIXED.\n" : "The mappings are FLOATING.\n");
3807	if (VBOX_FAILURE(rc))
3808	return rc;
3809
3810	/*
3811	* Dump the ranges.
3812	*/
3813	PPGMMAPPING pCur;
3814	for (pCur = pVM->pgm.s.pMappingsR3; pCur; pCur = pCur->pNextR3)
3815	{
3816	rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL,
3817	"%08x - %08x %s\n",
3818	pCur->GCPtr, pCur->GCPtrLast, pCur->pszDesc);
3819	if (VBOX_FAILURE(rc))
3820	return rc;
3821	}
3822
3823	return VINF_SUCCESS;
3824	}
3825
3826
3827	/**
3828	* The '.pgmsync' command.
3829	*
3830	* @returns VBox status.
3831	* @param pCmd Pointer to the command descriptor (as registered).
3832	* @param pCmdHlp Pointer to command helper functions.
3833	* @param pVM Pointer to the current VM (if any).
3834	* @param paArgs Pointer to (readonly) array of arguments.
3835	* @param cArgs Number of arguments in the array.
3836	*/
3837	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
3838	{
3839	/*
3840	* Validate input.
3841	*/
3842	if (!pVM)
3843	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires VM to be selected.\n");
3844
3845	/*
3846	* Force page directory sync.
3847	*/
3848	VM_FF_SET(pVM, VM_FF_PGM_SYNC_CR3);
3849
3850	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Forcing page directory sync.\n");
3851	if (VBOX_FAILURE(rc))
3852	return rc;
3853
3854	return VINF_SUCCESS;
3855	}
3856
3857
3858	/**
3859	* The '.pgmsyncalways' command.
3860	*
3861	* @returns VBox status.
3862	* @param pCmd Pointer to the command descriptor (as registered).
3863	* @param pCmdHlp Pointer to command helper functions.
3864	* @param pVM Pointer to the current VM (if any).
3865	* @param paArgs Pointer to (readonly) array of arguments.
3866	* @param cArgs Number of arguments in the array.
3867	*/
3868	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
3869	{
3870	/*
3871	* Validate input.
3872	*/
3873	if (!pVM)
3874	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires VM to be selected.\n");
3875
3876	/*
3877	* Force page directory sync.
3878	*/
3879	if (pVM->pgm.s.fSyncFlags & PGM_SYNC_ALWAYS)
3880	{
3881	ASMAtomicAndU32(&pVM->pgm.s.fSyncFlags, ~PGM_SYNC_ALWAYS);
3882	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Disabled permanent forced page directory syncing.\n");
3883	}
3884	else
3885	{
3886	ASMAtomicOrU32(&pVM->pgm.s.fSyncFlags, PGM_SYNC_ALWAYS);
3887	VM_FF_SET(pVM, VM_FF_PGM_SYNC_CR3);
3888	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Enabled permanent forced page directory syncing.\n");
3889	}
3890	}
3891
3892	#endif
3893
3894	/**
3895	* pvUser argument of the pgmR3CheckIntegrity*Node callbacks.
3896	*/
3897	typedef struct PGMCHECKINTARGS
3898	{
3899	bool fLeftToRight; /*< true: left-to-right; false: right-to-left. /
3900	PPGMPHYSHANDLER pPrevPhys;
3901	PPGMVIRTHANDLER pPrevVirt;
3902	PPGMPHYS2VIRTHANDLER pPrevPhys2Virt;
3903	PVM pVM;
3904	} PGMCHECKINTARGS, *PPGMCHECKINTARGS;
3905
3906	/**
3907	* Validate a node in the physical handler tree.
3908	*
3909	* @returns 0 on if ok, other wise 1.
3910	* @param pNode The handler node.
3911	* @param pvUser pVM.
3912	*/
3913	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
3914	{
3915	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3916	PPGMPHYSHANDLER pCur = (PPGMPHYSHANDLER)pNode;
3917	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
3918	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %VGp-%VGp %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3919	AssertReleaseMsg( !pArgs->pPrevPhys
3920	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys->Core.KeyLast > pCur->Core.Key),
3921	("pPrevPhys=%p %VGp-%VGp %s\n"
3922	" pCur=%p %VGp-%VGp %s\n",
3923	pArgs->pPrevPhys, pArgs->pPrevPhys->Core.Key, pArgs->pPrevPhys->Core.KeyLast, pArgs->pPrevPhys->pszDesc,
3924	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3925	pArgs->pPrevPhys = pCur;
3926	return 0;
3927	}
3928
3929
3930	/**
3931	* Validate a node in the virtual handler tree.
3932	*
3933	* @returns 0 on if ok, other wise 1.
3934	* @param pNode The handler node.
3935	* @param pvUser pVM.
3936	*/
3937	static DECLCALLBACK(int) pgmR3CheckIntegrityVirtHandlerNode(PAVLROGCPTRNODECORE pNode, void *pvUser)
3938	{
3939	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3940	PPGMVIRTHANDLER pCur = (PPGMVIRTHANDLER)pNode;
3941	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
3942	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %VGv-%VGv %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3943	AssertReleaseMsg( !pArgs->pPrevVirt
3944	\|\| (pArgs->fLeftToRight ? pArgs->pPrevVirt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevVirt->Core.KeyLast > pCur->Core.Key),
3945	("pPrevVirt=%p %VGv-%VGv %s\n"
3946	" pCur=%p %VGv-%VGv %s\n",
3947	pArgs->pPrevVirt, pArgs->pPrevVirt->Core.Key, pArgs->pPrevVirt->Core.KeyLast, pArgs->pPrevVirt->pszDesc,
3948	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3949	for (unsigned iPage = 0; iPage < pCur->cPages; iPage++)
3950	{
3951	AssertReleaseMsg(pCur->aPhysToVirt[iPage].offVirtHandler == -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage]),
3952	("pCur=%p %VGv-%VGv %s\n"
3953	"iPage=%d offVirtHandle=%#x expected %#x\n",
3954	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc,
3955	iPage, pCur->aPhysToVirt[iPage].offVirtHandler, -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage])));
3956	}
3957	pArgs->pPrevVirt = pCur;
3958	return 0;
3959	}
3960
3961
3962	/**
3963	* Validate a node in the virtual handler tree.
3964	*
3965	* @returns 0 on if ok, other wise 1.
3966	* @param pNode The handler node.
3967	* @param pvUser pVM.
3968	*/
3969	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysToVirtHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
3970	{
3971	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3972	PPGMPHYS2VIRTHANDLER pCur = (PPGMPHYS2VIRTHANDLER)pNode;
3973	AssertReleaseMsgReturn(!((uintptr_t)pCur & 3), ("\n"), 1);
3974	AssertReleaseMsgReturn(!(pCur->offVirtHandler & 3), ("\n"), 1);
3975	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %VGp-%VGp\n", pCur, pCur->Core.Key, pCur->Core.KeyLast));
3976	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
3977	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
3978	("pPrevPhys2Virt=%p %VGp-%VGp\n"
3979	" pCur=%p %VGp-%VGp\n",
3980	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
3981	pCur, pCur->Core.Key, pCur->Core.KeyLast));
3982	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
3983	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
3984	("pPrevPhys2Virt=%p %VGp-%VGp\n"
3985	" pCur=%p %VGp-%VGp\n",
3986	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
3987	pCur, pCur->Core.Key, pCur->Core.KeyLast));
3988	AssertReleaseMsg((pCur->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD),
3989	("pCur=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
3990	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
3991	if (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK)
3992	{
3993	PPGMPHYS2VIRTHANDLER pCur2 = pCur;
3994	for (;;)
3995	{
3996	pCur2 = (PPGMPHYS2VIRTHANDLER)((intptr_t)pCur + (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK));
3997	AssertReleaseMsg(pCur2 != pCur,
3998	(" pCur=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
3999	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
4000	AssertReleaseMsg((pCur2->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == PGMPHYS2VIRTHANDLER_IN_TREE,
4001	(" pCur=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4002	"pCur2=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4003	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4004	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4005	AssertReleaseMsg((pCur2->Core.Key ^ pCur->Core.Key) < PAGE_SIZE,
4006	(" pCur=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4007	"pCur2=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4008	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4009	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4010	AssertReleaseMsg((pCur2->Core.KeyLast ^ pCur->Core.KeyLast) < PAGE_SIZE,
4011	(" pCur=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4012	"pCur2=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4013	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4014	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4015	if (!(pCur2->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK))
4016	break;
4017	}
4018	}
4019
4020	pArgs->pPrevPhys2Virt = pCur;
4021	return 0;
4022	}
4023
4024
4025	/**
4026	* Perform an integrity check on the PGM component.
4027	*
4028	* @returns VINF_SUCCESS if everything is fine.
4029	* @returns VBox error status after asserting on integrity breach.
4030	* @param pVM The VM handle.
4031	*/
4032	PDMR3DECL(int) PGMR3CheckIntegrity(PVM pVM)
4033	{
4034	AssertReleaseReturn(pVM->pgm.s.offVM, VERR_INTERNAL_ERROR);
4035
4036	/*
4037	* Check the trees.
4038	*/
4039	int cErrors = 0;
4040	const static PGMCHECKINTARGS s_LeftToRight = { true, NULL, NULL, NULL, pVM };
4041	const static PGMCHECKINTARGS s_RightToLeft = { false, NULL, NULL, NULL, pVM };
4042	PGMCHECKINTARGS Args = s_LeftToRight;
4043	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesHC->PhysHandlers, true, pgmR3CheckIntegrityPhysHandlerNode, &Args);
4044	Args = s_RightToLeft;
4045	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesHC->PhysHandlers, false, pgmR3CheckIntegrityPhysHandlerNode, &Args);
4046	Args = s_LeftToRight;
4047	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesHC->VirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4048	Args = s_RightToLeft;
4049	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesHC->VirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4050	Args = s_LeftToRight;
4051	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesHC->HyperVirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4052	Args = s_RightToLeft;
4053	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesHC->HyperVirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4054	Args = s_LeftToRight;
4055	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesHC->PhysToVirtHandlers, true, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
4056	Args = s_RightToLeft;
4057	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesHC->PhysToVirtHandlers, false, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
4058
4059	return !cErrors ? VINF_SUCCESS : VERR_INTERNAL_ERROR;
4060	}
4061
4062
4063	/**
4064	* Inform PGM if we want all mappings to be put into the shadow page table. (necessary for e.g. VMX)
4065	*
4066	* @returns VBox status code.
4067	* @param pVM VM handle.
4068	* @param fEnable Enable or disable shadow mappings
4069	*/
4070	PGMR3DECL(int) PGMR3ChangeShwPDMappings(PVM pVM, bool fEnable)
4071	{
4072	pVM->pgm.s.fDisableMappings = !fEnable;
4073
4074	uint32_t cb;
4075	int rc = PGMR3MappingsSize(pVM, &cb);
4076	AssertRCReturn(rc, rc);
4077
4078	/* Pretend the mappings are now fixed; to force a refresh of the reserved PDEs. */
4079	rc = PGMR3MappingsFix(pVM, MM_HYPER_AREA_ADDRESS, cb);
4080	AssertRCReturn(rc, rc);
4081
4082	return VINF_SUCCESS;
4083	}

Note: See TracBrowser for help on using the repository browser.

source: vbox/trunk/src/VBox/VMM/PGM.cpp@ 9021

Download in other formats: