PGM.cpp@ 8536

Last change on this file since 8536 was 8536, checked in by vboxsync, 17 years ago
Split out the long mode only bits from the PDPE. Set accessed bits for the PDPE & PML4E
Property svn:eol-style set to `native` Property svn:keywords set to `Id`
File size: 184.4 KB

Line
1	/* $Id: PGM.cpp 8536 2008-05-02 16:46:51Z vboxsync $ */
2	/** @file
3	* PGM - Page Manager and Monitor. (Mixing stuff here, not good?)
4	*/
5
6	/*
7	* Copyright (C) 2006-2007 Sun Microsystems, Inc.
8	*
9	* This file is part of VirtualBox Open Source Edition (OSE), as
10	* available from http://www.virtualbox.org. This file is free software;
11	* you can redistribute it and/or modify it under the terms of the GNU
12	* General Public License (GPL) as published by the Free Software
13	* Foundation, in version 2 as it comes in the "COPYING" file of the
14	* VirtualBox OSE distribution. VirtualBox OSE is distributed in the
15	* hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
16	*
17	* Please contact Sun Microsystems, Inc., 4150 Network Circle, Santa
18	* Clara, CA 95054 USA or visit http://www.sun.com if you need
19	* additional information or have any questions.
20	*/
21
22
23	/** @page pg_pgm PGM - The Page Manager and Monitor
24	*
25	*
26	*
27	* @section sec_pgm_modes Paging Modes
28	*
29	* There are three memory contexts: Host Context (HC), Guest Context (GC)
30	* and intermediate context. When talking about paging HC can also be refered to
31	* as "host paging", and GC refered to as "shadow paging".
32	*
33	* We define three basic paging modes: 32-bit, PAE and AMD64. The host paging mode
34	* is defined by the host operating system. The mode used in the shadow paging mode
35	* depends on the host paging mode and what the mode the guest is currently in. The
36	* following relation between the two is defined:
37	*
38	* @verbatim
39	Host > 32-bit \| PAE \| AMD64 \|
40	Guest \| \| \| \|
41	==v================================
42	32-bit 32-bit PAE PAE
43	-------\|--------\|--------\|--------\|
44	PAE PAE PAE PAE
45	-------\|--------\|--------\|--------\|
46	AMD64 AMD64 AMD64 AMD64
47	-------\|--------\|--------\|--------\| @endverbatim
48	*
49	* All configuration except those in the diagonal (upper left) are expected to
50	* require special effort from the switcher (i.e. a bit slower).
51	*
52	*
53	*
54	*
55	* @section sec_pgm_shw The Shadow Memory Context
56	*
57	*
58	* [..]
59	*
60	* Because of guest context mappings requires PDPT and PML4 entries to allow
61	* writing on AMD64, the two upper levels will have fixed flags whatever the
62	* guest is thinking of using there. So, when shadowing the PD level we will
63	* calculate the effective flags of PD and all the higher levels. In legacy
64	* PAE mode this only applies to the PWT and PCD bits (the rest are
65	* ignored/reserved/MBZ). We will ignore those bits for the present.
66	*
67	*
68	*
69	* @section sec_pgm_int The Intermediate Memory Context
70	*
71	* The world switch goes thru an intermediate memory context which purpose it is
72	* to provide different mappings of the switcher code. All guest mappings are also
73	* present in this context.
74	*
75	* The switcher code is mapped at the same location as on the host, at an
76	* identity mapped location (physical equals virtual address), and at the
77	* hypervisor location.
78	*
79	* PGM maintain page tables for 32-bit, PAE and AMD64 paging modes. This
80	* simplifies switching guest CPU mode and consistency at the cost of more
81	* code to do the work. All memory use for those page tables is located below
82	* 4GB (this includes page tables for guest context mappings).
83	*
84	*
85	* @subsection subsec_pgm_int_gc Guest Context Mappings
86	*
87	* During assignment and relocation of a guest context mapping the intermediate
88	* memory context is used to verify the new location.
89	*
90	* Guest context mappings are currently restricted to below 4GB, for reasons
91	* of simplicity. This may change when we implement AMD64 support.
92	*
93	*
94	*
95	*
96	* @section sec_pgm_misc Misc
97	*
98	* @subsection subsec_pgm_misc_diff Differences Between Legacy PAE and Long Mode PAE
99	*
100	* The differences between legacy PAE and long mode PAE are:
101	* -# PDPE bits 1, 2, 5 and 6 are defined differently. In leagcy mode they are
102	* all marked down as must-be-zero, while in long mode 1, 2 and 5 have the
103	* usual meanings while 6 is ignored (AMD). This means that upon switching to
104	* legacy PAE mode we'll have to clear these bits and when going to long mode
105	* they must be set. This applies to both intermediate and shadow contexts,
106	* however we don't need to do it for the intermediate one since we're
107	* executing with CR0.WP at that time.
108	* -# CR3 allows a 32-byte aligned address in legacy mode, while in long mode
109	* a page aligned one is required.
110	*
111	*
112	* @section sec_pgm_handlers Access Handlers
113	*
114	* Placeholder.
115	*
116	*
117	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
118	*
119	* Placeholder.
120	*
121	*
122	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
123	*
124	* We currently implement three types of virtual access handlers: ALL, WRITE
125	* and HYPERVISOR (WRITE). See PGMVIRTHANDLERTYPE for some more details.
126	*
127	* The HYPERVISOR access handlers is kept in a separate tree since it doesn't apply
128	* to physical pages (PGMTREES::HyperVirtHandlers) and only needs to be consulted in
129	* a special \#PF case. The ALL and WRITE are in the PGMTREES::VirtHandlers tree, the
130	* rest of this section is going to be about these handlers.
131	*
132	* We'll go thru the life cycle of a handler and try make sense of it all, don't know
133	* how successfull this is gonna be...
134	*
135	* 1. A handler is registered thru the PGMR3HandlerVirtualRegister and
136	* PGMHandlerVirtualRegisterEx APIs. We check for conflicting virtual handlers
137	* and create a new node that is inserted into the AVL tree (range key). Then
138	* a full PGM resync is flagged (clear pool, sync cr3, update virtual bit of PGMPAGE).
139	*
140	* 2. The following PGMSyncCR3/SyncCR3 operation will first make invoke HandlerVirtualUpdate.
141	*
142	* 2a. HandlerVirtualUpdate will will lookup all the pages covered by virtual handlers
143	* via the current guest CR3 and update the physical page -> virtual handler
144	* translation. Needless to say, this doesn't exactly scale very well. If any changes
145	* are detected, it will flag a virtual bit update just like we did on registration.
146	* PGMPHYS pages with changes will have their virtual handler state reset to NONE.
147	*
148	* 2b. The virtual bit update process will iterate all the pages covered by all the
149	* virtual handlers and update the PGMPAGE virtual handler state to the max of all
150	* virtual handlers on that page.
151	*
152	* 2c. Back in SyncCR3 we will now flush the entire shadow page cache to make sure
153	* we don't miss any alias mappings of the monitored pages.
154	*
155	* 2d. SyncCR3 will then proceed with syncing the CR3 table.
156	*
157	* 3. \#PF(np,read) on a page in the range. This will cause it to be synced
158	* read-only and resumed if it's a WRITE handler. If it's an ALL handler we
159	* will call the handlers like in the next step. If the physical mapping has
160	* changed we will - some time in the future - perform a handler callback
161	* (optional) and update the physical -> virtual handler cache.
162	*
163	* 4. \#PF(,write) on a page in the range. This will cause the handler to
164	* be invoked.
165	*
166	* 5. The guest invalidates the page and changes the physical backing or
167	* unmaps it. This should cause the invalidation callback to be invoked
168	* (it might not yet be 100% perfect). Exactly what happens next... is
169	* this where we mess up and end up out of sync for a while?
170	*
171	* 6. The handler is deregistered by the client via PGMHandlerVirtualDeregister.
172	* We will then set all PGMPAGEs in the physical -> virtual handler cache for
173	* this handler to NONE and trigger a full PGM resync (basically the same
174	* as int step 1). Which means 2 is executed again.
175	*
176	*
177	* @subsubsection sub_sec_pgm_handler_virt_todo TODOs
178	*
179	* There is a bunch of things that needs to be done to make the virtual handlers
180	* work 100% correctly and work more efficiently.
181	*
182	* The first bit hasn't been implemented yet because it's going to slow the
183	* whole mess down even more, and besides it seems to be working reliably for
184	* our current uses. OTOH, some of the optimizations might end up more or less
185	* implementing the missing bits, so we'll see.
186	*
187	* On the optimization side, the first thing to do is to try avoid unnecessary
188	* cache flushing. Then try team up with the shadowing code to track changes
189	* in mappings by means of access to them (shadow in), updates to shadows pages,
190	* invlpg, and shadow PT discarding (perhaps).
191	*
192	* Some idea that have popped up for optimization for current and new features:
193	* - bitmap indicating where there are virtual handlers installed.
194	* (4KB => 220 pages, page 212 => covers 32-bit address space 1:1!)
195	* - Further optimize this by min/max (needs min/max avl getters).
196	* - Shadow page table entry bit (if any left)?
197	*
198	*/
199
200
201	/** @page pg_pgmPhys PGMPhys - Physical Guest Memory Management.
202	*
203	*
204	* Objectives:
205	* - Guest RAM over-commitment using memory ballooning,
206	* zero pages and general page sharing.
207	* - Moving or mirroring a VM onto a different physical machine.
208	*
209	*
210	* @subsection subsec_pgmPhys_Definitions Definitions
211	*
212	* Allocation chunk - A RTR0MemObjAllocPhysNC object and the tracking
213	* machinery assoicated with it.
214	*
215	*
216	*
217	*
218	* @subsection subsec_pgmPhys_AllocPage Allocating a page.
219	*
220	* Initially we map all guest memory to the (per VM) zero page, which
221	* means that none of the read functions will cause pages to be allocated.
222	*
223	* Exception, access bit in page tables that have been shared. This must
224	* be handled, but we must also make sure PGMGst*Modify doesn't make
225	* unnecessary modifications.
226	*
227	* Allocation points:
228	* - PGMPhysWriteGCPhys and PGMPhysWrite.
229	* - Replacing a zero page mapping at \#PF.
230	* - Replacing a shared page mapping at \#PF.
231	* - ROM registration (currently MMR3RomRegister).
232	* - VM restore (pgmR3Load).
233	*
234	* For the first three it would make sense to keep a few pages handy
235	* until we've reached the max memory commitment for the VM.
236	*
237	* For the ROM registration, we know exactly how many pages we need
238	* and will request these from ring-0. For restore, we will save
239	* the number of non-zero pages in the saved state and allocate
240	* them up front. This would allow the ring-0 component to refuse
241	* the request if the isn't sufficient memory available for VM use.
242	*
243	* Btw. for both ROM and restore allocations we won't be requiring
244	* zeroed pages as they are going to be filled instantly.
245	*
246	*
247	* @subsection subsec_pgmPhys_FreePage Freeing a page
248	*
249	* There are a few points where a page can be freed:
250	* - After being replaced by the zero page.
251	* - After being replaced by a shared page.
252	* - After being ballooned by the guest additions.
253	* - At reset.
254	* - At restore.
255	*
256	* When freeing one or more pages they will be returned to the ring-0
257	* component and replaced by the zero page.
258	*
259	* The reasoning for clearing out all the pages on reset is that it will
260	* return us to the exact same state as on power on, and may thereby help
261	* us reduce the memory load on the system. Further it might have a
262	* (temporary) positive influence on memory fragmentation (@see subsec_pgmPhys_Fragmentation).
263	*
264	* On restore, as mention under the allocation topic, pages should be
265	* freed / allocated depending on how many is actually required by the
266	* new VM state. The simplest approach is to do like on reset, and free
267	* all non-ROM pages and then allocate what we need.
268	*
269	* A measure to prevent some fragmentation, would be to let each allocation
270	* chunk have some affinity towards the VM having allocated the most pages
271	* from it. Also, try make sure to allocate from allocation chunks that
272	* are almost full. Admittedly, both these measures might work counter to
273	* our intentions and its probably not worth putting a lot of effort,
274	* cpu time or memory into this.
275	*
276	*
277	* @subsection subsec_pgmPhys_SharePage Sharing a page
278	*
279	* The basic idea is that there there will be a idle priority kernel
280	* thread walking the non-shared VM pages hashing them and looking for
281	* pages with the same checksum. If such pages are found, it will compare
282	* them byte-by-byte to see if they actually are identical. If found to be
283	* identical it will allocate a shared page, copy the content, check that
284	* the page didn't change while doing this, and finally request both the
285	* VMs to use the shared page instead. If the page is all zeros (special
286	* checksum and byte-by-byte check) it will request the VM that owns it
287	* to replace it with the zero page.
288	*
289	* To make this efficient, we will have to make sure not to try share a page
290	* that will change its contents soon. This part requires the most work.
291	* A simple idea would be to request the VM to write monitor the page for
292	* a while to make sure it isn't modified any time soon. Also, it may
293	* make sense to skip pages that are being write monitored since this
294	* information is readily available to the thread if it works on the
295	* per-VM guest memory structures (presently called PGMRAMRANGE).
296	*
297	*
298	* @subsection subsec_pgmPhys_Fragmentation Fragmentation Concerns and Counter Measures
299	*
300	* The pages are organized in allocation chunks in ring-0, this is a necessity
301	* if we wish to have an OS agnostic approach to this whole thing. (On Linux we
302	* could easily work on a page-by-page basis if we liked. Whether this is possible
303	* or efficient on NT I don't quite know.) Fragmentation within these chunks may
304	* become a problem as part of the idea here is that we wish to return memory to
305	* the host system.
306	*
307	* For instance, starting two VMs at the same time, they will both allocate the
308	* guest memory on-demand and if permitted their page allocations will be
309	* intermixed. Shut down one of the two VMs and it will be difficult to return
310	* any memory to the host system because the page allocation for the two VMs are
311	* mixed up in the same allocation chunks.
312	*
313	* To further complicate matters, when pages are freed because they have been
314	* ballooned or become shared/zero the whole idea is that the page is supposed
315	* to be reused by another VM or returned to the host system. This will cause
316	* allocation chunks to contain pages belonging to different VMs and prevent
317	* returning memory to the host when one of those VM shuts down.
318	*
319	* The only way to really deal with this problem is to move pages. This can
320	* either be done at VM shutdown and or by the idle priority worker thread
321	* that will be responsible for finding sharable/zero pages. The mechanisms
322	* involved for coercing a VM to move a page (or to do it for it) will be
323	* the same as when telling it to share/zero a page.
324	*
325	*
326	* @subsection subsec_pgmPhys_Tracking Tracking Structures And Their Cost
327	*
328	* There's a difficult balance between keeping the per-page tracking structures
329	* (global and guest page) easy to use and keeping them from eating too much
330	* memory. We have limited virtual memory resources available when operating in
331	* 32-bit kernel space (on 64-bit there'll it's quite a different story). The
332	* tracking structures will be attemted designed such that we can deal with up
333	* to 32GB of memory on a 32-bit system and essentially unlimited on 64-bit ones.
334	*
335	*
336	* @subsubsection subsubsec_pgmPhys_Tracking_Kernel Kernel Space
337	*
338	* @see pg_GMM
339	*
340	* @subsubsection subsubsec_pgmPhys_Tracking_PerVM Per-VM
341	*
342	* Fixed info is the physical address of the page (HCPhys) and the page id
343	* (described above). Theoretically we'll need 48(-12) bits for the HCPhys part.
344	* Today we've restricting ourselves to 40(-12) bits because this is the current
345	* restrictions of all AMD64 implementations (I think Barcelona will up this
346	* to 48(-12) bits, not that it really matters) and I needed the bits for
347	* tracking mappings of a page. 48-12 = 36. That leaves 28 bits, which means a
348	* decent range for the page id: 2^(28+12) = 1024TB.
349	*
350	* In additions to these, we'll have to keep maintaining the page flags as we
351	* currently do. Although it wouldn't harm to optimize these quite a bit, like
352	* for instance the ROM shouldn't depend on having a write handler installed
353	* in order for it to become read-only. A RO/RW bit should be considered so
354	* that the page syncing code doesn't have to mess about checking multiple
355	* flag combinations (ROM \|\| RW handler \|\| write monitored) in order to
356	* figure out how to setup a shadow PTE. But this of course, is second
357	* priority at present. Current this requires 12 bits, but could probably
358	* be optimized to ~8.
359	*
360	* Then there's the 24 bits used to track which shadow page tables are
361	* currently mapping a page for the purpose of speeding up physical
362	* access handlers, and thereby the page pool cache. More bit for this
363	* purpose wouldn't hurt IIRC.
364	*
365	* Then there is a new bit in which we need to record what kind of page
366	* this is, shared, zero, normal or write-monitored-normal. This'll
367	* require 2 bits. One bit might be needed for indicating whether a
368	* write monitored page has been written to. And yet another one or
369	* two for tracking migration status. 3-4 bits total then.
370	*
371	* Whatever is left will can be used to record the sharabilitiy of a
372	* page. The page checksum will not be stored in the per-VM table as
373	* the idle thread will not be permitted to do modifications to it.
374	* It will instead have to keep its own working set of potentially
375	* shareable pages and their check sums and stuff.
376	*
377	* For the present we'll keep the current packing of the
378	* PGMRAMRANGE::aHCPhys to keep the changes simple, only of course,
379	* we'll have to change it to a struct with a total of 128-bits at
380	* our disposal.
381	*
382	* The initial layout will be like this:
383	* @verbatim
384	RTHCPHYS HCPhys; The current stuff.
385	63:40 Current shadow PT tracking stuff.
386	39:12 The physical page frame number.
387	11:0 The current flags.
388	uint32_t u28PageId : 28; The page id.
389	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
390	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
391	uint32_t u1Reserved : 1; Reserved for later.
392	uint32_t u32Reserved; Reserved for later, mostly sharing stats.
393	@endverbatim
394	*
395	* The final layout will be something like this:
396	* @verbatim
397	RTHCPHYS HCPhys; The current stuff.
398	63:48 High page id (12+).
399	47:12 The physical page frame number.
400	11:0 Low page id.
401	uint32_t fReadOnly : 1; Whether it's readonly page (rom or monitored in some way).
402	uint32_t u3Type : 3; The page type {RESERVED, MMIO, MMIO2, ROM, shadowed ROM, RAM}.
403	uint32_t u2PhysMon : 2; Physical access handler type {none, read, write, all}.
404	uint32_t u2VirtMon : 2; Virtual access handler type {none, read, write, all}..
405	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
406	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
407	uint32_t u20Reserved : 20; Reserved for later, mostly sharing stats.
408	uint32_t u32Tracking; The shadow PT tracking stuff, roughly.
409	@endverbatim
410	*
411	* Cost wise, this means we'll double the cost for guest memory. There isn't anyway
412	* around that I'm afraid. It means that the cost of dealing out 32GB of memory
413	* to one or more VMs is: (32GB >> PAGE_SHIFT) * 16 bytes, or 128MBs. Or another
414	* example, the VM heap cost when assigning 1GB to a VM will be: 4MB.
415	*
416	* A couple of cost examples for the total cost per-VM + kernel.
417	* 32-bit Windows and 32-bit linux:
418	* 1GB guest ram, 256K pages: 4MB + 2MB(+) = 6MB
419	* 4GB guest ram, 1M pages: 16MB + 8MB(+) = 24MB
420	* 32GB guest ram, 8M pages: 128MB + 64MB(+) = 192MB
421	* 64-bit Windows and 64-bit linux:
422	* 1GB guest ram, 256K pages: 4MB + 3MB(+) = 7MB
423	* 4GB guest ram, 1M pages: 16MB + 12MB(+) = 28MB
424	* 32GB guest ram, 8M pages: 128MB + 96MB(+) = 224MB
425	*
426	* UPDATE - 2007-09-27:
427	* Will need a ballooned flag/state too because we cannot
428	* trust the guest 100% and reporting the same page as ballooned more
429	* than once will put the GMM off balance.
430	*
431	*
432	* @subsection subsec_pgmPhys_Serializing Serializing Access
433	*
434	* Initially, we'll try a simple scheme:
435	*
436	* - The per-VM RAM tracking structures (PGMRAMRANGE) is only modified
437	* by the EMT thread of that VM while in the pgm critsect.
438	* - Other threads in the VM process that needs to make reliable use of
439	* the per-VM RAM tracking structures will enter the critsect.
440	* - No process external thread or kernel thread will ever try enter
441	* the pgm critical section, as that just won't work.
442	* - The idle thread (and similar threads) doesn't not need 100% reliable
443	* data when performing it tasks as the EMT thread will be the one to
444	* do the actual changes later anyway. So, as long as it only accesses
445	* the main ram range, it can do so by somehow preventing the VM from
446	* being destroyed while it works on it...
447	*
448	* - The over-commitment management, including the allocating/freeing
449	* chunks, is serialized by a ring-0 mutex lock (a fast one since the
450	* more mundane mutex implementation is broken on Linux).
451	* - A separeate mutex is protecting the set of allocation chunks so
452	* that pages can be shared or/and freed up while some other VM is
453	* allocating more chunks. This mutex can be take from under the other
454	* one, but not the otherway around.
455	*
456	*
457	* @subsection subsec_pgmPhys_Request VM Request interface
458	*
459	* When in ring-0 it will become necessary to send requests to a VM so it can
460	* for instance move a page while defragmenting during VM destroy. The idle
461	* thread will make use of this interface to request VMs to setup shared
462	* pages and to perform write monitoring of pages.
463	*
464	* I would propose an interface similar to the current VMReq interface, similar
465	* in that it doesn't require locking and that the one sending the request may
466	* wait for completion if it wishes to. This shouldn't be very difficult to
467	* realize.
468	*
469	* The requests themselves are also pretty simple. They are basically:
470	* -# Check that some precondition is still true.
471	* -# Do the update.
472	* -# Update all shadow page tables involved with the page.
473	*
474	* The 3rd step is identical to what we're already doing when updating a
475	* physical handler, see pgmHandlerPhysicalSetRamFlagsAndFlushShadowPTs.
476	*
477	*
478	*
479	* @section sec_pgmPhys_MappingCaches Mapping Caches
480	*
481	* In order to be able to map in and out memory and to be able to support
482	* guest with more RAM than we've got virtual address space, we'll employing
483	* a mapping cache. There is already a tiny one for GC (see PGMGCDynMapGCPageEx)
484	* and we'll create a similar one for ring-0 unless we decide to setup a dedicate
485	* memory context for the HWACCM execution.
486	*
487	*
488	* @subsection subsec_pgmPhys_MappingCaches_R3 Ring-3
489	*
490	* We've considered implementing the ring-3 mapping cache page based but found
491	* that this was bother some when one had to take into account TLBs+SMP and
492	* portability (missing the necessary APIs on several platforms). There were
493	* also some performance concerns with this approach which hadn't quite been
494	* worked out.
495	*
496	* Instead, we'll be mapping allocation chunks into the VM process. This simplifies
497	* matters greatly quite a bit since we don't need to invent any new ring-0 stuff,
498	* only some minor RTR0MEMOBJ mapping stuff. The main concern here is that mapping
499	* compared to the previous idea is that mapping or unmapping a 1MB chunk is more
500	* costly than a single page, although how much more costly is uncertain. We'll
501	* try address this by using a very big cache, preferably bigger than the actual
502	* VM RAM size if possible. The current VM RAM sizes should give some idea for
503	* 32-bit boxes, while on 64-bit we can probably get away with employing an
504	* unlimited cache.
505	*
506	* The cache have to parts, as already indicated, the ring-3 side and the
507	* ring-0 side.
508	*
509	* The ring-0 will be tied to the page allocator since it will operate on the
510	* memory objects it contains. It will therefore require the first ring-0 mutex
511	* discussed in @ref subsec_pgmPhys_Serializing. We
512	* some double house keeping wrt to who has mapped what I think, since both
513	* VMMR0.r0 and RTR0MemObj will keep track of mapping relataions
514	*
515	* The ring-3 part will be protected by the pgm critsect. For simplicity, we'll
516	* require anyone that desires to do changes to the mapping cache to do that
517	* from within this critsect. Alternatively, we could employ a separate critsect
518	* for serializing changes to the mapping cache as this would reduce potential
519	* contention with other threads accessing mappings unrelated to the changes
520	* that are in process. We can see about this later, contention will show
521	* up in the statistics anyway, so it'll be simple to tell.
522	*
523	* The organization of the ring-3 part will be very much like how the allocation
524	* chunks are organized in ring-0, that is in an AVL tree by chunk id. To avoid
525	* having to walk the tree all the time, we'll have a couple of lookaside entries
526	* like in we do for I/O ports and MMIO in IOM.
527	*
528	* The simplified flow of a PGMPhysRead/Write function:
529	* -# Enter the PGM critsect.
530	* -# Lookup GCPhys in the ram ranges and get the Page ID.
531	* -# Calc the Allocation Chunk ID from the Page ID.
532	* -# Check the lookaside entries and then the AVL tree for the Chunk ID.
533	* If not found in cache:
534	* -# Call ring-0 and request it to be mapped and supply
535	* a chunk to be unmapped if the cache is maxed out already.
536	* -# Insert the new mapping into the AVL tree (id + R3 address).
537	* -# Update the relevant lookaside entry and return the mapping address.
538	* -# Do the read/write according to monitoring flags and everything.
539	* -# Leave the critsect.
540	*
541	*
542	* @section sec_pgmPhys_Fallback Fallback
543	*
544	* Current all the "second tier" hosts will not support the RTR0MemObjAllocPhysNC
545	* API and thus require a fallback.
546	*
547	* So, when RTR0MemObjAllocPhysNC returns VERR_NOT_SUPPORTED the page allocator
548	* will return to the ring-3 caller (and later ring-0) and asking it to seed
549	* the page allocator with some fresh pages (VERR_GMM_SEED_ME). Ring-3 will
550	* then perform an SUPPageAlloc(cbChunk >> PAGE_SHIFT) call and make a
551	* "SeededAllocPages" call to ring-0.
552	*
553	* The first time ring-0 sees the VERR_NOT_SUPPORTED failure it will disable
554	* all page sharing (zero page detection will continue). It will also force
555	* all allocations to come from the VM which seeded the page. Both these
556	* measures are taken to make sure that there will never be any need for
557	* mapping anything into ring-3 - everything will be mapped already.
558	*
559	* Whether we'll continue to use the current MM locked memory management
560	* for this I don't quite know (I'd prefer not to and just ditch that all
561	* togther), we'll see what's simplest to do.
562	*
563	*
564	*
565	* @section sec_pgmPhys_Changes Changes
566	*
567	* Breakdown of the changes involved?
568	*/
569
570
571	/** Saved state data unit version. */
572	#define PGM_SAVED_STATE_VERSION 6
573
574	/*******************************************************************************
575	* Header Files *
576	*******************************************************************************/
577	#define LOG_GROUP LOG_GROUP_PGM
578	#include <VBox/dbgf.h>
579	#include <VBox/pgm.h>
580	#include <VBox/cpum.h>
581	#include <VBox/iom.h>
582	#include <VBox/sup.h>
583	#include <VBox/mm.h>
584	#include <VBox/em.h>
585	#include <VBox/stam.h>
586	#include <VBox/rem.h>
587	#include <VBox/dbgf.h>
588	#include <VBox/rem.h>
589	#include <VBox/selm.h>
590	#include <VBox/ssm.h>
591	#include "PGMInternal.h"
592	#include <VBox/vm.h>
593	#include <VBox/dbg.h>
594	#include <VBox/hwaccm.h>
595
596	#include <iprt/assert.h>
597	#include <iprt/alloc.h>
598	#include <iprt/asm.h>
599	#include <iprt/thread.h>
600	#include <iprt/string.h>
601	#include <VBox/param.h>
602	#include <VBox/err.h>
603
604
605
606	/*******************************************************************************
607	* Internal Functions *
608	*******************************************************************************/
609	static int pgmR3InitPaging(PVM pVM);
610	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
611	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
612	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
613	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser);
614	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
615	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
616	#ifdef VBOX_STRICT
617	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser);
618	#endif
619	static DECLCALLBACK(int) pgmR3Save(PVM pVM, PSSMHANDLE pSSM);
620	static DECLCALLBACK(int) pgmR3Load(PVM pVM, PSSMHANDLE pSSM, uint32_t u32Version);
621	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0);
622	static void pgmR3ModeDataSwitch(PVM pVM, PGMMODE enmShw, PGMMODE enmGst);
623	static PGMMODE pgmR3CalcShadowMode(PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher);
624
625	#ifdef VBOX_WITH_STATISTICS
626	static void pgmR3InitStats(PVM pVM);
627	#endif
628
629	#ifdef VBOX_WITH_DEBUGGER
630	/** @todo all but the two last commands must be converted to 'info'. */
631	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
632	static DECLCALLBACK(int) pgmR3CmdMap(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
633	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
634	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
635	#endif
636
637
638	/*******************************************************************************
639	* Global Variables *
640	*******************************************************************************/
641	#ifdef VBOX_WITH_DEBUGGER
642	/** Command descriptors. */
643	static const DBGCCMD g_aCmds[] =
644	{
645	/* pszCmd, cArgsMin, cArgsMax, paArgDesc, cArgDescs, pResultDesc, fFlags, pfnHandler pszSyntax, ....pszDescription */
646	{ "pgmram", 0, 0, NULL, 0, NULL, 0, pgmR3CmdRam, "", "Display the ram ranges." },
647	{ "pgmmap", 0, 0, NULL, 0, NULL, 0, pgmR3CmdMap, "", "Display the mapping ranges." },
648	{ "pgmsync", 0, 0, NULL, 0, NULL, 0, pgmR3CmdSync, "", "Sync the CR3 page." },
649	{ "pgmsyncalways", 0, 0, NULL, 0, NULL, 0, pgmR3CmdSyncAlways, "", "Toggle permanent CR3 syncing." },
650	};
651	#endif
652
653
654
655
656	#if 1/// @todo ndef RT_ARCH_AMD64
657	/*
658	* Shadow - 32-bit mode
659	*/
660	#define PGM_SHW_TYPE PGM_TYPE_32BIT
661	#define PGM_SHW_NAME(name) PGM_SHW_NAME_32BIT(name)
662	#define PGM_SHW_NAME_GC_STR(name) PGM_SHW_NAME_GC_32BIT_STR(name)
663	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_32BIT_STR(name)
664	#include "PGMShw.h"
665
666	/* Guest - real mode */
667	#define PGM_GST_TYPE PGM_TYPE_REAL
668	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
669	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_REAL_STR(name)
670	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
671	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_REAL(name)
672	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_32BIT_REAL_STR(name)
673	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_REAL_STR(name)
674	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
675	#include "PGMGst.h"
676	#include "PGMBth.h"
677	#undef BTH_PGMPOOLKIND_PT_FOR_PT
678	#undef PGM_BTH_NAME
679	#undef PGM_BTH_NAME_GC_STR
680	#undef PGM_BTH_NAME_R0_STR
681	#undef PGM_GST_TYPE
682	#undef PGM_GST_NAME
683	#undef PGM_GST_NAME_GC_STR
684	#undef PGM_GST_NAME_R0_STR
685
686	/* Guest - protected mode */
687	#define PGM_GST_TYPE PGM_TYPE_PROT
688	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
689	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_PROT_STR(name)
690	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
691	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_PROT(name)
692	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_32BIT_PROT_STR(name)
693	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_PROT_STR(name)
694	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
695	#include "PGMGst.h"
696	#include "PGMBth.h"
697	#undef BTH_PGMPOOLKIND_PT_FOR_PT
698	#undef PGM_BTH_NAME
699	#undef PGM_BTH_NAME_GC_STR
700	#undef PGM_BTH_NAME_R0_STR
701	#undef PGM_GST_TYPE
702	#undef PGM_GST_NAME
703	#undef PGM_GST_NAME_GC_STR
704	#undef PGM_GST_NAME_R0_STR
705
706	/* Guest - 32-bit mode */
707	#define PGM_GST_TYPE PGM_TYPE_32BIT
708	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
709	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_32BIT_STR(name)
710	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
711	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_32BIT(name)
712	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_32BIT_32BIT_STR(name)
713	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_32BIT_STR(name)
714	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_32BIT_PT
715	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_32BIT_PT_FOR_32BIT_4MB
716	#include "PGMGst.h"
717	#include "PGMBth.h"
718	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
719	#undef BTH_PGMPOOLKIND_PT_FOR_PT
720	#undef PGM_BTH_NAME
721	#undef PGM_BTH_NAME_GC_STR
722	#undef PGM_BTH_NAME_R0_STR
723	#undef PGM_GST_TYPE
724	#undef PGM_GST_NAME
725	#undef PGM_GST_NAME_GC_STR
726	#undef PGM_GST_NAME_R0_STR
727
728	#undef PGM_SHW_TYPE
729	#undef PGM_SHW_NAME
730	#undef PGM_SHW_NAME_GC_STR
731	#undef PGM_SHW_NAME_R0_STR
732	#endif /* !RT_ARCH_AMD64 */
733
734
735	/*
736	* Shadow - PAE mode
737	*/
738	#define PGM_SHW_TYPE PGM_TYPE_PAE
739	#define PGM_SHW_NAME(name) PGM_SHW_NAME_PAE(name)
740	#define PGM_SHW_NAME_GC_STR(name) PGM_SHW_NAME_GC_PAE_STR(name)
741	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_PAE_STR(name)
742	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
743	#include "PGMShw.h"
744
745	/* Guest - real mode */
746	#define PGM_GST_TYPE PGM_TYPE_REAL
747	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
748	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_REAL_STR(name)
749	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
750	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
751	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_PAE_REAL_STR(name)
752	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_REAL_STR(name)
753	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
754	#include "PGMBth.h"
755	#undef BTH_PGMPOOLKIND_PT_FOR_PT
756	#undef PGM_BTH_NAME
757	#undef PGM_BTH_NAME_GC_STR
758	#undef PGM_BTH_NAME_R0_STR
759	#undef PGM_GST_TYPE
760	#undef PGM_GST_NAME
761	#undef PGM_GST_NAME_GC_STR
762	#undef PGM_GST_NAME_R0_STR
763
764	/* Guest - protected mode */
765	#define PGM_GST_TYPE PGM_TYPE_PROT
766	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
767	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_PROT_STR(name)
768	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
769	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PROT(name)
770	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_PAE_PROT_STR(name)
771	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PROT_STR(name)
772	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
773	#include "PGMBth.h"
774	#undef BTH_PGMPOOLKIND_PT_FOR_PT
775	#undef PGM_BTH_NAME
776	#undef PGM_BTH_NAME_GC_STR
777	#undef PGM_BTH_NAME_R0_STR
778	#undef PGM_GST_TYPE
779	#undef PGM_GST_NAME
780	#undef PGM_GST_NAME_GC_STR
781	#undef PGM_GST_NAME_R0_STR
782
783	/* Guest - 32-bit mode */
784	#define PGM_GST_TYPE PGM_TYPE_32BIT
785	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
786	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_32BIT_STR(name)
787	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
788	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_32BIT(name)
789	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_PAE_32BIT_STR(name)
790	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_32BIT_STR(name)
791	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
792	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
793	#include "PGMBth.h"
794	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
795	#undef BTH_PGMPOOLKIND_PT_FOR_PT
796	#undef PGM_BTH_NAME
797	#undef PGM_BTH_NAME_GC_STR
798	#undef PGM_BTH_NAME_R0_STR
799	#undef PGM_GST_TYPE
800	#undef PGM_GST_NAME
801	#undef PGM_GST_NAME_GC_STR
802	#undef PGM_GST_NAME_R0_STR
803
804	/* Guest - PAE mode */
805	#define PGM_GST_TYPE PGM_TYPE_PAE
806	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
807	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_PAE_STR(name)
808	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
809	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PAE(name)
810	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_PAE_PAE_STR(name)
811	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PAE_STR(name)
812	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
813	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
814	#include "PGMGst.h"
815	#include "PGMBth.h"
816	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
817	#undef BTH_PGMPOOLKIND_PT_FOR_PT
818	#undef PGM_BTH_NAME
819	#undef PGM_BTH_NAME_GC_STR
820	#undef PGM_BTH_NAME_R0_STR
821	#undef PGM_GST_TYPE
822	#undef PGM_GST_NAME
823	#undef PGM_GST_NAME_GC_STR
824	#undef PGM_GST_NAME_R0_STR
825
826	#undef PGM_SHW_TYPE
827	#undef PGM_SHW_NAME
828	#undef PGM_SHW_NAME_GC_STR
829	#undef PGM_SHW_NAME_R0_STR
830
831
832	/*
833	* Shadow - AMD64 mode
834	*/
835	#define PGM_SHW_TYPE PGM_TYPE_AMD64
836	#define PGM_SHW_NAME(name) PGM_SHW_NAME_AMD64(name)
837	#define PGM_SHW_NAME_GC_STR(name) PGM_SHW_NAME_GC_AMD64_STR(name)
838	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_AMD64_STR(name)
839	#include "PGMShw.h"
840
841	/* Guest - AMD64 mode */
842	#define PGM_GST_TYPE PGM_TYPE_AMD64
843	#define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
844	#define PGM_GST_NAME_GC_STR(name) PGM_GST_NAME_GC_AMD64_STR(name)
845	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
846	#define PGM_BTH_NAME(name) PGM_BTH_NAME_AMD64_AMD64(name)
847	#define PGM_BTH_NAME_GC_STR(name) PGM_BTH_NAME_GC_AMD64_AMD64_STR(name)
848	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_AMD64_AMD64_STR(name)
849	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
850	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
851	#include "PGMGst.h"
852	#include "PGMBth.h"
853	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
854	#undef BTH_PGMPOOLKIND_PT_FOR_PT
855	#undef PGM_BTH_NAME
856	#undef PGM_BTH_NAME_GC_STR
857	#undef PGM_BTH_NAME_R0_STR
858	#undef PGM_GST_TYPE
859	#undef PGM_GST_NAME
860	#undef PGM_GST_NAME_GC_STR
861	#undef PGM_GST_NAME_R0_STR
862
863	#undef PGM_SHW_TYPE
864	#undef PGM_SHW_NAME
865	#undef PGM_SHW_NAME_GC_STR
866	#undef PGM_SHW_NAME_R0_STR
867
868
869	/**
870	* Initiates the paging of VM.
871	*
872	* @returns VBox status code.
873	* @param pVM Pointer to VM structure.
874	*/
875	PGMR3DECL(int) PGMR3Init(PVM pVM)
876	{
877	LogFlow(("PGMR3Init:\n"));
878
879	/*
880	* Assert alignment and sizes.
881	*/
882	AssertRelease(sizeof(pVM->pgm.s) <= sizeof(pVM->pgm.padding));
883
884	/*
885	* Init the structure.
886	*/
887	pVM->pgm.s.offVM = RT_OFFSETOF(VM, pgm.s);
888	pVM->pgm.s.enmShadowMode = PGMMODE_INVALID;
889	pVM->pgm.s.enmGuestMode = PGMMODE_INVALID;
890	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
891	pVM->pgm.s.GCPhysCR3 = NIL_RTGCPHYS;
892	pVM->pgm.s.GCPhysGstCR3Monitored = NIL_RTGCPHYS;
893	pVM->pgm.s.fA20Enabled = true;
894	pVM->pgm.s.pGstPaePDPTHC = NULL;
895	pVM->pgm.s.pGstPaePDPTGC = 0;
896	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.apGstPaePDsHC); i++)
897	{
898	pVM->pgm.s.apGstPaePDsHC[i] = NULL;
899	pVM->pgm.s.apGstPaePDsGC[i] = 0;
900	pVM->pgm.s.aGCPhysGstPaePDs[i] = NIL_RTGCPHYS;
901	pVM->pgm.s.aGCPhysGstPaePDsMonitored[i] = NIL_RTGCPHYS;
902	}
903
904	#ifdef VBOX_STRICT
905	VMR3AtStateRegister(pVM, pgmR3ResetNoMorePhysWritesFlag, NULL);
906	#endif
907
908	/*
909	* Get the configured RAM size - to estimate saved state size.
910	*/
911	uint64_t cbRam;
912	int rc = CFGMR3QueryU64(CFGMR3GetRoot(pVM), "RamSize", &cbRam);
913	if (rc == VERR_CFGM_VALUE_NOT_FOUND)
914	cbRam = pVM->pgm.s.cbRamSize = 0;
915	else if (VBOX_SUCCESS(rc))
916	{
917	if (cbRam < PAGE_SIZE)
918	cbRam = 0;
919	cbRam = RT_ALIGN_64(cbRam, PAGE_SIZE);
920	pVM->pgm.s.cbRamSize = (RTUINT)cbRam;
921	}
922	else
923	{
924	AssertMsgFailed(("Configuration error: Failed to query integer \"RamSize\", rc=%Vrc.\n", rc));
925	return rc;
926	}
927
928	/*
929	* Register saved state data unit.
930	*/
931	rc = SSMR3RegisterInternal(pVM, "pgm", 1, PGM_SAVED_STATE_VERSION, (size_t)cbRam + sizeof(PGM),
932	NULL, pgmR3Save, NULL,
933	NULL, pgmR3Load, NULL);
934	if (VBOX_FAILURE(rc))
935	return rc;
936
937	/*
938	* Initialize the PGM critical section and flush the phys TLBs
939	*/
940	rc = PDMR3CritSectInit(pVM, &pVM->pgm.s.CritSect, "PGM");
941	AssertRCReturn(rc, rc);
942
943	PGMR3PhysChunkInvalidateTLB(pVM);
944	PGMPhysInvalidatePageR3MapTLB(pVM);
945	PGMPhysInvalidatePageR0MapTLB(pVM);
946	PGMPhysInvalidatePageGCMapTLB(pVM);
947
948	/*
949	* Trees
950	*/
951	rc = MMHyperAlloc(pVM, sizeof(PGMTREES), 0, MM_TAG_PGM, (void **)&pVM->pgm.s.pTreesHC);
952	if (VBOX_SUCCESS(rc))
953	{
954	pVM->pgm.s.pTreesGC = MMHyperHC2GC(pVM, pVM->pgm.s.pTreesHC);
955
956	/*
957	* Alocate the zero page.
958	*/
959	rc = MMHyperAlloc(pVM, PAGE_SIZE, PAGE_SIZE, MM_TAG_PGM, &pVM->pgm.s.pvZeroPgR3);
960	}
961	if (VBOX_SUCCESS(rc))
962	{
963	pVM->pgm.s.pvZeroPgGC = MMHyperR3ToGC(pVM, pVM->pgm.s.pvZeroPgR3);
964	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
965	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTHCPHYS);
966	pVM->pgm.s.HCPhysZeroPg = MMR3HyperHCVirt2HCPhys(pVM, pVM->pgm.s.pvZeroPgR3);
967	AssertRelease(pVM->pgm.s.HCPhysZeroPg != NIL_RTHCPHYS);
968
969	/*
970	* Init the paging.
971	*/
972	rc = pgmR3InitPaging(pVM);
973	}
974	if (VBOX_SUCCESS(rc))
975	{
976	/*
977	* Init the page pool.
978	*/
979	rc = pgmR3PoolInit(pVM);
980	}
981	if (VBOX_SUCCESS(rc))
982	{
983	/*
984	* Info & statistics
985	*/
986	DBGFR3InfoRegisterInternal(pVM, "mode",
987	"Shows the current paging mode. "
988	"Recognizes 'all', 'guest', 'shadow' and 'host' as arguments, defaulting to 'all' if nothing's given.",
989	pgmR3InfoMode);
990	DBGFR3InfoRegisterInternal(pVM, "pgmcr3",
991	"Dumps all the entries in the top level paging table. No arguments.",
992	pgmR3InfoCr3);
993	DBGFR3InfoRegisterInternal(pVM, "phys",
994	"Dumps all the physical address ranges. No arguments.",
995	pgmR3PhysInfo);
996	DBGFR3InfoRegisterInternal(pVM, "handlers",
997	"Dumps physical, virtual and hyper virtual handlers. "
998	"Pass 'phys', 'virt', 'hyper' as argument if only one kind is wanted."
999	"Add 'nost' if the statistics are unwanted, use together with 'all' or explicit selection.",
1000	pgmR3InfoHandlers);
1001	DBGFR3InfoRegisterInternal(pVM, "mappings",
1002	"Dumps guest mappings.",
1003	pgmR3MapInfo);
1004
1005	STAM_REL_REG(pVM, &pVM->pgm.s.cGuestModeChanges, STAMTYPE_COUNTER, "/PGM/cGuestModeChanges", STAMUNIT_OCCURENCES, "Number of guest mode changes.");
1006	#ifdef VBOX_WITH_STATISTICS
1007	pgmR3InitStats(pVM);
1008	#endif
1009	#ifdef VBOX_WITH_DEBUGGER
1010	/*
1011	* Debugger commands.
1012	*/
1013	static bool fRegisteredCmds = false;
1014	if (!fRegisteredCmds)
1015	{
1016	int rc = DBGCRegisterCommands(&g_aCmds[0], ELEMENTS(g_aCmds));
1017	if (VBOX_SUCCESS(rc))
1018	fRegisteredCmds = true;
1019	}
1020	#endif
1021	return VINF_SUCCESS;
1022	}
1023
1024	/* Almost no cleanup necessary, MM frees all memory. */
1025	PDMR3CritSectDelete(&pVM->pgm.s.CritSect);
1026
1027	return rc;
1028	}
1029
1030
1031	/**
1032	* Init paging.
1033	*
1034	* Since we need to check what mode the host is operating in before we can choose
1035	* the right paging functions for the host we have to delay this until R0 has
1036	* been initialized.
1037	*
1038	* @returns VBox status code.
1039	* @param pVM VM handle.
1040	*/
1041	static int pgmR3InitPaging(PVM pVM)
1042	{
1043	/*
1044	* Force a recalculation of modes and switcher so everyone gets notified.
1045	*/
1046	pVM->pgm.s.enmShadowMode = PGMMODE_INVALID;
1047	pVM->pgm.s.enmGuestMode = PGMMODE_INVALID;
1048	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1049
1050	/*
1051	* Allocate static mapping space for whatever the cr3 register
1052	* points to and in the case of PAE mode to the 4 PDs.
1053	*/
1054	int rc = MMR3HyperReserve(pVM, PAGE_SIZE * 5, "CR3 mapping", &pVM->pgm.s.GCPtrCR3Mapping);
1055	if (VBOX_FAILURE(rc))
1056	{
1057	AssertMsgFailed(("Failed to reserve two pages for cr mapping in HMA, rc=%Vrc\n", rc));
1058	return rc;
1059	}
1060	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
1061
1062	/*
1063	* Allocate pages for the three possible intermediate contexts
1064	* (AMD64, PAE and plain 32-Bit). We maintain all three contexts
1065	* for the sake of simplicity. The AMD64 uses the PAE for the
1066	* lower levels, making the total number of pages 11 (3 + 7 + 1).
1067	*
1068	* We assume that two page tables will be enought for the core code
1069	* mappings (HC virtual and identity).
1070	*/
1071	pVM->pgm.s.pInterPD = (PX86PD)MMR3PageAllocLow(pVM);
1072	pVM->pgm.s.apInterPTs[0] = (PX86PT)MMR3PageAllocLow(pVM);
1073	pVM->pgm.s.apInterPTs[1] = (PX86PT)MMR3PageAllocLow(pVM);
1074	pVM->pgm.s.apInterPaePTs[0] = (PX86PTPAE)MMR3PageAlloc(pVM);
1075	pVM->pgm.s.apInterPaePTs[1] = (PX86PTPAE)MMR3PageAlloc(pVM);
1076	pVM->pgm.s.apInterPaePDs[0] = (PX86PDPAE)MMR3PageAlloc(pVM);
1077	pVM->pgm.s.apInterPaePDs[1] = (PX86PDPAE)MMR3PageAlloc(pVM);
1078	pVM->pgm.s.apInterPaePDs[2] = (PX86PDPAE)MMR3PageAlloc(pVM);
1079	pVM->pgm.s.apInterPaePDs[3] = (PX86PDPAE)MMR3PageAlloc(pVM);
1080	pVM->pgm.s.pInterPaePDPT = (PX86PDPT)MMR3PageAllocLow(pVM);
1081	pVM->pgm.s.pInterPaePDPT64 = (PX86PDPT)MMR3PageAllocLow(pVM);
1082	pVM->pgm.s.pInterPaePML4 = (PX86PML4)MMR3PageAllocLow(pVM);
1083	if ( !pVM->pgm.s.pInterPD
1084	\|\| !pVM->pgm.s.apInterPTs[0]
1085	\|\| !pVM->pgm.s.apInterPTs[1]
1086	\|\| !pVM->pgm.s.apInterPaePTs[0]
1087	\|\| !pVM->pgm.s.apInterPaePTs[1]
1088	\|\| !pVM->pgm.s.apInterPaePDs[0]
1089	\|\| !pVM->pgm.s.apInterPaePDs[1]
1090	\|\| !pVM->pgm.s.apInterPaePDs[2]
1091	\|\| !pVM->pgm.s.apInterPaePDs[3]
1092	\|\| !pVM->pgm.s.pInterPaePDPT
1093	\|\| !pVM->pgm.s.pInterPaePDPT64
1094	\|\| !pVM->pgm.s.pInterPaePML4)
1095	{
1096	AssertMsgFailed(("Failed to allocate pages for the intermediate context!\n"));
1097	return VERR_NO_PAGE_MEMORY;
1098	}
1099
1100	pVM->pgm.s.HCPhysInterPD = MMPage2Phys(pVM, pVM->pgm.s.pInterPD);
1101	AssertRelease(pVM->pgm.s.HCPhysInterPD != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPD & PAGE_OFFSET_MASK));
1102	pVM->pgm.s.HCPhysInterPaePDPT = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT);
1103	AssertRelease(pVM->pgm.s.HCPhysInterPaePDPT != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePDPT & PAGE_OFFSET_MASK));
1104	pVM->pgm.s.HCPhysInterPaePML4 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePML4);
1105	AssertRelease(pVM->pgm.s.HCPhysInterPaePML4 != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePML4 & PAGE_OFFSET_MASK));
1106
1107	/*
1108	* Initialize the pages, setting up the PML4 and PDPT for repetitive 4GB action.
1109	*/
1110	ASMMemZeroPage(pVM->pgm.s.pInterPD);
1111	ASMMemZeroPage(pVM->pgm.s.apInterPTs[0]);
1112	ASMMemZeroPage(pVM->pgm.s.apInterPTs[1]);
1113
1114	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[0]);
1115	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[1]);
1116
1117	ASMMemZeroPage(pVM->pgm.s.pInterPaePDPT);
1118	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.apInterPaePDs); i++)
1119	{
1120	ASMMemZeroPage(pVM->pgm.s.apInterPaePDs[i]);
1121	pVM->pgm.s.pInterPaePDPT->a[i].u = X86_PDPE_P \| PGM_PLXFLAGS_PERMANENT
1122	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[i]);
1123	}
1124
1125	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.pInterPaePDPT64->a); i++)
1126	{
1127	const unsigned iPD = i % ELEMENTS(pVM->pgm.s.apInterPaePDs);
1128	pVM->pgm.s.pInterPaePDPT64->a[i].u = X86_PDPE_P \| X86_PDPE_RW \| X86_PDPE_US \| X86_PDPE_A \| PGM_PLXFLAGS_PERMANENT
1129	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[iPD]);
1130	}
1131
1132	RTHCPHYS HCPhysInterPaePDPT64 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64);
1133	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.pInterPaePML4->a); i++)
1134	pVM->pgm.s.pInterPaePML4->a[i].u = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_US \| X86_PML4E_A \| PGM_PLXFLAGS_PERMANENT
1135	\| HCPhysInterPaePDPT64;
1136
1137	/*
1138	* Allocate pages for the three possible guest contexts (AMD64, PAE and plain 32-Bit).
1139	* We allocate pages for all three posibilities to in order to simplify mappings and
1140	* avoid resource failure during mode switches. So, we need to cover all levels of the
1141	* of the first 4GB down to PD level.
1142	* As with the intermediate context, AMD64 uses the PAE PDPT and PDs.
1143	*/
1144	pVM->pgm.s.pHC32BitPD = (PX86PD)MMR3PageAllocLow(pVM);
1145	pVM->pgm.s.apHCPaePDs[0] = (PX86PDPAE)MMR3PageAlloc(pVM);
1146	pVM->pgm.s.apHCPaePDs[1] = (PX86PDPAE)MMR3PageAlloc(pVM);
1147	AssertRelease((uintptr_t)pVM->pgm.s.apHCPaePDs[0] + PAGE_SIZE == (uintptr_t)pVM->pgm.s.apHCPaePDs[1]);
1148	pVM->pgm.s.apHCPaePDs[2] = (PX86PDPAE)MMR3PageAlloc(pVM);
1149	AssertRelease((uintptr_t)pVM->pgm.s.apHCPaePDs[1] + PAGE_SIZE == (uintptr_t)pVM->pgm.s.apHCPaePDs[2]);
1150	pVM->pgm.s.apHCPaePDs[3] = (PX86PDPAE)MMR3PageAlloc(pVM);
1151	AssertRelease((uintptr_t)pVM->pgm.s.apHCPaePDs[2] + PAGE_SIZE == (uintptr_t)pVM->pgm.s.apHCPaePDs[3]);
1152	pVM->pgm.s.pHCPaePDPT = (PX86PDPT)MMR3PageAllocLow(pVM);
1153	pVM->pgm.s.pHCPaePML4 = (PX86PML4)MMR3PageAllocLow(pVM);
1154	if ( !pVM->pgm.s.pHC32BitPD
1155	\|\| !pVM->pgm.s.apHCPaePDs[0]
1156	\|\| !pVM->pgm.s.apHCPaePDs[1]
1157	\|\| !pVM->pgm.s.apHCPaePDs[2]
1158	\|\| !pVM->pgm.s.apHCPaePDs[3]
1159	\|\| !pVM->pgm.s.pHCPaePDPT
1160	\|\| !pVM->pgm.s.pHCPaePML4)
1161	{
1162	AssertMsgFailed(("Failed to allocate pages for the intermediate context!\n"));
1163	return VERR_NO_PAGE_MEMORY;
1164	}
1165
1166	/* get physical addresses. */
1167	pVM->pgm.s.HCPhys32BitPD = MMPage2Phys(pVM, pVM->pgm.s.pHC32BitPD);
1168	Assert(MMPagePhys2Page(pVM, pVM->pgm.s.HCPhys32BitPD) == pVM->pgm.s.pHC32BitPD);
1169	pVM->pgm.s.aHCPhysPaePDs[0] = MMPage2Phys(pVM, pVM->pgm.s.apHCPaePDs[0]);
1170	pVM->pgm.s.aHCPhysPaePDs[1] = MMPage2Phys(pVM, pVM->pgm.s.apHCPaePDs[1]);
1171	pVM->pgm.s.aHCPhysPaePDs[2] = MMPage2Phys(pVM, pVM->pgm.s.apHCPaePDs[2]);
1172	pVM->pgm.s.aHCPhysPaePDs[3] = MMPage2Phys(pVM, pVM->pgm.s.apHCPaePDs[3]);
1173	pVM->pgm.s.HCPhysPaePDPT = MMPage2Phys(pVM, pVM->pgm.s.pHCPaePDPT);
1174	pVM->pgm.s.HCPhysPaePML4 = MMPage2Phys(pVM, pVM->pgm.s.pHCPaePML4);
1175
1176	/*
1177	* Initialize the pages, setting up the PML4 and PDPT for action below 4GB.
1178	*/
1179	ASMMemZero32(pVM->pgm.s.pHC32BitPD, PAGE_SIZE);
1180
1181	ASMMemZero32(pVM->pgm.s.pHCPaePDPT, PAGE_SIZE);
1182	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.apHCPaePDs); i++)
1183	{
1184	ASMMemZero32(pVM->pgm.s.apHCPaePDs[i], PAGE_SIZE);
1185	pVM->pgm.s.pHCPaePDPT->a[i].u = X86_PDPE_P \| PGM_PLXFLAGS_PERMANENT \| pVM->pgm.s.aHCPhysPaePDs[i];
1186	/* The flags will be corrected when entering and leaving long mode. */
1187	}
1188
1189	ASMMemZero32(pVM->pgm.s.pHCPaePML4, PAGE_SIZE);
1190	pVM->pgm.s.pHCPaePML4->a[0].u = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_A
1191	\| PGM_PLXFLAGS_PERMANENT \| pVM->pgm.s.HCPhysPaePDPT;
1192
1193	CPUMSetHyperCR3(pVM, (uint32_t)pVM->pgm.s.HCPhys32BitPD);
1194
1195	/*
1196	* Initialize paging workers and mode from current host mode
1197	* and the guest running in real mode.
1198	*/
1199	pVM->pgm.s.enmHostMode = SUPGetPagingMode();
1200	switch (pVM->pgm.s.enmHostMode)
1201	{
1202	case SUPPAGINGMODE_32_BIT:
1203	case SUPPAGINGMODE_32_BIT_GLOBAL:
1204	case SUPPAGINGMODE_PAE:
1205	case SUPPAGINGMODE_PAE_GLOBAL:
1206	case SUPPAGINGMODE_PAE_NX:
1207	case SUPPAGINGMODE_PAE_GLOBAL_NX:
1208	break;
1209
1210	case SUPPAGINGMODE_AMD64:
1211	case SUPPAGINGMODE_AMD64_GLOBAL:
1212	case SUPPAGINGMODE_AMD64_NX:
1213	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
1214	#ifndef VBOX_WITH_HYBIRD_32BIT_KERNEL
1215	if (ARCH_BITS != 64)
1216	{
1217	AssertMsgFailed(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1218	LogRel(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1219	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1220	}
1221	#endif
1222	break;
1223	default:
1224	AssertMsgFailed(("Host mode %d is not supported\n", pVM->pgm.s.enmHostMode));
1225	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1226	}
1227	rc = pgmR3ModeDataInit(pVM, false /* don't resolve GC and R0 syms yet */);
1228	if (VBOX_SUCCESS(rc))
1229	rc = pgmR3ChangeMode(pVM, PGMMODE_REAL);
1230	if (VBOX_SUCCESS(rc))
1231	{
1232	LogFlow(("pgmR3InitPaging: returns successfully\n"));
1233	#if HC_ARCH_BITS == 64
1234	LogRel(("Debug: HCPhys32BitPD=%VHp aHCPhysPaePDs={%VHp,%VHp,%VHp,%VHp} HCPhysPaePDPT=%VHp HCPhysPaePML4=%VHp\n",
1235	pVM->pgm.s.HCPhys32BitPD, pVM->pgm.s.aHCPhysPaePDs[0], pVM->pgm.s.aHCPhysPaePDs[1], pVM->pgm.s.aHCPhysPaePDs[2], pVM->pgm.s.aHCPhysPaePDs[3],
1236	pVM->pgm.s.HCPhysPaePDPT, pVM->pgm.s.HCPhysPaePML4));
1237	LogRel(("Debug: HCPhysInterPD=%VHp HCPhysInterPaePDPT=%VHp HCPhysInterPaePML4=%VHp\n",
1238	pVM->pgm.s.HCPhysInterPD, pVM->pgm.s.HCPhysInterPaePDPT, pVM->pgm.s.HCPhysInterPaePML4));
1239	LogRel(("Debug: apInterPTs={%VHp,%VHp} apInterPaePTs={%VHp,%VHp} apInterPaePDs={%VHp,%VHp,%VHp,%VHp} pInterPaePDPT64=%VHp\n",
1240	MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[1]),
1241	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[1]),
1242	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[1]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[2]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[3]),
1243	MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64)));
1244	#endif
1245
1246	return VINF_SUCCESS;
1247	}
1248
1249	LogFlow(("pgmR3InitPaging: returns %Vrc\n", rc));
1250	return rc;
1251	}
1252
1253
1254	#ifdef VBOX_WITH_STATISTICS
1255	/**
1256	* Init statistics
1257	*/
1258	static void pgmR3InitStats(PVM pVM)
1259	{
1260	PPGM pPGM = &pVM->pgm.s;
1261	STAM_REG(pVM, &pPGM->StatGCInvalidatePage, STAMTYPE_PROFILE, "/PGM/GC/InvalidatePage", STAMUNIT_TICKS_PER_CALL, "PGMGCInvalidatePage() profiling.");
1262	STAM_REG(pVM, &pPGM->StatGCInvalidatePage4KBPages, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/4KBPages", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was called for a 4KB page.");
1263	STAM_REG(pVM, &pPGM->StatGCInvalidatePage4MBPages, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/4MBPages", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was called for a 4MB page.");
1264	STAM_REG(pVM, &pPGM->StatGCInvalidatePage4MBPagesSkip, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/4MBPagesSkip",STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() skipped a 4MB page.");
1265	STAM_REG(pVM, &pPGM->StatGCInvalidatePagePDMappings, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/PDMappings", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was called for a page directory containing mappings (no conflict).");
1266	STAM_REG(pVM, &pPGM->StatGCInvalidatePagePDNAs, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/PDNAs", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was called for a not accessed page directory.");
1267	STAM_REG(pVM, &pPGM->StatGCInvalidatePagePDNPs, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/PDNPs", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was called for a not present page directory.");
1268	STAM_REG(pVM, &pPGM->StatGCInvalidatePagePDOutOfSync, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/PDOutOfSync", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was called for an out of sync page directory.");
1269	STAM_REG(pVM, &pPGM->StatGCInvalidatePageSkipped, STAMTYPE_COUNTER, "/PGM/GC/InvalidatePage/Skipped", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
1270	STAM_REG(pVM, &pPGM->StatGCSyncPT, STAMTYPE_PROFILE, "/PGM/GC/SyncPT", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMGCSyncPT() body.");
1271	STAM_REG(pVM, &pPGM->StatGCAccessedPage, STAMTYPE_COUNTER, "/PGM/GC/AccessedPage", STAMUNIT_OCCURENCES, "The number of pages marked not present for accessed bit emulation.");
1272	STAM_REG(pVM, &pPGM->StatGCDirtyPage, STAMTYPE_COUNTER, "/PGM/GC/DirtyPage/Mark", STAMUNIT_OCCURENCES, "The number of pages marked read-only for dirty bit tracking.");
1273	STAM_REG(pVM, &pPGM->StatGCDirtyPageBig, STAMTYPE_COUNTER, "/PGM/GC/DirtyPage/MarkBig", STAMUNIT_OCCURENCES, "The number of 4MB pages marked read-only for dirty bit tracking.");
1274	STAM_REG(pVM, &pPGM->StatGCDirtyPageTrap, STAMTYPE_COUNTER, "/PGM/GC/DirtyPage/Trap", STAMUNIT_OCCURENCES, "The number of traps generated for dirty bit tracking.");
1275	STAM_REG(pVM, &pPGM->StatGCDirtyPageSkipped, STAMTYPE_COUNTER, "/PGM/GC/DirtyPage/Skipped", STAMUNIT_OCCURENCES, "The number of pages already dirty or readonly.");
1276	STAM_REG(pVM, &pPGM->StatGCDirtiedPage, STAMTYPE_COUNTER, "/PGM/GC/DirtyPage/SetDirty", STAMUNIT_OCCURENCES, "The number of pages marked dirty because of write accesses.");
1277	STAM_REG(pVM, &pPGM->StatGCDirtyTrackRealPF, STAMTYPE_COUNTER, "/PGM/GC/DirtyPage/RealPF", STAMUNIT_OCCURENCES, "The number of real pages faults during dirty bit tracking.");
1278	STAM_REG(pVM, &pPGM->StatGCPageAlreadyDirty, STAMTYPE_COUNTER, "/PGM/GC/DirtyPage/AlreadySet", STAMUNIT_OCCURENCES, "The number of pages already marked dirty because of write accesses.");
1279	STAM_REG(pVM, &pPGM->StatGCDirtyBitTracking, STAMTYPE_PROFILE, "/PGM/GC/DirtyPage", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMTrackDirtyBit() body.");
1280	STAM_REG(pVM, &pPGM->StatGCSyncPTAlloc, STAMTYPE_COUNTER, "/PGM/GC/SyncPT/Alloc", STAMUNIT_OCCURENCES, "The number of times PGMGCSyncPT() needed to allocate page tables.");
1281	STAM_REG(pVM, &pPGM->StatGCSyncPTConflict, STAMTYPE_COUNTER, "/PGM/GC/SyncPT/Conflicts", STAMUNIT_OCCURENCES, "The number of times PGMGCSyncPT() detected conflicts.");
1282	STAM_REG(pVM, &pPGM->StatGCSyncPTFailed, STAMTYPE_COUNTER, "/PGM/GC/SyncPT/Failed", STAMUNIT_OCCURENCES, "The number of times PGMGCSyncPT() failed.");
1283
1284	STAM_REG(pVM, &pPGM->StatGCTrap0e, STAMTYPE_PROFILE, "/PGM/GC/Trap0e", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMGCTrap0eHandler() body.");
1285	STAM_REG(pVM, &pPGM->StatCheckPageFault, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time/CheckPageFault", STAMUNIT_TICKS_PER_CALL, "Profiling of checking for dirty/access emulation faults.");
1286	STAM_REG(pVM, &pPGM->StatLazySyncPT, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time/SyncPT", STAMUNIT_TICKS_PER_CALL, "Profiling of lazy page table syncing.");
1287	STAM_REG(pVM, &pPGM->StatMapping, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time/Mapping", STAMUNIT_TICKS_PER_CALL, "Profiling of checking virtual mappings.");
1288	STAM_REG(pVM, &pPGM->StatOutOfSync, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time/OutOfSync", STAMUNIT_TICKS_PER_CALL, "Profiling of out of sync page handling.");
1289	STAM_REG(pVM, &pPGM->StatHandlers, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time/Handlers", STAMUNIT_TICKS_PER_CALL, "Profiling of checking handlers.");
1290	STAM_REG(pVM, &pPGM->StatEIPHandlers, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time/EIPHandlers", STAMUNIT_TICKS_PER_CALL, "Profiling of checking eip handlers.");
1291	STAM_REG(pVM, &pPGM->StatTrap0eCSAM, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/CSAM", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is CSAM.");
1292	STAM_REG(pVM, &pPGM->StatTrap0eDirtyAndAccessedBits, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/DirtyAndAccessedBits", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is dirty and/or accessed bit emulation.");
1293	STAM_REG(pVM, &pPGM->StatTrap0eGuestTrap, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/GuestTrap", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is a guest trap.");
1294	STAM_REG(pVM, &pPGM->StatTrap0eHndPhys, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/HandlerPhysical", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is a physical handler.");
1295	STAM_REG(pVM, &pPGM->StatTrap0eHndVirt, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/HandlerVirtual",STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is a virtual handler.");
1296	STAM_REG(pVM, &pPGM->StatTrap0eHndUnhandled, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/HandlerUnhandled", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is access outside the monitored areas of a monitored page.");
1297	STAM_REG(pVM, &pPGM->StatTrap0eMisc, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/Misc", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is not known.");
1298	STAM_REG(pVM, &pPGM->StatTrap0eOutOfSync, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/OutOfSync", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is an out-of-sync page.");
1299	STAM_REG(pVM, &pPGM->StatTrap0eOutOfSyncHndPhys, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/OutOfSyncHndPhys", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is an out-of-sync physical handler page.");
1300	STAM_REG(pVM, &pPGM->StatTrap0eOutOfSyncHndVirt, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/OutOfSyncHndVirt", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is an out-of-sync virtual handler page.");
1301	STAM_REG(pVM, &pPGM->StatTrap0eOutOfSyncObsHnd, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/OutOfSyncObsHnd", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is an obsolete handler page.");
1302	STAM_REG(pVM, &pPGM->StatTrap0eSyncPT, STAMTYPE_PROFILE, "/PGM/GC/Trap0e/Time2/SyncPT", STAMUNIT_TICKS_PER_CALL, "Profiling of the Trap0eHandler body when the cause is lazy syncing of a PT.");
1303
1304	STAM_REG(pVM, &pPGM->StatTrap0eMapHandler, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/Mapping", STAMUNIT_OCCURENCES, "Number of traps due to access handlers in mappings.");
1305	STAM_REG(pVM, &pPGM->StatHandlersOutOfSync, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/OutOfSync", STAMUNIT_OCCURENCES, "Number of traps due to out-of-sync handled pages.");
1306	STAM_REG(pVM, &pPGM->StatHandlersPhysical, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/Physical", STAMUNIT_OCCURENCES, "Number of traps due to physical access handlers.");
1307	STAM_REG(pVM, &pPGM->StatHandlersVirtual, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/Virtual", STAMUNIT_OCCURENCES, "Number of traps due to virtual access handlers.");
1308	STAM_REG(pVM, &pPGM->StatHandlersVirtualByPhys, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/VirtualByPhys", STAMUNIT_OCCURENCES, "Number of traps due to virtual access handlers by physical address.");
1309	STAM_REG(pVM, &pPGM->StatHandlersVirtualUnmarked, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/VirtualUnmarked", STAMUNIT_OCCURENCES,"Number of traps due to virtual access handlers by virtual address (without proper physical flags).");
1310	STAM_REG(pVM, &pPGM->StatHandlersUnhandled, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/Unhandled", STAMUNIT_OCCURENCES, "Number of traps due to access outside range of monitored page(s).");
1311	STAM_REG(pVM, &pPGM->StatHandlersInvalid, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Handlers/Invalid", STAMUNIT_OCCURENCES, "Number of traps due to access to invalid physical memory.");
1312
1313	STAM_REG(pVM, &pPGM->StatGCTrap0eConflicts, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Conflicts", STAMUNIT_OCCURENCES, "The number of times #PF was caused by an undetected conflict.");
1314	STAM_REG(pVM, &pPGM->StatGCTrap0eUSNotPresentRead, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/User/NPRead", STAMUNIT_OCCURENCES, "Number of user mode not present read page faults.");
1315	STAM_REG(pVM, &pPGM->StatGCTrap0eUSNotPresentWrite, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/User/NPWrite", STAMUNIT_OCCURENCES, "Number of user mode not present write page faults.");
1316	STAM_REG(pVM, &pPGM->StatGCTrap0eUSWrite, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/User/Write", STAMUNIT_OCCURENCES, "Number of user mode write page faults.");
1317	STAM_REG(pVM, &pPGM->StatGCTrap0eUSReserved, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/User/Reserved", STAMUNIT_OCCURENCES, "Number of user mode reserved bit page faults.");
1318	STAM_REG(pVM, &pPGM->StatGCTrap0eUSNXE, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/User/NXE", STAMUNIT_OCCURENCES, "Number of user mode NXE page faults.");
1319	STAM_REG(pVM, &pPGM->StatGCTrap0eUSRead, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/User/Read", STAMUNIT_OCCURENCES, "Number of user mode read page faults.");
1320
1321	STAM_REG(pVM, &pPGM->StatGCTrap0eSVNotPresentRead, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Supervisor/NPRead", STAMUNIT_OCCURENCES, "Number of supervisor mode not present read page faults.");
1322	STAM_REG(pVM, &pPGM->StatGCTrap0eSVNotPresentWrite, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Supervisor/NPWrite", STAMUNIT_OCCURENCES, "Number of supervisor mode not present write page faults.");
1323	STAM_REG(pVM, &pPGM->StatGCTrap0eSVWrite, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Supervisor/Write", STAMUNIT_OCCURENCES, "Number of supervisor mode write page faults.");
1324	STAM_REG(pVM, &pPGM->StatGCTrap0eSVReserved, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Supervisor/Reserved", STAMUNIT_OCCURENCES, "Number of supervisor mode reserved bit page faults.");
1325	STAM_REG(pVM, &pPGM->StatGCTrap0eSNXE, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/Supervisor/NXE", STAMUNIT_OCCURENCES, "Number of supervisor mode NXE page faults.");
1326	STAM_REG(pVM, &pPGM->StatGCTrap0eUnhandled, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/GuestPF/Unhandled", STAMUNIT_OCCURENCES, "Number of guest real page faults.");
1327	STAM_REG(pVM, &pPGM->StatGCTrap0eMap, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/GuestPF/Map", STAMUNIT_OCCURENCES, "Number of guest page faults due to map accesses.");
1328
1329	STAM_REG(pVM, &pPGM->StatTrap0eWPEmulGC, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/WP/InGC", STAMUNIT_OCCURENCES, "Number of guest page faults due to X86_CR0_WP emulation.");
1330	STAM_REG(pVM, &pPGM->StatTrap0eWPEmulR3, STAMTYPE_COUNTER, "/PGM/GC/Trap0e/WP/ToR3", STAMUNIT_OCCURENCES, "Number of guest page faults due to X86_CR0_WP emulation (forward to R3 for emulation).");
1331
1332	STAM_REG(pVM, &pPGM->StatGCGuestCR3WriteHandled, STAMTYPE_COUNTER, "/PGM/GC/CR3WriteInt", STAMUNIT_OCCURENCES, "The number of times the Guest CR3 change was successfully handled.");
1333	STAM_REG(pVM, &pPGM->StatGCGuestCR3WriteUnhandled, STAMTYPE_COUNTER, "/PGM/GC/CR3WriteEmu", STAMUNIT_OCCURENCES, "The number of times the Guest CR3 change was passed back to the recompiler.");
1334	STAM_REG(pVM, &pPGM->StatGCGuestCR3WriteConflict, STAMTYPE_COUNTER, "/PGM/GC/CR3WriteConflict", STAMUNIT_OCCURENCES, "The number of times the Guest CR3 monitoring detected a conflict.");
1335
1336	STAM_REG(pVM, &pPGM->StatGCPageOutOfSyncSupervisor, STAMTYPE_COUNTER, "/PGM/GC/OutOfSync/SuperVisor", STAMUNIT_OCCURENCES, "Number of traps due to pages out of sync.");
1337	STAM_REG(pVM, &pPGM->StatGCPageOutOfSyncUser, STAMTYPE_COUNTER, "/PGM/GC/OutOfSync/User", STAMUNIT_OCCURENCES, "Number of traps due to pages out of sync.");
1338
1339	STAM_REG(pVM, &pPGM->StatGCGuestROMWriteHandled, STAMTYPE_COUNTER, "/PGM/GC/ROMWriteInt", STAMUNIT_OCCURENCES, "The number of times the Guest ROM change was successfully handled.");
1340	STAM_REG(pVM, &pPGM->StatGCGuestROMWriteUnhandled, STAMTYPE_COUNTER, "/PGM/GC/ROMWriteEmu", STAMUNIT_OCCURENCES, "The number of times the Guest ROM change was passed back to the recompiler.");
1341
1342	STAM_REG(pVM, &pPGM->StatDynMapCacheHits, STAMTYPE_COUNTER, "/PGM/GC/DynMapCache/Hits" , STAMUNIT_OCCURENCES, "Number of dynamic page mapping cache hits.");
1343	STAM_REG(pVM, &pPGM->StatDynMapCacheMisses, STAMTYPE_COUNTER, "/PGM/GC/DynMapCache/Misses" , STAMUNIT_OCCURENCES, "Number of dynamic page mapping cache misses.");
1344
1345	STAM_REG(pVM, &pPGM->StatHCDetectedConflicts, STAMTYPE_COUNTER, "/PGM/HC/DetectedConflicts", STAMUNIT_OCCURENCES, "The number of times PGMR3CheckMappingConflicts() detected a conflict.");
1346	STAM_REG(pVM, &pPGM->StatHCGuestPDWrite, STAMTYPE_COUNTER, "/PGM/HC/PDWrite", STAMUNIT_OCCURENCES, "The total number of times pgmHCGuestPDWriteHandler() was called.");
1347	STAM_REG(pVM, &pPGM->StatHCGuestPDWriteConflict, STAMTYPE_COUNTER, "/PGM/HC/PDWriteConflict", STAMUNIT_OCCURENCES, "The number of times pgmHCGuestPDWriteHandler() detected a conflict.");
1348
1349	STAM_REG(pVM, &pPGM->StatHCInvalidatePage, STAMTYPE_PROFILE, "/PGM/HC/InvalidatePage", STAMUNIT_TICKS_PER_CALL, "PGMHCInvalidatePage() profiling.");
1350	STAM_REG(pVM, &pPGM->StatHCInvalidatePage4KBPages, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/4KBPages", STAMUNIT_OCCURENCES, "The number of times PGMHCInvalidatePage() was called for a 4KB page.");
1351	STAM_REG(pVM, &pPGM->StatHCInvalidatePage4MBPages, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/4MBPages", STAMUNIT_OCCURENCES, "The number of times PGMHCInvalidatePage() was called for a 4MB page.");
1352	STAM_REG(pVM, &pPGM->StatHCInvalidatePage4MBPagesSkip, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/4MBPagesSkip",STAMUNIT_OCCURENCES, "The number of times PGMHCInvalidatePage() skipped a 4MB page.");
1353	STAM_REG(pVM, &pPGM->StatHCInvalidatePagePDMappings, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/PDMappings", STAMUNIT_OCCURENCES, "The number of times PGMHCInvalidatePage() was called for a page directory containing mappings (no conflict).");
1354	STAM_REG(pVM, &pPGM->StatHCInvalidatePagePDNAs, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/PDNAs", STAMUNIT_OCCURENCES, "The number of times PGMHCInvalidatePage() was called for a not accessed page directory.");
1355	STAM_REG(pVM, &pPGM->StatHCInvalidatePagePDNPs, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/PDNPs", STAMUNIT_OCCURENCES, "The number of times PGMHCInvalidatePage() was called for a not present page directory.");
1356	STAM_REG(pVM, &pPGM->StatHCInvalidatePagePDOutOfSync, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/PDOutOfSync", STAMUNIT_OCCURENCES, "The number of times PGMGCInvalidatePage() was called for an out of sync page directory.");
1357	STAM_REG(pVM, &pPGM->StatHCInvalidatePageSkipped, STAMTYPE_COUNTER, "/PGM/HC/InvalidatePage/Skipped", STAMUNIT_OCCURENCES, "The number of times PGMHCInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
1358	STAM_REG(pVM, &pPGM->StatHCResolveConflict, STAMTYPE_PROFILE, "/PGM/HC/ResolveConflict", STAMUNIT_TICKS_PER_CALL, "pgmR3SyncPTResolveConflict() profiling (includes the entire relocation).");
1359	STAM_REG(pVM, &pPGM->StatHCPrefetch, STAMTYPE_PROFILE, "/PGM/HC/Prefetch", STAMUNIT_TICKS_PER_CALL, "PGMR3PrefetchPage profiling.");
1360
1361	STAM_REG(pVM, &pPGM->StatHCSyncPT, STAMTYPE_PROFILE, "/PGM/HC/SyncPT", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMR3SyncPT() body.");
1362	STAM_REG(pVM, &pPGM->StatHCAccessedPage, STAMTYPE_COUNTER, "/PGM/HC/AccessedPage", STAMUNIT_OCCURENCES, "The number of pages marked not present for accessed bit emulation.");
1363	STAM_REG(pVM, &pPGM->StatHCDirtyPage, STAMTYPE_COUNTER, "/PGM/HC/DirtyPage/Mark", STAMUNIT_OCCURENCES, "The number of pages marked read-only for dirty bit tracking.");
1364	STAM_REG(pVM, &pPGM->StatHCDirtyPageBig, STAMTYPE_COUNTER, "/PGM/HC/DirtyPage/MarkBig", STAMUNIT_OCCURENCES, "The number of 4MB pages marked read-only for dirty bit tracking.");
1365	STAM_REG(pVM, &pPGM->StatHCDirtyPageTrap, STAMTYPE_COUNTER, "/PGM/HC/DirtyPage/Trap", STAMUNIT_OCCURENCES, "The number of traps generated for dirty bit tracking.");
1366	STAM_REG(pVM, &pPGM->StatHCDirtyPageSkipped, STAMTYPE_COUNTER, "/PGM/HC/DirtyPage/Skipped", STAMUNIT_OCCURENCES, "The number of pages already dirty or readonly.");
1367	STAM_REG(pVM, &pPGM->StatHCDirtyBitTracking, STAMTYPE_PROFILE, "/PGM/HC/DirtyPage", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMTrackDirtyBit() body.");
1368
1369	STAM_REG(pVM, &pPGM->StatGCSyncPagePDNAs, STAMTYPE_COUNTER, "/PGM/GC/SyncPagePDNAs", STAMUNIT_OCCURENCES, "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1370	STAM_REG(pVM, &pPGM->StatGCSyncPagePDOutOfSync, STAMTYPE_COUNTER, "/PGM/GC/SyncPagePDOutOfSync", STAMUNIT_OCCURENCES, "The number of time we've encountered an out-of-sync PD in SyncPage.");
1371	STAM_REG(pVM, &pPGM->StatHCSyncPagePDNAs, STAMTYPE_COUNTER, "/PGM/HC/SyncPagePDNAs", STAMUNIT_OCCURENCES, "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1372	STAM_REG(pVM, &pPGM->StatHCSyncPagePDOutOfSync, STAMTYPE_COUNTER, "/PGM/HC/SyncPagePDOutOfSync", STAMUNIT_OCCURENCES, "The number of time we've encountered an out-of-sync PD in SyncPage.");
1373
1374	STAM_REG(pVM, &pPGM->StatFlushTLB, STAMTYPE_PROFILE, "/PGM/FlushTLB", STAMUNIT_OCCURENCES, "Profiling of the PGMFlushTLB() body.");
1375	STAM_REG(pVM, &pPGM->StatFlushTLBNewCR3, STAMTYPE_COUNTER, "/PGM/FlushTLB/NewCR3", STAMUNIT_OCCURENCES, "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
1376	STAM_REG(pVM, &pPGM->StatFlushTLBNewCR3Global, STAMTYPE_COUNTER, "/PGM/FlushTLB/NewCR3Global", STAMUNIT_OCCURENCES, "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
1377	STAM_REG(pVM, &pPGM->StatFlushTLBSameCR3, STAMTYPE_COUNTER, "/PGM/FlushTLB/SameCR3", STAMUNIT_OCCURENCES, "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
1378	STAM_REG(pVM, &pPGM->StatFlushTLBSameCR3Global, STAMTYPE_COUNTER, "/PGM/FlushTLB/SameCR3Global", STAMUNIT_OCCURENCES, "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
1379
1380	STAM_REG(pVM, &pPGM->StatGCSyncCR3, STAMTYPE_PROFILE, "/PGM/GC/SyncCR3", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMSyncCR3() body.");
1381	STAM_REG(pVM, &pPGM->StatGCSyncCR3Handlers, STAMTYPE_PROFILE, "/PGM/GC/SyncCR3/Handlers", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMSyncCR3() update handler section.");
1382	STAM_REG(pVM, &pPGM->StatGCSyncCR3HandlerVirtualUpdate, STAMTYPE_PROFILE, "/PGM/GC/SyncCR3/Handlers/VirtualUpdate",STAMUNIT_TICKS_PER_CALL, "Profiling of the virtual handler updates.");
1383	STAM_REG(pVM, &pPGM->StatGCSyncCR3HandlerVirtualReset, STAMTYPE_PROFILE, "/PGM/GC/SyncCR3/Handlers/VirtualReset", STAMUNIT_TICKS_PER_CALL, "Profiling of the virtual handler resets.");
1384	STAM_REG(pVM, &pPGM->StatGCSyncCR3Global, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/Global", STAMUNIT_OCCURENCES, "The number of global CR3 syncs.");
1385	STAM_REG(pVM, &pPGM->StatGCSyncCR3NotGlobal, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/NotGlobal", STAMUNIT_OCCURENCES, "The number of non-global CR3 syncs.");
1386	STAM_REG(pVM, &pPGM->StatGCSyncCR3DstCacheHit, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/DstChacheHit", STAMUNIT_OCCURENCES, "The number of times we got some kind of a cache hit.");
1387	STAM_REG(pVM, &pPGM->StatGCSyncCR3DstFreed, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/DstFreed", STAMUNIT_OCCURENCES, "The number of times we've had to free a shadow entry.");
1388	STAM_REG(pVM, &pPGM->StatGCSyncCR3DstFreedSrcNP, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/DstFreedSrcNP", STAMUNIT_OCCURENCES, "The number of times we've had to free a shadow entry for which the source entry was not present.");
1389	STAM_REG(pVM, &pPGM->StatGCSyncCR3DstNotPresent, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/DstNotPresent", STAMUNIT_OCCURENCES, "The number of times we've encountered a not present shadow entry for a present guest entry.");
1390	STAM_REG(pVM, &pPGM->StatGCSyncCR3DstSkippedGlobalPD, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/DstSkippedGlobalPD", STAMUNIT_OCCURENCES, "The number of times a global page directory wasn't flushed.");
1391	STAM_REG(pVM, &pPGM->StatGCSyncCR3DstSkippedGlobalPT, STAMTYPE_COUNTER, "/PGM/GC/SyncCR3/DstSkippedGlobalPT", STAMUNIT_OCCURENCES, "The number of times a page table with only global entries wasn't flushed.");
1392
1393	STAM_REG(pVM, &pPGM->StatHCSyncCR3, STAMTYPE_PROFILE, "/PGM/HC/SyncCR3", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMSyncCR3() body.");
1394	STAM_REG(pVM, &pPGM->StatHCSyncCR3Handlers, STAMTYPE_PROFILE, "/PGM/HC/SyncCR3/Handlers", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMSyncCR3() update handler section.");
1395	STAM_REG(pVM, &pPGM->StatHCSyncCR3HandlerVirtualUpdate, STAMTYPE_PROFILE, "/PGM/HC/SyncCR3/Handlers/VirtualUpdate",STAMUNIT_TICKS_PER_CALL, "Profiling of the virtual handler updates.");
1396	STAM_REG(pVM, &pPGM->StatHCSyncCR3HandlerVirtualReset, STAMTYPE_PROFILE, "/PGM/HC/SyncCR3/Handlers/VirtualReset", STAMUNIT_TICKS_PER_CALL, "Profiling of the virtual handler resets.");
1397	STAM_REG(pVM, &pPGM->StatHCSyncCR3Global, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/Global", STAMUNIT_OCCURENCES, "The number of global CR3 syncs.");
1398	STAM_REG(pVM, &pPGM->StatHCSyncCR3NotGlobal, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/NotGlobal", STAMUNIT_OCCURENCES, "The number of non-global CR3 syncs.");
1399	STAM_REG(pVM, &pPGM->StatHCSyncCR3DstCacheHit, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/DstChacheHit", STAMUNIT_OCCURENCES, "The number of times we got some kind of a cache hit.");
1400	STAM_REG(pVM, &pPGM->StatHCSyncCR3DstFreed, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/DstFreed", STAMUNIT_OCCURENCES, "The number of times we've had to free a shadow entry.");
1401	STAM_REG(pVM, &pPGM->StatHCSyncCR3DstFreedSrcNP, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/DstFreedSrcNP", STAMUNIT_OCCURENCES, "The number of times we've had to free a shadow entry for which the source entry was not present.");
1402	STAM_REG(pVM, &pPGM->StatHCSyncCR3DstNotPresent, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/DstNotPresent", STAMUNIT_OCCURENCES, "The number of times we've encountered a not present shadow entry for a present guest entry.");
1403	STAM_REG(pVM, &pPGM->StatHCSyncCR3DstSkippedGlobalPD, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/DstSkippedGlobalPD", STAMUNIT_OCCURENCES, "The number of times a global page directory wasn't flushed.");
1404	STAM_REG(pVM, &pPGM->StatHCSyncCR3DstSkippedGlobalPT, STAMTYPE_COUNTER, "/PGM/HC/SyncCR3/DstSkippedGlobalPT", STAMUNIT_OCCURENCES, "The number of times a page table with only global entries wasn't flushed.");
1405
1406	STAM_REG(pVM, &pPGM->StatVirtHandleSearchByPhysGC, STAMTYPE_PROFILE, "/PGM/VirtHandler/SearchByPhys/GC", STAMUNIT_TICKS_PER_CALL, "Profiling of pgmHandlerVirtualFindByPhysAddr in GC.");
1407	STAM_REG(pVM, &pPGM->StatVirtHandleSearchByPhysHC, STAMTYPE_PROFILE, "/PGM/VirtHandler/SearchByPhys/HC", STAMUNIT_TICKS_PER_CALL, "Profiling of pgmHandlerVirtualFindByPhysAddr in HC.");
1408	STAM_REG(pVM, &pPGM->StatHandlePhysicalReset, STAMTYPE_COUNTER, "/PGM/HC/HandlerPhysicalReset", STAMUNIT_OCCURENCES, "The number of times PGMR3HandlerPhysicalReset is called.");
1409
1410	STAM_REG(pVM, &pPGM->StatHCGstModifyPage, STAMTYPE_PROFILE, "/PGM/HC/GstModifyPage", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMGstModifyPage() body.");
1411	STAM_REG(pVM, &pPGM->StatGCGstModifyPage, STAMTYPE_PROFILE, "/PGM/GC/GstModifyPage", STAMUNIT_TICKS_PER_CALL, "Profiling of the PGMGstModifyPage() body.");
1412
1413	STAM_REG(pVM, &pPGM->StatSynPT4kGC, STAMTYPE_COUNTER, "/PGM/GC/SyncPT/4k", STAMUNIT_OCCURENCES, "Nr of 4k PT syncs");
1414	STAM_REG(pVM, &pPGM->StatSynPT4kHC, STAMTYPE_COUNTER, "/PGM/HC/SyncPT/4k", STAMUNIT_OCCURENCES, "Nr of 4k PT syncs");
1415	STAM_REG(pVM, &pPGM->StatSynPT4MGC, STAMTYPE_COUNTER, "/PGM/GC/SyncPT/4M", STAMUNIT_OCCURENCES, "Nr of 4M PT syncs");
1416	STAM_REG(pVM, &pPGM->StatSynPT4MHC, STAMTYPE_COUNTER, "/PGM/HC/SyncPT/4M", STAMUNIT_OCCURENCES, "Nr of 4M PT syncs");
1417
1418	STAM_REG(pVM, &pPGM->StatDynRamTotal, STAMTYPE_COUNTER, "/PGM/RAM/TotalAlloc", STAMUNIT_MEGABYTES, "Allocated mbs of guest ram.");
1419	STAM_REG(pVM, &pPGM->StatDynRamGrow, STAMTYPE_COUNTER, "/PGM/RAM/Grow", STAMUNIT_OCCURENCES, "Nr of pgmr3PhysGrowRange calls.");
1420
1421	STAM_REG(pVM, &pPGM->StatPageHCMapTlbHits, STAMTYPE_COUNTER, "/PGM/PageHCMap/TlbHits", STAMUNIT_OCCURENCES, "TLB hits.");
1422	STAM_REG(pVM, &pPGM->StatPageHCMapTlbMisses, STAMTYPE_COUNTER, "/PGM/PageHCMap/TlbMisses", STAMUNIT_OCCURENCES, "TLB misses.");
1423	STAM_REG(pVM, &pPGM->ChunkR3Map.c, STAMTYPE_U32, "/PGM/ChunkR3Map/c", STAMUNIT_OCCURENCES, "Number of mapped chunks.");
1424	STAM_REG(pVM, &pPGM->ChunkR3Map.cMax, STAMTYPE_U32, "/PGM/ChunkR3Map/cMax", STAMUNIT_OCCURENCES, "Maximum number of mapped chunks.");
1425	STAM_REG(pVM, &pPGM->StatChunkR3MapTlbHits, STAMTYPE_COUNTER, "/PGM/ChunkR3Map/TlbHits", STAMUNIT_OCCURENCES, "TLB hits.");
1426	STAM_REG(pVM, &pPGM->StatChunkR3MapTlbMisses, STAMTYPE_COUNTER, "/PGM/ChunkR3Map/TlbMisses", STAMUNIT_OCCURENCES, "TLB misses.");
1427	STAM_REG(pVM, &pPGM->StatPageReplaceShared, STAMTYPE_COUNTER, "/PGM/Page/ReplacedShared", STAMUNIT_OCCURENCES, "Times a shared page was replaced.");
1428	STAM_REG(pVM, &pPGM->StatPageReplaceZero, STAMTYPE_COUNTER, "/PGM/Page/ReplacedZero", STAMUNIT_OCCURENCES, "Times the zero page was replaced.");
1429	STAM_REG(pVM, &pPGM->StatPageHandyAllocs, STAMTYPE_COUNTER, "/PGM/Page/HandyAllocs", STAMUNIT_OCCURENCES, "Number of times we've allocated more handy pages.");
1430	STAM_REG(pVM, &pPGM->cAllPages, STAMTYPE_U32, "/PGM/Page/cAllPages", STAMUNIT_OCCURENCES, "The total number of pages.");
1431	STAM_REG(pVM, &pPGM->cPrivatePages, STAMTYPE_U32, "/PGM/Page/cPrivatePages", STAMUNIT_OCCURENCES, "The number of private pages.");
1432	STAM_REG(pVM, &pPGM->cSharedPages, STAMTYPE_U32, "/PGM/Page/cSharedPages", STAMUNIT_OCCURENCES, "The number of shared pages.");
1433	STAM_REG(pVM, &pPGM->cZeroPages, STAMTYPE_U32, "/PGM/Page/cZeroPages", STAMUNIT_OCCURENCES, "The number of zero backed pages.");
1434
1435	#ifdef PGMPOOL_WITH_GCPHYS_TRACKING
1436	STAM_REG(pVM, &pPGM->StatTrackVirgin, STAMTYPE_COUNTER, "/PGM/Track/Virgin", STAMUNIT_OCCURENCES, "The number of first time shadowings");
1437	STAM_REG(pVM, &pPGM->StatTrackAliased, STAMTYPE_COUNTER, "/PGM/Track/Aliased", STAMUNIT_OCCURENCES, "The number of times switching to cRef2, i.e. the page is being shadowed by two PTs.");
1438	STAM_REG(pVM, &pPGM->StatTrackAliasedMany, STAMTYPE_COUNTER, "/PGM/Track/AliasedMany", STAMUNIT_OCCURENCES, "The number of times we're tracking using cRef2.");
1439	STAM_REG(pVM, &pPGM->StatTrackAliasedLots, STAMTYPE_COUNTER, "/PGM/Track/AliasedLots", STAMUNIT_OCCURENCES, "The number of times we're hitting pages which has overflowed cRef2");
1440	STAM_REG(pVM, &pPGM->StatTrackOverflows, STAMTYPE_COUNTER, "/PGM/Track/Overflows", STAMUNIT_OCCURENCES, "The number of times the extent list grows to long.");
1441	STAM_REG(pVM, &pPGM->StatTrackDeref, STAMTYPE_PROFILE, "/PGM/Track/Deref", STAMUNIT_OCCURENCES, "Profiling of SyncPageWorkerTrackDeref (expensive).");
1442	#endif
1443
1444	for (unsigned i = 0; i < X86_PG_ENTRIES; i++)
1445	{
1446	/** @todo r=bird: We need a STAMR3RegisterF()! */
1447	char szName[32];
1448
1449	RTStrPrintf(szName, sizeof(szName), "/PGM/GC/PD/Trap0e/%04X", i);
1450	int rc = STAMR3Register(pVM, &pPGM->StatGCTrap0ePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, szName, STAMUNIT_OCCURENCES, "The number of traps in page directory n.");
1451	AssertRC(rc);
1452
1453	RTStrPrintf(szName, sizeof(szName), "/PGM/GC/PD/SyncPt/%04X", i);
1454	rc = STAMR3Register(pVM, &pPGM->StatGCSyncPtPD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, szName, STAMUNIT_OCCURENCES, "The number of syncs per PD n.");
1455	AssertRC(rc);
1456
1457	RTStrPrintf(szName, sizeof(szName), "/PGM/GC/PD/SyncPage/%04X", i);
1458	rc = STAMR3Register(pVM, &pPGM->StatGCSyncPagePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, szName, STAMUNIT_OCCURENCES, "The number of out of sync pages per page directory n.");
1459	AssertRC(rc);
1460	}
1461	}
1462	#endif /* VBOX_WITH_STATISTICS */
1463
1464	/**
1465	* Init the PGM bits that rely on VMMR0 and MM to be fully initialized.
1466	*
1467	* The dynamic mapping area will also be allocated and initialized at this
1468	* time. We could allocate it during PGMR3Init of course, but the mapping
1469	* wouldn't be allocated at that time preventing us from setting up the
1470	* page table entries with the dummy page.
1471	*
1472	* @returns VBox status code.
1473	* @param pVM VM handle.
1474	*/
1475	PGMR3DECL(int) PGMR3InitDynMap(PVM pVM)
1476	{
1477	/*
1478	* Reserve space for mapping the paging pages into guest context.
1479	*/
1480	int rc = MMR3HyperReserve(pVM, PAGE_SIZE * (2 + ELEMENTS(pVM->pgm.s.apHCPaePDs) + 1 + 2 + 2), "Paging", &pVM->pgm.s.pGC32BitPD);
1481	AssertRCReturn(rc, rc);
1482	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
1483
1484	/*
1485	* Reserve space for the dynamic mappings.
1486	*/
1487	/** @todo r=bird: Need to verify that the checks for crossing PTs are correct here. They seems to be assuming 4MB PTs.. */
1488	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping", &pVM->pgm.s.pbDynPageMapBaseGC);
1489	if ( VBOX_SUCCESS(rc)
1490	&& (pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_SHIFT) != ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_SHIFT))
1491	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping not crossing", &pVM->pgm.s.pbDynPageMapBaseGC);
1492	if (VBOX_SUCCESS(rc))
1493	{
1494	AssertRelease((pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_SHIFT) == ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_SHIFT));
1495	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
1496	}
1497	return rc;
1498	}
1499
1500
1501	/**
1502	* Ring-3 init finalizing.
1503	*
1504	* @returns VBox status code.
1505	* @param pVM The VM handle.
1506	*/
1507	PGMR3DECL(int) PGMR3InitFinalize(PVM pVM)
1508	{
1509	/*
1510	* Map the paging pages into the guest context.
1511	*/
1512	RTGCPTR GCPtr = pVM->pgm.s.pGC32BitPD;
1513	AssertReleaseReturn(GCPtr, VERR_INTERNAL_ERROR);
1514
1515	int rc = PGMMap(pVM, GCPtr, pVM->pgm.s.HCPhys32BitPD, PAGE_SIZE, 0);
1516	AssertRCReturn(rc, rc);
1517	pVM->pgm.s.pGC32BitPD = GCPtr;
1518	GCPtr += PAGE_SIZE;
1519	GCPtr += PAGE_SIZE; /* reserved page */
1520
1521	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.apHCPaePDs); i++)
1522	{
1523	rc = PGMMap(pVM, GCPtr, pVM->pgm.s.aHCPhysPaePDs[i], PAGE_SIZE, 0);
1524	AssertRCReturn(rc, rc);
1525	pVM->pgm.s.apGCPaePDs[i] = GCPtr;
1526	GCPtr += PAGE_SIZE;
1527	}
1528	/* A bit of paranoia is justified. */
1529	AssertRelease((RTGCUINTPTR)pVM->pgm.s.apGCPaePDs[0] + PAGE_SIZE == (RTGCUINTPTR)pVM->pgm.s.apGCPaePDs[1]);
1530	AssertRelease((RTGCUINTPTR)pVM->pgm.s.apGCPaePDs[1] + PAGE_SIZE == (RTGCUINTPTR)pVM->pgm.s.apGCPaePDs[2]);
1531	AssertRelease((RTGCUINTPTR)pVM->pgm.s.apGCPaePDs[2] + PAGE_SIZE == (RTGCUINTPTR)pVM->pgm.s.apGCPaePDs[3]);
1532	GCPtr += PAGE_SIZE; /* reserved page */
1533
1534	rc = PGMMap(pVM, GCPtr, pVM->pgm.s.HCPhysPaePDPT, PAGE_SIZE, 0);
1535	AssertRCReturn(rc, rc);
1536	pVM->pgm.s.pGCPaePDPT = GCPtr;
1537	GCPtr += PAGE_SIZE;
1538	GCPtr += PAGE_SIZE; /* reserved page */
1539
1540
1541	/*
1542	* Reserve space for the dynamic mappings.
1543	* Initialize the dynamic mapping pages with dummy pages to simply the cache.
1544	*/
1545	/* get the pointer to the page table entries. */
1546	PPGMMAPPING pMapping = pgmGetMapping(pVM, pVM->pgm.s.pbDynPageMapBaseGC);
1547	AssertRelease(pMapping);
1548	const uintptr_t off = pVM->pgm.s.pbDynPageMapBaseGC - pMapping->GCPtr;
1549	const unsigned iPT = off >> X86_PD_SHIFT;
1550	const unsigned iPG = (off >> X86_PT_SHIFT) & X86_PT_MASK;
1551	pVM->pgm.s.paDynPageMap32BitPTEsGC = pMapping->aPTs[iPT].pPTGC + iPG * sizeof(pMapping->aPTs[0].pPTR3->a[0]);
1552	pVM->pgm.s.paDynPageMapPaePTEsGC = pMapping->aPTs[iPT].paPaePTsGC + iPG * sizeof(pMapping->aPTs[0].paPaePTsR3->a[0]);
1553
1554	/* init cache */
1555	RTHCPHYS HCPhysDummy = MMR3PageDummyHCPhys(pVM);
1556	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.aHCPhysDynPageMapCache); i++)
1557	pVM->pgm.s.aHCPhysDynPageMapCache[i] = HCPhysDummy;
1558
1559	for (unsigned i = 0; i < MM_HYPER_DYNAMIC_SIZE; i += PAGE_SIZE)
1560	{
1561	rc = PGMMap(pVM, pVM->pgm.s.pbDynPageMapBaseGC + i, HCPhysDummy, PAGE_SIZE, 0);
1562	AssertRCReturn(rc, rc);
1563	}
1564
1565	return rc;
1566	}
1567
1568
1569	/**
1570	* Applies relocations to data and code managed by this
1571	* component. This function will be called at init and
1572	* whenever the VMM need to relocate it self inside the GC.
1573	*
1574	* @param pVM The VM.
1575	* @param offDelta Relocation delta relative to old location.
1576	*/
1577	PGMR3DECL(void) PGMR3Relocate(PVM pVM, RTGCINTPTR offDelta)
1578	{
1579	LogFlow(("PGMR3Relocate\n"));
1580
1581	/*
1582	* Paging stuff.
1583	*/
1584	pVM->pgm.s.GCPtrCR3Mapping += offDelta;
1585	/** @todo move this into shadow and guest specific relocation functions. */
1586	AssertMsg(pVM->pgm.s.pGC32BitPD, ("Init order, no relocation before paging is initialized!\n"));
1587	pVM->pgm.s.pGC32BitPD += offDelta;
1588	pVM->pgm.s.pGuestPDGC += offDelta;
1589	AssertCompile(ELEMENTS(pVM->pgm.s.apGCPaePDs) == ELEMENTS(pVM->pgm.s.apGstPaePDsGC));
1590	for (unsigned i = 0; i < ELEMENTS(pVM->pgm.s.apGCPaePDs); i++)
1591	{
1592	pVM->pgm.s.apGCPaePDs[i] += offDelta;
1593	pVM->pgm.s.apGstPaePDsGC[i] += offDelta;
1594	}
1595	pVM->pgm.s.pGstPaePDPTGC += offDelta;
1596	pVM->pgm.s.pGCPaePDPT += offDelta;
1597
1598	pgmR3ModeDataInit(pVM, true /* resolve GC/R0 symbols */);
1599	pgmR3ModeDataSwitch(pVM, pVM->pgm.s.enmShadowMode, pVM->pgm.s.enmGuestMode);
1600
1601	PGM_SHW_PFN(Relocate, pVM)(pVM, offDelta);
1602	PGM_GST_PFN(Relocate, pVM)(pVM, offDelta);
1603	PGM_BTH_PFN(Relocate, pVM)(pVM, offDelta);
1604
1605	/*
1606	* Trees.
1607	*/
1608	pVM->pgm.s.pTreesGC = MMHyperHC2GC(pVM, pVM->pgm.s.pTreesHC);
1609
1610	/*
1611	* Ram ranges.
1612	*/
1613	if (pVM->pgm.s.pRamRangesR3)
1614	{
1615	pVM->pgm.s.pRamRangesGC = MMHyperHC2GC(pVM, pVM->pgm.s.pRamRangesR3);
1616	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesR3; pCur->pNextR3; pCur = pCur->pNextR3)
1617	#ifdef VBOX_WITH_NEW_PHYS_CODE
1618	pCur->pNextGC = MMHyperR3ToGC(pVM, pCur->pNextR3);
1619	#else
1620	{
1621	pCur->pNextGC = MMHyperR3ToGC(pVM, pCur->pNextR3);
1622	if (pCur->pavHCChunkGC)
1623	pCur->pavHCChunkGC = MMHyperHC2GC(pVM, pCur->pavHCChunkHC);
1624	}
1625	#endif
1626	}
1627
1628	/*
1629	* Update the two page directories with all page table mappings.
1630	* (One or more of them have changed, that's why we're here.)
1631	*/
1632	pVM->pgm.s.pMappingsGC = MMHyperHC2GC(pVM, pVM->pgm.s.pMappingsR3);
1633	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur->pNextR3; pCur = pCur->pNextR3)
1634	pCur->pNextGC = MMHyperHC2GC(pVM, pCur->pNextR3);
1635
1636	/* Relocate GC addresses of Page Tables. */
1637	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur; pCur = pCur->pNextR3)
1638	{
1639	for (RTHCUINT i = 0; i < pCur->cPTs; i++)
1640	{
1641	pCur->aPTs[i].pPTGC = MMHyperR3ToGC(pVM, pCur->aPTs[i].pPTR3);
1642	pCur->aPTs[i].paPaePTsGC = MMHyperR3ToGC(pVM, pCur->aPTs[i].paPaePTsR3);
1643	}
1644	}
1645
1646	/*
1647	* Dynamic page mapping area.
1648	*/
1649	pVM->pgm.s.paDynPageMap32BitPTEsGC += offDelta;
1650	pVM->pgm.s.paDynPageMapPaePTEsGC += offDelta;
1651	pVM->pgm.s.pbDynPageMapBaseGC += offDelta;
1652
1653	/*
1654	* The Zero page.
1655	*/
1656	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
1657	AssertRelease(pVM->pgm.s.pvZeroPgR0);
1658
1659	/*
1660	* Physical and virtual handlers.
1661	*/
1662	RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesHC->PhysHandlers, true, pgmR3RelocatePhysHandler, &offDelta);
1663	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesHC->VirtHandlers, true, pgmR3RelocateVirtHandler, &offDelta);
1664	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesHC->HyperVirtHandlers, true, pgmR3RelocateHyperVirtHandler, &offDelta);
1665
1666	/*
1667	* The page pool.
1668	*/
1669	pgmR3PoolRelocate(pVM);
1670	}
1671
1672
1673	/**
1674	* Callback function for relocating a physical access handler.
1675	*
1676	* @returns 0 (continue enum)
1677	* @param pNode Pointer to a PGMPHYSHANDLER node.
1678	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
1679	* not certain the delta will fit in a void pointer for all possible configs.
1680	*/
1681	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser)
1682	{
1683	PPGMPHYSHANDLER pHandler = (PPGMPHYSHANDLER)pNode;
1684	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
1685	if (pHandler->pfnHandlerGC)
1686	pHandler->pfnHandlerGC += offDelta;
1687	if ((RTGCUINTPTR)pHandler->pvUserGC >= 0x10000)
1688	pHandler->pvUserGC += offDelta;
1689	return 0;
1690	}
1691
1692
1693	/**
1694	* Callback function for relocating a virtual access handler.
1695	*
1696	* @returns 0 (continue enum)
1697	* @param pNode Pointer to a PGMVIRTHANDLER node.
1698	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
1699	* not certain the delta will fit in a void pointer for all possible configs.
1700	*/
1701	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
1702	{
1703	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
1704	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
1705	Assert( pHandler->enmType == PGMVIRTHANDLERTYPE_ALL
1706	\|\| pHandler->enmType == PGMVIRTHANDLERTYPE_WRITE);
1707	Assert(pHandler->pfnHandlerGC);
1708	pHandler->pfnHandlerGC += offDelta;
1709	return 0;
1710	}
1711
1712
1713	/**
1714	* Callback function for relocating a virtual access handler for the hypervisor mapping.
1715	*
1716	* @returns 0 (continue enum)
1717	* @param pNode Pointer to a PGMVIRTHANDLER node.
1718	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
1719	* not certain the delta will fit in a void pointer for all possible configs.
1720	*/
1721	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
1722	{
1723	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
1724	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
1725	Assert(pHandler->enmType == PGMVIRTHANDLERTYPE_HYPERVISOR);
1726	Assert(pHandler->pfnHandlerGC);
1727	pHandler->pfnHandlerGC += offDelta;
1728	return 0;
1729	}
1730
1731
1732	/**
1733	* The VM is being reset.
1734	*
1735	* For the PGM component this means that any PD write monitors
1736	* needs to be removed.
1737	*
1738	* @param pVM VM handle.
1739	*/
1740	PGMR3DECL(void) PGMR3Reset(PVM pVM)
1741	{
1742	LogFlow(("PGMR3Reset:\n"));
1743	VM_ASSERT_EMT(pVM);
1744
1745	pgmLock(pVM);
1746
1747	/*
1748	* Unfix any fixed mappings and disable CR3 monitoring.
1749	*/
1750	pVM->pgm.s.fMappingsFixed = false;
1751	pVM->pgm.s.GCPtrMappingFixed = 0;
1752	pVM->pgm.s.cbMappingFixed = 0;
1753
1754	int rc = PGM_GST_PFN(UnmonitorCR3, pVM)(pVM);
1755	AssertRC(rc);
1756	#ifdef DEBUG
1757	DBGFR3InfoLog(pVM, "mappings", NULL);
1758	DBGFR3InfoLog(pVM, "handlers", "all nostat");
1759	#endif
1760
1761	/*
1762	* Reset the shadow page pool.
1763	*/
1764	pgmR3PoolReset(pVM);
1765
1766	/*
1767	* Re-init other members.
1768	*/
1769	pVM->pgm.s.fA20Enabled = true;
1770
1771	/*
1772	* Clear the FFs PGM owns.
1773	*/
1774	VM_FF_CLEAR(pVM, VM_FF_PGM_SYNC_CR3);
1775	VM_FF_CLEAR(pVM, VM_FF_PGM_SYNC_CR3_NON_GLOBAL);
1776
1777	/*
1778	* Reset (zero) RAM pages.
1779	*/
1780	rc = pgmR3PhysRamReset(pVM);
1781	if (RT_SUCCESS(rc))
1782	{
1783	#ifdef VBOX_WITH_NEW_PHYS_CODE
1784	/*
1785	* Reset (zero) shadow ROM pages.
1786	*/
1787	rc = pgmR3PhysRomReset(pVM);
1788	#endif
1789	if (RT_SUCCESS(rc))
1790	{
1791	/*
1792	* Switch mode back to real mode.
1793	*/
1794	rc = pgmR3ChangeMode(pVM, PGMMODE_REAL);
1795	STAM_REL_COUNTER_RESET(&pVM->pgm.s.cGuestModeChanges);
1796	}
1797	}
1798
1799	pgmUnlock(pVM);
1800	//return rc;
1801	AssertReleaseRC(rc);
1802	}
1803
1804
1805	#ifdef VBOX_STRICT
1806	/**
1807	* VM state change callback for clearing fNoMorePhysWrites after
1808	* a snapshot has been created.
1809	*/
1810	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser)
1811	{
1812	if (enmState == VMSTATE_RUNNING)
1813	pVM->pgm.s.fNoMorePhysWrites = false;
1814	}
1815	#endif
1816
1817
1818	/**
1819	* Terminates the PGM.
1820	*
1821	* @returns VBox status code.
1822	* @param pVM Pointer to VM structure.
1823	*/
1824	PGMR3DECL(int) PGMR3Term(PVM pVM)
1825	{
1826	return PDMR3CritSectDelete(&pVM->pgm.s.CritSect);
1827	}
1828
1829
1830	/**
1831	* Execute state save operation.
1832	*
1833	* @returns VBox status code.
1834	* @param pVM VM Handle.
1835	* @param pSSM SSM operation handle.
1836	*/
1837	static DECLCALLBACK(int) pgmR3Save(PVM pVM, PSSMHANDLE pSSM)
1838	{
1839	PPGM pPGM = &pVM->pgm.s;
1840
1841	/* No more writes to physical memory after this point! */
1842	pVM->pgm.s.fNoMorePhysWrites = true;
1843
1844	/*
1845	* Save basic data (required / unaffected by relocation).
1846	*/
1847	#if 1
1848	SSMR3PutBool(pSSM, pPGM->fMappingsFixed);
1849	#else
1850	SSMR3PutUInt(pSSM, pPGM->fMappingsFixed);
1851	#endif
1852	SSMR3PutGCPtr(pSSM, pPGM->GCPtrMappingFixed);
1853	SSMR3PutU32(pSSM, pPGM->cbMappingFixed);
1854	SSMR3PutUInt(pSSM, pPGM->cbRamSize);
1855	SSMR3PutGCPhys(pSSM, pPGM->GCPhysA20Mask);
1856	SSMR3PutUInt(pSSM, pPGM->fA20Enabled);
1857	SSMR3PutUInt(pSSM, pPGM->fSyncFlags);
1858	SSMR3PutUInt(pSSM, pPGM->enmGuestMode);
1859	SSMR3PutU32(pSSM, ~0); /* Separator. */
1860
1861	/*
1862	* The guest mappings.
1863	*/
1864	uint32_t i = 0;
1865	for (PPGMMAPPING pMapping = pPGM->pMappingsR3; pMapping; pMapping = pMapping->pNextR3, i++)
1866	{
1867	SSMR3PutU32(pSSM, i);
1868	SSMR3PutStrZ(pSSM, pMapping->pszDesc); /* This is the best unique id we have... */
1869	SSMR3PutGCPtr(pSSM, pMapping->GCPtr);
1870	SSMR3PutGCUIntPtr(pSSM, pMapping->cPTs);
1871	/* flags are done by the mapping owners! */
1872	}
1873	SSMR3PutU32(pSSM, ~0); /* terminator. */
1874
1875	/*
1876	* Ram range flags and bits.
1877	*/
1878	i = 0;
1879	for (PPGMRAMRANGE pRam = pPGM->pRamRangesR3; pRam; pRam = pRam->pNextR3, i++)
1880	{
1881	/** @todo MMIO ranges may move (PCI reconfig), we currently assume they don't. */
1882
1883	SSMR3PutU32(pSSM, i);
1884	SSMR3PutGCPhys(pSSM, pRam->GCPhys);
1885	SSMR3PutGCPhys(pSSM, pRam->GCPhysLast);
1886	SSMR3PutGCPhys(pSSM, pRam->cb);
1887	SSMR3PutU8(pSSM, !!pRam->pvHC); /* boolean indicating memory or not. */
1888
1889	/* Flags. */
1890	const unsigned cPages = pRam->cb >> PAGE_SHIFT;
1891	for (unsigned iPage = 0; iPage < cPages; iPage++)
1892	SSMR3PutU16(pSSM, (uint16_t)(pRam->aPages[iPage].HCPhys & ~X86_PTE_PAE_PG_MASK)); /** @todo PAGE FLAGS */
1893
1894	/* any memory associated with the range. */
1895	if (pRam->fFlags & MM_RAM_FLAGS_DYNAMIC_ALLOC)
1896	{
1897	for (unsigned iChunk = 0; iChunk < (pRam->cb >> PGM_DYNAMIC_CHUNK_SHIFT); iChunk++)
1898	{
1899	if (pRam->pavHCChunkHC[iChunk])
1900	{
1901	SSMR3PutU8(pSSM, 1); /* chunk present */
1902	SSMR3PutMem(pSSM, pRam->pavHCChunkHC[iChunk], PGM_DYNAMIC_CHUNK_SIZE);
1903	}
1904	else
1905	SSMR3PutU8(pSSM, 0); /* no chunk present */
1906	}
1907	}
1908	else if (pRam->pvHC)
1909	{
1910	int rc = SSMR3PutMem(pSSM, pRam->pvHC, pRam->cb);
1911	if (VBOX_FAILURE(rc))
1912	{
1913	Log(("pgmR3Save: SSMR3PutMem(, %p, %#x) -> %Vrc\n", pRam->pvHC, pRam->cb, rc));
1914	return rc;
1915	}
1916	}
1917	}
1918	return SSMR3PutU32(pSSM, ~0); /* terminator. */
1919	}
1920
1921
1922	/**
1923	* Execute state load operation.
1924	*
1925	* @returns VBox status code.
1926	* @param pVM VM Handle.
1927	* @param pSSM SSM operation handle.
1928	* @param u32Version Data layout version.
1929	*/
1930	static DECLCALLBACK(int) pgmR3Load(PVM pVM, PSSMHANDLE pSSM, uint32_t u32Version)
1931	{
1932	/*
1933	* Validate version.
1934	*/
1935	if (u32Version != PGM_SAVED_STATE_VERSION)
1936	{
1937	Log(("pgmR3Load: Invalid version u32Version=%d (current %d)!\n", u32Version, PGM_SAVED_STATE_VERSION));
1938	return VERR_SSM_UNSUPPORTED_DATA_UNIT_VERSION;
1939	}
1940
1941	/*
1942	* Call the reset function to make sure all the memory is cleared.
1943	*/
1944	PGMR3Reset(pVM);
1945
1946	/*
1947	* Load basic data (required / unaffected by relocation).
1948	*/
1949	PPGM pPGM = &pVM->pgm.s;
1950	#if 1
1951	SSMR3GetBool(pSSM, &pPGM->fMappingsFixed);
1952	#else
1953	uint32_t u;
1954	SSMR3GetU32(pSSM, &u);
1955	pPGM->fMappingsFixed = u;
1956	#endif
1957	SSMR3GetGCPtr(pSSM, &pPGM->GCPtrMappingFixed);
1958	SSMR3GetU32(pSSM, &pPGM->cbMappingFixed);
1959
1960	RTUINT cbRamSize;
1961	int rc = SSMR3GetU32(pSSM, &cbRamSize);
1962	if (VBOX_FAILURE(rc))
1963	return rc;
1964	if (cbRamSize != pPGM->cbRamSize)
1965	return VERR_SSM_LOAD_MEMORY_SIZE_MISMATCH;
1966	SSMR3GetGCPhys(pSSM, &pPGM->GCPhysA20Mask);
1967	SSMR3GetUInt(pSSM, &pPGM->fA20Enabled);
1968	SSMR3GetUInt(pSSM, &pPGM->fSyncFlags);
1969	RTUINT uGuestMode;
1970	SSMR3GetUInt(pSSM, &uGuestMode);
1971	pPGM->enmGuestMode = (PGMMODE)uGuestMode;
1972
1973	/* check separator. */
1974	uint32_t u32Sep;
1975	SSMR3GetU32(pSSM, &u32Sep);
1976	if (VBOX_FAILURE(rc))
1977	return rc;
1978	if (u32Sep != (uint32_t)~0)
1979	{
1980	AssertMsgFailed(("u32Sep=%#x (first)\n", u32Sep));
1981	return VERR_SSM_DATA_UNIT_FORMAT_CHANGED;
1982	}
1983
1984	/*
1985	* The guest mappings.
1986	*/
1987	uint32_t i = 0;
1988	for (;; i++)
1989	{
1990	/* Check the seqence number / separator. */
1991	rc = SSMR3GetU32(pSSM, &u32Sep);
1992	if (VBOX_FAILURE(rc))
1993	return rc;
1994	if (u32Sep == ~0U)
1995	break;
1996	if (u32Sep != i)
1997	{
1998	AssertMsgFailed(("u32Sep=%#x (last)\n", u32Sep));
1999	return VERR_SSM_DATA_UNIT_FORMAT_CHANGED;
2000	}
2001
2002	/* get the mapping details. */
2003	char szDesc[256];
2004	szDesc[0] = '\0';
2005	rc = SSMR3GetStrZ(pSSM, szDesc, sizeof(szDesc));
2006	if (VBOX_FAILURE(rc))
2007	return rc;
2008	RTGCPTR GCPtr;
2009	SSMR3GetGCPtr(pSSM, &GCPtr);
2010	RTGCUINTPTR cPTs;
2011	rc = SSMR3GetU32(pSSM, &cPTs);
2012	if (VBOX_FAILURE(rc))
2013	return rc;
2014
2015	/* find matching range. */
2016	PPGMMAPPING pMapping;
2017	for (pMapping = pPGM->pMappingsR3; pMapping; pMapping = pMapping->pNextR3)
2018	if ( pMapping->cPTs == cPTs
2019	&& !strcmp(pMapping->pszDesc, szDesc))
2020	break;
2021	if (!pMapping)
2022	{
2023	LogRel(("Couldn't find mapping: cPTs=%#x szDesc=%s (GCPtr=%VGv)\n",
2024	cPTs, szDesc, GCPtr));
2025	AssertFailed();
2026	return VERR_SSM_LOAD_CONFIG_MISMATCH;
2027	}
2028
2029	/* relocate it. */
2030	if (pMapping->GCPtr != GCPtr)
2031	{
2032	AssertMsg((GCPtr >> X86_PD_SHIFT << X86_PD_SHIFT) == GCPtr, ("GCPtr=%VGv\n", GCPtr));
2033	#if HC_ARCH_BITS == 64
2034	LogRel(("Mapping: %VGv -> %VGv %s\n", pMapping->GCPtr, GCPtr, pMapping->pszDesc));
2035	#endif
2036	pgmR3MapRelocate(pVM, pMapping, pMapping->GCPtr, GCPtr);
2037	}
2038	else
2039	Log(("pgmR3Load: '%s' needed no relocation (%VGv)\n", szDesc, GCPtr));
2040	}
2041
2042	/*
2043	* Ram range flags and bits.
2044	*/
2045	i = 0;
2046	for (PPGMRAMRANGE pRam = pPGM->pRamRangesR3; pRam; pRam = pRam->pNextR3, i++)
2047	{
2048	/** @todo MMIO ranges may move (PCI reconfig), we currently assume they don't. */
2049	/* Check the seqence number / separator. */
2050	rc = SSMR3GetU32(pSSM, &u32Sep);
2051	if (VBOX_FAILURE(rc))
2052	return rc;
2053	if (u32Sep == ~0U)
2054	break;
2055	if (u32Sep != i)
2056	{
2057	AssertMsgFailed(("u32Sep=%#x (last)\n", u32Sep));
2058	return VERR_SSM_DATA_UNIT_FORMAT_CHANGED;
2059	}
2060
2061	/* Get the range details. */
2062	RTGCPHYS GCPhys;
2063	SSMR3GetGCPhys(pSSM, &GCPhys);
2064	RTGCPHYS GCPhysLast;
2065	SSMR3GetGCPhys(pSSM, &GCPhysLast);
2066	RTGCPHYS cb;
2067	SSMR3GetGCPhys(pSSM, &cb);
2068	uint8_t fHaveBits;
2069	rc = SSMR3GetU8(pSSM, &fHaveBits);
2070	if (VBOX_FAILURE(rc))
2071	return rc;
2072	if (fHaveBits & ~1)
2073	{
2074	AssertMsgFailed(("u32Sep=%#x (last)\n", u32Sep));
2075	return VERR_SSM_DATA_UNIT_FORMAT_CHANGED;
2076	}
2077
2078	/* Match it up with the current range. */
2079	if ( GCPhys != pRam->GCPhys
2080	\|\| GCPhysLast != pRam->GCPhysLast
2081	\|\| cb != pRam->cb
2082	\|\| fHaveBits != !!pRam->pvHC)
2083	{
2084	LogRel(("Ram range: %VGp-%VGp %VGp bytes %s\n"
2085	"State : %VGp-%VGp %VGp bytes %s\n",
2086	pRam->GCPhys, pRam->GCPhysLast, pRam->cb, pRam->pvHC ? "bits" : "nobits",
2087	GCPhys, GCPhysLast, cb, fHaveBits ? "bits" : "nobits"));
2088	/*
2089	* If we're loading a state for debugging purpose, don't make a fuss if
2090	* the MMIO[2] and ROM stuff isn't 100% right, just skip the mismatches.
2091	*/
2092	if ( SSMR3HandleGetAfter(pSSM) != SSMAFTER_DEBUG_IT
2093	\|\| GCPhys < 8 * _1M)
2094	AssertFailedReturn(VERR_SSM_LOAD_CONFIG_MISMATCH);
2095
2096	RTGCPHYS cPages = ((GCPhysLast - GCPhys) + 1) >> PAGE_SHIFT;
2097	while (cPages-- > 0)
2098	{
2099	uint16_t u16Ignore;
2100	SSMR3GetU16(pSSM, &u16Ignore);
2101	}
2102	continue;
2103	}
2104
2105	/* Flags. */
2106	const unsigned cPages = pRam->cb >> PAGE_SHIFT;
2107	for (unsigned iPage = 0; iPage < cPages; iPage++)
2108	{
2109	uint16_t u16 = 0;
2110	SSMR3GetU16(pSSM, &u16);
2111	u16 &= PAGE_OFFSET_MASK & ~( RT_BIT(4) \| RT_BIT(5) \| RT_BIT(6)
2112	\| RT_BIT(7) \| RT_BIT(8) \| RT_BIT(9) \| RT_BIT(10) );
2113	// &= MM_RAM_FLAGS_DYNAMIC_ALLOC \| MM_RAM_FLAGS_RESERVED \| MM_RAM_FLAGS_ROM \| MM_RAM_FLAGS_MMIO \| MM_RAM_FLAGS_MMIO2
2114	pRam->aPages[iPage].HCPhys = PGM_PAGE_GET_HCPHYS(&pRam->aPages[iPage]) \| (RTHCPHYS)u16; /** @todo PAGE FLAGS */
2115	}
2116
2117	/* any memory associated with the range. */
2118	if (pRam->fFlags & MM_RAM_FLAGS_DYNAMIC_ALLOC)
2119	{
2120	for (unsigned iChunk = 0; iChunk < (pRam->cb >> PGM_DYNAMIC_CHUNK_SHIFT); iChunk++)
2121	{
2122	uint8_t fValidChunk;
2123
2124	rc = SSMR3GetU8(pSSM, &fValidChunk);
2125	if (VBOX_FAILURE(rc))
2126	return rc;
2127	if (fValidChunk > 1)
2128	return VERR_SSM_DATA_UNIT_FORMAT_CHANGED;
2129
2130	if (fValidChunk)
2131	{
2132	if (!pRam->pavHCChunkHC[iChunk])
2133	{
2134	rc = pgmr3PhysGrowRange(pVM, pRam->GCPhys + iChunk * PGM_DYNAMIC_CHUNK_SIZE);
2135	if (VBOX_FAILURE(rc))
2136	return rc;
2137	}
2138	Assert(pRam->pavHCChunkHC[iChunk]);
2139
2140	SSMR3GetMem(pSSM, pRam->pavHCChunkHC[iChunk], PGM_DYNAMIC_CHUNK_SIZE);
2141	}
2142	/* else nothing to do */
2143	}
2144	}
2145	else if (pRam->pvHC)
2146	{
2147	int rc = SSMR3GetMem(pSSM, pRam->pvHC, pRam->cb);
2148	if (VBOX_FAILURE(rc))
2149	{
2150	Log(("pgmR3Save: SSMR3GetMem(, %p, %#x) -> %Vrc\n", pRam->pvHC, pRam->cb, rc));
2151	return rc;
2152	}
2153	}
2154	}
2155
2156	/*
2157	* We require a full resync now.
2158	*/
2159	VM_FF_SET(pVM, VM_FF_PGM_SYNC_CR3_NON_GLOBAL);
2160	VM_FF_SET(pVM, VM_FF_PGM_SYNC_CR3);
2161	pPGM->fSyncFlags \|= PGM_SYNC_UPDATE_PAGE_BIT_VIRTUAL;
2162	pPGM->fPhysCacheFlushPending = true;
2163	pgmR3HandlerPhysicalUpdateAll(pVM);
2164
2165	/*
2166	* Change the paging mode.
2167	*/
2168	return pgmR3ChangeMode(pVM, pPGM->enmGuestMode);
2169	}
2170
2171
2172	/**
2173	* Show paging mode.
2174	*
2175	* @param pVM VM Handle.
2176	* @param pHlp The info helpers.
2177	* @param pszArgs "all" (default), "guest", "shadow" or "host".
2178	*/
2179	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2180	{
2181	/* digest argument. */
2182	bool fGuest, fShadow, fHost;
2183	if (pszArgs)
2184	pszArgs = RTStrStripL(pszArgs);
2185	if (!pszArgs \|\| !*pszArgs \|\| strstr(pszArgs, "all"))
2186	fShadow = fHost = fGuest = true;
2187	else
2188	{
2189	fShadow = fHost = fGuest = false;
2190	if (strstr(pszArgs, "guest"))
2191	fGuest = true;
2192	if (strstr(pszArgs, "shadow"))
2193	fShadow = true;
2194	if (strstr(pszArgs, "host"))
2195	fHost = true;
2196	}
2197
2198	/* print info. */
2199	if (fGuest)
2200	pHlp->pfnPrintf(pHlp, "Guest paging mode: %s, changed %RU64 times, A20 %s\n",
2201	PGMGetModeName(pVM->pgm.s.enmGuestMode), pVM->pgm.s.cGuestModeChanges.c,
2202	pVM->pgm.s.fA20Enabled ? "enabled" : "disabled");
2203	if (fShadow)
2204	pHlp->pfnPrintf(pHlp, "Shadow paging mode: %s\n", PGMGetModeName(pVM->pgm.s.enmShadowMode));
2205	if (fHost)
2206	{
2207	const char *psz;
2208	switch (pVM->pgm.s.enmHostMode)
2209	{
2210	case SUPPAGINGMODE_INVALID: psz = "invalid"; break;
2211	case SUPPAGINGMODE_32_BIT: psz = "32-bit"; break;
2212	case SUPPAGINGMODE_32_BIT_GLOBAL: psz = "32-bit+G"; break;
2213	case SUPPAGINGMODE_PAE: psz = "PAE"; break;
2214	case SUPPAGINGMODE_PAE_GLOBAL: psz = "PAE+G"; break;
2215	case SUPPAGINGMODE_PAE_NX: psz = "PAE+NX"; break;
2216	case SUPPAGINGMODE_PAE_GLOBAL_NX: psz = "PAE+G+NX"; break;
2217	case SUPPAGINGMODE_AMD64: psz = "AMD64"; break;
2218	case SUPPAGINGMODE_AMD64_GLOBAL: psz = "AMD64+G"; break;
2219	case SUPPAGINGMODE_AMD64_NX: psz = "AMD64+NX"; break;
2220	case SUPPAGINGMODE_AMD64_GLOBAL_NX: psz = "AMD64+G+NX"; break;
2221	default: psz = "unknown"; break;
2222	}
2223	pHlp->pfnPrintf(pHlp, "Host paging mode: %s\n", psz);
2224	}
2225	}
2226
2227
2228	/**
2229	* Dump registered MMIO ranges to the log.
2230	*
2231	* @param pVM VM Handle.
2232	* @param pHlp The info helpers.
2233	* @param pszArgs Arguments, ignored.
2234	*/
2235	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2236	{
2237	NOREF(pszArgs);
2238	pHlp->pfnPrintf(pHlp,
2239	"RAM ranges (pVM=%p)\n"
2240	"%.s %.s\n",
2241	pVM,
2242	sizeof(RTGCPHYS) * 4 + 1, "GC Phys Range ",
2243	sizeof(RTHCPTR) * 2, "pvHC ");
2244
2245	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesR3; pCur; pCur = pCur->pNextR3)
2246	pHlp->pfnPrintf(pHlp,
2247	"%RGp-%RGp %RHv %s\n",
2248	pCur->GCPhys,
2249	pCur->GCPhysLast,
2250	pCur->pvHC,
2251	pCur->pszDesc);
2252	}
2253
2254	/**
2255	* Dump the page directory to the log.
2256	*
2257	* @param pVM VM Handle.
2258	* @param pHlp The info helpers.
2259	* @param pszArgs Arguments, ignored.
2260	*/
2261	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2262	{
2263	/** @todo fix this! Convert the PGMR3DumpHierarchyHC functions to do guest stuff. */
2264	/* Big pages supported? */
2265	const bool fPSE = !!(CPUMGetGuestCR4(pVM) & X86_CR4_PSE);
2266	/* Global pages supported? */
2267	const bool fPGE = !!(CPUMGetGuestCR4(pVM) & X86_CR4_PGE);
2268
2269	NOREF(pszArgs);
2270
2271	/*
2272	* Get page directory addresses.
2273	*/
2274	PX86PD pPDSrc = pVM->pgm.s.pGuestPDHC;
2275	Assert(pPDSrc);
2276	Assert(MMPhysGCPhys2HCVirt(pVM, (RTGCPHYS)(CPUMGetGuestCR3(pVM) & X86_CR3_PAGE_MASK), sizeof(*pPDSrc)) == pPDSrc);
2277
2278	/*
2279	* Iterate the page directory.
2280	*/
2281	for (unsigned iPD = 0; iPD < ELEMENTS(pPDSrc->a); iPD++)
2282	{
2283	X86PDE PdeSrc = pPDSrc->a[iPD];
2284	if (PdeSrc.n.u1Present)
2285	{
2286	if (PdeSrc.b.u1Size && fPSE)
2287	{
2288	pHlp->pfnPrintf(pHlp,
2289	"%04X - %VGp P=%d U=%d RW=%d G=%d - BIG\n",
2290	iPD,
2291	PdeSrc.u & X86_PDE_PG_MASK,
2292	PdeSrc.b.u1Present, PdeSrc.b.u1User, PdeSrc.b.u1Write, PdeSrc.b.u1Global && fPGE);
2293	}
2294	else
2295	{
2296	pHlp->pfnPrintf(pHlp,
2297	"%04X - %VGp P=%d U=%d RW=%d [G=%d]\n",
2298	iPD,
2299	PdeSrc.u & X86_PDE4M_PG_MASK,
2300	PdeSrc.n.u1Present, PdeSrc.n.u1User, PdeSrc.n.u1Write, PdeSrc.b.u1Global && fPGE);
2301	}
2302	}
2303	}
2304	}
2305
2306
2307	/**
2308	* Serivce a VMMCALLHOST_PGM_LOCK call.
2309	*
2310	* @returns VBox status code.
2311	* @param pVM The VM handle.
2312	*/
2313	PDMR3DECL(int) PGMR3LockCall(PVM pVM)
2314	{
2315	return pgmLock(pVM);
2316	}
2317
2318
2319	/**
2320	* Converts a PGMMODE value to a PGM_TYPE_* \#define.
2321	*
2322	* @returns PGM_TYPE_*.
2323	* @param pgmMode The mode value to convert.
2324	*/
2325	DECLINLINE(unsigned) pgmModeToType(PGMMODE pgmMode)
2326	{
2327	switch (pgmMode)
2328	{
2329	case PGMMODE_REAL: return PGM_TYPE_REAL;
2330	case PGMMODE_PROTECTED: return PGM_TYPE_PROT;
2331	case PGMMODE_32_BIT: return PGM_TYPE_32BIT;
2332	case PGMMODE_PAE:
2333	case PGMMODE_PAE_NX: return PGM_TYPE_PAE;
2334	case PGMMODE_AMD64:
2335	case PGMMODE_AMD64_NX: return PGM_TYPE_AMD64;
2336	default:
2337	AssertFatalMsgFailed(("pgmMode=%d\n", pgmMode));
2338	}
2339	}
2340
2341
2342	/**
2343	* Gets the index into the paging mode data array of a SHW+GST mode.
2344	*
2345	* @returns PGM::paPagingData index.
2346	* @param uShwType The shadow paging mode type.
2347	* @param uGstType The guest paging mode type.
2348	*/
2349	DECLINLINE(unsigned) pgmModeDataIndex(unsigned uShwType, unsigned uGstType)
2350	{
2351	Assert(uShwType >= PGM_TYPE_32BIT && uShwType <= PGM_TYPE_AMD64);
2352	Assert(uGstType >= PGM_TYPE_REAL && uGstType <= PGM_TYPE_AMD64);
2353	return (uShwType - PGM_TYPE_32BIT) * (PGM_TYPE_AMD64 - PGM_TYPE_32BIT + 1)
2354	+ (uGstType - PGM_TYPE_REAL);
2355	}
2356
2357
2358	/**
2359	* Gets the index into the paging mode data array of a SHW+GST mode.
2360	*
2361	* @returns PGM::paPagingData index.
2362	* @param enmShw The shadow paging mode.
2363	* @param enmGst The guest paging mode.
2364	*/
2365	DECLINLINE(unsigned) pgmModeDataIndexByMode(PGMMODE enmShw, PGMMODE enmGst)
2366	{
2367	Assert(enmShw >= PGMMODE_32_BIT && enmShw <= PGMMODE_MAX);
2368	Assert(enmGst > PGMMODE_INVALID && enmGst < PGMMODE_MAX);
2369	return pgmModeDataIndex(pgmModeToType(enmShw), pgmModeToType(enmGst));
2370	}
2371
2372
2373	/**
2374	* Calculates the max data index.
2375	* @returns The number of entries in the pagaing data array.
2376	*/
2377	DECLINLINE(unsigned) pgmModeDataMaxIndex(void)
2378	{
2379	return pgmModeDataIndex(PGM_TYPE_AMD64, PGM_TYPE_AMD64) + 1;
2380	}
2381
2382
2383	/**
2384	* Initializes the paging mode data kept in PGM::paModeData.
2385	*
2386	* @param pVM The VM handle.
2387	* @param fResolveGCAndR0 Indicate whether or not GC and Ring-0 symbols can be resolved now.
2388	* This is used early in the init process to avoid trouble with PDM
2389	* not being initialized yet.
2390	*/
2391	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0)
2392	{
2393	PPGMMODEDATA pModeData;
2394	int rc;
2395
2396	/*
2397	* Allocate the array on the first call.
2398	*/
2399	if (!pVM->pgm.s.paModeData)
2400	{
2401	pVM->pgm.s.paModeData = (PPGMMODEDATA)MMR3HeapAllocZ(pVM, MM_TAG_PGM, sizeof(PGMMODEDATA) * pgmModeDataMaxIndex());
2402	AssertReturn(pVM->pgm.s.paModeData, VERR_NO_MEMORY);
2403	}
2404
2405	/*
2406	* Initialize the array entries.
2407	*/
2408	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_REAL)];
2409	pModeData->uShwType = PGM_TYPE_32BIT;
2410	pModeData->uGstType = PGM_TYPE_REAL;
2411	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2412	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2413	rc = PGM_BTH_NAME_32BIT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2414
2415	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGMMODE_PROTECTED)];
2416	pModeData->uShwType = PGM_TYPE_32BIT;
2417	pModeData->uGstType = PGM_TYPE_PROT;
2418	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2419	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2420	rc = PGM_BTH_NAME_32BIT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2421
2422	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_32BIT)];
2423	pModeData->uShwType = PGM_TYPE_32BIT;
2424	pModeData->uGstType = PGM_TYPE_32BIT;
2425	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2426	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2427	rc = PGM_BTH_NAME_32BIT_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2428
2429	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_REAL)];
2430	pModeData->uShwType = PGM_TYPE_PAE;
2431	pModeData->uGstType = PGM_TYPE_REAL;
2432	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2433	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2434	rc = PGM_BTH_NAME_PAE_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2435
2436	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PROT)];
2437	pModeData->uShwType = PGM_TYPE_PAE;
2438	pModeData->uGstType = PGM_TYPE_PROT;
2439	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2440	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2441	rc = PGM_BTH_NAME_PAE_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2442
2443	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_32BIT)];
2444	pModeData->uShwType = PGM_TYPE_PAE;
2445	pModeData->uGstType = PGM_TYPE_32BIT;
2446	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2447	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2448	rc = PGM_BTH_NAME_PAE_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2449
2450	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PAE)];
2451	pModeData->uShwType = PGM_TYPE_PAE;
2452	pModeData->uGstType = PGM_TYPE_PAE;
2453	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2454	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2455	rc = PGM_BTH_NAME_PAE_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2456
2457	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_AMD64, PGM_TYPE_AMD64)];
2458	pModeData->uShwType = PGM_TYPE_AMD64;
2459	pModeData->uGstType = PGM_TYPE_AMD64;
2460	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2461	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2462
2463	return VINF_SUCCESS;
2464	}
2465
2466
2467	/**
2468	* Swtich to different (or relocated in the relocate case) mode data.
2469	*
2470	* @param pVM The VM handle.
2471	* @param enmShw The the shadow paging mode.
2472	* @param enmGst The the guest paging mode.
2473	*/
2474	static void pgmR3ModeDataSwitch(PVM pVM, PGMMODE enmShw, PGMMODE enmGst)
2475	{
2476	PPGMMODEDATA pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(enmShw, enmGst)];
2477
2478	Assert(pModeData->uGstType == pgmModeToType(enmGst));
2479	Assert(pModeData->uShwType == pgmModeToType(enmShw));
2480
2481	/* shadow */
2482	pVM->pgm.s.pfnR3ShwRelocate = pModeData->pfnR3ShwRelocate;
2483	pVM->pgm.s.pfnR3ShwExit = pModeData->pfnR3ShwExit;
2484	pVM->pgm.s.pfnR3ShwGetPage = pModeData->pfnR3ShwGetPage;
2485	Assert(pVM->pgm.s.pfnR3ShwGetPage);
2486	pVM->pgm.s.pfnR3ShwModifyPage = pModeData->pfnR3ShwModifyPage;
2487	pVM->pgm.s.pfnR3ShwGetPDEByIndex = pModeData->pfnR3ShwGetPDEByIndex;
2488	pVM->pgm.s.pfnR3ShwSetPDEByIndex = pModeData->pfnR3ShwSetPDEByIndex;
2489	pVM->pgm.s.pfnR3ShwModifyPDEByIndex = pModeData->pfnR3ShwModifyPDEByIndex;
2490
2491	pVM->pgm.s.pfnGCShwGetPage = pModeData->pfnGCShwGetPage;
2492	pVM->pgm.s.pfnGCShwModifyPage = pModeData->pfnGCShwModifyPage;
2493	pVM->pgm.s.pfnGCShwGetPDEByIndex = pModeData->pfnGCShwGetPDEByIndex;
2494	pVM->pgm.s.pfnGCShwSetPDEByIndex = pModeData->pfnGCShwSetPDEByIndex;
2495	pVM->pgm.s.pfnGCShwModifyPDEByIndex = pModeData->pfnGCShwModifyPDEByIndex;
2496
2497	pVM->pgm.s.pfnR0ShwGetPage = pModeData->pfnR0ShwGetPage;
2498	pVM->pgm.s.pfnR0ShwModifyPage = pModeData->pfnR0ShwModifyPage;
2499	pVM->pgm.s.pfnR0ShwGetPDEByIndex = pModeData->pfnR0ShwGetPDEByIndex;
2500	pVM->pgm.s.pfnR0ShwSetPDEByIndex = pModeData->pfnR0ShwSetPDEByIndex;
2501	pVM->pgm.s.pfnR0ShwModifyPDEByIndex = pModeData->pfnR0ShwModifyPDEByIndex;
2502
2503
2504	/* guest */
2505	pVM->pgm.s.pfnR3GstRelocate = pModeData->pfnR3GstRelocate;
2506	pVM->pgm.s.pfnR3GstExit = pModeData->pfnR3GstExit;
2507	pVM->pgm.s.pfnR3GstGetPage = pModeData->pfnR3GstGetPage;
2508	Assert(pVM->pgm.s.pfnR3GstGetPage);
2509	pVM->pgm.s.pfnR3GstModifyPage = pModeData->pfnR3GstModifyPage;
2510	pVM->pgm.s.pfnR3GstGetPDE = pModeData->pfnR3GstGetPDE;
2511	pVM->pgm.s.pfnR3GstMonitorCR3 = pModeData->pfnR3GstMonitorCR3;
2512	pVM->pgm.s.pfnR3GstUnmonitorCR3 = pModeData->pfnR3GstUnmonitorCR3;
2513	pVM->pgm.s.pfnR3GstMapCR3 = pModeData->pfnR3GstMapCR3;
2514	pVM->pgm.s.pfnR3GstUnmapCR3 = pModeData->pfnR3GstUnmapCR3;
2515	pVM->pgm.s.pfnR3GstWriteHandlerCR3 = pModeData->pfnR3GstWriteHandlerCR3;
2516	pVM->pgm.s.pszR3GstWriteHandlerCR3 = pModeData->pszR3GstWriteHandlerCR3;
2517	pVM->pgm.s.pfnR3GstPAEWriteHandlerCR3 = pModeData->pfnR3GstPAEWriteHandlerCR3;
2518	pVM->pgm.s.pszR3GstPAEWriteHandlerCR3 = pModeData->pszR3GstPAEWriteHandlerCR3;
2519
2520	pVM->pgm.s.pfnGCGstGetPage = pModeData->pfnGCGstGetPage;
2521	pVM->pgm.s.pfnGCGstModifyPage = pModeData->pfnGCGstModifyPage;
2522	pVM->pgm.s.pfnGCGstGetPDE = pModeData->pfnGCGstGetPDE;
2523	pVM->pgm.s.pfnGCGstMonitorCR3 = pModeData->pfnGCGstMonitorCR3;
2524	pVM->pgm.s.pfnGCGstUnmonitorCR3 = pModeData->pfnGCGstUnmonitorCR3;
2525	pVM->pgm.s.pfnGCGstMapCR3 = pModeData->pfnGCGstMapCR3;
2526	pVM->pgm.s.pfnGCGstUnmapCR3 = pModeData->pfnGCGstUnmapCR3;
2527	pVM->pgm.s.pfnGCGstWriteHandlerCR3 = pModeData->pfnGCGstWriteHandlerCR3;
2528	pVM->pgm.s.pfnGCGstPAEWriteHandlerCR3 = pModeData->pfnGCGstPAEWriteHandlerCR3;
2529
2530	pVM->pgm.s.pfnR0GstGetPage = pModeData->pfnR0GstGetPage;
2531	pVM->pgm.s.pfnR0GstModifyPage = pModeData->pfnR0GstModifyPage;
2532	pVM->pgm.s.pfnR0GstGetPDE = pModeData->pfnR0GstGetPDE;
2533	pVM->pgm.s.pfnR0GstMonitorCR3 = pModeData->pfnR0GstMonitorCR3;
2534	pVM->pgm.s.pfnR0GstUnmonitorCR3 = pModeData->pfnR0GstUnmonitorCR3;
2535	pVM->pgm.s.pfnR0GstMapCR3 = pModeData->pfnR0GstMapCR3;
2536	pVM->pgm.s.pfnR0GstUnmapCR3 = pModeData->pfnR0GstUnmapCR3;
2537	pVM->pgm.s.pfnR0GstWriteHandlerCR3 = pModeData->pfnR0GstWriteHandlerCR3;
2538	pVM->pgm.s.pfnR0GstPAEWriteHandlerCR3 = pModeData->pfnR0GstPAEWriteHandlerCR3;
2539
2540
2541	/* both */
2542	pVM->pgm.s.pfnR3BthRelocate = pModeData->pfnR3BthRelocate;
2543	pVM->pgm.s.pfnR3BthTrap0eHandler = pModeData->pfnR3BthTrap0eHandler;
2544	pVM->pgm.s.pfnR3BthInvalidatePage = pModeData->pfnR3BthInvalidatePage;
2545	pVM->pgm.s.pfnR3BthSyncCR3 = pModeData->pfnR3BthSyncCR3;
2546	Assert(pVM->pgm.s.pfnR3BthSyncCR3);
2547	pVM->pgm.s.pfnR3BthSyncPage = pModeData->pfnR3BthSyncPage;
2548	pVM->pgm.s.pfnR3BthPrefetchPage = pModeData->pfnR3BthPrefetchPage;
2549	pVM->pgm.s.pfnR3BthVerifyAccessSyncPage = pModeData->pfnR3BthVerifyAccessSyncPage;
2550	#ifdef VBOX_STRICT
2551	pVM->pgm.s.pfnR3BthAssertCR3 = pModeData->pfnR3BthAssertCR3;
2552	#endif
2553
2554	pVM->pgm.s.pfnGCBthTrap0eHandler = pModeData->pfnGCBthTrap0eHandler;
2555	pVM->pgm.s.pfnGCBthInvalidatePage = pModeData->pfnGCBthInvalidatePage;
2556	pVM->pgm.s.pfnGCBthSyncCR3 = pModeData->pfnGCBthSyncCR3;
2557	pVM->pgm.s.pfnGCBthSyncPage = pModeData->pfnGCBthSyncPage;
2558	pVM->pgm.s.pfnGCBthPrefetchPage = pModeData->pfnGCBthPrefetchPage;
2559	pVM->pgm.s.pfnGCBthVerifyAccessSyncPage = pModeData->pfnGCBthVerifyAccessSyncPage;
2560	#ifdef VBOX_STRICT
2561	pVM->pgm.s.pfnGCBthAssertCR3 = pModeData->pfnGCBthAssertCR3;
2562	#endif
2563
2564	pVM->pgm.s.pfnR0BthTrap0eHandler = pModeData->pfnR0BthTrap0eHandler;
2565	pVM->pgm.s.pfnR0BthInvalidatePage = pModeData->pfnR0BthInvalidatePage;
2566	pVM->pgm.s.pfnR0BthSyncCR3 = pModeData->pfnR0BthSyncCR3;
2567	pVM->pgm.s.pfnR0BthSyncPage = pModeData->pfnR0BthSyncPage;
2568	pVM->pgm.s.pfnR0BthPrefetchPage = pModeData->pfnR0BthPrefetchPage;
2569	pVM->pgm.s.pfnR0BthVerifyAccessSyncPage = pModeData->pfnR0BthVerifyAccessSyncPage;
2570	#ifdef VBOX_STRICT
2571	pVM->pgm.s.pfnR0BthAssertCR3 = pModeData->pfnR0BthAssertCR3;
2572	#endif
2573	}
2574
2575
2576	#ifdef DEBUG_bird
2577	#include <stdlib.h> /* getenv() remove me! */
2578	#endif
2579
2580	/**
2581	* Calculates the shadow paging mode.
2582	*
2583	* @returns The shadow paging mode.
2584	* @param enmGuestMode The guest mode.
2585	* @param enmHostMode The host mode.
2586	* @param enmShadowMode The current shadow mode.
2587	* @param penmSwitcher Where to store the switcher to use.
2588	* VMMSWITCHER_INVALID means no change.
2589	*/
2590	static PGMMODE pgmR3CalcShadowMode(PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher)
2591	{
2592	VMMSWITCHER enmSwitcher = VMMSWITCHER_INVALID;
2593	switch (enmGuestMode)
2594	{
2595	/*
2596	* When switching to real or protected mode we don't change
2597	* anything since it's likely that we'll switch back pretty soon.
2598	*
2599	* During pgmR3InitPaging we'll end up here with PGMMODE_INVALID
2600	* and is supposed to determin which shadow paging and switcher to
2601	* use during init.
2602	*/
2603	case PGMMODE_REAL:
2604	case PGMMODE_PROTECTED:
2605	if (enmShadowMode != PGMMODE_INVALID)
2606	break; /* (no change) */
2607	switch (enmHostMode)
2608	{
2609	case SUPPAGINGMODE_32_BIT:
2610	case SUPPAGINGMODE_32_BIT_GLOBAL:
2611	enmShadowMode = PGMMODE_32_BIT;
2612	enmSwitcher = VMMSWITCHER_32_TO_32;
2613	break;
2614
2615	case SUPPAGINGMODE_PAE:
2616	case SUPPAGINGMODE_PAE_NX:
2617	case SUPPAGINGMODE_PAE_GLOBAL:
2618	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2619	enmShadowMode = PGMMODE_PAE;
2620	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
2621	#ifdef DEBUG_bird
2622	if (getenv("VBOX_32BIT"))
2623	{
2624	enmShadowMode = PGMMODE_32_BIT;
2625	enmSwitcher = VMMSWITCHER_PAE_TO_32;
2626	}
2627	#endif
2628	break;
2629
2630	case SUPPAGINGMODE_AMD64:
2631	case SUPPAGINGMODE_AMD64_GLOBAL:
2632	case SUPPAGINGMODE_AMD64_NX:
2633	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2634	enmShadowMode = PGMMODE_PAE;
2635	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
2636	break;
2637
2638	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
2639	}
2640	break;
2641
2642	case PGMMODE_32_BIT:
2643	switch (enmHostMode)
2644	{
2645	case SUPPAGINGMODE_32_BIT:
2646	case SUPPAGINGMODE_32_BIT_GLOBAL:
2647	enmShadowMode = PGMMODE_32_BIT;
2648	enmSwitcher = VMMSWITCHER_32_TO_32;
2649	break;
2650
2651	case SUPPAGINGMODE_PAE:
2652	case SUPPAGINGMODE_PAE_NX:
2653	case SUPPAGINGMODE_PAE_GLOBAL:
2654	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2655	enmShadowMode = PGMMODE_PAE;
2656	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
2657	#ifdef DEBUG_bird
2658	if (getenv("VBOX_32BIT"))
2659	{
2660	enmShadowMode = PGMMODE_32_BIT;
2661	enmSwitcher = VMMSWITCHER_PAE_TO_32;
2662	}
2663	#endif
2664	break;
2665
2666	case SUPPAGINGMODE_AMD64:
2667	case SUPPAGINGMODE_AMD64_GLOBAL:
2668	case SUPPAGINGMODE_AMD64_NX:
2669	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2670	enmShadowMode = PGMMODE_PAE;
2671	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
2672	break;
2673
2674	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
2675	}
2676	break;
2677
2678	case PGMMODE_PAE:
2679	case PGMMODE_PAE_NX: /** @todo This might require more switchers and guest+both modes. */
2680	switch (enmHostMode)
2681	{
2682	case SUPPAGINGMODE_32_BIT:
2683	case SUPPAGINGMODE_32_BIT_GLOBAL:
2684	enmShadowMode = PGMMODE_PAE;
2685	enmSwitcher = VMMSWITCHER_32_TO_PAE;
2686	break;
2687
2688	case SUPPAGINGMODE_PAE:
2689	case SUPPAGINGMODE_PAE_NX:
2690	case SUPPAGINGMODE_PAE_GLOBAL:
2691	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2692	enmShadowMode = PGMMODE_PAE;
2693	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
2694	break;
2695
2696	case SUPPAGINGMODE_AMD64:
2697	case SUPPAGINGMODE_AMD64_GLOBAL:
2698	case SUPPAGINGMODE_AMD64_NX:
2699	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2700	enmShadowMode = PGMMODE_PAE;
2701	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
2702	break;
2703
2704	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
2705	}
2706	break;
2707
2708	case PGMMODE_AMD64:
2709	case PGMMODE_AMD64_NX:
2710	switch (enmHostMode)
2711	{
2712	case SUPPAGINGMODE_32_BIT:
2713	case SUPPAGINGMODE_32_BIT_GLOBAL:
2714	enmShadowMode = PGMMODE_PAE;
2715	enmSwitcher = VMMSWITCHER_32_TO_AMD64;
2716	break;
2717
2718	case SUPPAGINGMODE_PAE:
2719	case SUPPAGINGMODE_PAE_NX:
2720	case SUPPAGINGMODE_PAE_GLOBAL:
2721	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2722	enmShadowMode = PGMMODE_PAE;
2723	enmSwitcher = VMMSWITCHER_PAE_TO_AMD64;
2724	break;
2725
2726	case SUPPAGINGMODE_AMD64:
2727	case SUPPAGINGMODE_AMD64_GLOBAL:
2728	case SUPPAGINGMODE_AMD64_NX:
2729	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2730	enmShadowMode = PGMMODE_AMD64;
2731	enmSwitcher = VMMSWITCHER_AMD64_TO_AMD64;
2732	break;
2733
2734	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
2735	}
2736	break;
2737
2738
2739	default:
2740	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
2741	return PGMMODE_INVALID;
2742	}
2743
2744	*penmSwitcher = enmSwitcher;
2745	return enmShadowMode;
2746	}
2747
2748
2749	/**
2750	* Performs the actual mode change.
2751	* This is called by PGMChangeMode and pgmR3InitPaging().
2752	*
2753	* @returns VBox status code.
2754	* @param pVM VM handle.
2755	* @param enmGuestMode The new guest mode. This is assumed to be different from
2756	* the current mode.
2757	*/
2758	int pgmR3ChangeMode(PVM pVM, PGMMODE enmGuestMode)
2759	{
2760	LogFlow(("pgmR3ChangeMode: Guest mode: %d -> %d\n", pVM->pgm.s.enmGuestMode, enmGuestMode));
2761	STAM_REL_COUNTER_INC(&pVM->pgm.s.cGuestModeChanges);
2762
2763	/*
2764	* Calc the shadow mode and switcher.
2765	*/
2766	VMMSWITCHER enmSwitcher;
2767	PGMMODE enmShadowMode = pgmR3CalcShadowMode(enmGuestMode, pVM->pgm.s.enmHostMode, pVM->pgm.s.enmShadowMode, &enmSwitcher);
2768	if (enmSwitcher != VMMSWITCHER_INVALID)
2769	{
2770	/*
2771	* Select new switcher.
2772	*/
2773	int rc = VMMR3SelectSwitcher(pVM, enmSwitcher);
2774	if (VBOX_FAILURE(rc))
2775	{
2776	AssertReleaseMsgFailed(("VMMR3SelectSwitcher(%d) -> %Vrc\n", enmSwitcher, rc));
2777	return rc;
2778	}
2779	}
2780
2781	/*
2782	* Exit old mode(s).
2783	*/
2784	/* shadow */
2785	if (enmShadowMode != pVM->pgm.s.enmShadowMode)
2786	{
2787	LogFlow(("pgmR3ChangeMode: Shadow mode: %d -> %d\n", pVM->pgm.s.enmShadowMode, enmShadowMode));
2788	if (PGM_SHW_PFN(Exit, pVM))
2789	{
2790	int rc = PGM_SHW_PFN(Exit, pVM)(pVM);
2791	if (VBOX_FAILURE(rc))
2792	{
2793	AssertMsgFailed(("Exit failed for shadow mode %d: %Vrc\n", pVM->pgm.s.enmShadowMode, rc));
2794	return rc;
2795	}
2796	}
2797
2798	}
2799
2800	/* guest */
2801	if (PGM_GST_PFN(Exit, pVM))
2802	{
2803	int rc = PGM_GST_PFN(Exit, pVM)(pVM);
2804	if (VBOX_FAILURE(rc))
2805	{
2806	AssertMsgFailed(("Exit failed for guest mode %d: %Vrc\n", pVM->pgm.s.enmGuestMode, rc));
2807	return rc;
2808	}
2809	}
2810
2811	/*
2812	* Load new paging mode data.
2813	*/
2814	pgmR3ModeDataSwitch(pVM, enmShadowMode, enmGuestMode);
2815
2816	/*
2817	* Enter new shadow mode (if changed).
2818	*/
2819	if (enmShadowMode != pVM->pgm.s.enmShadowMode)
2820	{
2821	int rc;
2822	pVM->pgm.s.enmShadowMode = enmShadowMode;
2823	switch (enmShadowMode)
2824	{
2825	case PGMMODE_32_BIT:
2826	rc = PGM_SHW_NAME_32BIT(Enter)(pVM);
2827	break;
2828	case PGMMODE_PAE:
2829	case PGMMODE_PAE_NX:
2830	rc = PGM_SHW_NAME_PAE(Enter)(pVM);
2831	break;
2832	case PGMMODE_AMD64:
2833	case PGMMODE_AMD64_NX:
2834	rc = PGM_SHW_NAME_AMD64(Enter)(pVM);
2835	break;
2836	case PGMMODE_REAL:
2837	case PGMMODE_PROTECTED:
2838	default:
2839	AssertReleaseMsgFailed(("enmShadowMode=%d\n", enmShadowMode));
2840	return VERR_INTERNAL_ERROR;
2841	}
2842	if (VBOX_FAILURE(rc))
2843	{
2844	AssertReleaseMsgFailed(("Entering enmShadowMode=%d failed: %Vrc\n", enmShadowMode, rc));
2845	pVM->pgm.s.enmShadowMode = PGMMODE_INVALID;
2846	return rc;
2847	}
2848	}
2849
2850	/*
2851	* Enter the new guest and shadow+guest modes.
2852	*/
2853	int rc = -1;
2854	int rc2 = -1;
2855	RTGCPHYS GCPhysCR3 = NIL_RTGCPHYS;
2856	pVM->pgm.s.enmGuestMode = enmGuestMode;
2857	switch (enmGuestMode)
2858	{
2859	case PGMMODE_REAL:
2860	rc = PGM_GST_NAME_REAL(Enter)(pVM, NIL_RTGCPHYS);
2861	switch (pVM->pgm.s.enmShadowMode)
2862	{
2863	case PGMMODE_32_BIT:
2864	rc2 = PGM_BTH_NAME_32BIT_REAL(Enter)(pVM, NIL_RTGCPHYS);
2865	break;
2866	case PGMMODE_PAE:
2867	case PGMMODE_PAE_NX:
2868	rc2 = PGM_BTH_NAME_PAE_REAL(Enter)(pVM, NIL_RTGCPHYS);
2869	break;
2870	case PGMMODE_AMD64:
2871	case PGMMODE_AMD64_NX:
2872	AssertMsgFailed(("Should use PAE shadow mode!\n"));
2873	default: AssertFailed(); break;
2874	}
2875	break;
2876
2877	case PGMMODE_PROTECTED:
2878	rc = PGM_GST_NAME_PROT(Enter)(pVM, NIL_RTGCPHYS);
2879	switch (pVM->pgm.s.enmShadowMode)
2880	{
2881	case PGMMODE_32_BIT:
2882	rc2 = PGM_BTH_NAME_32BIT_PROT(Enter)(pVM, NIL_RTGCPHYS);
2883	break;
2884	case PGMMODE_PAE:
2885	case PGMMODE_PAE_NX:
2886	rc2 = PGM_BTH_NAME_PAE_PROT(Enter)(pVM, NIL_RTGCPHYS);
2887	break;
2888	case PGMMODE_AMD64:
2889	case PGMMODE_AMD64_NX:
2890	AssertMsgFailed(("Should use PAE shadow mode!\n"));
2891	default: AssertFailed(); break;
2892	}
2893	break;
2894
2895	case PGMMODE_32_BIT:
2896	GCPhysCR3 = CPUMGetGuestCR3(pVM) & X86_CR3_PAGE_MASK;
2897	rc = PGM_GST_NAME_32BIT(Enter)(pVM, GCPhysCR3);
2898	switch (pVM->pgm.s.enmShadowMode)
2899	{
2900	case PGMMODE_32_BIT:
2901	rc2 = PGM_BTH_NAME_32BIT_32BIT(Enter)(pVM, GCPhysCR3);
2902	break;
2903	case PGMMODE_PAE:
2904	case PGMMODE_PAE_NX:
2905	rc2 = PGM_BTH_NAME_PAE_32BIT(Enter)(pVM, GCPhysCR3);
2906	break;
2907	case PGMMODE_AMD64:
2908	case PGMMODE_AMD64_NX:
2909	AssertMsgFailed(("Should use PAE shadow mode!\n"));
2910	default: AssertFailed(); break;
2911	}
2912	break;
2913
2914	//case PGMMODE_PAE_NX:
2915	case PGMMODE_PAE:
2916	{
2917	uint32_t u32Dummy, u32Features;
2918
2919	CPUMGetGuestCpuId(pVM, 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
2920	if (!(u32Features & X86_CPUID_FEATURE_EDX_PAE))
2921	{
2922	/* Pause first, then inform Main. */
2923	rc = VMR3SuspendNoSave(pVM);
2924	AssertRC(rc);
2925
2926	VMSetRuntimeError(pVM, true, "PAEmode",
2927	N_("The guest is trying to switch to the PAE mode which is currently disabled by default in VirtualBox. Experimental PAE support can be enabled using the -pae option with VBoxManage."));
2928	/* we must return TRUE here otherwise the recompiler will assert */
2929	return VINF_SUCCESS;
2930	}
2931	GCPhysCR3 = CPUMGetGuestCR3(pVM) & X86_CR3_PAE_PAGE_MASK;
2932	rc = PGM_GST_NAME_PAE(Enter)(pVM, GCPhysCR3);
2933	switch (pVM->pgm.s.enmShadowMode)
2934	{
2935	case PGMMODE_PAE:
2936	case PGMMODE_PAE_NX:
2937	rc2 = PGM_BTH_NAME_PAE_PAE(Enter)(pVM, GCPhysCR3);
2938	break;
2939	case PGMMODE_32_BIT:
2940	case PGMMODE_AMD64:
2941	case PGMMODE_AMD64_NX:
2942	AssertMsgFailed(("Should use PAE shadow mode!\n"));
2943	default: AssertFailed(); break;
2944	}
2945	break;
2946	}
2947
2948	//case PGMMODE_AMD64_NX:
2949	case PGMMODE_AMD64:
2950	GCPhysCR3 = CPUMGetGuestCR3(pVM) & 0xfffffffffffff000ULL; /** @todo define this mask and make CR3 64-bit in this case! */
2951	rc = PGM_GST_NAME_AMD64(Enter)(pVM, GCPhysCR3);
2952	switch (pVM->pgm.s.enmShadowMode)
2953	{
2954	case PGMMODE_AMD64:
2955	case PGMMODE_AMD64_NX:
2956	rc2 = PGM_BTH_NAME_AMD64_AMD64(Enter)(pVM, GCPhysCR3);
2957	break;
2958	case PGMMODE_32_BIT:
2959	case PGMMODE_PAE:
2960	case PGMMODE_PAE_NX:
2961	AssertMsgFailed(("Should use AMD64 shadow mode!\n"));
2962	default: AssertFailed(); break;
2963	}
2964	break;
2965
2966	default:
2967	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
2968	rc = VERR_NOT_IMPLEMENTED;
2969	break;
2970	}
2971
2972	/* status codes. */
2973	AssertRC(rc);
2974	AssertRC(rc2);
2975	if (VBOX_SUCCESS(rc))
2976	{
2977	rc = rc2;
2978	if (VBOX_SUCCESS(rc)) /* no informational status codes. */
2979	rc = VINF_SUCCESS;
2980	}
2981
2982	/*
2983	* Notify SELM so it can update the TSSes with correct CR3s.
2984	*/
2985	SELMR3PagingModeChanged(pVM);
2986
2987	/* Notify HWACCM as well. */
2988	HWACCMR3PagingModeChanged(pVM, pVM->pgm.s.enmShadowMode);
2989	return rc;
2990	}
2991
2992
2993	/**
2994	* Dumps a PAE shadow page table.
2995	*
2996	* @returns VBox status code (VINF_SUCCESS).
2997	* @param pVM The VM handle.
2998	* @param pPT Pointer to the page table.
2999	* @param u64Address The virtual address of the page table starts.
3000	* @param fLongMode Set if this a long mode table; clear if it's a legacy mode table.
3001	* @param cMaxDepth The maxium depth.
3002	* @param pHlp Pointer to the output functions.
3003	*/
3004	static int pgmR3DumpHierarchyHCPaePT(PVM pVM, PX86PTPAE pPT, uint64_t u64Address, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3005	{
3006	for (unsigned i = 0; i < ELEMENTS(pPT->a); i++)
3007	{
3008	X86PTEPAE Pte = pPT->a[i];
3009	if (Pte.n.u1Present)
3010	{
3011	pHlp->pfnPrintf(pHlp,
3012	fLongMode /P R S A D G WT CD AT NX 4M a p ? /
3013	? "%016llx 3 \| P %c %c %c %c %c %s %s %s %s 4K %c%c%c %016llx\n"
3014	: "%08llx 2 \| P %c %c %c %c %c %s %s %s %s 4K %c%c%c %016llx\n",
3015	u64Address + ((uint64_t)i << X86_PT_PAE_SHIFT),
3016	Pte.n.u1Write ? 'W' : 'R',
3017	Pte.n.u1User ? 'U' : 'S',
3018	Pte.n.u1Accessed ? 'A' : '-',
3019	Pte.n.u1Dirty ? 'D' : '-',
3020	Pte.n.u1Global ? 'G' : '-',
3021	Pte.n.u1WriteThru ? "WT" : "--",
3022	Pte.n.u1CacheDisable? "CD" : "--",
3023	Pte.n.u1PAT ? "AT" : "--",
3024	Pte.n.u1NoExecute ? "NX" : "--",
3025	Pte.u & PGM_PTFLAGS_TRACK_DIRTY ? 'd' : '-',
3026	Pte.u & RT_BIT(10) ? '1' : '0',
3027	Pte.u & PGM_PTFLAGS_CSAM_VALIDATED? 'v' : '-',
3028	Pte.u & X86_PTE_PAE_PG_MASK);
3029	}
3030	}
3031	return VINF_SUCCESS;
3032	}
3033
3034
3035	/**
3036	* Dumps a PAE shadow page directory table.
3037	*
3038	* @returns VBox status code (VINF_SUCCESS).
3039	* @param pVM The VM handle.
3040	* @param HCPhys The physical address of the page directory table.
3041	* @param u64Address The virtual address of the page table starts.
3042	* @param cr4 The CR4, PSE is currently used.
3043	* @param fLongMode Set if this a long mode table; clear if it's a legacy mode table.
3044	* @param cMaxDepth The maxium depth.
3045	* @param pHlp Pointer to the output functions.
3046	*/
3047	static int pgmR3DumpHierarchyHCPaePD(PVM pVM, RTHCPHYS HCPhys, uint64_t u64Address, uint32_t cr4, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3048	{
3049	PX86PDPAE pPD = (PX86PDPAE)MMPagePhys2Page(pVM, HCPhys);
3050	if (!pPD)
3051	{
3052	pHlp->pfnPrintf(pHlp, "%0*llx error! Page directory at HCPhys=%#VHp was not found in the page pool!\n",
3053	fLongMode ? 16 : 8, u64Address, HCPhys);
3054	return VERR_INVALID_PARAMETER;
3055	}
3056	int rc = VINF_SUCCESS;
3057	for (unsigned i = 0; i < ELEMENTS(pPD->a); i++)
3058	{
3059	X86PDEPAE Pde = pPD->a[i];
3060	if (Pde.n.u1Present)
3061	{
3062	if ((cr4 & X86_CR4_PSE) && Pde.b.u1Size)
3063	pHlp->pfnPrintf(pHlp,
3064	fLongMode /P R S A D G WT CD AT NX 4M a p ? /
3065	? "%016llx 2 \| P %c %c %c %c %c %s %s %s %s 4M %c%c%c %016llx\n"
3066	: "%08llx 1 \| P %c %c %c %c %c %s %s %s %s 4M %c%c%c %016llx\n",
3067	u64Address + ((uint64_t)i << X86_PD_PAE_SHIFT),
3068	Pde.b.u1Write ? 'W' : 'R',
3069	Pde.b.u1User ? 'U' : 'S',
3070	Pde.b.u1Accessed ? 'A' : '-',
3071	Pde.b.u1Dirty ? 'D' : '-',
3072	Pde.b.u1Global ? 'G' : '-',
3073	Pde.b.u1WriteThru ? "WT" : "--",
3074	Pde.b.u1CacheDisable? "CD" : "--",
3075	Pde.b.u1PAT ? "AT" : "--",
3076	Pde.b.u1NoExecute ? "NX" : "--",
3077	Pde.u & RT_BIT_64(9) ? '1' : '0',
3078	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
3079	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
3080	Pde.u & X86_PDE_PAE_PG_MASK);
3081	else
3082	{
3083	pHlp->pfnPrintf(pHlp,
3084	fLongMode /P R S A D G WT CD AT NX 4M a p ? /
3085	? "%016llx 2 \| P %c %c %c %c %c %s %s .. %s 4K %c%c%c %016llx\n"
3086	: "%08llx 1 \| P %c %c %c %c %c %s %s .. %s 4K %c%c%c %016llx\n",
3087	u64Address + ((uint64_t)i << X86_PD_PAE_SHIFT),
3088	Pde.n.u1Write ? 'W' : 'R',
3089	Pde.n.u1User ? 'U' : 'S',
3090	Pde.n.u1Accessed ? 'A' : '-',
3091	Pde.n.u1Reserved0 ? '?' : '.', /* ignored */
3092	Pde.n.u1Reserved1 ? '?' : '.', /* ignored */
3093	Pde.n.u1WriteThru ? "WT" : "--",
3094	Pde.n.u1CacheDisable? "CD" : "--",
3095	Pde.n.u1NoExecute ? "NX" : "--",
3096	Pde.u & RT_BIT_64(9) ? '1' : '0',
3097	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
3098	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
3099	Pde.u & X86_PDE_PAE_PG_MASK);
3100	if (cMaxDepth >= 1)
3101	{
3102	/** @todo what about using the page pool for mapping PTs? */
3103	uint64_t u64AddressPT = u64Address + ((uint64_t)i << X86_PD_PAE_SHIFT);
3104	RTHCPHYS HCPhysPT = Pde.u & X86_PDE_PAE_PG_MASK;
3105	PX86PTPAE pPT = NULL;
3106	if (!(Pde.u & PGM_PDFLAGS_MAPPING))
3107	pPT = (PX86PTPAE)MMPagePhys2Page(pVM, HCPhysPT);
3108	else
3109	{
3110	for (PPGMMAPPING pMap = pVM->pgm.s.pMappingsR3; pMap; pMap = pMap->pNextR3)
3111	{
3112	uint64_t off = u64AddressPT - pMap->GCPtr;
3113	if (off < pMap->cb)
3114	{
3115	const int iPDE = (uint32_t)(off >> X86_PD_SHIFT);
3116	const int iSub = (int)((off >> X86_PD_PAE_SHIFT) & 1); /* MSC is a pain sometimes */
3117	if ((iSub ? pMap->aPTs[iPDE].HCPhysPaePT1 : pMap->aPTs[iPDE].HCPhysPaePT0) != HCPhysPT)
3118	pHlp->pfnPrintf(pHlp, "%0*llx error! Mapping error! PT %d has HCPhysPT=%VHp not %VHp is in the PD.\n",
3119	fLongMode ? 16 : 8, u64AddressPT, iPDE,
3120	iSub ? pMap->aPTs[iPDE].HCPhysPaePT1 : pMap->aPTs[iPDE].HCPhysPaePT0, HCPhysPT);
3121	pPT = &pMap->aPTs[iPDE].paPaePTsR3[iSub];
3122	}
3123	}
3124	}
3125	int rc2 = VERR_INVALID_PARAMETER;
3126	if (pPT)
3127	rc2 = pgmR3DumpHierarchyHCPaePT(pVM, pPT, u64AddressPT, fLongMode, cMaxDepth - 1, pHlp);
3128	else
3129	pHlp->pfnPrintf(pHlp, "%0*llx error! Page table at HCPhys=%#VHp was not found in the page pool!\n",
3130	fLongMode ? 16 : 8, u64AddressPT, HCPhysPT);
3131	if (rc2 < rc && VBOX_SUCCESS(rc))
3132	rc = rc2;
3133	}
3134	}
3135	}
3136	}
3137	return rc;
3138	}
3139
3140
3141	/**
3142	* Dumps a PAE shadow page directory pointer table.
3143	*
3144	* @returns VBox status code (VINF_SUCCESS).
3145	* @param pVM The VM handle.
3146	* @param HCPhys The physical address of the page directory pointer table.
3147	* @param u64Address The virtual address of the page table starts.
3148	* @param cr4 The CR4, PSE is currently used.
3149	* @param fLongMode Set if this a long mode table; clear if it's a legacy mode table.
3150	* @param cMaxDepth The maxium depth.
3151	* @param pHlp Pointer to the output functions.
3152	*/
3153	static int pgmR3DumpHierarchyHCPaePDPT(PVM pVM, RTHCPHYS HCPhys, uint64_t u64Address, uint32_t cr4, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3154	{
3155	PX86PDPT pPDPT = (PX86PDPT)MMPagePhys2Page(pVM, HCPhys);
3156	if (!pPDPT)
3157	{
3158	pHlp->pfnPrintf(pHlp, "%0*llx error! Page directory pointer table at HCPhys=%#VHp was not found in the page pool!\n",
3159	fLongMode ? 16 : 8, u64Address, HCPhys);
3160	return VERR_INVALID_PARAMETER;
3161	}
3162
3163	int rc = VINF_SUCCESS;
3164	const unsigned c = fLongMode ? ELEMENTS(pPDPT->a) : X86_PG_PAE_PDPE_ENTRIES;
3165	for (unsigned i = 0; i < c; i++)
3166	{
3167	X86PDPE Pdpe = pPDPT->a[i];
3168	if (Pdpe.n.u1Present)
3169	{
3170	if (fLongMode)
3171	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a p ? /
3172	"%016llx 1 \| P %c %c %c %c %c %s %s %s %s .. %c%c%c %016llx\n",
3173	u64Address + ((uint64_t)i << X86_PDPT_SHIFT),
3174	Pdpe.lm.u1Write ? 'W' : 'R',
3175	Pdpe.lm.u1User ? 'U' : 'S',
3176	Pdpe.lm.u1Accessed ? 'A' : '-',
3177	Pdpe.lm.u3Reserved & 1? '?' : '.', /* ignored */
3178	Pdpe.lm.u3Reserved & 4? '!' : '.', /* mbz */
3179	Pdpe.lm.u1WriteThru ? "WT" : "--",
3180	Pdpe.lm.u1CacheDisable? "CD" : "--",
3181	Pdpe.lm.u3Reserved & 2? "!" : "..",/* mbz */
3182	Pdpe.lm.u1NoExecute ? "NX" : "--",
3183	Pdpe.u & RT_BIT(9) ? '1' : '0',
3184	Pdpe.u & PGM_PLXFLAGS_PERMANENT ? 'p' : '-',
3185	Pdpe.u & RT_BIT(11) ? '1' : '0',
3186	Pdpe.u & X86_PDPE_PG_MASK);
3187	else
3188	pHlp->pfnPrintf(pHlp, /P G WT CD AT NX 4M a p ? /
3189	"%08x 0 \| P %c %s %s %s %s .. %c%c%c %016llx\n",
3190	i << X86_PDPT_SHIFT,
3191	Pdpe.n.u4Reserved & 1? '!' : '.', /* mbz */
3192	Pdpe.n.u4Reserved & 4? '!' : '.', /* mbz */
3193	Pdpe.n.u1WriteThru ? "WT" : "--",
3194	Pdpe.n.u1CacheDisable? "CD" : "--",
3195	Pdpe.n.u4Reserved & 2? "!" : "..",/* mbz */
3196	Pdpe.u & RT_BIT(9) ? '1' : '0',
3197	Pdpe.u & PGM_PLXFLAGS_PERMANENT ? 'p' : '-',
3198	Pdpe.u & RT_BIT(11) ? '1' : '0',
3199	Pdpe.u & X86_PDPE_PG_MASK);
3200	if (cMaxDepth >= 1)
3201	{
3202	int rc2 = pgmR3DumpHierarchyHCPaePD(pVM, Pdpe.u & X86_PDPE_PG_MASK, u64Address + ((uint64_t)i << X86_PDPT_SHIFT),
3203	cr4, fLongMode, cMaxDepth - 1, pHlp);
3204	if (rc2 < rc && VBOX_SUCCESS(rc))
3205	rc = rc2;
3206	}
3207	}
3208	}
3209	return rc;
3210	}
3211
3212
3213	/**
3214	* Dumps a 32-bit shadow page table.
3215	*
3216	* @returns VBox status code (VINF_SUCCESS).
3217	* @param pVM The VM handle.
3218	* @param HCPhys The physical address of the table.
3219	* @param cr4 The CR4, PSE is currently used.
3220	* @param cMaxDepth The maxium depth.
3221	* @param pHlp Pointer to the output functions.
3222	*/
3223	static int pgmR3DumpHierarchyHcPaePML4(PVM pVM, RTHCPHYS HCPhys, uint32_t cr4, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3224	{
3225	PX86PML4 pPML4 = (PX86PML4)MMPagePhys2Page(pVM, HCPhys);
3226	if (!pPML4)
3227	{
3228	pHlp->pfnPrintf(pHlp, "Page map level 4 at HCPhys=%#VHp was not found in the page pool!\n", HCPhys);
3229	return VERR_INVALID_PARAMETER;
3230	}
3231
3232	int rc = VINF_SUCCESS;
3233	for (unsigned i = 0; i < ELEMENTS(pPML4->a); i++)
3234	{
3235	X86PML4E Pml4e = pPML4->a[i];
3236	if (Pml4e.n.u1Present)
3237	{
3238	uint64_t u64Address = ((uint64_t)i << X86_PML4_SHIFT) \| (((uint64_t)i >> (X86_PML4_SHIFT - X86_PDPT_SHIFT - 1)) * 0xffff000000000000ULL);
3239	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a p ? /
3240	"%016llx 0 \| P %c %c %c %c %c %s %s %s %s .. %c%c%c %016llx\n",
3241	u64Address,
3242	Pml4e.n.u1Write ? 'W' : 'R',
3243	Pml4e.n.u1User ? 'U' : 'S',
3244	Pml4e.n.u1Accessed ? 'A' : '-',
3245	Pml4e.n.u3Reserved & 1? '?' : '.', /* ignored */
3246	Pml4e.n.u3Reserved & 4? '!' : '.', /* mbz */
3247	Pml4e.n.u1WriteThru ? "WT" : "--",
3248	Pml4e.n.u1CacheDisable? "CD" : "--",
3249	Pml4e.n.u3Reserved & 2? "!" : "..",/* mbz */
3250	Pml4e.n.u1NoExecute ? "NX" : "--",
3251	Pml4e.u & RT_BIT(9) ? '1' : '0',
3252	Pml4e.u & PGM_PLXFLAGS_PERMANENT ? 'p' : '-',
3253	Pml4e.u & RT_BIT(11) ? '1' : '0',
3254	Pml4e.u & X86_PML4E_PG_MASK);
3255
3256	if (cMaxDepth >= 1)
3257	{
3258	int rc2 = pgmR3DumpHierarchyHCPaePDPT(pVM, Pml4e.u & X86_PML4E_PG_MASK, u64Address, cr4, true, cMaxDepth - 1, pHlp);
3259	if (rc2 < rc && VBOX_SUCCESS(rc))
3260	rc = rc2;
3261	}
3262	}
3263	}
3264	return rc;
3265	}
3266
3267
3268	/**
3269	* Dumps a 32-bit shadow page table.
3270	*
3271	* @returns VBox status code (VINF_SUCCESS).
3272	* @param pVM The VM handle.
3273	* @param pPT Pointer to the page table.
3274	* @param u32Address The virtual address this table starts at.
3275	* @param pHlp Pointer to the output functions.
3276	*/
3277	int pgmR3DumpHierarchyHC32BitPT(PVM pVM, PX86PT pPT, uint32_t u32Address, PCDBGFINFOHLP pHlp)
3278	{
3279	for (unsigned i = 0; i < ELEMENTS(pPT->a); i++)
3280	{
3281	X86PTE Pte = pPT->a[i];
3282	if (Pte.n.u1Present)
3283	{
3284	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a m d /
3285	"%08x 1 \| P %c %c %c %c %c %s %s %s .. 4K %c%c%c %08x\n",
3286	u32Address + (i << X86_PT_SHIFT),
3287	Pte.n.u1Write ? 'W' : 'R',
3288	Pte.n.u1User ? 'U' : 'S',
3289	Pte.n.u1Accessed ? 'A' : '-',
3290	Pte.n.u1Dirty ? 'D' : '-',
3291	Pte.n.u1Global ? 'G' : '-',
3292	Pte.n.u1WriteThru ? "WT" : "--",
3293	Pte.n.u1CacheDisable? "CD" : "--",
3294	Pte.n.u1PAT ? "AT" : "--",
3295	Pte.u & PGM_PTFLAGS_TRACK_DIRTY ? 'd' : '-',
3296	Pte.u & RT_BIT(10) ? '1' : '0',
3297	Pte.u & PGM_PTFLAGS_CSAM_VALIDATED ? 'v' : '-',
3298	Pte.u & X86_PDE_PG_MASK);
3299	}
3300	}
3301	return VINF_SUCCESS;
3302	}
3303
3304
3305	/**
3306	* Dumps a 32-bit shadow page directory and page tables.
3307	*
3308	* @returns VBox status code (VINF_SUCCESS).
3309	* @param pVM The VM handle.
3310	* @param cr3 The root of the hierarchy.
3311	* @param cr4 The CR4, PSE is currently used.
3312	* @param cMaxDepth How deep into the hierarchy the dumper should go.
3313	* @param pHlp Pointer to the output functions.
3314	*/
3315	int pgmR3DumpHierarchyHC32BitPD(PVM pVM, uint32_t cr3, uint32_t cr4, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3316	{
3317	PX86PD pPD = (PX86PD)MMPagePhys2Page(pVM, cr3 & X86_CR3_PAGE_MASK);
3318	if (!pPD)
3319	{
3320	pHlp->pfnPrintf(pHlp, "Page directory at %#x was not found in the page pool!\n", cr3 & X86_CR3_PAGE_MASK);
3321	return VERR_INVALID_PARAMETER;
3322	}
3323
3324	int rc = VINF_SUCCESS;
3325	for (unsigned i = 0; i < ELEMENTS(pPD->a); i++)
3326	{
3327	X86PDE Pde = pPD->a[i];
3328	if (Pde.n.u1Present)
3329	{
3330	const uint32_t u32Address = i << X86_PD_SHIFT;
3331	if ((cr4 & X86_CR4_PSE) && Pde.b.u1Size)
3332	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a m d /
3333	"%08x 0 \| P %c %c %c %c %c %s %s %s .. 4M %c%c%c %08x\n",
3334	u32Address,
3335	Pde.b.u1Write ? 'W' : 'R',
3336	Pde.b.u1User ? 'U' : 'S',
3337	Pde.b.u1Accessed ? 'A' : '-',
3338	Pde.b.u1Dirty ? 'D' : '-',
3339	Pde.b.u1Global ? 'G' : '-',
3340	Pde.b.u1WriteThru ? "WT" : "--",
3341	Pde.b.u1CacheDisable? "CD" : "--",
3342	Pde.b.u1PAT ? "AT" : "--",
3343	Pde.u & RT_BIT_64(9) ? '1' : '0',
3344	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
3345	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
3346	Pde.u & X86_PDE4M_PG_MASK);
3347	else
3348	{
3349	pHlp->pfnPrintf(pHlp, /P R S A D G WT CD AT NX 4M a m d /
3350	"%08x 0 \| P %c %c %c %c %c %s %s .. .. 4K %c%c%c %08x\n",
3351	u32Address,
3352	Pde.n.u1Write ? 'W' : 'R',
3353	Pde.n.u1User ? 'U' : 'S',
3354	Pde.n.u1Accessed ? 'A' : '-',
3355	Pde.n.u1Reserved0 ? '?' : '.', /* ignored */
3356	Pde.n.u1Reserved1 ? '?' : '.', /* ignored */
3357	Pde.n.u1WriteThru ? "WT" : "--",
3358	Pde.n.u1CacheDisable? "CD" : "--",
3359	Pde.u & RT_BIT_64(9) ? '1' : '0',
3360	Pde.u & PGM_PDFLAGS_MAPPING ? 'm' : '-',
3361	Pde.u & PGM_PDFLAGS_TRACK_DIRTY ? 'd' : '-',
3362	Pde.u & X86_PDE_PG_MASK);
3363	if (cMaxDepth >= 1)
3364	{
3365	/** @todo what about using the page pool for mapping PTs? */
3366	RTHCPHYS HCPhys = Pde.u & X86_PDE_PG_MASK;
3367	PX86PT pPT = NULL;
3368	if (!(Pde.u & PGM_PDFLAGS_MAPPING))
3369	pPT = (PX86PT)MMPagePhys2Page(pVM, HCPhys);
3370	else
3371	{
3372	for (PPGMMAPPING pMap = pVM->pgm.s.pMappingsR3; pMap; pMap = pMap->pNextR3)
3373	if (u32Address - pMap->GCPtr < pMap->cb)
3374	{
3375	int iPDE = (u32Address - pMap->GCPtr) >> X86_PD_SHIFT;
3376	if (pMap->aPTs[iPDE].HCPhysPT != HCPhys)
3377	pHlp->pfnPrintf(pHlp, "%08x error! Mapping error! PT %d has HCPhysPT=%VHp not %VHp is in the PD.\n",
3378	u32Address, iPDE, pMap->aPTs[iPDE].HCPhysPT, HCPhys);
3379	pPT = pMap->aPTs[iPDE].pPTR3;
3380	}
3381	}
3382	int rc2 = VERR_INVALID_PARAMETER;
3383	if (pPT)
3384	rc2 = pgmR3DumpHierarchyHC32BitPT(pVM, pPT, u32Address, pHlp);
3385	else
3386	pHlp->pfnPrintf(pHlp, "%08x error! Page table at %#x was not found in the page pool!\n", u32Address, HCPhys);
3387	if (rc2 < rc && VBOX_SUCCESS(rc))
3388	rc = rc2;
3389	}
3390	}
3391	}
3392	}
3393
3394	return rc;
3395	}
3396
3397
3398	/**
3399	* Dumps a 32-bit shadow page table.
3400	*
3401	* @returns VBox status code (VINF_SUCCESS).
3402	* @param pVM The VM handle.
3403	* @param pPT Pointer to the page table.
3404	* @param u32Address The virtual address this table starts at.
3405	* @param PhysSearch Address to search for.
3406	*/
3407	int pgmR3DumpHierarchyGC32BitPT(PVM pVM, PX86PT pPT, uint32_t u32Address, RTGCPHYS PhysSearch)
3408	{
3409	for (unsigned i = 0; i < ELEMENTS(pPT->a); i++)
3410	{
3411	X86PTE Pte = pPT->a[i];
3412	if (Pte.n.u1Present)
3413	{
3414	Log(( /P R S A D G WT CD AT NX 4M a m d /
3415	"%08x 1 \| P %c %c %c %c %c %s %s %s .. 4K %c%c%c %08x\n",
3416	u32Address + (i << X86_PT_SHIFT),
3417	Pte.n.u1Write ? 'W' : 'R',
3418	Pte.n.u1User ? 'U' : 'S',
3419	Pte.n.u1Accessed ? 'A' : '-',
3420	Pte.n.u1Dirty ? 'D' : '-',
3421	Pte.n.u1Global ? 'G' : '-',
3422	Pte.n.u1WriteThru ? "WT" : "--",
3423	Pte.n.u1CacheDisable? "CD" : "--",
3424	Pte.n.u1PAT ? "AT" : "--",
3425	Pte.u & PGM_PTFLAGS_TRACK_DIRTY ? 'd' : '-',
3426	Pte.u & RT_BIT(10) ? '1' : '0',
3427	Pte.u & PGM_PTFLAGS_CSAM_VALIDATED ? 'v' : '-',
3428	Pte.u & X86_PDE_PG_MASK));
3429
3430	if ((Pte.u & X86_PDE_PG_MASK) == PhysSearch)
3431	{
3432	uint64_t fPageShw = 0;
3433	RTHCPHYS pPhysHC = 0;
3434
3435	PGMShwGetPage(pVM, (RTGCPTR)(u32Address + (i << X86_PT_SHIFT)), &fPageShw, &pPhysHC);
3436	Log(("Found %VGp at %VGv -> flags=%llx\n", PhysSearch, (RTGCPTR)(u32Address + (i << X86_PT_SHIFT)), fPageShw));
3437	}
3438	}
3439	}
3440	return VINF_SUCCESS;
3441	}
3442
3443
3444	/**
3445	* Dumps a 32-bit guest page directory and page tables.
3446	*
3447	* @returns VBox status code (VINF_SUCCESS).
3448	* @param pVM The VM handle.
3449	* @param cr3 The root of the hierarchy.
3450	* @param cr4 The CR4, PSE is currently used.
3451	* @param PhysSearch Address to search for.
3452	*/
3453	PGMR3DECL(int) PGMR3DumpHierarchyGC(PVM pVM, uint64_t cr3, uint64_t cr4, RTGCPHYS PhysSearch)
3454	{
3455	bool fLongMode = false;
3456	const unsigned cch = fLongMode ? 16 : 8; NOREF(cch);
3457	PX86PD pPD = 0;
3458
3459	int rc = PGM_GCPHYS_2_PTR(pVM, cr3 & X86_CR3_PAGE_MASK, &pPD);
3460	if (VBOX_FAILURE(rc) \|\| !pPD)
3461	{
3462	Log(("Page directory at %#x was not found in the page pool!\n", cr3 & X86_CR3_PAGE_MASK));
3463	return VERR_INVALID_PARAMETER;
3464	}
3465
3466	Log(("cr3=%08x cr4=%08x%s\n"
3467	"%-*s P - Present\n"
3468	"%-*s \| R/W - Read (0) / Write (1)\n"
3469	"%-*s \| \| U/S - User (1) / Supervisor (0)\n"
3470	"%-*s \| \| \| A - Accessed\n"
3471	"%-*s \| \| \| \| D - Dirty\n"
3472	"%-*s \| \| \| \| \| G - Global\n"
3473	"%-*s \| \| \| \| \| \| WT - Write thru\n"
3474	"%-*s \| \| \| \| \| \| \| CD - Cache disable\n"
3475	"%-*s \| \| \| \| \| \| \| \| AT - Attribute table (PAT)\n"
3476	"%-*s \| \| \| \| \| \| \| \| \| NX - No execute (K8)\n"
3477	"%-*s \| \| \| \| \| \| \| \| \| \| 4K/4M/2M - Page size.\n"
3478	"%-*s \| \| \| \| \| \| \| \| \| \| \| AVL - a=allocated; m=mapping; d=track dirty;\n"
3479	"%-*s \| \| \| \| \| \| \| \| \| \| \| \| p=permanent; v=validated;\n"
3480	"%-*s Level \| \| \| \| \| \| \| \| \| \| \| \| Page\n"
3481	/* xxxx n **** P R S A D G WT CD AT NX 4M AVL xxxxxxxxxxxxx
3482	- W U - - - -- -- -- -- -- 010 */
3483	, cr3, cr4, fLongMode ? " Long Mode" : "",
3484	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "",
3485	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "Address"));
3486
3487	for (unsigned i = 0; i < ELEMENTS(pPD->a); i++)
3488	{
3489	X86PDE Pde = pPD->a[i];
3490	if (Pde.n.u1Present)
3491	{
3492	const uint32_t u32Address = i << X86_PD_SHIFT;
3493
3494	if ((cr4 & X86_CR4_PSE) && Pde.b.u1Size)
3495	Log(( /P R S A D G WT CD AT NX 4M a m d /
3496	"%08x 0 \| P %c %c %c %c %c %s %s %s .. 4M %c%c%c %08x\n",
3497	u32Address,
3498	Pde.b.u1Write ? 'W' : 'R',
3499	Pde.b.u1User ? 'U' : 'S',
3500	Pde.b.u1Accessed ? 'A' : '-',
3501	Pde.b.u1Dirty ? 'D' : '-',
3502	Pde.b.u1Global ? 'G' : '-',
3503	Pde.b.u1WriteThru ? "WT" : "--",
3504	Pde.b.u1CacheDisable? "CD" : "--",
3505	Pde.b.u1PAT ? "AT" : "--",
3506	Pde.u & RT_BIT(9) ? '1' : '0',
3507	Pde.u & RT_BIT(10) ? '1' : '0',
3508	Pde.u & RT_BIT(11) ? '1' : '0',
3509	Pde.u & X86_PDE4M_PG_MASK));
3510	/** @todo PhysSearch */
3511	else
3512	{
3513	Log(( /P R S A D G WT CD AT NX 4M a m d /
3514	"%08x 0 \| P %c %c %c %c %c %s %s .. .. 4K %c%c%c %08x\n",
3515	u32Address,
3516	Pde.n.u1Write ? 'W' : 'R',
3517	Pde.n.u1User ? 'U' : 'S',
3518	Pde.n.u1Accessed ? 'A' : '-',
3519	Pde.n.u1Reserved0 ? '?' : '.', /* ignored */
3520	Pde.n.u1Reserved1 ? '?' : '.', /* ignored */
3521	Pde.n.u1WriteThru ? "WT" : "--",
3522	Pde.n.u1CacheDisable? "CD" : "--",
3523	Pde.u & RT_BIT(9) ? '1' : '0',
3524	Pde.u & RT_BIT(10) ? '1' : '0',
3525	Pde.u & RT_BIT(11) ? '1' : '0',
3526	Pde.u & X86_PDE_PG_MASK));
3527	////if (cMaxDepth >= 1)
3528	{
3529	/** @todo what about using the page pool for mapping PTs? */
3530	RTGCPHYS GCPhys = Pde.u & X86_PDE_PG_MASK;
3531	PX86PT pPT = NULL;
3532
3533	rc = PGM_GCPHYS_2_PTR(pVM, GCPhys, &pPT);
3534
3535	int rc2 = VERR_INVALID_PARAMETER;
3536	if (pPT)
3537	rc2 = pgmR3DumpHierarchyGC32BitPT(pVM, pPT, u32Address, PhysSearch);
3538	else
3539	Log(("%08x error! Page table at %#x was not found in the page pool!\n", u32Address, GCPhys));
3540	if (rc2 < rc && VBOX_SUCCESS(rc))
3541	rc = rc2;
3542	}
3543	}
3544	}
3545	}
3546
3547	return rc;
3548	}
3549
3550
3551	/**
3552	* Dumps a page table hierarchy use only physical addresses and cr4/lm flags.
3553	*
3554	* @returns VBox status code (VINF_SUCCESS).
3555	* @param pVM The VM handle.
3556	* @param cr3 The root of the hierarchy.
3557	* @param cr4 The cr4, only PAE and PSE is currently used.
3558	* @param fLongMode Set if long mode, false if not long mode.
3559	* @param cMaxDepth Number of levels to dump.
3560	* @param pHlp Pointer to the output functions.
3561	*/
3562	PGMR3DECL(int) PGMR3DumpHierarchyHC(PVM pVM, uint64_t cr3, uint64_t cr4, bool fLongMode, unsigned cMaxDepth, PCDBGFINFOHLP pHlp)
3563	{
3564	if (!pHlp)
3565	pHlp = DBGFR3InfoLogHlp();
3566	if (!cMaxDepth)
3567	return VINF_SUCCESS;
3568	const unsigned cch = fLongMode ? 16 : 8;
3569	pHlp->pfnPrintf(pHlp,
3570	"cr3=%08x cr4=%08x%s\n"
3571	"%-*s P - Present\n"
3572	"%-*s \| R/W - Read (0) / Write (1)\n"
3573	"%-*s \| \| U/S - User (1) / Supervisor (0)\n"
3574	"%-*s \| \| \| A - Accessed\n"
3575	"%-*s \| \| \| \| D - Dirty\n"
3576	"%-*s \| \| \| \| \| G - Global\n"
3577	"%-*s \| \| \| \| \| \| WT - Write thru\n"
3578	"%-*s \| \| \| \| \| \| \| CD - Cache disable\n"
3579	"%-*s \| \| \| \| \| \| \| \| AT - Attribute table (PAT)\n"
3580	"%-*s \| \| \| \| \| \| \| \| \| NX - No execute (K8)\n"
3581	"%-*s \| \| \| \| \| \| \| \| \| \| 4K/4M/2M - Page size.\n"
3582	"%-*s \| \| \| \| \| \| \| \| \| \| \| AVL - a=allocated; m=mapping; d=track dirty;\n"
3583	"%-*s \| \| \| \| \| \| \| \| \| \| \| \| p=permanent; v=validated;\n"
3584	"%-*s Level \| \| \| \| \| \| \| \| \| \| \| \| Page\n"
3585	/* xxxx n **** P R S A D G WT CD AT NX 4M AVL xxxxxxxxxxxxx
3586	- W U - - - -- -- -- -- -- 010 */
3587	, cr3, cr4, fLongMode ? " Long Mode" : "",
3588	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "",
3589	cch, "", cch, "", cch, "", cch, "", cch, "", cch, "", cch, "Address");
3590	if (cr4 & X86_CR4_PAE)
3591	{
3592	if (fLongMode)
3593	return pgmR3DumpHierarchyHcPaePML4(pVM, cr3 & X86_CR3_PAGE_MASK, cr4, cMaxDepth, pHlp);
3594	return pgmR3DumpHierarchyHCPaePDPT(pVM, cr3 & X86_CR3_PAE_PAGE_MASK, 0, cr4, false, cMaxDepth, pHlp);
3595	}
3596	return pgmR3DumpHierarchyHC32BitPD(pVM, cr3 & X86_CR3_PAGE_MASK, cr4, cMaxDepth, pHlp);
3597	}
3598
3599
3600
3601	#ifdef VBOX_WITH_DEBUGGER
3602	/**
3603	* The '.pgmram' command.
3604	*
3605	* @returns VBox status.
3606	* @param pCmd Pointer to the command descriptor (as registered).
3607	* @param pCmdHlp Pointer to command helper functions.
3608	* @param pVM Pointer to the current VM (if any).
3609	* @param paArgs Pointer to (readonly) array of arguments.
3610	* @param cArgs Number of arguments in the array.
3611	*/
3612	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
3613	{
3614	/*
3615	* Validate input.
3616	*/
3617	if (!pVM)
3618	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires VM to be selected.\n");
3619	if (!pVM->pgm.s.pRamRangesGC)
3620	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Sorry, no Ram is registered.\n");
3621
3622	/*
3623	* Dump the ranges.
3624	*/
3625	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "From - To (incl) pvHC\n");
3626	PPGMRAMRANGE pRam;
3627	for (pRam = pVM->pgm.s.pRamRangesR3; pRam; pRam = pRam->pNextR3)
3628	{
3629	rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL,
3630	"%VGp - %VGp %p\n",
3631	pRam->GCPhys, pRam->GCPhysLast, pRam->pvHC);
3632	if (VBOX_FAILURE(rc))
3633	return rc;
3634	}
3635
3636	return VINF_SUCCESS;
3637	}
3638
3639
3640	/**
3641	* The '.pgmmap' command.
3642	*
3643	* @returns VBox status.
3644	* @param pCmd Pointer to the command descriptor (as registered).
3645	* @param pCmdHlp Pointer to command helper functions.
3646	* @param pVM Pointer to the current VM (if any).
3647	* @param paArgs Pointer to (readonly) array of arguments.
3648	* @param cArgs Number of arguments in the array.
3649	*/
3650	static DECLCALLBACK(int) pgmR3CmdMap(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
3651	{
3652	/*
3653	* Validate input.
3654	*/
3655	if (!pVM)
3656	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires VM to be selected.\n");
3657	if (!pVM->pgm.s.pMappingsR3)
3658	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Sorry, no mappings are registered.\n");
3659
3660	/*
3661	* Print message about the fixedness of the mappings.
3662	*/
3663	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, pVM->pgm.s.fMappingsFixed ? "The mappings are FIXED.\n" : "The mappings are FLOATING.\n");
3664	if (VBOX_FAILURE(rc))
3665	return rc;
3666
3667	/*
3668	* Dump the ranges.
3669	*/
3670	PPGMMAPPING pCur;
3671	for (pCur = pVM->pgm.s.pMappingsR3; pCur; pCur = pCur->pNextR3)
3672	{
3673	rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL,
3674	"%08x - %08x %s\n",
3675	pCur->GCPtr, pCur->GCPtrLast, pCur->pszDesc);
3676	if (VBOX_FAILURE(rc))
3677	return rc;
3678	}
3679
3680	return VINF_SUCCESS;
3681	}
3682
3683
3684	/**
3685	* The '.pgmsync' command.
3686	*
3687	* @returns VBox status.
3688	* @param pCmd Pointer to the command descriptor (as registered).
3689	* @param pCmdHlp Pointer to command helper functions.
3690	* @param pVM Pointer to the current VM (if any).
3691	* @param paArgs Pointer to (readonly) array of arguments.
3692	* @param cArgs Number of arguments in the array.
3693	*/
3694	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
3695	{
3696	/*
3697	* Validate input.
3698	*/
3699	if (!pVM)
3700	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires VM to be selected.\n");
3701
3702	/*
3703	* Force page directory sync.
3704	*/
3705	VM_FF_SET(pVM, VM_FF_PGM_SYNC_CR3);
3706
3707	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Forcing page directory sync.\n");
3708	if (VBOX_FAILURE(rc))
3709	return rc;
3710
3711	return VINF_SUCCESS;
3712	}
3713
3714
3715	/**
3716	* The '.pgmsyncalways' command.
3717	*
3718	* @returns VBox status.
3719	* @param pCmd Pointer to the command descriptor (as registered).
3720	* @param pCmdHlp Pointer to command helper functions.
3721	* @param pVM Pointer to the current VM (if any).
3722	* @param paArgs Pointer to (readonly) array of arguments.
3723	* @param cArgs Number of arguments in the array.
3724	*/
3725	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
3726	{
3727	/*
3728	* Validate input.
3729	*/
3730	if (!pVM)
3731	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires VM to be selected.\n");
3732
3733	/*
3734	* Force page directory sync.
3735	*/
3736	if (pVM->pgm.s.fSyncFlags & PGM_SYNC_ALWAYS)
3737	{
3738	ASMAtomicAndU32(&pVM->pgm.s.fSyncFlags, ~PGM_SYNC_ALWAYS);
3739	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Disabled permanent forced page directory syncing.\n");
3740	}
3741	else
3742	{
3743	ASMAtomicOrU32(&pVM->pgm.s.fSyncFlags, PGM_SYNC_ALWAYS);
3744	VM_FF_SET(pVM, VM_FF_PGM_SYNC_CR3);
3745	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Enabled permanent forced page directory syncing.\n");
3746	}
3747	}
3748
3749	#endif
3750
3751	/**
3752	* pvUser argument of the pgmR3CheckIntegrity*Node callbacks.
3753	*/
3754	typedef struct PGMCHECKINTARGS
3755	{
3756	bool fLeftToRight; /*< true: left-to-right; false: right-to-left. /
3757	PPGMPHYSHANDLER pPrevPhys;
3758	PPGMVIRTHANDLER pPrevVirt;
3759	PPGMPHYS2VIRTHANDLER pPrevPhys2Virt;
3760	PVM pVM;
3761	} PGMCHECKINTARGS, *PPGMCHECKINTARGS;
3762
3763	/**
3764	* Validate a node in the physical handler tree.
3765	*
3766	* @returns 0 on if ok, other wise 1.
3767	* @param pNode The handler node.
3768	* @param pvUser pVM.
3769	*/
3770	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
3771	{
3772	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3773	PPGMPHYSHANDLER pCur = (PPGMPHYSHANDLER)pNode;
3774	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
3775	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %VGp-%VGp %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3776	AssertReleaseMsg( !pArgs->pPrevPhys
3777	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys->Core.KeyLast > pCur->Core.Key),
3778	("pPrevPhys=%p %VGp-%VGp %s\n"
3779	" pCur=%p %VGp-%VGp %s\n",
3780	pArgs->pPrevPhys, pArgs->pPrevPhys->Core.Key, pArgs->pPrevPhys->Core.KeyLast, pArgs->pPrevPhys->pszDesc,
3781	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3782	pArgs->pPrevPhys = pCur;
3783	return 0;
3784	}
3785
3786
3787	/**
3788	* Validate a node in the virtual handler tree.
3789	*
3790	* @returns 0 on if ok, other wise 1.
3791	* @param pNode The handler node.
3792	* @param pvUser pVM.
3793	*/
3794	static DECLCALLBACK(int) pgmR3CheckIntegrityVirtHandlerNode(PAVLROGCPTRNODECORE pNode, void *pvUser)
3795	{
3796	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3797	PPGMVIRTHANDLER pCur = (PPGMVIRTHANDLER)pNode;
3798	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
3799	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %VGv-%VGv %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3800	AssertReleaseMsg( !pArgs->pPrevVirt
3801	\|\| (pArgs->fLeftToRight ? pArgs->pPrevVirt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevVirt->Core.KeyLast > pCur->Core.Key),
3802	("pPrevVirt=%p %VGv-%VGv %s\n"
3803	" pCur=%p %VGv-%VGv %s\n",
3804	pArgs->pPrevVirt, pArgs->pPrevVirt->Core.Key, pArgs->pPrevVirt->Core.KeyLast, pArgs->pPrevVirt->pszDesc,
3805	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3806	for (unsigned iPage = 0; iPage < pCur->cPages; iPage++)
3807	{
3808	AssertReleaseMsg(pCur->aPhysToVirt[iPage].offVirtHandler == -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage]),
3809	("pCur=%p %VGv-%VGv %s\n"
3810	"iPage=%d offVirtHandle=%#x expected %#x\n",
3811	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc,
3812	iPage, pCur->aPhysToVirt[iPage].offVirtHandler, -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage])));
3813	}
3814	pArgs->pPrevVirt = pCur;
3815	return 0;
3816	}
3817
3818
3819	/**
3820	* Validate a node in the virtual handler tree.
3821	*
3822	* @returns 0 on if ok, other wise 1.
3823	* @param pNode The handler node.
3824	* @param pvUser pVM.
3825	*/
3826	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysToVirtHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
3827	{
3828	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3829	PPGMPHYS2VIRTHANDLER pCur = (PPGMPHYS2VIRTHANDLER)pNode;
3830	AssertReleaseMsgReturn(!((uintptr_t)pCur & 3), ("\n"), 1);
3831	AssertReleaseMsgReturn(!(pCur->offVirtHandler & 3), ("\n"), 1);
3832	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %VGp-%VGp\n", pCur, pCur->Core.Key, pCur->Core.KeyLast));
3833	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
3834	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
3835	("pPrevPhys2Virt=%p %VGp-%VGp\n"
3836	" pCur=%p %VGp-%VGp\n",
3837	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
3838	pCur, pCur->Core.Key, pCur->Core.KeyLast));
3839	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
3840	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
3841	("pPrevPhys2Virt=%p %VGp-%VGp\n"
3842	" pCur=%p %VGp-%VGp\n",
3843	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
3844	pCur, pCur->Core.Key, pCur->Core.KeyLast));
3845	AssertReleaseMsg((pCur->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD),
3846	("pCur=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
3847	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
3848	if (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK)
3849	{
3850	PPGMPHYS2VIRTHANDLER pCur2 = pCur;
3851	for (;;)
3852	{
3853	pCur2 = (PPGMPHYS2VIRTHANDLER)((intptr_t)pCur + (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK));
3854	AssertReleaseMsg(pCur2 != pCur,
3855	(" pCur=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
3856	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
3857	AssertReleaseMsg((pCur2->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == PGMPHYS2VIRTHANDLER_IN_TREE,
3858	(" pCur=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
3859	"pCur2=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
3860	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
3861	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
3862	AssertReleaseMsg((pCur2->Core.Key ^ pCur->Core.Key) < PAGE_SIZE,
3863	(" pCur=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
3864	"pCur2=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
3865	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
3866	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
3867	AssertReleaseMsg((pCur2->Core.KeyLast ^ pCur->Core.KeyLast) < PAGE_SIZE,
3868	(" pCur=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
3869	"pCur2=%p:{.Core.Key=%VGp, .Core.KeyLast=%VGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
3870	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
3871	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
3872	if (!(pCur2->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK))
3873	break;
3874	}
3875	}
3876
3877	pArgs->pPrevPhys2Virt = pCur;
3878	return 0;
3879	}
3880
3881
3882	/**
3883	* Perform an integrity check on the PGM component.
3884	*
3885	* @returns VINF_SUCCESS if everything is fine.
3886	* @returns VBox error status after asserting on integrity breach.
3887	* @param pVM The VM handle.
3888	*/
3889	PDMR3DECL(int) PGMR3CheckIntegrity(PVM pVM)
3890	{
3891	AssertReleaseReturn(pVM->pgm.s.offVM, VERR_INTERNAL_ERROR);
3892
3893	/*
3894	* Check the trees.
3895	*/
3896	int cErrors = 0;
3897	PGMCHECKINTARGS Args = { true, NULL, NULL, NULL, pVM };
3898	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesHC->PhysHandlers, true, pgmR3CheckIntegrityPhysHandlerNode, &Args);
3899	Args.fLeftToRight = false;
3900	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesHC->PhysHandlers, false, pgmR3CheckIntegrityPhysHandlerNode, &Args);
3901	Args.fLeftToRight = true;
3902	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesHC->VirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
3903	Args.fLeftToRight = false;
3904	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesHC->VirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
3905	Args.fLeftToRight = true;
3906	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesHC->HyperVirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
3907	Args.fLeftToRight = false;
3908	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesHC->HyperVirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
3909	Args.fLeftToRight = true;
3910	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesHC->PhysToVirtHandlers, true, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
3911	Args.fLeftToRight = false;
3912	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesHC->PhysToVirtHandlers, false, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
3913
3914	return !cErrors ? VINF_SUCCESS : VERR_INTERNAL_ERROR;
3915	}
3916
3917
3918	/**
3919	* Inform PGM if we want all mappings to be put into the shadow page table. (necessary for e.g. VMX)
3920	*
3921	* @returns VBox status code.
3922	* @param pVM VM handle.
3923	* @param fEnable Enable or disable shadow mappings
3924	*/
3925	PGMR3DECL(int) PGMR3ChangeShwPDMappings(PVM pVM, bool fEnable)
3926	{
3927	pVM->pgm.s.fDisableMappings = !fEnable;
3928
3929	uint32_t cb;
3930	int rc = PGMR3MappingsSize(pVM, &cb);
3931	AssertRCReturn(rc, rc);
3932
3933	/* Pretend the mappings are now fixed; to force a refresh of the reserved PDEs. */
3934	rc = PGMR3MappingsFix(pVM, MM_HYPER_AREA_ADDRESS, cb);
3935	AssertRCReturn(rc, rc);
3936
3937	return VINF_SUCCESS;
3938	}

Note: See TracBrowser for help on using the repository browser.

source: vbox/trunk/src/VBox/VMM/PGM.cpp@ 8536

Download in other formats: