PGM.cpp@ 48568

Last change on this file since 48568 was 47786, checked in by vboxsync, 11 years ago
PGM: Added a new page type for the VT-x APIC access page MMIO alias instead of abusing the MMIO2 aliasing. There are important differences, we can safely access the MMIO2 page when aliased and save time doing so, while the alias created by IOMMMIOMapMMIOHCPage must not be accessed outside the VT-x execution AFAIK.
Property svn:eol-style set to `native` Property svn:keywords set to `Id Revision`
File size: 203.3 KB

Line
1	/* $Id: PGM.cpp 47786 2013-08-16 08:59:32Z vboxsync $ */
2	/** @file
3	* PGM - Page Manager and Monitor. (Mixing stuff here, not good?)
4	*/
5
6	/*
7	* Copyright (C) 2006-2013 Oracle Corporation
8	*
9	* This file is part of VirtualBox Open Source Edition (OSE), as
10	* available from http://www.virtualbox.org. This file is free software;
11	* you can redistribute it and/or modify it under the terms of the GNU
12	* General Public License (GPL) as published by the Free Software
13	* Foundation, in version 2 as it comes in the "COPYING" file of the
14	* VirtualBox OSE distribution. VirtualBox OSE is distributed in the
15	* hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
16	*/
17
18
19	/** @page pg_pgm PGM - The Page Manager and Monitor
20	*
21	* @see grp_pgm,
22	* @ref pg_pgm_pool,
23	* @ref pg_pgm_phys.
24	*
25	*
26	* @section sec_pgm_modes Paging Modes
27	*
28	* There are three memory contexts: Host Context (HC), Guest Context (GC)
29	* and intermediate context. When talking about paging HC can also be referred
30	* to as "host paging", and GC referred to as "shadow paging".
31	*
32	* We define three basic paging modes: 32-bit, PAE and AMD64. The host paging mode
33	* is defined by the host operating system. The mode used in the shadow paging mode
34	* depends on the host paging mode and what the mode the guest is currently in. The
35	* following relation between the two is defined:
36	*
37	* @verbatim
38	Host > 32-bit \| PAE \| AMD64 \|
39	Guest \| \| \| \|
40	==v================================
41	32-bit 32-bit PAE PAE
42	-------\|--------\|--------\|--------\|
43	PAE PAE PAE PAE
44	-------\|--------\|--------\|--------\|
45	AMD64 AMD64 AMD64 AMD64
46	-------\|--------\|--------\|--------\| @endverbatim
47	*
48	* All configuration except those in the diagonal (upper left) are expected to
49	* require special effort from the switcher (i.e. a bit slower).
50	*
51	*
52	*
53	*
54	* @section sec_pgm_shw The Shadow Memory Context
55	*
56	*
57	* [..]
58	*
59	* Because of guest context mappings requires PDPT and PML4 entries to allow
60	* writing on AMD64, the two upper levels will have fixed flags whatever the
61	* guest is thinking of using there. So, when shadowing the PD level we will
62	* calculate the effective flags of PD and all the higher levels. In legacy
63	* PAE mode this only applies to the PWT and PCD bits (the rest are
64	* ignored/reserved/MBZ). We will ignore those bits for the present.
65	*
66	*
67	*
68	* @section sec_pgm_int The Intermediate Memory Context
69	*
70	* The world switch goes thru an intermediate memory context which purpose it is
71	* to provide different mappings of the switcher code. All guest mappings are also
72	* present in this context.
73	*
74	* The switcher code is mapped at the same location as on the host, at an
75	* identity mapped location (physical equals virtual address), and at the
76	* hypervisor location. The identity mapped location is for when the world
77	* switches that involves disabling paging.
78	*
79	* PGM maintain page tables for 32-bit, PAE and AMD64 paging modes. This
80	* simplifies switching guest CPU mode and consistency at the cost of more
81	* code to do the work. All memory use for those page tables is located below
82	* 4GB (this includes page tables for guest context mappings).
83	*
84	* Note! The intermediate memory context is also used for 64-bit guest
85	* execution on 32-bit hosts. Because we need to load 64-bit registers
86	* prior to switching to guest context, we need to be in 64-bit mode
87	* first. So, HM has some 64-bit worker routines in VMMRC.rc that get
88	* invoked via the special world switcher code in LegacyToAMD64.asm.
89	*
90	*
91	* @subsection subsec_pgm_int_gc Guest Context Mappings
92	*
93	* During assignment and relocation of a guest context mapping the intermediate
94	* memory context is used to verify the new location.
95	*
96	* Guest context mappings are currently restricted to below 4GB, for reasons
97	* of simplicity. This may change when we implement AMD64 support.
98	*
99	*
100	*
101	*
102	* @section sec_pgm_misc Misc
103	*
104	*
105	* @subsection sec_pgm_misc_A20 The A20 Gate
106	*
107	* PGM implements the A20 gate masking when translating a virtual guest address
108	* into a physical address for CPU access, i.e. PGMGstGetPage (and friends) and
109	* the code reading the guest page table entries during shadowing. The masking
110	* is done consistenly for all CPU modes, paged ones included. Large pages are
111	* also masked correctly. (On current CPUs, experiments indicates that AMD does
112	* not apply A20M in paged modes and intel only does it for the 2nd MB of
113	* memory.)
114	*
115	* The A20 gate implementation is per CPU core. It can be configured on a per
116	* core basis via the keyboard device and PC architecture device. This is
117	* probably not exactly how real CPUs do it, but SMP and A20 isn't a place where
118	* guest OSes try pushing things anyway, so who cares. (On current real systems
119	* the A20M signal is probably only sent to the boot CPU and it affects all
120	* thread and probably all cores in that package.)
121	*
122	* The keyboard device and the PC architecture device doesn't OR their A20
123	* config bits together, rather they are currently implemented such that they
124	* mirror the CPU state. So, flipping the bit in either of them will change the
125	* A20 state. (On real hardware the bits of the two devices should probably be
126	* ORed together to indicate enabled, i.e. both needs to be cleared to disable
127	* A20 masking.)
128	*
129	* The A20 state will change immediately, transmeta fashion. There is no delays
130	* due to buses, wiring or other physical stuff. (On real hardware there are
131	* normally delays, the delays differs between the two devices and probably also
132	* between chipsets and CPU generations. Note that it's said that transmeta CPUs
133	* does the change immediately like us, they apparently intercept/handles the
134	* port accesses in microcode. Neat.)
135	*
136	* @sa http://en.wikipedia.org/wiki/A20_line#The_80286_and_the_high_memory_area
137	*
138	*
139	* @subsection subsec_pgm_misc_diff Differences Between Legacy PAE and Long Mode PAE
140	*
141	* The differences between legacy PAE and long mode PAE are:
142	* -# PDPE bits 1, 2, 5 and 6 are defined differently. In leagcy mode they are
143	* all marked down as must-be-zero, while in long mode 1, 2 and 5 have the
144	* usual meanings while 6 is ignored (AMD). This means that upon switching to
145	* legacy PAE mode we'll have to clear these bits and when going to long mode
146	* they must be set. This applies to both intermediate and shadow contexts,
147	* however we don't need to do it for the intermediate one since we're
148	* executing with CR0.WP at that time.
149	* -# CR3 allows a 32-byte aligned address in legacy mode, while in long mode
150	* a page aligned one is required.
151	*
152	*
153	* @section sec_pgm_handlers Access Handlers
154	*
155	* Placeholder.
156	*
157	*
158	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
159	*
160	* Placeholder.
161	*
162	*
163	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
164	*
165	* We currently implement three types of virtual access handlers: ALL, WRITE
166	* and HYPERVISOR (WRITE). See PGMVIRTHANDLERTYPE for some more details.
167	*
168	* The HYPERVISOR access handlers is kept in a separate tree since it doesn't apply
169	* to physical pages (PGMTREES::HyperVirtHandlers) and only needs to be consulted in
170	* a special \#PF case. The ALL and WRITE are in the PGMTREES::VirtHandlers tree, the
171	* rest of this section is going to be about these handlers.
172	*
173	* We'll go thru the life cycle of a handler and try make sense of it all, don't know
174	* how successful this is gonna be...
175	*
176	* 1. A handler is registered thru the PGMR3HandlerVirtualRegister and
177	* PGMHandlerVirtualRegisterEx APIs. We check for conflicting virtual handlers
178	* and create a new node that is inserted into the AVL tree (range key). Then
179	* a full PGM resync is flagged (clear pool, sync cr3, update virtual bit of PGMPAGE).
180	*
181	* 2. The following PGMSyncCR3/SyncCR3 operation will first make invoke HandlerVirtualUpdate.
182	*
183	* 2a. HandlerVirtualUpdate will will lookup all the pages covered by virtual handlers
184	* via the current guest CR3 and update the physical page -> virtual handler
185	* translation. Needless to say, this doesn't exactly scale very well. If any changes
186	* are detected, it will flag a virtual bit update just like we did on registration.
187	* PGMPHYS pages with changes will have their virtual handler state reset to NONE.
188	*
189	* 2b. The virtual bit update process will iterate all the pages covered by all the
190	* virtual handlers and update the PGMPAGE virtual handler state to the max of all
191	* virtual handlers on that page.
192	*
193	* 2c. Back in SyncCR3 we will now flush the entire shadow page cache to make sure
194	* we don't miss any alias mappings of the monitored pages.
195	*
196	* 2d. SyncCR3 will then proceed with syncing the CR3 table.
197	*
198	* 3. \#PF(np,read) on a page in the range. This will cause it to be synced
199	* read-only and resumed if it's a WRITE handler. If it's an ALL handler we
200	* will call the handlers like in the next step. If the physical mapping has
201	* changed we will - some time in the future - perform a handler callback
202	* (optional) and update the physical -> virtual handler cache.
203	*
204	* 4. \#PF(,write) on a page in the range. This will cause the handler to
205	* be invoked.
206	*
207	* 5. The guest invalidates the page and changes the physical backing or
208	* unmaps it. This should cause the invalidation callback to be invoked
209	* (it might not yet be 100% perfect). Exactly what happens next... is
210	* this where we mess up and end up out of sync for a while?
211	*
212	* 6. The handler is deregistered by the client via PGMHandlerVirtualDeregister.
213	* We will then set all PGMPAGEs in the physical -> virtual handler cache for
214	* this handler to NONE and trigger a full PGM resync (basically the same
215	* as int step 1). Which means 2 is executed again.
216	*
217	*
218	* @subsubsection sub_sec_pgm_handler_virt_todo TODOs
219	*
220	* There is a bunch of things that needs to be done to make the virtual handlers
221	* work 100% correctly and work more efficiently.
222	*
223	* The first bit hasn't been implemented yet because it's going to slow the
224	* whole mess down even more, and besides it seems to be working reliably for
225	* our current uses. OTOH, some of the optimizations might end up more or less
226	* implementing the missing bits, so we'll see.
227	*
228	* On the optimization side, the first thing to do is to try avoid unnecessary
229	* cache flushing. Then try team up with the shadowing code to track changes
230	* in mappings by means of access to them (shadow in), updates to shadows pages,
231	* invlpg, and shadow PT discarding (perhaps).
232	*
233	* Some idea that have popped up for optimization for current and new features:
234	* - bitmap indicating where there are virtual handlers installed.
235	* (4KB => 220 pages, page 212 => covers 32-bit address space 1:1!)
236	* - Further optimize this by min/max (needs min/max avl getters).
237	* - Shadow page table entry bit (if any left)?
238	*
239	*/
240
241
242	/** @page pg_pgm_phys PGM Physical Guest Memory Management
243	*
244	*
245	* Objectives:
246	* - Guest RAM over-commitment using memory ballooning,
247	* zero pages and general page sharing.
248	* - Moving or mirroring a VM onto a different physical machine.
249	*
250	*
251	* @subsection subsec_pgmPhys_Definitions Definitions
252	*
253	* Allocation chunk - A RTR0MemObjAllocPhysNC object and the tracking
254	* machinery associated with it.
255	*
256	*
257	*
258	*
259	* @subsection subsec_pgmPhys_AllocPage Allocating a page.
260	*
261	* Initially we map all guest memory to the (per VM) zero page, which
262	* means that none of the read functions will cause pages to be allocated.
263	*
264	* Exception, access bit in page tables that have been shared. This must
265	* be handled, but we must also make sure PGMGst*Modify doesn't make
266	* unnecessary modifications.
267	*
268	* Allocation points:
269	* - PGMPhysSimpleWriteGCPhys and PGMPhysWrite.
270	* - Replacing a zero page mapping at \#PF.
271	* - Replacing a shared page mapping at \#PF.
272	* - ROM registration (currently MMR3RomRegister).
273	* - VM restore (pgmR3Load).
274	*
275	* For the first three it would make sense to keep a few pages handy
276	* until we've reached the max memory commitment for the VM.
277	*
278	* For the ROM registration, we know exactly how many pages we need
279	* and will request these from ring-0. For restore, we will save
280	* the number of non-zero pages in the saved state and allocate
281	* them up front. This would allow the ring-0 component to refuse
282	* the request if the isn't sufficient memory available for VM use.
283	*
284	* Btw. for both ROM and restore allocations we won't be requiring
285	* zeroed pages as they are going to be filled instantly.
286	*
287	*
288	* @subsection subsec_pgmPhys_FreePage Freeing a page
289	*
290	* There are a few points where a page can be freed:
291	* - After being replaced by the zero page.
292	* - After being replaced by a shared page.
293	* - After being ballooned by the guest additions.
294	* - At reset.
295	* - At restore.
296	*
297	* When freeing one or more pages they will be returned to the ring-0
298	* component and replaced by the zero page.
299	*
300	* The reasoning for clearing out all the pages on reset is that it will
301	* return us to the exact same state as on power on, and may thereby help
302	* us reduce the memory load on the system. Further it might have a
303	* (temporary) positive influence on memory fragmentation (@see subsec_pgmPhys_Fragmentation).
304	*
305	* On restore, as mention under the allocation topic, pages should be
306	* freed / allocated depending on how many is actually required by the
307	* new VM state. The simplest approach is to do like on reset, and free
308	* all non-ROM pages and then allocate what we need.
309	*
310	* A measure to prevent some fragmentation, would be to let each allocation
311	* chunk have some affinity towards the VM having allocated the most pages
312	* from it. Also, try make sure to allocate from allocation chunks that
313	* are almost full. Admittedly, both these measures might work counter to
314	* our intentions and its probably not worth putting a lot of effort,
315	* cpu time or memory into this.
316	*
317	*
318	* @subsection subsec_pgmPhys_SharePage Sharing a page
319	*
320	* The basic idea is that there there will be a idle priority kernel
321	* thread walking the non-shared VM pages hashing them and looking for
322	* pages with the same checksum. If such pages are found, it will compare
323	* them byte-by-byte to see if they actually are identical. If found to be
324	* identical it will allocate a shared page, copy the content, check that
325	* the page didn't change while doing this, and finally request both the
326	* VMs to use the shared page instead. If the page is all zeros (special
327	* checksum and byte-by-byte check) it will request the VM that owns it
328	* to replace it with the zero page.
329	*
330	* To make this efficient, we will have to make sure not to try share a page
331	* that will change its contents soon. This part requires the most work.
332	* A simple idea would be to request the VM to write monitor the page for
333	* a while to make sure it isn't modified any time soon. Also, it may
334	* make sense to skip pages that are being write monitored since this
335	* information is readily available to the thread if it works on the
336	* per-VM guest memory structures (presently called PGMRAMRANGE).
337	*
338	*
339	* @subsection subsec_pgmPhys_Fragmentation Fragmentation Concerns and Counter Measures
340	*
341	* The pages are organized in allocation chunks in ring-0, this is a necessity
342	* if we wish to have an OS agnostic approach to this whole thing. (On Linux we
343	* could easily work on a page-by-page basis if we liked. Whether this is possible
344	* or efficient on NT I don't quite know.) Fragmentation within these chunks may
345	* become a problem as part of the idea here is that we wish to return memory to
346	* the host system.
347	*
348	* For instance, starting two VMs at the same time, they will both allocate the
349	* guest memory on-demand and if permitted their page allocations will be
350	* intermixed. Shut down one of the two VMs and it will be difficult to return
351	* any memory to the host system because the page allocation for the two VMs are
352	* mixed up in the same allocation chunks.
353	*
354	* To further complicate matters, when pages are freed because they have been
355	* ballooned or become shared/zero the whole idea is that the page is supposed
356	* to be reused by another VM or returned to the host system. This will cause
357	* allocation chunks to contain pages belonging to different VMs and prevent
358	* returning memory to the host when one of those VM shuts down.
359	*
360	* The only way to really deal with this problem is to move pages. This can
361	* either be done at VM shutdown and or by the idle priority worker thread
362	* that will be responsible for finding sharable/zero pages. The mechanisms
363	* involved for coercing a VM to move a page (or to do it for it) will be
364	* the same as when telling it to share/zero a page.
365	*
366	*
367	* @subsection subsec_pgmPhys_Tracking Tracking Structures And Their Cost
368	*
369	* There's a difficult balance between keeping the per-page tracking structures
370	* (global and guest page) easy to use and keeping them from eating too much
371	* memory. We have limited virtual memory resources available when operating in
372	* 32-bit kernel space (on 64-bit there'll it's quite a different story). The
373	* tracking structures will be attempted designed such that we can deal with up
374	* to 32GB of memory on a 32-bit system and essentially unlimited on 64-bit ones.
375	*
376	*
377	* @subsubsection subsubsec_pgmPhys_Tracking_Kernel Kernel Space
378	*
379	* @see pg_GMM
380	*
381	* @subsubsection subsubsec_pgmPhys_Tracking_PerVM Per-VM
382	*
383	* Fixed info is the physical address of the page (HCPhys) and the page id
384	* (described above). Theoretically we'll need 48(-12) bits for the HCPhys part.
385	* Today we've restricting ourselves to 40(-12) bits because this is the current
386	* restrictions of all AMD64 implementations (I think Barcelona will up this
387	* to 48(-12) bits, not that it really matters) and I needed the bits for
388	* tracking mappings of a page. 48-12 = 36. That leaves 28 bits, which means a
389	* decent range for the page id: 2^(28+12) = 1024TB.
390	*
391	* In additions to these, we'll have to keep maintaining the page flags as we
392	* currently do. Although it wouldn't harm to optimize these quite a bit, like
393	* for instance the ROM shouldn't depend on having a write handler installed
394	* in order for it to become read-only. A RO/RW bit should be considered so
395	* that the page syncing code doesn't have to mess about checking multiple
396	* flag combinations (ROM \|\| RW handler \|\| write monitored) in order to
397	* figure out how to setup a shadow PTE. But this of course, is second
398	* priority at present. Current this requires 12 bits, but could probably
399	* be optimized to ~8.
400	*
401	* Then there's the 24 bits used to track which shadow page tables are
402	* currently mapping a page for the purpose of speeding up physical
403	* access handlers, and thereby the page pool cache. More bit for this
404	* purpose wouldn't hurt IIRC.
405	*
406	* Then there is a new bit in which we need to record what kind of page
407	* this is, shared, zero, normal or write-monitored-normal. This'll
408	* require 2 bits. One bit might be needed for indicating whether a
409	* write monitored page has been written to. And yet another one or
410	* two for tracking migration status. 3-4 bits total then.
411	*
412	* Whatever is left will can be used to record the sharabilitiy of a
413	* page. The page checksum will not be stored in the per-VM table as
414	* the idle thread will not be permitted to do modifications to it.
415	* It will instead have to keep its own working set of potentially
416	* shareable pages and their check sums and stuff.
417	*
418	* For the present we'll keep the current packing of the
419	* PGMRAMRANGE::aHCPhys to keep the changes simple, only of course,
420	* we'll have to change it to a struct with a total of 128-bits at
421	* our disposal.
422	*
423	* The initial layout will be like this:
424	* @verbatim
425	RTHCPHYS HCPhys; The current stuff.
426	63:40 Current shadow PT tracking stuff.
427	39:12 The physical page frame number.
428	11:0 The current flags.
429	uint32_t u28PageId : 28; The page id.
430	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
431	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
432	uint32_t u1Reserved : 1; Reserved for later.
433	uint32_t u32Reserved; Reserved for later, mostly sharing stats.
434	@endverbatim
435	*
436	* The final layout will be something like this:
437	* @verbatim
438	RTHCPHYS HCPhys; The current stuff.
439	63:48 High page id (12+).
440	47:12 The physical page frame number.
441	11:0 Low page id.
442	uint32_t fReadOnly : 1; Whether it's readonly page (rom or monitored in some way).
443	uint32_t u3Type : 3; The page type {RESERVED, MMIO, MMIO2, ROM, shadowed ROM, RAM}.
444	uint32_t u2PhysMon : 2; Physical access handler type {none, read, write, all}.
445	uint32_t u2VirtMon : 2; Virtual access handler type {none, read, write, all}..
446	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
447	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
448	uint32_t u20Reserved : 20; Reserved for later, mostly sharing stats.
449	uint32_t u32Tracking; The shadow PT tracking stuff, roughly.
450	@endverbatim
451	*
452	* Cost wise, this means we'll double the cost for guest memory. There isn't anyway
453	* around that I'm afraid. It means that the cost of dealing out 32GB of memory
454	* to one or more VMs is: (32GB >> PAGE_SHIFT) * 16 bytes, or 128MBs. Or another
455	* example, the VM heap cost when assigning 1GB to a VM will be: 4MB.
456	*
457	* A couple of cost examples for the total cost per-VM + kernel.
458	* 32-bit Windows and 32-bit linux:
459	* 1GB guest ram, 256K pages: 4MB + 2MB(+) = 6MB
460	* 4GB guest ram, 1M pages: 16MB + 8MB(+) = 24MB
461	* 32GB guest ram, 8M pages: 128MB + 64MB(+) = 192MB
462	* 64-bit Windows and 64-bit linux:
463	* 1GB guest ram, 256K pages: 4MB + 3MB(+) = 7MB
464	* 4GB guest ram, 1M pages: 16MB + 12MB(+) = 28MB
465	* 32GB guest ram, 8M pages: 128MB + 96MB(+) = 224MB
466	*
467	* UPDATE - 2007-09-27:
468	* Will need a ballooned flag/state too because we cannot
469	* trust the guest 100% and reporting the same page as ballooned more
470	* than once will put the GMM off balance.
471	*
472	*
473	* @subsection subsec_pgmPhys_Serializing Serializing Access
474	*
475	* Initially, we'll try a simple scheme:
476	*
477	* - The per-VM RAM tracking structures (PGMRAMRANGE) is only modified
478	* by the EMT thread of that VM while in the pgm critsect.
479	* - Other threads in the VM process that needs to make reliable use of
480	* the per-VM RAM tracking structures will enter the critsect.
481	* - No process external thread or kernel thread will ever try enter
482	* the pgm critical section, as that just won't work.
483	* - The idle thread (and similar threads) doesn't not need 100% reliable
484	* data when performing it tasks as the EMT thread will be the one to
485	* do the actual changes later anyway. So, as long as it only accesses
486	* the main ram range, it can do so by somehow preventing the VM from
487	* being destroyed while it works on it...
488	*
489	* - The over-commitment management, including the allocating/freeing
490	* chunks, is serialized by a ring-0 mutex lock (a fast one since the
491	* more mundane mutex implementation is broken on Linux).
492	* - A separate mutex is protecting the set of allocation chunks so
493	* that pages can be shared or/and freed up while some other VM is
494	* allocating more chunks. This mutex can be take from under the other
495	* one, but not the other way around.
496	*
497	*
498	* @subsection subsec_pgmPhys_Request VM Request interface
499	*
500	* When in ring-0 it will become necessary to send requests to a VM so it can
501	* for instance move a page while defragmenting during VM destroy. The idle
502	* thread will make use of this interface to request VMs to setup shared
503	* pages and to perform write monitoring of pages.
504	*
505	* I would propose an interface similar to the current VMReq interface, similar
506	* in that it doesn't require locking and that the one sending the request may
507	* wait for completion if it wishes to. This shouldn't be very difficult to
508	* realize.
509	*
510	* The requests themselves are also pretty simple. They are basically:
511	* -# Check that some precondition is still true.
512	* -# Do the update.
513	* -# Update all shadow page tables involved with the page.
514	*
515	* The 3rd step is identical to what we're already doing when updating a
516	* physical handler, see pgmHandlerPhysicalSetRamFlagsAndFlushShadowPTs.
517	*
518	*
519	*
520	* @section sec_pgmPhys_MappingCaches Mapping Caches
521	*
522	* In order to be able to map in and out memory and to be able to support
523	* guest with more RAM than we've got virtual address space, we'll employing
524	* a mapping cache. Normally ring-0 and ring-3 can share the same cache,
525	* however on 32-bit darwin the ring-0 code is running in a different memory
526	* context and therefore needs a separate cache. In raw-mode context we also
527	* need a separate cache. The 32-bit darwin mapping cache and the one for
528	* raw-mode context share a lot of code, see PGMRZDYNMAP.
529	*
530	*
531	* @subsection subsec_pgmPhys_MappingCaches_R3 Ring-3
532	*
533	* We've considered implementing the ring-3 mapping cache page based but found
534	* that this was bother some when one had to take into account TLBs+SMP and
535	* portability (missing the necessary APIs on several platforms). There were
536	* also some performance concerns with this approach which hadn't quite been
537	* worked out.
538	*
539	* Instead, we'll be mapping allocation chunks into the VM process. This simplifies
540	* matters greatly quite a bit since we don't need to invent any new ring-0 stuff,
541	* only some minor RTR0MEMOBJ mapping stuff. The main concern here is that mapping
542	* compared to the previous idea is that mapping or unmapping a 1MB chunk is more
543	* costly than a single page, although how much more costly is uncertain. We'll
544	* try address this by using a very big cache, preferably bigger than the actual
545	* VM RAM size if possible. The current VM RAM sizes should give some idea for
546	* 32-bit boxes, while on 64-bit we can probably get away with employing an
547	* unlimited cache.
548	*
549	* The cache have to parts, as already indicated, the ring-3 side and the
550	* ring-0 side.
551	*
552	* The ring-0 will be tied to the page allocator since it will operate on the
553	* memory objects it contains. It will therefore require the first ring-0 mutex
554	* discussed in @ref subsec_pgmPhys_Serializing. We
555	* some double house keeping wrt to who has mapped what I think, since both
556	* VMMR0.r0 and RTR0MemObj will keep track of mapping relations
557	*
558	* The ring-3 part will be protected by the pgm critsect. For simplicity, we'll
559	* require anyone that desires to do changes to the mapping cache to do that
560	* from within this critsect. Alternatively, we could employ a separate critsect
561	* for serializing changes to the mapping cache as this would reduce potential
562	* contention with other threads accessing mappings unrelated to the changes
563	* that are in process. We can see about this later, contention will show
564	* up in the statistics anyway, so it'll be simple to tell.
565	*
566	* The organization of the ring-3 part will be very much like how the allocation
567	* chunks are organized in ring-0, that is in an AVL tree by chunk id. To avoid
568	* having to walk the tree all the time, we'll have a couple of lookaside entries
569	* like in we do for I/O ports and MMIO in IOM.
570	*
571	* The simplified flow of a PGMPhysRead/Write function:
572	* -# Enter the PGM critsect.
573	* -# Lookup GCPhys in the ram ranges and get the Page ID.
574	* -# Calc the Allocation Chunk ID from the Page ID.
575	* -# Check the lookaside entries and then the AVL tree for the Chunk ID.
576	* If not found in cache:
577	* -# Call ring-0 and request it to be mapped and supply
578	* a chunk to be unmapped if the cache is maxed out already.
579	* -# Insert the new mapping into the AVL tree (id + R3 address).
580	* -# Update the relevant lookaside entry and return the mapping address.
581	* -# Do the read/write according to monitoring flags and everything.
582	* -# Leave the critsect.
583	*
584	*
585	* @section sec_pgmPhys_Fallback Fallback
586	*
587	* Current all the "second tier" hosts will not support the RTR0MemObjAllocPhysNC
588	* API and thus require a fallback.
589	*
590	* So, when RTR0MemObjAllocPhysNC returns VERR_NOT_SUPPORTED the page allocator
591	* will return to the ring-3 caller (and later ring-0) and asking it to seed
592	* the page allocator with some fresh pages (VERR_GMM_SEED_ME). Ring-3 will
593	* then perform an SUPR3PageAlloc(cbChunk >> PAGE_SHIFT) call and make a
594	* "SeededAllocPages" call to ring-0.
595	*
596	* The first time ring-0 sees the VERR_NOT_SUPPORTED failure it will disable
597	* all page sharing (zero page detection will continue). It will also force
598	* all allocations to come from the VM which seeded the page. Both these
599	* measures are taken to make sure that there will never be any need for
600	* mapping anything into ring-3 - everything will be mapped already.
601	*
602	* Whether we'll continue to use the current MM locked memory management
603	* for this I don't quite know (I'd prefer not to and just ditch that all
604	* together), we'll see what's simplest to do.
605	*
606	*
607	*
608	* @section sec_pgmPhys_Changes Changes
609	*
610	* Breakdown of the changes involved?
611	*/
612
613	/*******************************************************************************
614	* Header Files *
615	*******************************************************************************/
616	#define LOG_GROUP LOG_GROUP_PGM
617	#include <VBox/vmm/dbgf.h>
618	#include <VBox/vmm/pgm.h>
619	#include <VBox/vmm/cpum.h>
620	#include <VBox/vmm/iom.h>
621	#include <VBox/sup.h>
622	#include <VBox/vmm/mm.h>
623	#include <VBox/vmm/em.h>
624	#include <VBox/vmm/stam.h>
625	#ifdef VBOX_WITH_REM
626	# include <VBox/vmm/rem.h>
627	#endif
628	#include <VBox/vmm/selm.h>
629	#include <VBox/vmm/ssm.h>
630	#include <VBox/vmm/hm.h>
631	#include "PGMInternal.h"
632	#include <VBox/vmm/vm.h>
633	#include <VBox/vmm/uvm.h>
634	#include "PGMInline.h"
635
636	#include <VBox/dbg.h>
637	#include <VBox/param.h>
638	#include <VBox/err.h>
639
640	#include <iprt/asm.h>
641	#include <iprt/asm-amd64-x86.h>
642	#include <iprt/assert.h>
643	#include <iprt/env.h>
644	#include <iprt/mem.h>
645	#include <iprt/file.h>
646	#include <iprt/string.h>
647	#include <iprt/thread.h>
648
649
650	/*******************************************************************************
651	* Internal Functions *
652	*******************************************************************************/
653	static int pgmR3InitPaging(PVM pVM);
654	static int pgmR3InitStats(PVM pVM);
655	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
656	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
657	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
658	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser);
659	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
660	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
661	#ifdef VBOX_STRICT
662	static FNVMATSTATE pgmR3ResetNoMorePhysWritesFlag;
663	#endif
664	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0);
665	static void pgmR3ModeDataSwitch(PVM pVM, PVMCPU pVCpu, PGMMODE enmShw, PGMMODE enmGst);
666	static PGMMODE pgmR3CalcShadowMode(PVM pVM, PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher);
667
668	#ifdef VBOX_WITH_DEBUGGER
669	static FNDBGCCMD pgmR3CmdError;
670	static FNDBGCCMD pgmR3CmdSync;
671	static FNDBGCCMD pgmR3CmdSyncAlways;
672	# ifdef VBOX_STRICT
673	static FNDBGCCMD pgmR3CmdAssertCR3;
674	# endif
675	static FNDBGCCMD pgmR3CmdPhysToFile;
676	#endif
677
678
679	/*******************************************************************************
680	* Global Variables *
681	*******************************************************************************/
682	#ifdef VBOX_WITH_DEBUGGER
683	/** Argument descriptors for '.pgmerror' and '.pgmerroroff'. */
684	static const DBGCVARDESC g_aPgmErrorArgs[] =
685	{
686	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
687	{ 0, 1, DBGCVAR_CAT_STRING, 0, "where", "Error injection location." },
688	};
689
690	static const DBGCVARDESC g_aPgmPhysToFileArgs[] =
691	{
692	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
693	{ 1, 1, DBGCVAR_CAT_STRING, 0, "file", "The file name." },
694	{ 0, 1, DBGCVAR_CAT_STRING, 0, "nozero", "If present, zero pages are skipped." },
695	};
696
697	# ifdef DEBUG_sandervl
698	static const DBGCVARDESC g_aPgmCountPhysWritesArgs[] =
699	{
700	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
701	{ 1, 1, DBGCVAR_CAT_STRING, 0, "enabled", "on/off." },
702	{ 1, 1, DBGCVAR_CAT_NUMBER_NO_RANGE, 0, "interval", "Interval in ms." },
703	};
704	# endif
705
706	/** Command descriptors. */
707	static const DBGCCMD g_aCmds[] =
708	{
709	/* pszCmd, cArgsMin, cArgsMax, paArgDesc, cArgDescs, fFlags, pfnHandler pszSyntax, ....pszDescription */
710	{ "pgmsync", 0, 0, NULL, 0, 0, pgmR3CmdSync, "", "Sync the CR3 page." },
711	{ "pgmerror", 0, 1, &g_aPgmErrorArgs[0], 1, 0, pgmR3CmdError, "", "Enables inject runtime of errors into parts of PGM." },
712	{ "pgmerroroff", 0, 1, &g_aPgmErrorArgs[0], 1, 0, pgmR3CmdError, "", "Disables inject runtime errors into parts of PGM." },
713	# ifdef VBOX_STRICT
714	{ "pgmassertcr3", 0, 0, NULL, 0, 0, pgmR3CmdAssertCR3, "", "Check the shadow CR3 mapping." },
715	# ifdef VBOX_WITH_PAGE_SHARING
716	{ "pgmcheckduppages", 0, 0, NULL, 0, 0, pgmR3CmdCheckDuplicatePages, "", "Check for duplicate pages in all running VMs." },
717	{ "pgmsharedmodules", 0, 0, NULL, 0, 0, pgmR3CmdShowSharedModules, "", "Print shared modules info." },
718	# endif
719	# endif
720	{ "pgmsyncalways", 0, 0, NULL, 0, 0, pgmR3CmdSyncAlways, "", "Toggle permanent CR3 syncing." },
721	{ "pgmphystofile", 1, 2, &g_aPgmPhysToFileArgs[0], 2, 0, pgmR3CmdPhysToFile, "", "Save the physical memory to file." },
722	};
723	#endif
724
725
726
727
728	/*
729	* Shadow - 32-bit mode
730	*/
731	#define PGM_SHW_TYPE PGM_TYPE_32BIT
732	#define PGM_SHW_NAME(name) PGM_SHW_NAME_32BIT(name)
733	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_32BIT_STR(name)
734	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_32BIT_STR(name)
735	#include "PGMShw.h"
736
737	/* Guest - real mode */
738	#define PGM_GST_TYPE PGM_TYPE_REAL
739	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
740	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
741	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
742	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_REAL(name)
743	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_REAL_STR(name)
744	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_REAL_STR(name)
745	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
746	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD_PHYS
747	#include "PGMBth.h"
748	#include "PGMGstDefs.h"
749	#include "PGMGst.h"
750	#undef BTH_PGMPOOLKIND_PT_FOR_PT
751	#undef BTH_PGMPOOLKIND_ROOT
752	#undef PGM_BTH_NAME
753	#undef PGM_BTH_NAME_RC_STR
754	#undef PGM_BTH_NAME_R0_STR
755	#undef PGM_GST_TYPE
756	#undef PGM_GST_NAME
757	#undef PGM_GST_NAME_RC_STR
758	#undef PGM_GST_NAME_R0_STR
759
760	/* Guest - protected mode */
761	#define PGM_GST_TYPE PGM_TYPE_PROT
762	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
763	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
764	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
765	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_PROT(name)
766	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_PROT_STR(name)
767	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_PROT_STR(name)
768	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
769	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD_PHYS
770	#include "PGMBth.h"
771	#include "PGMGstDefs.h"
772	#include "PGMGst.h"
773	#undef BTH_PGMPOOLKIND_PT_FOR_PT
774	#undef BTH_PGMPOOLKIND_ROOT
775	#undef PGM_BTH_NAME
776	#undef PGM_BTH_NAME_RC_STR
777	#undef PGM_BTH_NAME_R0_STR
778	#undef PGM_GST_TYPE
779	#undef PGM_GST_NAME
780	#undef PGM_GST_NAME_RC_STR
781	#undef PGM_GST_NAME_R0_STR
782
783	/* Guest - 32-bit mode */
784	#define PGM_GST_TYPE PGM_TYPE_32BIT
785	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
786	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
787	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
788	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_32BIT(name)
789	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_32BIT_STR(name)
790	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_32BIT_STR(name)
791	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_32BIT_PT
792	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_32BIT_PT_FOR_32BIT_4MB
793	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD
794	#include "PGMBth.h"
795	#include "PGMGstDefs.h"
796	#include "PGMGst.h"
797	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
798	#undef BTH_PGMPOOLKIND_PT_FOR_PT
799	#undef BTH_PGMPOOLKIND_ROOT
800	#undef PGM_BTH_NAME
801	#undef PGM_BTH_NAME_RC_STR
802	#undef PGM_BTH_NAME_R0_STR
803	#undef PGM_GST_TYPE
804	#undef PGM_GST_NAME
805	#undef PGM_GST_NAME_RC_STR
806	#undef PGM_GST_NAME_R0_STR
807
808	#undef PGM_SHW_TYPE
809	#undef PGM_SHW_NAME
810	#undef PGM_SHW_NAME_RC_STR
811	#undef PGM_SHW_NAME_R0_STR
812
813
814	/*
815	* Shadow - PAE mode
816	*/
817	#define PGM_SHW_TYPE PGM_TYPE_PAE
818	#define PGM_SHW_NAME(name) PGM_SHW_NAME_PAE(name)
819	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_PAE_STR(name)
820	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_PAE_STR(name)
821	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
822	#include "PGMShw.h"
823
824	/* Guest - real mode */
825	#define PGM_GST_TYPE PGM_TYPE_REAL
826	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
827	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
828	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
829	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
830	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_REAL_STR(name)
831	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_REAL_STR(name)
832	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
833	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_PHYS
834	#include "PGMGstDefs.h"
835	#include "PGMBth.h"
836	#undef BTH_PGMPOOLKIND_PT_FOR_PT
837	#undef BTH_PGMPOOLKIND_ROOT
838	#undef PGM_BTH_NAME
839	#undef PGM_BTH_NAME_RC_STR
840	#undef PGM_BTH_NAME_R0_STR
841	#undef PGM_GST_TYPE
842	#undef PGM_GST_NAME
843	#undef PGM_GST_NAME_RC_STR
844	#undef PGM_GST_NAME_R0_STR
845
846	/* Guest - protected mode */
847	#define PGM_GST_TYPE PGM_TYPE_PROT
848	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
849	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
850	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
851	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PROT(name)
852	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_PROT_STR(name)
853	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PROT_STR(name)
854	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
855	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_PHYS
856	#include "PGMGstDefs.h"
857	#include "PGMBth.h"
858	#undef BTH_PGMPOOLKIND_PT_FOR_PT
859	#undef BTH_PGMPOOLKIND_ROOT
860	#undef PGM_BTH_NAME
861	#undef PGM_BTH_NAME_RC_STR
862	#undef PGM_BTH_NAME_R0_STR
863	#undef PGM_GST_TYPE
864	#undef PGM_GST_NAME
865	#undef PGM_GST_NAME_RC_STR
866	#undef PGM_GST_NAME_R0_STR
867
868	/* Guest - 32-bit mode */
869	#define PGM_GST_TYPE PGM_TYPE_32BIT
870	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
871	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
872	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
873	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_32BIT(name)
874	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_32BIT_STR(name)
875	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_32BIT_STR(name)
876	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
877	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
878	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_FOR_32BIT
879	#include "PGMGstDefs.h"
880	#include "PGMBth.h"
881	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
882	#undef BTH_PGMPOOLKIND_PT_FOR_PT
883	#undef BTH_PGMPOOLKIND_ROOT
884	#undef PGM_BTH_NAME
885	#undef PGM_BTH_NAME_RC_STR
886	#undef PGM_BTH_NAME_R0_STR
887	#undef PGM_GST_TYPE
888	#undef PGM_GST_NAME
889	#undef PGM_GST_NAME_RC_STR
890	#undef PGM_GST_NAME_R0_STR
891
892	/* Guest - PAE mode */
893	#define PGM_GST_TYPE PGM_TYPE_PAE
894	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
895	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
896	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
897	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PAE(name)
898	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_PAE_STR(name)
899	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PAE_STR(name)
900	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
901	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
902	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT
903	#include "PGMBth.h"
904	#include "PGMGstDefs.h"
905	#include "PGMGst.h"
906	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
907	#undef BTH_PGMPOOLKIND_PT_FOR_PT
908	#undef BTH_PGMPOOLKIND_ROOT
909	#undef PGM_BTH_NAME
910	#undef PGM_BTH_NAME_RC_STR
911	#undef PGM_BTH_NAME_R0_STR
912	#undef PGM_GST_TYPE
913	#undef PGM_GST_NAME
914	#undef PGM_GST_NAME_RC_STR
915	#undef PGM_GST_NAME_R0_STR
916
917	#undef PGM_SHW_TYPE
918	#undef PGM_SHW_NAME
919	#undef PGM_SHW_NAME_RC_STR
920	#undef PGM_SHW_NAME_R0_STR
921
922
923	/*
924	* Shadow - AMD64 mode
925	*/
926	#define PGM_SHW_TYPE PGM_TYPE_AMD64
927	#define PGM_SHW_NAME(name) PGM_SHW_NAME_AMD64(name)
928	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_AMD64_STR(name)
929	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_AMD64_STR(name)
930	#include "PGMShw.h"
931
932	#ifdef VBOX_WITH_64_BITS_GUESTS
933	/* Guest - AMD64 mode */
934	# define PGM_GST_TYPE PGM_TYPE_AMD64
935	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
936	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
937	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
938	# define PGM_BTH_NAME(name) PGM_BTH_NAME_AMD64_AMD64(name)
939	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_AMD64_AMD64_STR(name)
940	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_AMD64_AMD64_STR(name)
941	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
942	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
943	# define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_64BIT_PML4
944	# include "PGMBth.h"
945	# include "PGMGstDefs.h"
946	# include "PGMGst.h"
947	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
948	# undef BTH_PGMPOOLKIND_PT_FOR_PT
949	# undef BTH_PGMPOOLKIND_ROOT
950	# undef PGM_BTH_NAME
951	# undef PGM_BTH_NAME_RC_STR
952	# undef PGM_BTH_NAME_R0_STR
953	# undef PGM_GST_TYPE
954	# undef PGM_GST_NAME
955	# undef PGM_GST_NAME_RC_STR
956	# undef PGM_GST_NAME_R0_STR
957	#endif /* VBOX_WITH_64_BITS_GUESTS */
958
959	#undef PGM_SHW_TYPE
960	#undef PGM_SHW_NAME
961	#undef PGM_SHW_NAME_RC_STR
962	#undef PGM_SHW_NAME_R0_STR
963
964
965	/*
966	* Shadow - Nested paging mode
967	*/
968	#define PGM_SHW_TYPE PGM_TYPE_NESTED
969	#define PGM_SHW_NAME(name) PGM_SHW_NAME_NESTED(name)
970	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_NESTED_STR(name)
971	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_NESTED_STR(name)
972	#include "PGMShw.h"
973
974	/* Guest - real mode */
975	#define PGM_GST_TYPE PGM_TYPE_REAL
976	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
977	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
978	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
979	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_REAL(name)
980	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_REAL_STR(name)
981	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_REAL_STR(name)
982	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
983	#include "PGMGstDefs.h"
984	#include "PGMBth.h"
985	#undef BTH_PGMPOOLKIND_PT_FOR_PT
986	#undef PGM_BTH_NAME
987	#undef PGM_BTH_NAME_RC_STR
988	#undef PGM_BTH_NAME_R0_STR
989	#undef PGM_GST_TYPE
990	#undef PGM_GST_NAME
991	#undef PGM_GST_NAME_RC_STR
992	#undef PGM_GST_NAME_R0_STR
993
994	/* Guest - protected mode */
995	#define PGM_GST_TYPE PGM_TYPE_PROT
996	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
997	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
998	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
999	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PROT(name)
1000	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_PROT_STR(name)
1001	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PROT_STR(name)
1002	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
1003	#include "PGMGstDefs.h"
1004	#include "PGMBth.h"
1005	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1006	#undef PGM_BTH_NAME
1007	#undef PGM_BTH_NAME_RC_STR
1008	#undef PGM_BTH_NAME_R0_STR
1009	#undef PGM_GST_TYPE
1010	#undef PGM_GST_NAME
1011	#undef PGM_GST_NAME_RC_STR
1012	#undef PGM_GST_NAME_R0_STR
1013
1014	/* Guest - 32-bit mode */
1015	#define PGM_GST_TYPE PGM_TYPE_32BIT
1016	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
1017	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
1018	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
1019	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_32BIT(name)
1020	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_32BIT_STR(name)
1021	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_32BIT_STR(name)
1022	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
1023	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
1024	#include "PGMGstDefs.h"
1025	#include "PGMBth.h"
1026	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1027	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1028	#undef PGM_BTH_NAME
1029	#undef PGM_BTH_NAME_RC_STR
1030	#undef PGM_BTH_NAME_R0_STR
1031	#undef PGM_GST_TYPE
1032	#undef PGM_GST_NAME
1033	#undef PGM_GST_NAME_RC_STR
1034	#undef PGM_GST_NAME_R0_STR
1035
1036	/* Guest - PAE mode */
1037	#define PGM_GST_TYPE PGM_TYPE_PAE
1038	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
1039	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
1040	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
1041	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PAE(name)
1042	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_PAE_STR(name)
1043	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PAE_STR(name)
1044	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1045	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1046	#include "PGMGstDefs.h"
1047	#include "PGMBth.h"
1048	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1049	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1050	#undef PGM_BTH_NAME
1051	#undef PGM_BTH_NAME_RC_STR
1052	#undef PGM_BTH_NAME_R0_STR
1053	#undef PGM_GST_TYPE
1054	#undef PGM_GST_NAME
1055	#undef PGM_GST_NAME_RC_STR
1056	#undef PGM_GST_NAME_R0_STR
1057
1058	#ifdef VBOX_WITH_64_BITS_GUESTS
1059	/* Guest - AMD64 mode */
1060	# define PGM_GST_TYPE PGM_TYPE_AMD64
1061	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
1062	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
1063	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
1064	# define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_AMD64(name)
1065	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_AMD64_STR(name)
1066	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_AMD64_STR(name)
1067	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1068	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1069	# include "PGMGstDefs.h"
1070	# include "PGMBth.h"
1071	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
1072	# undef BTH_PGMPOOLKIND_PT_FOR_PT
1073	# undef PGM_BTH_NAME
1074	# undef PGM_BTH_NAME_RC_STR
1075	# undef PGM_BTH_NAME_R0_STR
1076	# undef PGM_GST_TYPE
1077	# undef PGM_GST_NAME
1078	# undef PGM_GST_NAME_RC_STR
1079	# undef PGM_GST_NAME_R0_STR
1080	#endif /* VBOX_WITH_64_BITS_GUESTS */
1081
1082	#undef PGM_SHW_TYPE
1083	#undef PGM_SHW_NAME
1084	#undef PGM_SHW_NAME_RC_STR
1085	#undef PGM_SHW_NAME_R0_STR
1086
1087
1088	/*
1089	* Shadow - EPT
1090	*/
1091	#define PGM_SHW_TYPE PGM_TYPE_EPT
1092	#define PGM_SHW_NAME(name) PGM_SHW_NAME_EPT(name)
1093	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_EPT_STR(name)
1094	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_EPT_STR(name)
1095	#include "PGMShw.h"
1096
1097	/* Guest - real mode */
1098	#define PGM_GST_TYPE PGM_TYPE_REAL
1099	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
1100	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
1101	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
1102	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_REAL(name)
1103	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_REAL_STR(name)
1104	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_REAL_STR(name)
1105	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
1106	#include "PGMGstDefs.h"
1107	#include "PGMBth.h"
1108	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1109	#undef PGM_BTH_NAME
1110	#undef PGM_BTH_NAME_RC_STR
1111	#undef PGM_BTH_NAME_R0_STR
1112	#undef PGM_GST_TYPE
1113	#undef PGM_GST_NAME
1114	#undef PGM_GST_NAME_RC_STR
1115	#undef PGM_GST_NAME_R0_STR
1116
1117	/* Guest - protected mode */
1118	#define PGM_GST_TYPE PGM_TYPE_PROT
1119	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
1120	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
1121	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
1122	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_PROT(name)
1123	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_PROT_STR(name)
1124	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_PROT_STR(name)
1125	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
1126	#include "PGMGstDefs.h"
1127	#include "PGMBth.h"
1128	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1129	#undef PGM_BTH_NAME
1130	#undef PGM_BTH_NAME_RC_STR
1131	#undef PGM_BTH_NAME_R0_STR
1132	#undef PGM_GST_TYPE
1133	#undef PGM_GST_NAME
1134	#undef PGM_GST_NAME_RC_STR
1135	#undef PGM_GST_NAME_R0_STR
1136
1137	/* Guest - 32-bit mode */
1138	#define PGM_GST_TYPE PGM_TYPE_32BIT
1139	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
1140	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
1141	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
1142	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_32BIT(name)
1143	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_32BIT_STR(name)
1144	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_32BIT_STR(name)
1145	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
1146	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
1147	#include "PGMGstDefs.h"
1148	#include "PGMBth.h"
1149	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1150	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1151	#undef PGM_BTH_NAME
1152	#undef PGM_BTH_NAME_RC_STR
1153	#undef PGM_BTH_NAME_R0_STR
1154	#undef PGM_GST_TYPE
1155	#undef PGM_GST_NAME
1156	#undef PGM_GST_NAME_RC_STR
1157	#undef PGM_GST_NAME_R0_STR
1158
1159	/* Guest - PAE mode */
1160	#define PGM_GST_TYPE PGM_TYPE_PAE
1161	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
1162	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
1163	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
1164	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_PAE(name)
1165	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_PAE_STR(name)
1166	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_PAE_STR(name)
1167	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1168	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1169	#include "PGMGstDefs.h"
1170	#include "PGMBth.h"
1171	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1172	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1173	#undef PGM_BTH_NAME
1174	#undef PGM_BTH_NAME_RC_STR
1175	#undef PGM_BTH_NAME_R0_STR
1176	#undef PGM_GST_TYPE
1177	#undef PGM_GST_NAME
1178	#undef PGM_GST_NAME_RC_STR
1179	#undef PGM_GST_NAME_R0_STR
1180
1181	#ifdef VBOX_WITH_64_BITS_GUESTS
1182	/* Guest - AMD64 mode */
1183	# define PGM_GST_TYPE PGM_TYPE_AMD64
1184	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
1185	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
1186	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
1187	# define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_AMD64(name)
1188	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_AMD64_STR(name)
1189	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_AMD64_STR(name)
1190	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1191	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1192	# include "PGMGstDefs.h"
1193	# include "PGMBth.h"
1194	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
1195	# undef BTH_PGMPOOLKIND_PT_FOR_PT
1196	# undef PGM_BTH_NAME
1197	# undef PGM_BTH_NAME_RC_STR
1198	# undef PGM_BTH_NAME_R0_STR
1199	# undef PGM_GST_TYPE
1200	# undef PGM_GST_NAME
1201	# undef PGM_GST_NAME_RC_STR
1202	# undef PGM_GST_NAME_R0_STR
1203	#endif /* VBOX_WITH_64_BITS_GUESTS */
1204
1205	#undef PGM_SHW_TYPE
1206	#undef PGM_SHW_NAME
1207	#undef PGM_SHW_NAME_RC_STR
1208	#undef PGM_SHW_NAME_R0_STR
1209
1210
1211
1212	/**
1213	* Initiates the paging of VM.
1214	*
1215	* @returns VBox status code.
1216	* @param pVM Pointer to VM structure.
1217	*/
1218	VMMR3DECL(int) PGMR3Init(PVM pVM)
1219	{
1220	LogFlow(("PGMR3Init:\n"));
1221	PCFGMNODE pCfgPGM = CFGMR3GetChild(CFGMR3GetRoot(pVM), "/PGM");
1222	int rc;
1223
1224	/*
1225	* Assert alignment and sizes.
1226	*/
1227	AssertCompile(sizeof(pVM->pgm.s) <= sizeof(pVM->pgm.padding));
1228	AssertCompile(sizeof(pVM->aCpus[0].pgm.s) <= sizeof(pVM->aCpus[0].pgm.padding));
1229	AssertCompileMemberAlignment(PGM, CritSectX, sizeof(uintptr_t));
1230
1231	/*
1232	* Init the structure.
1233	*/
1234	pVM->pgm.s.offVM = RT_OFFSETOF(VM, pgm.s);
1235	pVM->pgm.s.offVCpuPGM = RT_OFFSETOF(VMCPU, pgm.s);
1236
1237	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.aHandyPages); i++)
1238	{
1239	pVM->pgm.s.aHandyPages[i].HCPhysGCPhys = NIL_RTHCPHYS;
1240	pVM->pgm.s.aHandyPages[i].idPage = NIL_GMM_PAGEID;
1241	pVM->pgm.s.aHandyPages[i].idSharedPage = NIL_GMM_PAGEID;
1242	}
1243
1244	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.aLargeHandyPage); i++)
1245	{
1246	pVM->pgm.s.aLargeHandyPage[i].HCPhysGCPhys = NIL_RTHCPHYS;
1247	pVM->pgm.s.aLargeHandyPage[i].idPage = NIL_GMM_PAGEID;
1248	pVM->pgm.s.aLargeHandyPage[i].idSharedPage = NIL_GMM_PAGEID;
1249	}
1250
1251	/* Init the per-CPU part. */
1252	for (VMCPUID idCpu = 0; idCpu < pVM->cCpus; idCpu++)
1253	{
1254	PVMCPU pVCpu = &pVM->aCpus[idCpu];
1255	PPGMCPU pPGM = &pVCpu->pgm.s;
1256
1257	pPGM->offVM = (uintptr_t)&pVCpu->pgm.s - (uintptr_t)pVM;
1258	pPGM->offVCpu = RT_OFFSETOF(VMCPU, pgm.s);
1259	pPGM->offPGM = (uintptr_t)&pVCpu->pgm.s - (uintptr_t)&pVM->pgm.s;
1260
1261	pPGM->enmShadowMode = PGMMODE_INVALID;
1262	pPGM->enmGuestMode = PGMMODE_INVALID;
1263
1264	pPGM->GCPhysCR3 = NIL_RTGCPHYS;
1265
1266	pPGM->pGst32BitPdR3 = NULL;
1267	pPGM->pGstPaePdptR3 = NULL;
1268	pPGM->pGstAmd64Pml4R3 = NULL;
1269	#ifndef VBOX_WITH_2X_4GB_ADDR_SPACE
1270	pPGM->pGst32BitPdR0 = NIL_RTR0PTR;
1271	pPGM->pGstPaePdptR0 = NIL_RTR0PTR;
1272	pPGM->pGstAmd64Pml4R0 = NIL_RTR0PTR;
1273	#endif
1274	pPGM->pGst32BitPdRC = NIL_RTRCPTR;
1275	pPGM->pGstPaePdptRC = NIL_RTRCPTR;
1276	for (unsigned i = 0; i < RT_ELEMENTS(pVCpu->pgm.s.apGstPaePDsR3); i++)
1277	{
1278	pPGM->apGstPaePDsR3[i] = NULL;
1279	#ifndef VBOX_WITH_2X_4GB_ADDR_SPACE
1280	pPGM->apGstPaePDsR0[i] = NIL_RTR0PTR;
1281	#endif
1282	pPGM->apGstPaePDsRC[i] = NIL_RTRCPTR;
1283	pPGM->aGCPhysGstPaePDs[i] = NIL_RTGCPHYS;
1284	pPGM->aGstPaePdpeRegs[i].u = UINT64_MAX;
1285	pPGM->aGCPhysGstPaePDsMonitored[i] = NIL_RTGCPHYS;
1286	}
1287
1288	pPGM->fA20Enabled = true;
1289	pPGM->GCPhysA20Mask = ~((RTGCPHYS)!pPGM->fA20Enabled << 20);
1290	}
1291
1292	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1293	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(32) - 1; /* default; checked later */
1294	pVM->pgm.s.GCPtrPrevRamRangeMapping = MM_HYPER_AREA_ADDRESS;
1295
1296	rc = CFGMR3QueryBoolDef(CFGMR3GetRoot(pVM), "RamPreAlloc", &pVM->pgm.s.fRamPreAlloc,
1297	#ifdef VBOX_WITH_PREALLOC_RAM_BY_DEFAULT
1298	true
1299	#else
1300	false
1301	#endif
1302	);
1303	AssertLogRelRCReturn(rc, rc);
1304
1305	#if HC_ARCH_BITS == 32
1306	# ifdef RT_OS_DARWIN
1307	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, _1G / GMM_CHUNK_SIZE * 3);
1308	# else
1309	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, _1G / GMM_CHUNK_SIZE);
1310	# endif
1311	#else
1312	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, UINT32_MAX);
1313	#endif
1314	AssertLogRelRCReturn(rc, rc);
1315	for (uint32_t i = 0; i < RT_ELEMENTS(pVM->pgm.s.ChunkR3Map.Tlb.aEntries); i++)
1316	pVM->pgm.s.ChunkR3Map.Tlb.aEntries[i].idChunk = NIL_GMM_CHUNKID;
1317
1318	/*
1319	* Get the configured RAM size - to estimate saved state size.
1320	*/
1321	uint64_t cbRam;
1322	rc = CFGMR3QueryU64(CFGMR3GetRoot(pVM), "RamSize", &cbRam);
1323	if (rc == VERR_CFGM_VALUE_NOT_FOUND)
1324	cbRam = 0;
1325	else if (RT_SUCCESS(rc))
1326	{
1327	if (cbRam < PAGE_SIZE)
1328	cbRam = 0;
1329	cbRam = RT_ALIGN_64(cbRam, PAGE_SIZE);
1330	}
1331	else
1332	{
1333	AssertMsgFailed(("Configuration error: Failed to query integer \"RamSize\", rc=%Rrc.\n", rc));
1334	return rc;
1335	}
1336
1337	/*
1338	* Check for PCI pass-through.
1339	*/
1340	rc = CFGMR3QueryBoolDef(pCfgPGM, "PciPassThrough", &pVM->pgm.s.fPciPassthrough, false);
1341	AssertMsgRCReturn(rc, ("Configuration error: Failed to query integer \"PciPassThrough\", rc=%Rrc.\n", rc), rc);
1342	AssertLogRelReturn(!pVM->pgm.s.fPciPassthrough \|\| pVM->pgm.s.fRamPreAlloc, VERR_INVALID_PARAMETER);
1343
1344	#ifdef VBOX_WITH_STATISTICS
1345	/*
1346	* Allocate memory for the statistics before someone tries to use them.
1347	*/
1348	size_t cbTotalStats = RT_ALIGN_Z(sizeof(PGMSTATS), 64) + RT_ALIGN_Z(sizeof(PGMCPUSTATS), 64) * pVM->cCpus;
1349	void *pv;
1350	rc = MMHyperAlloc(pVM, RT_ALIGN_Z(cbTotalStats, PAGE_SIZE), PAGE_SIZE, MM_TAG_PGM, &pv);
1351	AssertRCReturn(rc, rc);
1352
1353	pVM->pgm.s.pStatsR3 = (PGMSTATS *)pv;
1354	pVM->pgm.s.pStatsR0 = MMHyperCCToR0(pVM, pv);
1355	pVM->pgm.s.pStatsRC = MMHyperCCToRC(pVM, pv);
1356	pv = (uint8_t *)pv + RT_ALIGN_Z(sizeof(PGMSTATS), 64);
1357
1358	for (VMCPUID iCpu = 0; iCpu < pVM->cCpus; iCpu++)
1359	{
1360	pVM->aCpus[iCpu].pgm.s.pStatsR3 = (PGMCPUSTATS *)pv;
1361	pVM->aCpus[iCpu].pgm.s.pStatsR0 = MMHyperCCToR0(pVM, pv);
1362	pVM->aCpus[iCpu].pgm.s.pStatsRC = MMHyperCCToRC(pVM, pv);
1363
1364	pv = (uint8_t *)pv + RT_ALIGN_Z(sizeof(PGMCPUSTATS), 64);
1365	}
1366	#endif /* VBOX_WITH_STATISTICS */
1367
1368	/*
1369	* Register callbacks, string formatters and the saved state data unit.
1370	*/
1371	#ifdef VBOX_STRICT
1372	VMR3AtStateRegister(pVM->pUVM, pgmR3ResetNoMorePhysWritesFlag, NULL);
1373	#endif
1374	PGMRegisterStringFormatTypes();
1375
1376	rc = pgmR3InitSavedState(pVM, cbRam);
1377	if (RT_FAILURE(rc))
1378	return rc;
1379
1380	/*
1381	* Initialize the PGM critical section and flush the phys TLBs
1382	*/
1383	rc = PDMR3CritSectInit(pVM, &pVM->pgm.s.CritSectX, RT_SRC_POS, "PGM");
1384	AssertRCReturn(rc, rc);
1385
1386	PGMR3PhysChunkInvalidateTLB(pVM);
1387	pgmPhysInvalidatePageMapTLB(pVM);
1388
1389	/*
1390	* For the time being we sport a full set of handy pages in addition to the base
1391	* memory to simplify things.
1392	*/
1393	rc = MMR3ReserveHandyPages(pVM, RT_ELEMENTS(pVM->pgm.s.aHandyPages)); /** @todo this should be changed to PGM_HANDY_PAGES_MIN but this needs proper testing... */
1394	AssertRCReturn(rc, rc);
1395
1396	/*
1397	* Trees
1398	*/
1399	rc = MMHyperAlloc(pVM, sizeof(PGMTREES), 0, MM_TAG_PGM, (void **)&pVM->pgm.s.pTreesR3);
1400	if (RT_SUCCESS(rc))
1401	{
1402	pVM->pgm.s.pTreesR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pTreesR3);
1403	pVM->pgm.s.pTreesRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pTreesR3);
1404
1405	/*
1406	* Allocate the zero page.
1407	*/
1408	rc = MMHyperAlloc(pVM, PAGE_SIZE, PAGE_SIZE, MM_TAG_PGM, &pVM->pgm.s.pvZeroPgR3);
1409	}
1410	if (RT_SUCCESS(rc))
1411	{
1412	pVM->pgm.s.pvZeroPgRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pvZeroPgR3);
1413	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
1414	pVM->pgm.s.HCPhysZeroPg = MMR3HyperHCVirt2HCPhys(pVM, pVM->pgm.s.pvZeroPgR3);
1415	AssertRelease(pVM->pgm.s.HCPhysZeroPg != NIL_RTHCPHYS);
1416
1417	/*
1418	* Allocate the invalid MMIO page.
1419	* (The invalid bits in HCPhysInvMmioPg are set later on init complete.)
1420	*/
1421	rc = MMHyperAlloc(pVM, PAGE_SIZE, PAGE_SIZE, MM_TAG_PGM, &pVM->pgm.s.pvMmioPgR3);
1422	}
1423	if (RT_SUCCESS(rc))
1424	{
1425	ASMMemFill32(pVM->pgm.s.pvMmioPgR3, PAGE_SIZE, 0xfeedface);
1426	pVM->pgm.s.HCPhysMmioPg = MMR3HyperHCVirt2HCPhys(pVM, pVM->pgm.s.pvMmioPgR3);
1427	AssertRelease(pVM->pgm.s.HCPhysMmioPg != NIL_RTHCPHYS);
1428	pVM->pgm.s.HCPhysInvMmioPg = pVM->pgm.s.HCPhysMmioPg;
1429
1430	/*
1431	* Init the paging.
1432	*/
1433	rc = pgmR3InitPaging(pVM);
1434	}
1435	if (RT_SUCCESS(rc))
1436	{
1437	/*
1438	* Init the page pool.
1439	*/
1440	rc = pgmR3PoolInit(pVM);
1441	}
1442	if (RT_SUCCESS(rc))
1443	{
1444	for (VMCPUID i = 0; i < pVM->cCpus; i++)
1445	{
1446	PVMCPU pVCpu = &pVM->aCpus[i];
1447	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
1448	if (RT_FAILURE(rc))
1449	break;
1450	}
1451	}
1452
1453	if (RT_SUCCESS(rc))
1454	{
1455	/*
1456	* Info & statistics
1457	*/
1458	DBGFR3InfoRegisterInternal(pVM, "mode",
1459	"Shows the current paging mode. "
1460	"Recognizes 'all', 'guest', 'shadow' and 'host' as arguments, defaulting to 'all' if nothing is given.",
1461	pgmR3InfoMode);
1462	DBGFR3InfoRegisterInternal(pVM, "pgmcr3",
1463	"Dumps all the entries in the top level paging table. No arguments.",
1464	pgmR3InfoCr3);
1465	DBGFR3InfoRegisterInternal(pVM, "phys",
1466	"Dumps all the physical address ranges. No arguments.",
1467	pgmR3PhysInfo);
1468	DBGFR3InfoRegisterInternal(pVM, "handlers",
1469	"Dumps physical, virtual and hyper virtual handlers. "
1470	"Pass 'phys', 'virt', 'hyper' as argument if only one kind is wanted."
1471	"Add 'nost' if the statistics are unwanted, use together with 'all' or explicit selection.",
1472	pgmR3InfoHandlers);
1473	DBGFR3InfoRegisterInternal(pVM, "mappings",
1474	"Dumps guest mappings.",
1475	pgmR3MapInfo);
1476
1477	pgmR3InitStats(pVM);
1478
1479	#ifdef VBOX_WITH_DEBUGGER
1480	/*
1481	* Debugger commands.
1482	*/
1483	static bool s_fRegisteredCmds = false;
1484	if (!s_fRegisteredCmds)
1485	{
1486	int rc2 = DBGCRegisterCommands(&g_aCmds[0], RT_ELEMENTS(g_aCmds));
1487	if (RT_SUCCESS(rc2))
1488	s_fRegisteredCmds = true;
1489	}
1490	#endif
1491	return VINF_SUCCESS;
1492	}
1493
1494	/* Almost no cleanup necessary, MM frees all memory. */
1495	PDMR3CritSectDelete(&pVM->pgm.s.CritSectX);
1496
1497	return rc;
1498	}
1499
1500
1501	/**
1502	* Init paging.
1503	*
1504	* Since we need to check what mode the host is operating in before we can choose
1505	* the right paging functions for the host we have to delay this until R0 has
1506	* been initialized.
1507	*
1508	* @returns VBox status code.
1509	* @param pVM Pointer to the VM.
1510	*/
1511	static int pgmR3InitPaging(PVM pVM)
1512	{
1513	/*
1514	* Force a recalculation of modes and switcher so everyone gets notified.
1515	*/
1516	for (VMCPUID i = 0; i < pVM->cCpus; i++)
1517	{
1518	PVMCPU pVCpu = &pVM->aCpus[i];
1519
1520	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
1521	pVCpu->pgm.s.enmGuestMode = PGMMODE_INVALID;
1522	}
1523
1524	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1525
1526	/*
1527	* Allocate static mapping space for whatever the cr3 register
1528	* points to and in the case of PAE mode to the 4 PDs.
1529	*/
1530	int rc = MMR3HyperReserve(pVM, PAGE_SIZE * 5, "CR3 mapping", &pVM->pgm.s.GCPtrCR3Mapping);
1531	if (RT_FAILURE(rc))
1532	{
1533	AssertMsgFailed(("Failed to reserve two pages for cr mapping in HMA, rc=%Rrc\n", rc));
1534	return rc;
1535	}
1536	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
1537
1538	/*
1539	* Allocate pages for the three possible intermediate contexts
1540	* (AMD64, PAE and plain 32-Bit). We maintain all three contexts
1541	* for the sake of simplicity. The AMD64 uses the PAE for the
1542	* lower levels, making the total number of pages 11 (3 + 7 + 1).
1543	*
1544	* We assume that two page tables will be enought for the core code
1545	* mappings (HC virtual and identity).
1546	*/
1547	pVM->pgm.s.pInterPD = (PX86PD)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPD, VERR_NO_PAGE_MEMORY);
1548	pVM->pgm.s.apInterPTs[0] = (PX86PT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.apInterPTs[0], VERR_NO_PAGE_MEMORY);
1549	pVM->pgm.s.apInterPTs[1] = (PX86PT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.apInterPTs[1], VERR_NO_PAGE_MEMORY);
1550	pVM->pgm.s.apInterPaePTs[0] = (PX86PTPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePTs[0], VERR_NO_PAGE_MEMORY);
1551	pVM->pgm.s.apInterPaePTs[1] = (PX86PTPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePTs[1], VERR_NO_PAGE_MEMORY);
1552	pVM->pgm.s.apInterPaePDs[0] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[0], VERR_NO_PAGE_MEMORY);
1553	pVM->pgm.s.apInterPaePDs[1] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[1], VERR_NO_PAGE_MEMORY);
1554	pVM->pgm.s.apInterPaePDs[2] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[2], VERR_NO_PAGE_MEMORY);
1555	pVM->pgm.s.apInterPaePDs[3] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[3], VERR_NO_PAGE_MEMORY);
1556	pVM->pgm.s.pInterPaePDPT = (PX86PDPT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePDPT, VERR_NO_PAGE_MEMORY);
1557	pVM->pgm.s.pInterPaePDPT64 = (PX86PDPT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePDPT64, VERR_NO_PAGE_MEMORY);
1558	pVM->pgm.s.pInterPaePML4 = (PX86PML4)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePML4, VERR_NO_PAGE_MEMORY);
1559
1560	pVM->pgm.s.HCPhysInterPD = MMPage2Phys(pVM, pVM->pgm.s.pInterPD);
1561	AssertRelease(pVM->pgm.s.HCPhysInterPD != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPD & PAGE_OFFSET_MASK));
1562	pVM->pgm.s.HCPhysInterPaePDPT = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT);
1563	AssertRelease(pVM->pgm.s.HCPhysInterPaePDPT != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePDPT & PAGE_OFFSET_MASK));
1564	pVM->pgm.s.HCPhysInterPaePML4 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePML4);
1565	AssertRelease(pVM->pgm.s.HCPhysInterPaePML4 != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePML4 & PAGE_OFFSET_MASK) && pVM->pgm.s.HCPhysInterPaePML4 < 0xffffffff);
1566
1567	/*
1568	* Initialize the pages, setting up the PML4 and PDPT for repetitive 4GB action.
1569	*/
1570	ASMMemZeroPage(pVM->pgm.s.pInterPD);
1571	ASMMemZeroPage(pVM->pgm.s.apInterPTs[0]);
1572	ASMMemZeroPage(pVM->pgm.s.apInterPTs[1]);
1573
1574	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[0]);
1575	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[1]);
1576
1577	ASMMemZeroPage(pVM->pgm.s.pInterPaePDPT);
1578	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.apInterPaePDs); i++)
1579	{
1580	ASMMemZeroPage(pVM->pgm.s.apInterPaePDs[i]);
1581	pVM->pgm.s.pInterPaePDPT->a[i].u = X86_PDPE_P \| PGM_PLXFLAGS_PERMANENT
1582	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[i]);
1583	}
1584
1585	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.pInterPaePDPT64->a); i++)
1586	{
1587	const unsigned iPD = i % RT_ELEMENTS(pVM->pgm.s.apInterPaePDs);
1588	pVM->pgm.s.pInterPaePDPT64->a[i].u = X86_PDPE_P \| X86_PDPE_RW \| X86_PDPE_US \| X86_PDPE_A \| PGM_PLXFLAGS_PERMANENT
1589	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[iPD]);
1590	}
1591
1592	RTHCPHYS HCPhysInterPaePDPT64 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64);
1593	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.pInterPaePML4->a); i++)
1594	pVM->pgm.s.pInterPaePML4->a[i].u = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_US \| X86_PML4E_A \| PGM_PLXFLAGS_PERMANENT
1595	\| HCPhysInterPaePDPT64;
1596
1597	/*
1598	* Initialize paging workers and mode from current host mode
1599	* and the guest running in real mode.
1600	*/
1601	pVM->pgm.s.enmHostMode = SUPR3GetPagingMode();
1602	switch (pVM->pgm.s.enmHostMode)
1603	{
1604	case SUPPAGINGMODE_32_BIT:
1605	case SUPPAGINGMODE_32_BIT_GLOBAL:
1606	case SUPPAGINGMODE_PAE:
1607	case SUPPAGINGMODE_PAE_GLOBAL:
1608	case SUPPAGINGMODE_PAE_NX:
1609	case SUPPAGINGMODE_PAE_GLOBAL_NX:
1610	break;
1611
1612	case SUPPAGINGMODE_AMD64:
1613	case SUPPAGINGMODE_AMD64_GLOBAL:
1614	case SUPPAGINGMODE_AMD64_NX:
1615	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
1616	#ifndef VBOX_WITH_HYBRID_32BIT_KERNEL
1617	if (ARCH_BITS != 64)
1618	{
1619	AssertMsgFailed(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1620	LogRel(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1621	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1622	}
1623	#endif
1624	break;
1625	default:
1626	AssertMsgFailed(("Host mode %d is not supported\n", pVM->pgm.s.enmHostMode));
1627	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1628	}
1629	rc = pgmR3ModeDataInit(pVM, false /* don't resolve GC and R0 syms yet */);
1630	if (RT_SUCCESS(rc))
1631	{
1632	LogFlow(("pgmR3InitPaging: returns successfully\n"));
1633	#if HC_ARCH_BITS == 64
1634	LogRel(("Debug: HCPhysInterPD=%RHp HCPhysInterPaePDPT=%RHp HCPhysInterPaePML4=%RHp\n",
1635	pVM->pgm.s.HCPhysInterPD, pVM->pgm.s.HCPhysInterPaePDPT, pVM->pgm.s.HCPhysInterPaePML4));
1636	LogRel(("Debug: apInterPTs={%RHp,%RHp} apInterPaePTs={%RHp,%RHp} apInterPaePDs={%RHp,%RHp,%RHp,%RHp} pInterPaePDPT64=%RHp\n",
1637	MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[1]),
1638	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[1]),
1639	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[1]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[2]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[3]),
1640	MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64)));
1641	#endif
1642
1643	/*
1644	* Log the host paging mode. It may come in handy.
1645	*/
1646	const char *pszHostMode;
1647	switch (pVM->pgm.s.enmHostMode)
1648	{
1649	case SUPPAGINGMODE_32_BIT: pszHostMode = "32-bit"; break;
1650	case SUPPAGINGMODE_32_BIT_GLOBAL: pszHostMode = "32-bit+PGE"; break;
1651	case SUPPAGINGMODE_PAE: pszHostMode = "PAE"; break;
1652	case SUPPAGINGMODE_PAE_GLOBAL: pszHostMode = "PAE+PGE"; break;
1653	case SUPPAGINGMODE_PAE_NX: pszHostMode = "PAE+NXE"; break;
1654	case SUPPAGINGMODE_PAE_GLOBAL_NX: pszHostMode = "PAE+PGE+NXE"; break;
1655	case SUPPAGINGMODE_AMD64: pszHostMode = "AMD64"; break;
1656	case SUPPAGINGMODE_AMD64_GLOBAL: pszHostMode = "AMD64+PGE"; break;
1657	case SUPPAGINGMODE_AMD64_NX: pszHostMode = "AMD64+NX"; break;
1658	case SUPPAGINGMODE_AMD64_GLOBAL_NX: pszHostMode = "AMD64+PGE+NX"; break;
1659	default: pszHostMode = "???"; break;
1660	}
1661	LogRel(("Host paging mode: %s\n", pszHostMode));
1662
1663	return VINF_SUCCESS;
1664	}
1665
1666	LogFlow(("pgmR3InitPaging: returns %Rrc\n", rc));
1667	return rc;
1668	}
1669
1670
1671	/**
1672	* Init statistics
1673	* @returns VBox status code.
1674	*/
1675	static int pgmR3InitStats(PVM pVM)
1676	{
1677	PPGM pPGM = &pVM->pgm.s;
1678	int rc;
1679
1680	/*
1681	* Release statistics.
1682	*/
1683	/* Common - misc variables */
1684	STAM_REL_REG(pVM, &pPGM->cAllPages, STAMTYPE_U32, "/PGM/Page/cAllPages", STAMUNIT_COUNT, "The total number of pages.");
1685	STAM_REL_REG(pVM, &pPGM->cPrivatePages, STAMTYPE_U32, "/PGM/Page/cPrivatePages", STAMUNIT_COUNT, "The number of private pages.");
1686	STAM_REL_REG(pVM, &pPGM->cSharedPages, STAMTYPE_U32, "/PGM/Page/cSharedPages", STAMUNIT_COUNT, "The number of shared pages.");
1687	STAM_REL_REG(pVM, &pPGM->cReusedSharedPages, STAMTYPE_U32, "/PGM/Page/cReusedSharedPages", STAMUNIT_COUNT, "The number of reused shared pages.");
1688	STAM_REL_REG(pVM, &pPGM->cZeroPages, STAMTYPE_U32, "/PGM/Page/cZeroPages", STAMUNIT_COUNT, "The number of zero backed pages.");
1689	STAM_REL_REG(pVM, &pPGM->cPureMmioPages, STAMTYPE_U32, "/PGM/Page/cPureMmioPages", STAMUNIT_COUNT, "The number of pure MMIO pages.");
1690	STAM_REL_REG(pVM, &pPGM->cMonitoredPages, STAMTYPE_U32, "/PGM/Page/cMonitoredPages", STAMUNIT_COUNT, "The number of write monitored pages.");
1691	STAM_REL_REG(pVM, &pPGM->cWrittenToPages, STAMTYPE_U32, "/PGM/Page/cWrittenToPages", STAMUNIT_COUNT, "The number of previously write monitored pages that have been written to.");
1692	STAM_REL_REG(pVM, &pPGM->cWriteLockedPages, STAMTYPE_U32, "/PGM/Page/cWriteLockedPages", STAMUNIT_COUNT, "The number of write(/read) locked pages.");
1693	STAM_REL_REG(pVM, &pPGM->cReadLockedPages, STAMTYPE_U32, "/PGM/Page/cReadLockedPages", STAMUNIT_COUNT, "The number of read (only) locked pages.");
1694	STAM_REL_REG(pVM, &pPGM->cBalloonedPages, STAMTYPE_U32, "/PGM/Page/cBalloonedPages", STAMUNIT_COUNT, "The number of ballooned pages.");
1695	STAM_REL_REG(pVM, &pPGM->cHandyPages, STAMTYPE_U32, "/PGM/Page/cHandyPages", STAMUNIT_COUNT, "The number of handy pages (not included in cAllPages).");
1696	STAM_REL_REG(pVM, &pPGM->cLargePages, STAMTYPE_U32, "/PGM/Page/cLargePages", STAMUNIT_COUNT, "The number of large pages allocated (includes disabled).");
1697	STAM_REL_REG(pVM, &pPGM->cLargePagesDisabled, STAMTYPE_U32, "/PGM/Page/cLargePagesDisabled", STAMUNIT_COUNT, "The number of disabled large pages.");
1698	STAM_REL_REG(pVM, &pPGM->cRelocations, STAMTYPE_COUNTER, "/PGM/cRelocations", STAMUNIT_OCCURENCES,"Number of hypervisor relocations.");
1699	STAM_REL_REG(pVM, &pPGM->ChunkR3Map.c, STAMTYPE_U32, "/PGM/ChunkR3Map/c", STAMUNIT_COUNT, "Number of mapped chunks.");
1700	STAM_REL_REG(pVM, &pPGM->ChunkR3Map.cMax, STAMTYPE_U32, "/PGM/ChunkR3Map/cMax", STAMUNIT_COUNT, "Maximum number of mapped chunks.");
1701	STAM_REL_REG(pVM, &pPGM->cMappedChunks, STAMTYPE_U32, "/PGM/ChunkR3Map/Mapped", STAMUNIT_COUNT, "Number of times we mapped a chunk.");
1702	STAM_REL_REG(pVM, &pPGM->cUnmappedChunks, STAMTYPE_U32, "/PGM/ChunkR3Map/Unmapped", STAMUNIT_COUNT, "Number of times we unmapped a chunk.");
1703
1704	STAM_REL_REG(pVM, &pPGM->StatLargePageReused, STAMTYPE_COUNTER, "/PGM/LargePage/Reused", STAMUNIT_OCCURENCES, "The number of times we've reused a large page.");
1705	STAM_REL_REG(pVM, &pPGM->StatLargePageRefused, STAMTYPE_COUNTER, "/PGM/LargePage/Refused", STAMUNIT_OCCURENCES, "The number of times we couldn't use a large page.");
1706	STAM_REL_REG(pVM, &pPGM->StatLargePageRecheck, STAMTYPE_COUNTER, "/PGM/LargePage/Recheck", STAMUNIT_OCCURENCES, "The number of times we've rechecked a disabled large page.");
1707
1708	STAM_REL_REG(pVM, &pPGM->StatShModCheck, STAMTYPE_PROFILE, "/PGM/ShMod/Check", STAMUNIT_TICKS_PER_CALL, "Profiles the shared module checking.");
1709
1710	/* Live save */
1711	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.fActive, STAMTYPE_U8, "/PGM/LiveSave/fActive", STAMUNIT_COUNT, "Active or not.");
1712	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cIgnoredPages, STAMTYPE_U32, "/PGM/LiveSave/cIgnoredPages", STAMUNIT_COUNT, "The number of ignored pages in the RAM ranges (i.e. MMIO, MMIO2 and ROM).");
1713	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cDirtyPagesLong, STAMTYPE_U32, "/PGM/LiveSave/cDirtyPagesLong", STAMUNIT_COUNT, "Longer term dirty page average.");
1714	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cDirtyPagesShort, STAMTYPE_U32, "/PGM/LiveSave/cDirtyPagesShort", STAMUNIT_COUNT, "Short term dirty page average.");
1715	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cPagesPerSecond, STAMTYPE_U32, "/PGM/LiveSave/cPagesPerSecond", STAMUNIT_COUNT, "Pages per second.");
1716	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cSavedPages, STAMTYPE_U64, "/PGM/LiveSave/cSavedPages", STAMUNIT_COUNT, "The total number of saved pages.");
1717	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cReadPages", STAMUNIT_COUNT, "RAM: Ready pages.");
1718	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cDirtyPages", STAMUNIT_COUNT, "RAM: Dirty pages.");
1719	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cZeroPages", STAMUNIT_COUNT, "RAM: Ready zero pages.");
1720	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cMonitoredPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cMonitoredPages", STAMUNIT_COUNT, "RAM: Write monitored pages.");
1721	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cReadPages", STAMUNIT_COUNT, "ROM: Ready pages.");
1722	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cDirtyPages", STAMUNIT_COUNT, "ROM: Dirty pages.");
1723	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cZeroPages", STAMUNIT_COUNT, "ROM: Ready zero pages.");
1724	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cMonitoredPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cMonitoredPages", STAMUNIT_COUNT, "ROM: Write monitored pages.");
1725	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cReadPages", STAMUNIT_COUNT, "MMIO2: Ready pages.");
1726	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cDirtyPages", STAMUNIT_COUNT, "MMIO2: Dirty pages.");
1727	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cZeroPages", STAMUNIT_COUNT, "MMIO2: Ready zero pages.");
1728	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cMonitoredPages,STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cMonitoredPages",STAMUNIT_COUNT, "MMIO2: Write monitored pages.");
1729
1730	#ifdef VBOX_WITH_STATISTICS
1731
1732	# define PGM_REG_COUNTER(a, b, c) \
1733	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_OCCURENCES, c, b); \
1734	AssertRC(rc);
1735
1736	# define PGM_REG_COUNTER_BYTES(a, b, c) \
1737	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_BYTES, c, b); \
1738	AssertRC(rc);
1739
1740	# define PGM_REG_PROFILE(a, b, c) \
1741	rc = STAMR3RegisterF(pVM, a, STAMTYPE_PROFILE, STAMVISIBILITY_ALWAYS, STAMUNIT_TICKS_PER_CALL, c, b); \
1742	AssertRC(rc);
1743
1744	PGMSTATS *pStats = pVM->pgm.s.pStatsR3;
1745
1746	PGM_REG_PROFILE(&pStats->StatAllocLargePage, "/PGM/LargePage/Prof/Alloc", "Time spent by the host OS for large page allocation.");
1747	PGM_REG_PROFILE(&pStats->StatClearLargePage, "/PGM/LargePage/Prof/Clear", "Time spent clearing the newly allocated large pages.");
1748	PGM_REG_COUNTER(&pStats->StatLargePageOverflow, "/PGM/LargePage/Overflow", "The number of times allocating a large page took too long.");
1749	PGM_REG_PROFILE(&pStats->StatR3IsValidLargePage, "/PGM/LargePage/Prof/R3/IsValid", "pgmPhysIsValidLargePage profiling - R3.");
1750	PGM_REG_PROFILE(&pStats->StatRZIsValidLargePage, "/PGM/LargePage/Prof/RZ/IsValid", "pgmPhysIsValidLargePage profiling - RZ.");
1751
1752	PGM_REG_COUNTER(&pStats->StatR3DetectedConflicts, "/PGM/R3/DetectedConflicts", "The number of times PGMR3CheckMappingConflicts() detected a conflict.");
1753	PGM_REG_PROFILE(&pStats->StatR3ResolveConflict, "/PGM/R3/ResolveConflict", "pgmR3SyncPTResolveConflict() profiling (includes the entire relocation).");
1754	PGM_REG_COUNTER(&pStats->StatR3PhysRead, "/PGM/R3/Phys/Read", "The number of times PGMPhysRead was called.");
1755	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysReadBytes, "/PGM/R3/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1756	PGM_REG_COUNTER(&pStats->StatR3PhysWrite, "/PGM/R3/Phys/Write", "The number of times PGMPhysWrite was called.");
1757	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysWriteBytes, "/PGM/R3/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1758	PGM_REG_COUNTER(&pStats->StatR3PhysSimpleRead, "/PGM/R3/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1759	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysSimpleReadBytes, "/PGM/R3/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1760	PGM_REG_COUNTER(&pStats->StatR3PhysSimpleWrite, "/PGM/R3/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1761	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysSimpleWriteBytes, "/PGM/R3/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1762
1763	PGM_REG_COUNTER(&pStats->StatRZChunkR3MapTlbHits, "/PGM/ChunkR3Map/TlbHitsRZ", "TLB hits.");
1764	PGM_REG_COUNTER(&pStats->StatRZChunkR3MapTlbMisses, "/PGM/ChunkR3Map/TlbMissesRZ", "TLB misses.");
1765	PGM_REG_PROFILE(&pStats->StatChunkAging, "/PGM/ChunkR3Map/Map/Aging", "Chunk aging profiling.");
1766	PGM_REG_PROFILE(&pStats->StatChunkFindCandidate, "/PGM/ChunkR3Map/Map/Find", "Chunk unmap find profiling.");
1767	PGM_REG_PROFILE(&pStats->StatChunkUnmap, "/PGM/ChunkR3Map/Map/Unmap", "Chunk unmap of address space profiling.");
1768	PGM_REG_PROFILE(&pStats->StatChunkMap, "/PGM/ChunkR3Map/Map/Map", "Chunk map of address space profiling.");
1769
1770	PGM_REG_COUNTER(&pStats->StatRZPageMapTlbHits, "/PGM/RZ/Page/MapTlbHits", "TLB hits.");
1771	PGM_REG_COUNTER(&pStats->StatRZPageMapTlbMisses, "/PGM/RZ/Page/MapTlbMisses", "TLB misses.");
1772	PGM_REG_COUNTER(&pStats->StatR3ChunkR3MapTlbHits, "/PGM/ChunkR3Map/TlbHitsR3", "TLB hits.");
1773	PGM_REG_COUNTER(&pStats->StatR3ChunkR3MapTlbMisses, "/PGM/ChunkR3Map/TlbMissesR3", "TLB misses.");
1774	PGM_REG_COUNTER(&pStats->StatR3PageMapTlbHits, "/PGM/R3/Page/MapTlbHits", "TLB hits.");
1775	PGM_REG_COUNTER(&pStats->StatR3PageMapTlbMisses, "/PGM/R3/Page/MapTlbMisses", "TLB misses.");
1776	PGM_REG_COUNTER(&pStats->StatPageMapTlbFlushes, "/PGM/R3/Page/MapTlbFlushes", "TLB flushes (all contexts).");
1777	PGM_REG_COUNTER(&pStats->StatPageMapTlbFlushEntry, "/PGM/R3/Page/MapTlbFlushEntry", "TLB entry flushes (all contexts).");
1778
1779	PGM_REG_COUNTER(&pStats->StatRZRamRangeTlbHits, "/PGM/RZ/RamRange/TlbHits", "TLB hits.");
1780	PGM_REG_COUNTER(&pStats->StatRZRamRangeTlbMisses, "/PGM/RZ/RamRange/TlbMisses", "TLB misses.");
1781	PGM_REG_COUNTER(&pStats->StatR3RamRangeTlbHits, "/PGM/R3/RamRange/TlbHits", "TLB hits.");
1782	PGM_REG_COUNTER(&pStats->StatR3RamRangeTlbMisses, "/PGM/R3/RamRange/TlbMisses", "TLB misses.");
1783
1784	PGM_REG_PROFILE(&pStats->StatRZSyncCR3HandlerVirtualUpdate, "/PGM/RZ/SyncCR3/Handlers/VirtualUpdate", "Profiling of the virtual handler updates.");
1785	PGM_REG_PROFILE(&pStats->StatRZSyncCR3HandlerVirtualReset, "/PGM/RZ/SyncCR3/Handlers/VirtualReset", "Profiling of the virtual handler resets.");
1786	PGM_REG_PROFILE(&pStats->StatR3SyncCR3HandlerVirtualUpdate, "/PGM/R3/SyncCR3/Handlers/VirtualUpdate", "Profiling of the virtual handler updates.");
1787	PGM_REG_PROFILE(&pStats->StatR3SyncCR3HandlerVirtualReset, "/PGM/R3/SyncCR3/Handlers/VirtualReset", "Profiling of the virtual handler resets.");
1788
1789	PGM_REG_COUNTER(&pStats->StatRZPhysHandlerReset, "/PGM/RZ/PhysHandlerReset", "The number of times PGMHandlerPhysicalReset is called.");
1790	PGM_REG_COUNTER(&pStats->StatR3PhysHandlerReset, "/PGM/R3/PhysHandlerReset", "The number of times PGMHandlerPhysicalReset is called.");
1791	PGM_REG_COUNTER(&pStats->StatRZPhysHandlerLookupHits, "/PGM/RZ/PhysHandlerLookupHits", "The number of cache hits when looking up physical handlers.");
1792	PGM_REG_COUNTER(&pStats->StatR3PhysHandlerLookupHits, "/PGM/R3/PhysHandlerLookupHits", "The number of cache hits when looking up physical handlers.");
1793	PGM_REG_COUNTER(&pStats->StatRZPhysHandlerLookupMisses, "/PGM/RZ/PhysHandlerLookupMisses", "The number of cache misses when looking up physical handlers.");
1794	PGM_REG_COUNTER(&pStats->StatR3PhysHandlerLookupMisses, "/PGM/R3/PhysHandlerLookupMisses", "The number of cache misses when looking up physical handlers.");
1795	PGM_REG_PROFILE(&pStats->StatRZVirtHandlerSearchByPhys, "/PGM/RZ/VirtHandlerSearchByPhys", "Profiling of pgmHandlerVirtualFindByPhysAddr.");
1796	PGM_REG_PROFILE(&pStats->StatR3VirtHandlerSearchByPhys, "/PGM/R3/VirtHandlerSearchByPhys", "Profiling of pgmHandlerVirtualFindByPhysAddr.");
1797
1798	PGM_REG_COUNTER(&pStats->StatRZPageReplaceShared, "/PGM/RZ/Page/ReplacedShared", "Times a shared page was replaced.");
1799	PGM_REG_COUNTER(&pStats->StatRZPageReplaceZero, "/PGM/RZ/Page/ReplacedZero", "Times the zero page was replaced.");
1800	/// @todo PGM_REG_COUNTER(&pStats->StatRZPageHandyAllocs, "/PGM/RZ/Page/HandyAllocs", "Number of times we've allocated more handy pages.");
1801	PGM_REG_COUNTER(&pStats->StatR3PageReplaceShared, "/PGM/R3/Page/ReplacedShared", "Times a shared page was replaced.");
1802	PGM_REG_COUNTER(&pStats->StatR3PageReplaceZero, "/PGM/R3/Page/ReplacedZero", "Times the zero page was replaced.");
1803	/// @todo PGM_REG_COUNTER(&pStats->StatR3PageHandyAllocs, "/PGM/R3/Page/HandyAllocs", "Number of times we've allocated more handy pages.");
1804
1805	PGM_REG_COUNTER(&pStats->StatRZPhysRead, "/PGM/RZ/Phys/Read", "The number of times PGMPhysRead was called.");
1806	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysReadBytes, "/PGM/RZ/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1807	PGM_REG_COUNTER(&pStats->StatRZPhysWrite, "/PGM/RZ/Phys/Write", "The number of times PGMPhysWrite was called.");
1808	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysWriteBytes, "/PGM/RZ/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1809	PGM_REG_COUNTER(&pStats->StatRZPhysSimpleRead, "/PGM/RZ/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1810	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysSimpleReadBytes, "/PGM/RZ/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1811	PGM_REG_COUNTER(&pStats->StatRZPhysSimpleWrite, "/PGM/RZ/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1812	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysSimpleWriteBytes, "/PGM/RZ/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1813
1814	/* GC only: */
1815	PGM_REG_COUNTER(&pStats->StatRCInvlPgConflict, "/PGM/RC/InvlPgConflict", "Number of times PGMInvalidatePage() detected a mapping conflict.");
1816	PGM_REG_COUNTER(&pStats->StatRCInvlPgSyncMonCR3, "/PGM/RC/InvlPgSyncMonitorCR3", "Number of times PGMInvalidatePage() ran into PGM_SYNC_MONITOR_CR3.");
1817
1818	PGM_REG_COUNTER(&pStats->StatRCPhysRead, "/PGM/RC/Phys/Read", "The number of times PGMPhysRead was called.");
1819	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysReadBytes, "/PGM/RC/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1820	PGM_REG_COUNTER(&pStats->StatRCPhysWrite, "/PGM/RC/Phys/Write", "The number of times PGMPhysWrite was called.");
1821	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysWriteBytes, "/PGM/RC/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1822	PGM_REG_COUNTER(&pStats->StatRCPhysSimpleRead, "/PGM/RC/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1823	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysSimpleReadBytes, "/PGM/RC/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1824	PGM_REG_COUNTER(&pStats->StatRCPhysSimpleWrite, "/PGM/RC/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1825	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysSimpleWriteBytes, "/PGM/RC/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1826
1827	PGM_REG_COUNTER(&pStats->StatTrackVirgin, "/PGM/Track/Virgin", "The number of first time shadowings");
1828	PGM_REG_COUNTER(&pStats->StatTrackAliased, "/PGM/Track/Aliased", "The number of times switching to cRef2, i.e. the page is being shadowed by two PTs.");
1829	PGM_REG_COUNTER(&pStats->StatTrackAliasedMany, "/PGM/Track/AliasedMany", "The number of times we're tracking using cRef2.");
1830	PGM_REG_COUNTER(&pStats->StatTrackAliasedLots, "/PGM/Track/AliasedLots", "The number of times we're hitting pages which has overflowed cRef2");
1831	PGM_REG_COUNTER(&pStats->StatTrackOverflows, "/PGM/Track/Overflows", "The number of times the extent list grows too long.");
1832	PGM_REG_COUNTER(&pStats->StatTrackNoExtentsLeft, "/PGM/Track/NoExtentLeft", "The number of times the extent list was exhausted.");
1833	PGM_REG_PROFILE(&pStats->StatTrackDeref, "/PGM/Track/Deref", "Profiling of SyncPageWorkerTrackDeref (expensive).");
1834
1835	# undef PGM_REG_COUNTER
1836	# undef PGM_REG_PROFILE
1837	#endif
1838
1839	/*
1840	* Note! The layout below matches the member layout exactly!
1841	*/
1842
1843	/*
1844	* Common - stats
1845	*/
1846	for (VMCPUID idCpu = 0; idCpu < pVM->cCpus; idCpu++)
1847	{
1848	PPGMCPU pPgmCpu = &pVM->aCpus[idCpu].pgm.s;
1849
1850	#define PGM_REG_COUNTER(a, b, c) \
1851	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_OCCURENCES, c, b, idCpu); \
1852	AssertRC(rc);
1853	#define PGM_REG_PROFILE(a, b, c) \
1854	rc = STAMR3RegisterF(pVM, a, STAMTYPE_PROFILE, STAMVISIBILITY_ALWAYS, STAMUNIT_TICKS_PER_CALL, c, b, idCpu); \
1855	AssertRC(rc);
1856
1857	PGM_REG_COUNTER(&pPgmCpu->cGuestModeChanges, "/PGM/CPU%u/cGuestModeChanges", "Number of guest mode changes.");
1858	PGM_REG_COUNTER(&pPgmCpu->cA20Changes, "/PGM/CPU%u/cA20Changes", "Number of A20 gate changes.");
1859
1860	#ifdef VBOX_WITH_STATISTICS
1861	PGMCPUSTATS *pCpuStats = pVM->aCpus[idCpu].pgm.s.pStatsR3;
1862
1863	# if 0 /* rarely useful; leave for debugging. */
1864	for (unsigned j = 0; j < RT_ELEMENTS(pPgmCpu->StatSyncPtPD); j++)
1865	STAMR3RegisterF(pVM, &pCpuStats->StatSyncPtPD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1866	"The number of SyncPT per PD n.", "/PGM/CPU%u/PDSyncPT/%04X", i, j);
1867	for (unsigned j = 0; j < RT_ELEMENTS(pCpuStats->StatSyncPagePD); j++)
1868	STAMR3RegisterF(pVM, &pCpuStats->StatSyncPagePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1869	"The number of SyncPage per PD n.", "/PGM/CPU%u/PDSyncPage/%04X", i, j);
1870	# endif
1871	/* R0 only: */
1872	PGM_REG_PROFILE(&pCpuStats->StatR0NpMiscfg, "/PGM/CPU%u/R0/NpMiscfg", "PGMR0Trap0eHandlerNPMisconfig() profiling.");
1873	PGM_REG_COUNTER(&pCpuStats->StatR0NpMiscfgSyncPage, "/PGM/CPU%u/R0/NpMiscfgSyncPage", "SyncPage calls from PGMR0Trap0eHandlerNPMisconfig().");
1874
1875	/* RZ only: */
1876	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0e, "/PGM/CPU%u/RZ/Trap0e", "Profiling of the PGMTrap0eHandler() body.");
1877	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Ballooned, "/PGM/CPU%u/RZ/Trap0e/Time2/Ballooned", "Profiling of the Trap0eHandler body when the cause is read access to a ballooned page.");
1878	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2CSAM, "/PGM/CPU%u/RZ/Trap0e/Time2/CSAM", "Profiling of the Trap0eHandler body when the cause is CSAM.");
1879	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2DirtyAndAccessed, "/PGM/CPU%u/RZ/Trap0e/Time2/DirtyAndAccessedBits", "Profiling of the Trap0eHandler body when the cause is dirty and/or accessed bit emulation.");
1880	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2GuestTrap, "/PGM/CPU%u/RZ/Trap0e/Time2/GuestTrap", "Profiling of the Trap0eHandler body when the cause is a guest trap.");
1881	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2HndPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerPhysical", "Profiling of the Trap0eHandler body when the cause is a physical handler.");
1882	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2HndVirt, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerVirtual", "Profiling of the Trap0eHandler body when the cause is a virtual handler.");
1883	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2HndUnhandled, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerUnhandled", "Profiling of the Trap0eHandler body when the cause is access outside the monitored areas of a monitored page.");
1884	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2InvalidPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/InvalidPhys", "Profiling of the Trap0eHandler body when the cause is access to an invalid physical guest address.");
1885	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2MakeWritable, "/PGM/CPU%u/RZ/Trap0e/Time2/MakeWritable", "Profiling of the Trap0eHandler body when the cause is that a page needed to be made writeable.");
1886	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Mapping, "/PGM/CPU%u/RZ/Trap0e/Time2/Mapping", "Profiling of the Trap0eHandler body when the cause is related to the guest mappings.");
1887	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Misc, "/PGM/CPU%u/RZ/Trap0e/Time2/Misc", "Profiling of the Trap0eHandler body when the cause is not known.");
1888	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSync, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSync", "Profiling of the Trap0eHandler body when the cause is an out-of-sync page.");
1889	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSyncHndPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncHndPhys", "Profiling of the Trap0eHandler body when the cause is an out-of-sync physical handler page.");
1890	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSyncHndVirt, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncHndVirt", "Profiling of the Trap0eHandler body when the cause is an out-of-sync virtual handler page.");
1891	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSyncHndObs, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncObsHnd", "Profiling of the Trap0eHandler body when the cause is an obsolete handler page.");
1892	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2SyncPT, "/PGM/CPU%u/RZ/Trap0e/Time2/SyncPT", "Profiling of the Trap0eHandler body when the cause is lazy syncing of a PT.");
1893	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2WPEmulation, "/PGM/CPU%u/RZ/Trap0e/Time2/WPEmulation", "Profiling of the Trap0eHandler body when the cause is CR0.WP emulation.");
1894	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Wp0RoUsHack, "/PGM/CPU%u/RZ/Trap0e/Time2/WP0R0USHack", "Profiling of the Trap0eHandler body when the cause is CR0.WP and netware hack to be enabled.");
1895	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Wp0RoUsUnhack, "/PGM/CPU%u/RZ/Trap0e/Time2/WP0R0USUnhack", "Profiling of the Trap0eHandler body when the cause is CR0.WP and netware hack to be disabled.");
1896	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eConflicts, "/PGM/CPU%u/RZ/Trap0e/Conflicts", "The number of times #PF was caused by an undetected conflict.");
1897	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersMapping, "/PGM/CPU%u/RZ/Trap0e/Handlers/Mapping", "Number of traps due to access handlers in mappings.");
1898	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersOutOfSync, "/PGM/CPU%u/RZ/Trap0e/Handlers/OutOfSync", "Number of traps due to out-of-sync handled pages.");
1899	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersPhysAll, "/PGM/CPU%u/RZ/Trap0e/Handlers/PhysAll", "Number of traps due to physical all-access handlers.");
1900	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersPhysAllOpt, "/PGM/CPU%u/RZ/Trap0e/Handlers/PhysAllOpt", "Number of the physical all-access handler traps using the optimization.");
1901	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersPhysWrite, "/PGM/CPU%u/RZ/Trap0e/Handlers/PhysWrite", "Number of traps due to physical write-access handlers.");
1902	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersVirtual, "/PGM/CPU%u/RZ/Trap0e/Handlers/Virtual", "Number of traps due to virtual access handlers.");
1903	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersVirtualByPhys, "/PGM/CPU%u/RZ/Trap0e/Handlers/VirtualByPhys", "Number of traps due to virtual access handlers by physical address.");
1904	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersVirtualUnmarked,"/PGM/CPU%u/RZ/Trap0e/Handlers/VirtualUnmarked","Number of traps due to virtual access handlers by virtual address (without proper physical flags).");
1905	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersUnhandled, "/PGM/CPU%u/RZ/Trap0e/Handlers/Unhandled", "Number of traps due to access outside range of monitored page(s).");
1906	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersInvalid, "/PGM/CPU%u/RZ/Trap0e/Handlers/Invalid", "Number of traps due to access to invalid physical memory.");
1907	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSNotPresentRead, "/PGM/CPU%u/RZ/Trap0e/Err/User/NPRead", "Number of user mode not present read page faults.");
1908	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSNotPresentWrite, "/PGM/CPU%u/RZ/Trap0e/Err/User/NPWrite", "Number of user mode not present write page faults.");
1909	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSWrite, "/PGM/CPU%u/RZ/Trap0e/Err/User/Write", "Number of user mode write page faults.");
1910	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSReserved, "/PGM/CPU%u/RZ/Trap0e/Err/User/Reserved", "Number of user mode reserved bit page faults.");
1911	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSNXE, "/PGM/CPU%u/RZ/Trap0e/Err/User/NXE", "Number of user mode NXE page faults.");
1912	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSRead, "/PGM/CPU%u/RZ/Trap0e/Err/User/Read", "Number of user mode read page faults.");
1913	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVNotPresentRead, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NPRead", "Number of supervisor mode not present read page faults.");
1914	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVNotPresentWrite, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NPWrite", "Number of supervisor mode not present write page faults.");
1915	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVWrite, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/Write", "Number of supervisor mode write page faults.");
1916	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVReserved, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/Reserved", "Number of supervisor mode reserved bit page faults.");
1917	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSNXE, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NXE", "Number of supervisor mode NXE page faults.");
1918	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eGuestPF, "/PGM/CPU%u/RZ/Trap0e/GuestPF", "Number of real guest page faults.");
1919	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eGuestPFMapping, "/PGM/CPU%u/RZ/Trap0e/GuestPF/InMapping", "Number of real guest page faults in a mapping.");
1920	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eWPEmulInRZ, "/PGM/CPU%u/RZ/Trap0e/WP/InRZ", "Number of guest page faults due to X86_CR0_WP emulation.");
1921	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eWPEmulToR3, "/PGM/CPU%u/RZ/Trap0e/WP/ToR3", "Number of guest page faults due to X86_CR0_WP emulation (forward to R3 for emulation).");
1922	#if 0 /* rarely useful; leave for debugging. */
1923	for (unsigned j = 0; j < RT_ELEMENTS(pCpuStats->StatRZTrap0ePD); j++)
1924	STAMR3RegisterF(pVM, &pCpuStats->StatRZTrap0ePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1925	"The number of traps in page directory n.", "/PGM/CPU%u/RZ/Trap0e/PD/%04X", i, j);
1926	#endif
1927	PGM_REG_COUNTER(&pCpuStats->StatRZGuestCR3WriteHandled, "/PGM/CPU%u/RZ/CR3WriteHandled", "The number of times the Guest CR3 change was successfully handled.");
1928	PGM_REG_COUNTER(&pCpuStats->StatRZGuestCR3WriteUnhandled, "/PGM/CPU%u/RZ/CR3WriteUnhandled", "The number of times the Guest CR3 change was passed back to the recompiler.");
1929	PGM_REG_COUNTER(&pCpuStats->StatRZGuestCR3WriteConflict, "/PGM/CPU%u/RZ/CR3WriteConflict", "The number of times the Guest CR3 monitoring detected a conflict.");
1930	PGM_REG_COUNTER(&pCpuStats->StatRZGuestROMWriteHandled, "/PGM/CPU%u/RZ/ROMWriteHandled", "The number of times the Guest ROM change was successfully handled.");
1931	PGM_REG_COUNTER(&pCpuStats->StatRZGuestROMWriteUnhandled, "/PGM/CPU%u/RZ/ROMWriteUnhandled", "The number of times the Guest ROM change was passed back to the recompiler.");
1932
1933	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapMigrateInvlPg, "/PGM/CPU%u/RZ/DynMap/MigrateInvlPg", "invlpg count in PGMR0DynMapMigrateAutoSet.");
1934	PGM_REG_PROFILE(&pCpuStats->StatRZDynMapGCPageInl, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl", "Calls to pgmR0DynMapGCPageInlined.");
1935	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlHits, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/Hits", "Hash table lookup hits.");
1936	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlMisses, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/Misses", "Misses that falls back to the code common.");
1937	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlRamHits, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/RamHits", "1st ram range hits.");
1938	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlRamMisses, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/RamMisses", "1st ram range misses, takes slow path.");
1939	PGM_REG_PROFILE(&pCpuStats->StatRZDynMapHCPageInl, "/PGM/CPU%u/RZ/DynMap/PageHCPageInl", "Calls to pgmRZDynMapHCPageInlined.");
1940	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapHCPageInlHits, "/PGM/CPU%u/RZ/DynMap/PageHCPageInl/Hits", "Hash table lookup hits.");
1941	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapHCPageInlMisses, "/PGM/CPU%u/RZ/DynMap/PageHCPageInl/Misses", "Misses that falls back to the code common.");
1942	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPage, "/PGM/CPU%u/RZ/DynMap/Page", "Calls to pgmR0DynMapPage");
1943	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetOptimize, "/PGM/CPU%u/RZ/DynMap/Page/SetOptimize", "Calls to pgmRZDynMapOptimizeAutoSet.");
1944	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetSearchFlushes, "/PGM/CPU%u/RZ/DynMap/Page/SetSearchFlushes", "Set search restoring to subset flushes.");
1945	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetSearchHits, "/PGM/CPU%u/RZ/DynMap/Page/SetSearchHits", "Set search hits.");
1946	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetSearchMisses, "/PGM/CPU%u/RZ/DynMap/Page/SetSearchMisses", "Set search misses.");
1947	PGM_REG_PROFILE(&pCpuStats->StatRZDynMapHCPage, "/PGM/CPU%u/RZ/DynMap/Page/HCPage", "Calls to pgmRZDynMapHCPageCommon (ring-0).");
1948	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageHits0, "/PGM/CPU%u/RZ/DynMap/Page/Hits0", "Hits at iPage+0");
1949	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageHits1, "/PGM/CPU%u/RZ/DynMap/Page/Hits1", "Hits at iPage+1");
1950	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageHits2, "/PGM/CPU%u/RZ/DynMap/Page/Hits2", "Hits at iPage+2");
1951	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageInvlPg, "/PGM/CPU%u/RZ/DynMap/Page/InvlPg", "invlpg count in pgmR0DynMapPageSlow.");
1952	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlow, "/PGM/CPU%u/RZ/DynMap/Page/Slow", "Calls to pgmR0DynMapPageSlow - subtract this from pgmR0DynMapPage to get 1st level hits.");
1953	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlowLoopHits, "/PGM/CPU%u/RZ/DynMap/Page/SlowLoopHits" , "Hits in the loop path.");
1954	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlowLoopMisses, "/PGM/CPU%u/RZ/DynMap/Page/SlowLoopMisses", "Misses in the loop path. NonLoopMisses = Slow - SlowLoopHit - SlowLoopMisses");
1955	//PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlowLostHits, "/PGM/CPU%u/R0/DynMap/Page/SlowLostHits", "Lost hits.");
1956	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSubsets, "/PGM/CPU%u/RZ/DynMap/Subsets", "Times PGMRZDynMapPushAutoSubset was called.");
1957	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPopFlushes, "/PGM/CPU%u/RZ/DynMap/SubsetPopFlushes", "Times PGMRZDynMapPopAutoSubset flushes the subset.");
1958	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[0], "/PGM/CPU%u/RZ/DynMap/SetFilledPct000..09", "00-09% filled (RC: min(set-size, dynmap-size))");
1959	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[1], "/PGM/CPU%u/RZ/DynMap/SetFilledPct010..19", "10-19% filled (RC: min(set-size, dynmap-size))");
1960	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[2], "/PGM/CPU%u/RZ/DynMap/SetFilledPct020..29", "20-29% filled (RC: min(set-size, dynmap-size))");
1961	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[3], "/PGM/CPU%u/RZ/DynMap/SetFilledPct030..39", "30-39% filled (RC: min(set-size, dynmap-size))");
1962	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[4], "/PGM/CPU%u/RZ/DynMap/SetFilledPct040..49", "40-49% filled (RC: min(set-size, dynmap-size))");
1963	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[5], "/PGM/CPU%u/RZ/DynMap/SetFilledPct050..59", "50-59% filled (RC: min(set-size, dynmap-size))");
1964	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[6], "/PGM/CPU%u/RZ/DynMap/SetFilledPct060..69", "60-69% filled (RC: min(set-size, dynmap-size))");
1965	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[7], "/PGM/CPU%u/RZ/DynMap/SetFilledPct070..79", "70-79% filled (RC: min(set-size, dynmap-size))");
1966	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[8], "/PGM/CPU%u/RZ/DynMap/SetFilledPct080..89", "80-89% filled (RC: min(set-size, dynmap-size))");
1967	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[9], "/PGM/CPU%u/RZ/DynMap/SetFilledPct090..99", "90-99% filled (RC: min(set-size, dynmap-size))");
1968	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[10], "/PGM/CPU%u/RZ/DynMap/SetFilledPct100", "100% filled (RC: min(set-size, dynmap-size))");
1969
1970	/* HC only: */
1971
1972	/* RZ & R3: */
1973	PGM_REG_PROFILE(&pCpuStats->StatRZSyncCR3, "/PGM/CPU%u/RZ/SyncCR3", "Profiling of the PGMSyncCR3() body.");
1974	PGM_REG_PROFILE(&pCpuStats->StatRZSyncCR3Handlers, "/PGM/CPU%u/RZ/SyncCR3/Handlers", "Profiling of the PGMSyncCR3() update handler section.");
1975	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3Global, "/PGM/CPU%u/RZ/SyncCR3/Global", "The number of global CR3 syncs.");
1976	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3NotGlobal, "/PGM/CPU%u/RZ/SyncCR3/NotGlobal", "The number of non-global CR3 syncs.");
1977	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstCacheHit, "/PGM/CPU%u/RZ/SyncCR3/DstChacheHit", "The number of times we got some kind of a cache hit.");
1978	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstFreed, "/PGM/CPU%u/RZ/SyncCR3/DstFreed", "The number of times we've had to free a shadow entry.");
1979	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstFreedSrcNP, "/PGM/CPU%u/RZ/SyncCR3/DstFreedSrcNP", "The number of times we've had to free a shadow entry for which the source entry was not present.");
1980	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstNotPresent, "/PGM/CPU%u/RZ/SyncCR3/DstNotPresent", "The number of times we've encountered a not present shadow entry for a present guest entry.");
1981	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstSkippedGlobalPD, "/PGM/CPU%u/RZ/SyncCR3/DstSkippedGlobalPD", "The number of times a global page directory wasn't flushed.");
1982	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstSkippedGlobalPT, "/PGM/CPU%u/RZ/SyncCR3/DstSkippedGlobalPT", "The number of times a page table with only global entries wasn't flushed.");
1983	PGM_REG_PROFILE(&pCpuStats->StatRZSyncPT, "/PGM/CPU%u/RZ/SyncPT", "Profiling of the pfnSyncPT() body.");
1984	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPTFailed, "/PGM/CPU%u/RZ/SyncPT/Failed", "The number of times pfnSyncPT() failed.");
1985	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPT4K, "/PGM/CPU%u/RZ/SyncPT/4K", "Nr of 4K PT syncs");
1986	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPT4M, "/PGM/CPU%u/RZ/SyncPT/4M", "Nr of 4M PT syncs");
1987	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPagePDNAs, "/PGM/CPU%u/RZ/SyncPagePDNAs", "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1988	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPagePDOutOfSync, "/PGM/CPU%u/RZ/SyncPagePDOutOfSync", "The number of time we've encountered an out-of-sync PD in SyncPage.");
1989	PGM_REG_COUNTER(&pCpuStats->StatRZAccessedPage, "/PGM/CPU%u/RZ/AccessedPage", "The number of pages marked not present for accessed bit emulation.");
1990	PGM_REG_PROFILE(&pCpuStats->StatRZDirtyBitTracking, "/PGM/CPU%u/RZ/DirtyPage", "Profiling the dirty bit tracking in CheckPageFault().");
1991	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPage, "/PGM/CPU%u/RZ/DirtyPage/Mark", "The number of pages marked read-only for dirty bit tracking.");
1992	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageBig, "/PGM/CPU%u/RZ/DirtyPage/MarkBig", "The number of 4MB pages marked read-only for dirty bit tracking.");
1993	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageSkipped, "/PGM/CPU%u/RZ/DirtyPage/Skipped", "The number of pages already dirty or readonly.");
1994	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageTrap, "/PGM/CPU%u/RZ/DirtyPage/Trap", "The number of traps generated for dirty bit tracking.");
1995	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageStale, "/PGM/CPU%u/RZ/DirtyPage/Stale", "The number of traps generated for dirty bit tracking (stale tlb entries).");
1996	PGM_REG_COUNTER(&pCpuStats->StatRZDirtiedPage, "/PGM/CPU%u/RZ/DirtyPage/SetDirty", "The number of pages marked dirty because of write accesses.");
1997	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyTrackRealPF, "/PGM/CPU%u/RZ/DirtyPage/RealPF", "The number of real pages faults during dirty bit tracking.");
1998	PGM_REG_COUNTER(&pCpuStats->StatRZPageAlreadyDirty, "/PGM/CPU%u/RZ/DirtyPage/AlreadySet", "The number of pages already marked dirty because of write accesses.");
1999	PGM_REG_PROFILE(&pCpuStats->StatRZInvalidatePage, "/PGM/CPU%u/RZ/InvalidatePage", "PGMInvalidatePage() profiling.");
2000	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePage4KBPages, "/PGM/CPU%u/RZ/InvalidatePage/4KBPages", "The number of times PGMInvalidatePage() was called for a 4KB page.");
2001	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePage4MBPages, "/PGM/CPU%u/RZ/InvalidatePage/4MBPages", "The number of times PGMInvalidatePage() was called for a 4MB page.");
2002	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePage4MBPagesSkip, "/PGM/CPU%u/RZ/InvalidatePage/4MBPagesSkip","The number of times PGMInvalidatePage() skipped a 4MB page.");
2003	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDMappings, "/PGM/CPU%u/RZ/InvalidatePage/PDMappings", "The number of times PGMInvalidatePage() was called for a page directory containing mappings (no conflict).");
2004	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDNAs, "/PGM/CPU%u/RZ/InvalidatePage/PDNAs", "The number of times PGMInvalidatePage() was called for a not accessed page directory.");
2005	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDNPs, "/PGM/CPU%u/RZ/InvalidatePage/PDNPs", "The number of times PGMInvalidatePage() was called for a not present page directory.");
2006	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDOutOfSync, "/PGM/CPU%u/RZ/InvalidatePage/PDOutOfSync", "The number of times PGMInvalidatePage() was called for an out of sync page directory.");
2007	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePageSkipped, "/PGM/CPU%u/RZ/InvalidatePage/Skipped", "The number of times PGMInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
2008	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncSupervisor, "/PGM/CPU%u/RZ/OutOfSync/SuperVisor", "Number of traps due to pages out of sync (P) and times VerifyAccessSyncPage calls SyncPage.");
2009	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncUser, "/PGM/CPU%u/RZ/OutOfSync/User", "Number of traps due to pages out of sync (P) and times VerifyAccessSyncPage calls SyncPage.");
2010	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncSupervisorWrite,"/PGM/CPU%u/RZ/OutOfSync/SuperVisorWrite", "Number of traps due to pages out of sync (RW) and times VerifyAccessSyncPage calls SyncPage.");
2011	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncUserWrite, "/PGM/CPU%u/RZ/OutOfSync/UserWrite", "Number of traps due to pages out of sync (RW) and times VerifyAccessSyncPage calls SyncPage.");
2012	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncBallloon, "/PGM/CPU%u/RZ/OutOfSync/Balloon", "The number of times a ballooned page was accessed (read).");
2013	PGM_REG_PROFILE(&pCpuStats->StatRZPrefetch, "/PGM/CPU%u/RZ/Prefetch", "PGMPrefetchPage profiling.");
2014	PGM_REG_PROFILE(&pCpuStats->StatRZFlushTLB, "/PGM/CPU%u/RZ/FlushTLB", "Profiling of the PGMFlushTLB() body.");
2015	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBNewCR3, "/PGM/CPU%u/RZ/FlushTLB/NewCR3", "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
2016	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBNewCR3Global, "/PGM/CPU%u/RZ/FlushTLB/NewCR3Global", "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
2017	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBSameCR3, "/PGM/CPU%u/RZ/FlushTLB/SameCR3", "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
2018	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBSameCR3Global, "/PGM/CPU%u/RZ/FlushTLB/SameCR3Global", "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
2019	PGM_REG_PROFILE(&pCpuStats->StatRZGstModifyPage, "/PGM/CPU%u/RZ/GstModifyPage", "Profiling of the PGMGstModifyPage() body.");
2020
2021	PGM_REG_PROFILE(&pCpuStats->StatR3SyncCR3, "/PGM/CPU%u/R3/SyncCR3", "Profiling of the PGMSyncCR3() body.");
2022	PGM_REG_PROFILE(&pCpuStats->StatR3SyncCR3Handlers, "/PGM/CPU%u/R3/SyncCR3/Handlers", "Profiling of the PGMSyncCR3() update handler section.");
2023	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3Global, "/PGM/CPU%u/R3/SyncCR3/Global", "The number of global CR3 syncs.");
2024	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3NotGlobal, "/PGM/CPU%u/R3/SyncCR3/NotGlobal", "The number of non-global CR3 syncs.");
2025	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstCacheHit, "/PGM/CPU%u/R3/SyncCR3/DstChacheHit", "The number of times we got some kind of a cache hit.");
2026	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstFreed, "/PGM/CPU%u/R3/SyncCR3/DstFreed", "The number of times we've had to free a shadow entry.");
2027	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstFreedSrcNP, "/PGM/CPU%u/R3/SyncCR3/DstFreedSrcNP", "The number of times we've had to free a shadow entry for which the source entry was not present.");
2028	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstNotPresent, "/PGM/CPU%u/R3/SyncCR3/DstNotPresent", "The number of times we've encountered a not present shadow entry for a present guest entry.");
2029	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstSkippedGlobalPD, "/PGM/CPU%u/R3/SyncCR3/DstSkippedGlobalPD", "The number of times a global page directory wasn't flushed.");
2030	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstSkippedGlobalPT, "/PGM/CPU%u/R3/SyncCR3/DstSkippedGlobalPT", "The number of times a page table with only global entries wasn't flushed.");
2031	PGM_REG_PROFILE(&pCpuStats->StatR3SyncPT, "/PGM/CPU%u/R3/SyncPT", "Profiling of the pfnSyncPT() body.");
2032	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPTFailed, "/PGM/CPU%u/R3/SyncPT/Failed", "The number of times pfnSyncPT() failed.");
2033	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPT4K, "/PGM/CPU%u/R3/SyncPT/4K", "Nr of 4K PT syncs");
2034	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPT4M, "/PGM/CPU%u/R3/SyncPT/4M", "Nr of 4M PT syncs");
2035	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPagePDNAs, "/PGM/CPU%u/R3/SyncPagePDNAs", "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
2036	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPagePDOutOfSync, "/PGM/CPU%u/R3/SyncPagePDOutOfSync", "The number of time we've encountered an out-of-sync PD in SyncPage.");
2037	PGM_REG_COUNTER(&pCpuStats->StatR3AccessedPage, "/PGM/CPU%u/R3/AccessedPage", "The number of pages marked not present for accessed bit emulation.");
2038	PGM_REG_PROFILE(&pCpuStats->StatR3DirtyBitTracking, "/PGM/CPU%u/R3/DirtyPage", "Profiling the dirty bit tracking in CheckPageFault().");
2039	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPage, "/PGM/CPU%u/R3/DirtyPage/Mark", "The number of pages marked read-only for dirty bit tracking.");
2040	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPageBig, "/PGM/CPU%u/R3/DirtyPage/MarkBig", "The number of 4MB pages marked read-only for dirty bit tracking.");
2041	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPageSkipped, "/PGM/CPU%u/R3/DirtyPage/Skipped", "The number of pages already dirty or readonly.");
2042	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPageTrap, "/PGM/CPU%u/R3/DirtyPage/Trap", "The number of traps generated for dirty bit tracking.");
2043	PGM_REG_COUNTER(&pCpuStats->StatR3DirtiedPage, "/PGM/CPU%u/R3/DirtyPage/SetDirty", "The number of pages marked dirty because of write accesses.");
2044	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyTrackRealPF, "/PGM/CPU%u/R3/DirtyPage/RealPF", "The number of real pages faults during dirty bit tracking.");
2045	PGM_REG_COUNTER(&pCpuStats->StatR3PageAlreadyDirty, "/PGM/CPU%u/R3/DirtyPage/AlreadySet", "The number of pages already marked dirty because of write accesses.");
2046	PGM_REG_PROFILE(&pCpuStats->StatR3InvalidatePage, "/PGM/CPU%u/R3/InvalidatePage", "PGMInvalidatePage() profiling.");
2047	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePage4KBPages, "/PGM/CPU%u/R3/InvalidatePage/4KBPages", "The number of times PGMInvalidatePage() was called for a 4KB page.");
2048	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePage4MBPages, "/PGM/CPU%u/R3/InvalidatePage/4MBPages", "The number of times PGMInvalidatePage() was called for a 4MB page.");
2049	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePage4MBPagesSkip, "/PGM/CPU%u/R3/InvalidatePage/4MBPagesSkip","The number of times PGMInvalidatePage() skipped a 4MB page.");
2050	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDMappings, "/PGM/CPU%u/R3/InvalidatePage/PDMappings", "The number of times PGMInvalidatePage() was called for a page directory containing mappings (no conflict).");
2051	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDNAs, "/PGM/CPU%u/R3/InvalidatePage/PDNAs", "The number of times PGMInvalidatePage() was called for a not accessed page directory.");
2052	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDNPs, "/PGM/CPU%u/R3/InvalidatePage/PDNPs", "The number of times PGMInvalidatePage() was called for a not present page directory.");
2053	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDOutOfSync, "/PGM/CPU%u/R3/InvalidatePage/PDOutOfSync", "The number of times PGMInvalidatePage() was called for an out of sync page directory.");
2054	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePageSkipped, "/PGM/CPU%u/R3/InvalidatePage/Skipped", "The number of times PGMInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
2055	PGM_REG_COUNTER(&pCpuStats->StatR3PageOutOfSyncSupervisor, "/PGM/CPU%u/R3/OutOfSync/SuperVisor", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
2056	PGM_REG_COUNTER(&pCpuStats->StatR3PageOutOfSyncUser, "/PGM/CPU%u/R3/OutOfSync/User", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
2057	PGM_REG_COUNTER(&pCpuStats->StatR3PageOutOfSyncBallloon, "/PGM/CPU%u/R3/OutOfSync/Balloon", "The number of times a ballooned page was accessed (read).");
2058	PGM_REG_PROFILE(&pCpuStats->StatR3Prefetch, "/PGM/CPU%u/R3/Prefetch", "PGMPrefetchPage profiling.");
2059	PGM_REG_PROFILE(&pCpuStats->StatR3FlushTLB, "/PGM/CPU%u/R3/FlushTLB", "Profiling of the PGMFlushTLB() body.");
2060	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBNewCR3, "/PGM/CPU%u/R3/FlushTLB/NewCR3", "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
2061	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBNewCR3Global, "/PGM/CPU%u/R3/FlushTLB/NewCR3Global", "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
2062	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBSameCR3, "/PGM/CPU%u/R3/FlushTLB/SameCR3", "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
2063	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBSameCR3Global, "/PGM/CPU%u/R3/FlushTLB/SameCR3Global", "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
2064	PGM_REG_PROFILE(&pCpuStats->StatR3GstModifyPage, "/PGM/CPU%u/R3/GstModifyPage", "Profiling of the PGMGstModifyPage() body.");
2065	#endif /* VBOX_WITH_STATISTICS */
2066
2067	#undef PGM_REG_PROFILE
2068	#undef PGM_REG_COUNTER
2069
2070	}
2071
2072	return VINF_SUCCESS;
2073	}
2074
2075
2076	/**
2077	* Init the PGM bits that rely on VMMR0 and MM to be fully initialized.
2078	*
2079	* The dynamic mapping area will also be allocated and initialized at this
2080	* time. We could allocate it during PGMR3Init of course, but the mapping
2081	* wouldn't be allocated at that time preventing us from setting up the
2082	* page table entries with the dummy page.
2083	*
2084	* @returns VBox status code.
2085	* @param pVM Pointer to the VM.
2086	*/
2087	VMMR3DECL(int) PGMR3InitDynMap(PVM pVM)
2088	{
2089	RTGCPTR GCPtr;
2090	int rc;
2091
2092	/*
2093	* Reserve space for the dynamic mappings.
2094	*/
2095	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping", &GCPtr);
2096	if (RT_SUCCESS(rc))
2097	pVM->pgm.s.pbDynPageMapBaseGC = GCPtr;
2098
2099	if ( RT_SUCCESS(rc)
2100	&& (pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_PAE_SHIFT) != ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_PAE_SHIFT))
2101	{
2102	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping not crossing", &GCPtr);
2103	if (RT_SUCCESS(rc))
2104	pVM->pgm.s.pbDynPageMapBaseGC = GCPtr;
2105	}
2106	if (RT_SUCCESS(rc))
2107	{
2108	AssertRelease((pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_PAE_SHIFT) == ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_PAE_SHIFT));
2109	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
2110	}
2111	return rc;
2112	}
2113
2114
2115	/**
2116	* Ring-3 init finalizing.
2117	*
2118	* @returns VBox status code.
2119	* @param pVM Pointer to the VM.
2120	*/
2121	VMMR3DECL(int) PGMR3InitFinalize(PVM pVM)
2122	{
2123	int rc;
2124
2125	/*
2126	* Reserve space for the dynamic mappings.
2127	* Initialize the dynamic mapping pages with dummy pages to simply the cache.
2128	*/
2129	/* get the pointer to the page table entries. */
2130	PPGMMAPPING pMapping = pgmGetMapping(pVM, pVM->pgm.s.pbDynPageMapBaseGC);
2131	AssertRelease(pMapping);
2132	const uintptr_t off = pVM->pgm.s.pbDynPageMapBaseGC - pMapping->GCPtr;
2133	const unsigned iPT = off >> X86_PD_SHIFT;
2134	const unsigned iPG = (off >> X86_PT_SHIFT) & X86_PT_MASK;
2135	pVM->pgm.s.paDynPageMap32BitPTEsGC = pMapping->aPTs[iPT].pPTRC + iPG * sizeof(pMapping->aPTs[0].pPTR3->a[0]);
2136	pVM->pgm.s.paDynPageMapPaePTEsGC = pMapping->aPTs[iPT].paPaePTsRC + iPG * sizeof(pMapping->aPTs[0].paPaePTsR3->a[0]);
2137
2138	/* init cache area */
2139	RTHCPHYS HCPhysDummy = MMR3PageDummyHCPhys(pVM);
2140	for (uint32_t offDynMap = 0; offDynMap < MM_HYPER_DYNAMIC_SIZE; offDynMap += PAGE_SIZE)
2141	{
2142	rc = PGMMap(pVM, pVM->pgm.s.pbDynPageMapBaseGC + offDynMap, HCPhysDummy, PAGE_SIZE, 0);
2143	AssertRCReturn(rc, rc);
2144	}
2145
2146	/*
2147	* Determine the max physical address width (MAXPHYADDR) and apply it to
2148	* all the mask members and stuff.
2149	*/
2150	uint32_t cMaxPhysAddrWidth;
2151	uint32_t uMaxExtLeaf = ASMCpuId_EAX(0x80000000);
2152	if ( uMaxExtLeaf >= 0x80000008
2153	&& uMaxExtLeaf <= 0x80000fff)
2154	{
2155	cMaxPhysAddrWidth = ASMCpuId_EAX(0x80000008) & 0xff;
2156	LogRel(("PGM: The CPU physical address width is %u bits\n", cMaxPhysAddrWidth));
2157	cMaxPhysAddrWidth = RT_MIN(52, cMaxPhysAddrWidth);
2158	pVM->pgm.s.fLessThan52PhysicalAddressBits = cMaxPhysAddrWidth < 52;
2159	for (uint32_t iBit = cMaxPhysAddrWidth; iBit < 52; iBit++)
2160	pVM->pgm.s.HCPhysInvMmioPg \|= RT_BIT_64(iBit);
2161	}
2162	else
2163	{
2164	LogRel(("PGM: ASSUMING CPU physical address width of 48 bits (uMaxExtLeaf=%#x)\n", uMaxExtLeaf));
2165	cMaxPhysAddrWidth = 48;
2166	pVM->pgm.s.fLessThan52PhysicalAddressBits = true;
2167	pVM->pgm.s.HCPhysInvMmioPg \|= UINT64_C(0x000f0000000000);
2168	}
2169
2170	pVM->pgm.s.GCPhysInvAddrMask = 0;
2171	for (uint32_t iBit = cMaxPhysAddrWidth; iBit < 64; iBit++)
2172	pVM->pgm.s.GCPhysInvAddrMask \|= RT_BIT_64(iBit);
2173
2174	/*
2175	* Initialize the invalid paging entry masks, assuming NX is disabled.
2176	*/
2177	uint64_t fMbzPageFrameMask = pVM->pgm.s.GCPhysInvAddrMask & UINT64_C(0x000ffffffffff000);
2178	for (VMCPUID iCpu = 0; iCpu < pVM->cCpus; iCpu++)
2179	{
2180	PVMCPU pVCpu = &pVM->aCpus[iCpu];
2181
2182	/** @todo The manuals are not entirely clear whether the physical
2183	* address width is relevant. See table 5-9 in the intel
2184	* manual vs the PDE4M descriptions. Write testcase (NP). */
2185	pVCpu->pgm.s.fGst32BitMbzBigPdeMask = ((uint32_t)(fMbzPageFrameMask >> (32 - 13)) & X86_PDE4M_PG_HIGH_MASK)
2186	\| X86_PDE4M_MBZ_MASK;
2187
2188	pVCpu->pgm.s.fGstPaeMbzPteMask = fMbzPageFrameMask \| X86_PTE_PAE_MBZ_MASK_NO_NX;
2189	pVCpu->pgm.s.fGstPaeMbzPdeMask = fMbzPageFrameMask \| X86_PDE_PAE_MBZ_MASK_NO_NX;
2190	pVCpu->pgm.s.fGstPaeMbzBigPdeMask = fMbzPageFrameMask \| X86_PDE2M_PAE_MBZ_MASK_NO_NX;
2191	pVCpu->pgm.s.fGstPaeMbzPdpeMask = fMbzPageFrameMask \| X86_PDPE_PAE_MBZ_MASK;
2192
2193	pVCpu->pgm.s.fGstAmd64MbzPteMask = fMbzPageFrameMask \| X86_PTE_LM_MBZ_MASK_NO_NX;
2194	pVCpu->pgm.s.fGstAmd64MbzPdeMask = fMbzPageFrameMask \| X86_PDE_LM_MBZ_MASK_NX;
2195	pVCpu->pgm.s.fGstAmd64MbzBigPdeMask = fMbzPageFrameMask \| X86_PDE2M_LM_MBZ_MASK_NX;
2196	pVCpu->pgm.s.fGstAmd64MbzPdpeMask = fMbzPageFrameMask \| X86_PDPE_LM_MBZ_MASK_NO_NX;
2197	pVCpu->pgm.s.fGstAmd64MbzBigPdpeMask = fMbzPageFrameMask \| X86_PDPE1G_LM_MBZ_MASK_NO_NX;
2198	pVCpu->pgm.s.fGstAmd64MbzPml4eMask = fMbzPageFrameMask \| X86_PML4E_MBZ_MASK_NO_NX;
2199
2200	pVCpu->pgm.s.fGst64ShadowedPteMask = X86_PTE_P \| X86_PTE_RW \| X86_PTE_US \| X86_PTE_G \| X86_PTE_A \| X86_PTE_D;
2201	pVCpu->pgm.s.fGst64ShadowedPdeMask = X86_PDE_P \| X86_PDE_RW \| X86_PDE_US \| X86_PDE_A;
2202	pVCpu->pgm.s.fGst64ShadowedBigPdeMask = X86_PDE4M_P \| X86_PDE4M_RW \| X86_PDE4M_US \| X86_PDE4M_A;
2203	pVCpu->pgm.s.fGst64ShadowedBigPde4PteMask =
2204	X86_PDE4M_P \| X86_PDE4M_RW \| X86_PDE4M_US \| X86_PDE4M_G \| X86_PDE4M_A \| X86_PDE4M_D;
2205	pVCpu->pgm.s.fGstAmd64ShadowedPdpeMask = X86_PDPE_P \| X86_PDPE_RW \| X86_PDPE_US \| X86_PDPE_A;
2206	pVCpu->pgm.s.fGstAmd64ShadowedPml4eMask = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_US \| X86_PML4E_A;
2207	}
2208
2209	/*
2210	* Note that AMD uses all the 8 reserved bits for the address (so 40 bits in total);
2211	* Intel only goes up to 36 bits, so we stick to 36 as well.
2212	* Update: More recent intel manuals specifies 40 bits just like AMD.
2213	*/
2214	uint32_t u32Dummy, u32Features;
2215	CPUMGetGuestCpuId(VMMGetCpu(pVM), 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
2216	if (u32Features & X86_CPUID_FEATURE_EDX_PSE36)
2217	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(RT_MAX(36, cMaxPhysAddrWidth)) - 1;
2218	else
2219	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(32) - 1;
2220
2221	/*
2222	* Allocate memory if we're supposed to do that.
2223	*/
2224	if (pVM->pgm.s.fRamPreAlloc)
2225	rc = pgmR3PhysRamPreAllocate(pVM);
2226
2227	LogRel(("PGMR3InitFinalize: 4 MB PSE mask %RGp\n", pVM->pgm.s.GCPhys4MBPSEMask));
2228	return rc;
2229	}
2230
2231
2232	/**
2233	* Init phase completed callback.
2234	*
2235	* @returns VBox status code.
2236	* @param pVM Pointer to the VM.
2237	* @param enmWhat What has been completed.
2238	* @thread EMT(0)
2239	*/
2240	VMMR3_INT_DECL(int) PGMR3InitCompleted(PVM pVM, VMINITCOMPLETED enmWhat)
2241	{
2242	switch (enmWhat)
2243	{
2244	case VMINITCOMPLETED_HM:
2245	#ifdef VBOX_WITH_PCI_PASSTHROUGH
2246	if (pVM->pgm.s.fPciPassthrough)
2247	{
2248	AssertLogRelReturn(pVM->pgm.s.fRamPreAlloc, VERR_PCI_PASSTHROUGH_NO_RAM_PREALLOC);
2249	AssertLogRelReturn(HMIsEnabled(pVM), VERR_PCI_PASSTHROUGH_NO_HM);
2250	AssertLogRelReturn(HMIsNestedPagingActive(pVM), VERR_PCI_PASSTHROUGH_NO_NESTED_PAGING);
2251
2252	/*
2253	* Report assignments to the IOMMU (hope that's good enough for now).
2254	*/
2255	if (pVM->pgm.s.fPciPassthrough)
2256	{
2257	int rc = VMMR3CallR0(pVM, VMMR0_DO_PGM_PHYS_SETUP_IOMMU, 0, NULL);
2258	AssertRCReturn(rc, rc);
2259	}
2260	}
2261	#else
2262	AssertLogRelReturn(!pVM->pgm.s.fPciPassthrough, VERR_PGM_PCI_PASSTHRU_MISCONFIG);
2263	#endif
2264	break;
2265
2266	default:
2267	/* shut up gcc */
2268	break;
2269	}
2270
2271	return VINF_SUCCESS;
2272	}
2273
2274
2275	/**
2276	* Applies relocations to data and code managed by this component.
2277	*
2278	* This function will be called at init and whenever the VMM need to relocate it
2279	* self inside the GC.
2280	*
2281	* @param pVM The VM.
2282	* @param offDelta Relocation delta relative to old location.
2283	*/
2284	VMMR3DECL(void) PGMR3Relocate(PVM pVM, RTGCINTPTR offDelta)
2285	{
2286	LogFlow(("PGMR3Relocate %RGv to %RGv\n", pVM->pgm.s.GCPtrCR3Mapping, pVM->pgm.s.GCPtrCR3Mapping + offDelta));
2287
2288	/*
2289	* Paging stuff.
2290	*/
2291	pVM->pgm.s.GCPtrCR3Mapping += offDelta;
2292
2293	pgmR3ModeDataInit(pVM, true /* resolve GC/R0 symbols */);
2294
2295	/* Shadow, guest and both mode switch & relocation for each VCPU. */
2296	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2297	{
2298	PVMCPU pVCpu = &pVM->aCpus[i];
2299
2300	pgmR3ModeDataSwitch(pVM, pVCpu, pVCpu->pgm.s.enmShadowMode, pVCpu->pgm.s.enmGuestMode);
2301
2302	PGM_SHW_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2303	PGM_GST_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2304	PGM_BTH_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2305	}
2306
2307	/*
2308	* Trees.
2309	*/
2310	pVM->pgm.s.pTreesRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pTreesR3);
2311
2312	/*
2313	* Ram ranges.
2314	*/
2315	if (pVM->pgm.s.pRamRangesXR3)
2316	{
2317	/* Update the pSelfRC pointers and relink them. */
2318	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesXR3; pCur; pCur = pCur->pNextR3)
2319	if (!(pCur->fFlags & PGM_RAM_RANGE_FLAGS_FLOATING))
2320	pCur->pSelfRC = MMHyperCCToRC(pVM, pCur);
2321	pgmR3PhysRelinkRamRanges(pVM);
2322
2323	/* Flush the RC TLB. */
2324	for (unsigned i = 0; i < PGM_RAMRANGE_TLB_ENTRIES; i++)
2325	pVM->pgm.s.apRamRangesTlbRC[i] = NIL_RTRCPTR;
2326	}
2327
2328	/*
2329	* Update the pSelfRC pointer of the MMIO2 ram ranges since they might not
2330	* be mapped and thus not included in the above exercise.
2331	*/
2332	for (PPGMMMIO2RANGE pCur = pVM->pgm.s.pMmio2RangesR3; pCur; pCur = pCur->pNextR3)
2333	if (!(pCur->RamRange.fFlags & PGM_RAM_RANGE_FLAGS_FLOATING))
2334	pCur->RamRange.pSelfRC = MMHyperCCToRC(pVM, &pCur->RamRange);
2335
2336	/*
2337	* Update the two page directories with all page table mappings.
2338	* (One or more of them have changed, that's why we're here.)
2339	*/
2340	pVM->pgm.s.pMappingsRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pMappingsR3);
2341	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur->pNextR3; pCur = pCur->pNextR3)
2342	pCur->pNextRC = MMHyperR3ToRC(pVM, pCur->pNextR3);
2343
2344	/* Relocate GC addresses of Page Tables. */
2345	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur; pCur = pCur->pNextR3)
2346	{
2347	for (RTHCUINT i = 0; i < pCur->cPTs; i++)
2348	{
2349	pCur->aPTs[i].pPTRC = MMHyperR3ToRC(pVM, pCur->aPTs[i].pPTR3);
2350	pCur->aPTs[i].paPaePTsRC = MMHyperR3ToRC(pVM, pCur->aPTs[i].paPaePTsR3);
2351	}
2352	}
2353
2354	/*
2355	* Dynamic page mapping area.
2356	*/
2357	pVM->pgm.s.paDynPageMap32BitPTEsGC += offDelta;
2358	pVM->pgm.s.paDynPageMapPaePTEsGC += offDelta;
2359	pVM->pgm.s.pbDynPageMapBaseGC += offDelta;
2360
2361	if (pVM->pgm.s.pRCDynMap)
2362	{
2363	pVM->pgm.s.pRCDynMap += offDelta;
2364	PPGMRCDYNMAP pDynMap = (PPGMRCDYNMAP)MMHyperRCToCC(pVM, pVM->pgm.s.pRCDynMap);
2365
2366	pDynMap->paPages += offDelta;
2367	PPGMRCDYNMAPENTRY paPages = (PPGMRCDYNMAPENTRY)MMHyperRCToCC(pVM, pDynMap->paPages);
2368
2369	for (uint32_t iPage = 0; iPage < pDynMap->cPages; iPage++)
2370	{
2371	paPages[iPage].pvPage += offDelta;
2372	paPages[iPage].uPte.pLegacy += offDelta;
2373	paPages[iPage].uPte.pPae += offDelta;
2374	}
2375	}
2376
2377	/*
2378	* The Zero page.
2379	*/
2380	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
2381	#ifdef VBOX_WITH_2X_4GB_ADDR_SPACE
2382	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTR0PTR \|\| !HMIsEnabled(pVM));
2383	#else
2384	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTR0PTR);
2385	#endif
2386
2387	/*
2388	* Physical and virtual handlers.
2389	*/
2390	RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, true, pgmR3RelocatePhysHandler, &offDelta);
2391	pVM->pgm.s.pLastPhysHandlerRC = NIL_RTRCPTR;
2392	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesR3->VirtHandlers, true, pgmR3RelocateVirtHandler, &offDelta);
2393	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesR3->HyperVirtHandlers, true, pgmR3RelocateHyperVirtHandler, &offDelta);
2394
2395	/*
2396	* The page pool.
2397	*/
2398	pgmR3PoolRelocate(pVM);
2399
2400	#ifdef VBOX_WITH_STATISTICS
2401	/*
2402	* Statistics.
2403	*/
2404	pVM->pgm.s.pStatsRC = MMHyperCCToRC(pVM, pVM->pgm.s.pStatsR3);
2405	for (VMCPUID iCpu = 0; iCpu < pVM->cCpus; iCpu++)
2406	pVM->aCpus[iCpu].pgm.s.pStatsRC = MMHyperCCToRC(pVM, pVM->aCpus[iCpu].pgm.s.pStatsR3);
2407	#endif
2408	}
2409
2410
2411	/**
2412	* Callback function for relocating a physical access handler.
2413	*
2414	* @returns 0 (continue enum)
2415	* @param pNode Pointer to a PGMPHYSHANDLER node.
2416	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2417	* not certain the delta will fit in a void pointer for all possible configs.
2418	*/
2419	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser)
2420	{
2421	PPGMPHYSHANDLER pHandler = (PPGMPHYSHANDLER)pNode;
2422	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2423	if (pHandler->pfnHandlerRC)
2424	pHandler->pfnHandlerRC += offDelta;
2425	if (pHandler->pvUserRC >= 0x10000)
2426	pHandler->pvUserRC += offDelta;
2427	return 0;
2428	}
2429
2430
2431	/**
2432	* Callback function for relocating a virtual access handler.
2433	*
2434	* @returns 0 (continue enum)
2435	* @param pNode Pointer to a PGMVIRTHANDLER node.
2436	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2437	* not certain the delta will fit in a void pointer for all possible configs.
2438	*/
2439	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
2440	{
2441	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
2442	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2443	Assert( pHandler->enmType == PGMVIRTHANDLERTYPE_ALL
2444	\|\| pHandler->enmType == PGMVIRTHANDLERTYPE_WRITE);
2445	Assert(pHandler->pfnHandlerRC);
2446	pHandler->pfnHandlerRC += offDelta;
2447	return 0;
2448	}
2449
2450
2451	/**
2452	* Callback function for relocating a virtual access handler for the hypervisor mapping.
2453	*
2454	* @returns 0 (continue enum)
2455	* @param pNode Pointer to a PGMVIRTHANDLER node.
2456	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2457	* not certain the delta will fit in a void pointer for all possible configs.
2458	*/
2459	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
2460	{
2461	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
2462	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2463	Assert(pHandler->enmType == PGMVIRTHANDLERTYPE_HYPERVISOR);
2464	Assert(pHandler->pfnHandlerRC);
2465	pHandler->pfnHandlerRC += offDelta;
2466	return 0;
2467	}
2468
2469
2470	/**
2471	* Resets a virtual CPU when unplugged.
2472	*
2473	* @param pVM Pointer to the VM.
2474	* @param pVCpu Pointer to the VMCPU.
2475	*/
2476	VMMR3DECL(void) PGMR3ResetCpu(PVM pVM, PVMCPU pVCpu)
2477	{
2478	int rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
2479	AssertRC(rc);
2480
2481	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
2482	AssertRC(rc);
2483
2484	STAM_REL_COUNTER_RESET(&pVCpu->pgm.s.cGuestModeChanges);
2485
2486	pgmR3PoolResetUnpluggedCpu(pVM, pVCpu);
2487
2488	/*
2489	* Re-init other members.
2490	*/
2491	pVCpu->pgm.s.fA20Enabled = true;
2492	pVCpu->pgm.s.GCPhysA20Mask = ~((RTGCPHYS)!pVCpu->pgm.s.fA20Enabled << 20);
2493
2494	/*
2495	* Clear the FFs PGM owns.
2496	*/
2497	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
2498	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3_NON_GLOBAL);
2499	}
2500
2501
2502	/**
2503	* The VM is being reset.
2504	*
2505	* For the PGM component this means that any PD write monitors
2506	* needs to be removed.
2507	*
2508	* @param pVM Pointer to the VM.
2509	*/
2510	VMMR3_INT_DECL(void) PGMR3Reset(PVM pVM)
2511	{
2512	LogFlow(("PGMR3Reset:\n"));
2513	VM_ASSERT_EMT(pVM);
2514
2515	pgmLock(pVM);
2516
2517	/*
2518	* Unfix any fixed mappings and disable CR3 monitoring.
2519	*/
2520	pVM->pgm.s.fMappingsFixed = false;
2521	pVM->pgm.s.fMappingsFixedRestored = false;
2522	pVM->pgm.s.GCPtrMappingFixed = NIL_RTGCPTR;
2523	pVM->pgm.s.cbMappingFixed = 0;
2524
2525	/*
2526	* Exit the guest paging mode before the pgm pool gets reset.
2527	* Important to clean up the amd64 case.
2528	*/
2529	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2530	{
2531	PVMCPU pVCpu = &pVM->aCpus[i];
2532	int rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
2533	AssertReleaseRC(rc);
2534	}
2535
2536	#ifdef DEBUG
2537	DBGFR3_INFO_LOG(pVM, "mappings", NULL);
2538	DBGFR3_INFO_LOG(pVM, "handlers", "all nostat");
2539	#endif
2540
2541	/*
2542	* Switch mode back to real mode. (before resetting the pgm pool!)
2543	*/
2544	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2545	{
2546	PVMCPU pVCpu = &pVM->aCpus[i];
2547
2548	int rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
2549	AssertReleaseRC(rc);
2550
2551	STAM_REL_COUNTER_RESET(&pVCpu->pgm.s.cGuestModeChanges);
2552	STAM_REL_COUNTER_RESET(&pVCpu->pgm.s.cA20Changes);
2553	}
2554
2555	/*
2556	* Reset the shadow page pool.
2557	*/
2558	pgmR3PoolReset(pVM);
2559
2560	/*
2561	* Re-init various other members and clear the FFs that PGM owns.
2562	*/
2563	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2564	{
2565	PVMCPU pVCpu = &pVM->aCpus[i];
2566
2567	pVCpu->pgm.s.fGst32BitPageSizeExtension = false;
2568	PGMNotifyNxeChanged(pVCpu, false);
2569
2570	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
2571	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3_NON_GLOBAL);
2572
2573	if (!pVCpu->pgm.s.fA20Enabled)
2574	{
2575	pVCpu->pgm.s.fA20Enabled = true;
2576	pVCpu->pgm.s.GCPhysA20Mask = ~((RTGCPHYS)!pVCpu->pgm.s.fA20Enabled << 20);
2577	#ifdef PGM_WITH_A20
2578	pVCpu->pgm.s.fSyncFlags \|= PGM_SYNC_UPDATE_PAGE_BIT_VIRTUAL;
2579	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
2580	pgmR3RefreshShadowModeAfterA20Change(pVCpu);
2581	HMFlushTLB(pVCpu);
2582	#endif
2583	}
2584	}
2585
2586	pgmUnlock(pVM);
2587	}
2588
2589
2590	/**
2591	* Memory setup after VM construction or reset.
2592	*
2593	* @param pVM Pointer to the VM.
2594	* @param fAtReset Indicates the context, after reset if @c true or after
2595	* construction if @c false.
2596	*/
2597	VMMR3_INT_DECL(void) PGMR3MemSetup(PVM pVM, bool fAtReset)
2598	{
2599	if (fAtReset)
2600	{
2601	pgmLock(pVM);
2602
2603	int rc = pgmR3PhysRamZeroAll(pVM);
2604	AssertReleaseRC(rc);
2605
2606	rc = pgmR3PhysRomReset(pVM);
2607	AssertReleaseRC(rc);
2608
2609	pgmUnlock(pVM);
2610	}
2611	}
2612
2613
2614	#ifdef VBOX_STRICT
2615	/**
2616	* VM state change callback for clearing fNoMorePhysWrites after
2617	* a snapshot has been created.
2618	*/
2619	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PUVM pUVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser)
2620	{
2621	if ( enmState == VMSTATE_RUNNING
2622	\|\| enmState == VMSTATE_RESUMING)
2623	pUVM->pVM->pgm.s.fNoMorePhysWrites = false;
2624	NOREF(enmOldState); NOREF(pvUser);
2625	}
2626	#endif
2627
2628	/**
2629	* Private API to reset fNoMorePhysWrites.
2630	*/
2631	VMMR3DECL(void) PGMR3ResetNoMorePhysWritesFlag(PVM pVM)
2632	{
2633	pVM->pgm.s.fNoMorePhysWrites = false;
2634	}
2635
2636	/**
2637	* Terminates the PGM.
2638	*
2639	* @returns VBox status code.
2640	* @param pVM Pointer to VM structure.
2641	*/
2642	VMMR3DECL(int) PGMR3Term(PVM pVM)
2643	{
2644	/* Must free shared pages here. */
2645	pgmLock(pVM);
2646	pgmR3PhysRamTerm(pVM);
2647	pgmR3PhysRomTerm(pVM);
2648	pgmUnlock(pVM);
2649
2650	PGMDeregisterStringFormatTypes();
2651	return PDMR3CritSectDelete(&pVM->pgm.s.CritSectX);
2652	}
2653
2654
2655	/**
2656	* Show paging mode.
2657	*
2658	* @param pVM Pointer to the VM.
2659	* @param pHlp The info helpers.
2660	* @param pszArgs "all" (default), "guest", "shadow" or "host".
2661	*/
2662	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2663	{
2664	/* digest argument. */
2665	bool fGuest, fShadow, fHost;
2666	if (pszArgs)
2667	pszArgs = RTStrStripL(pszArgs);
2668	if (!pszArgs \|\| !*pszArgs \|\| strstr(pszArgs, "all"))
2669	fShadow = fHost = fGuest = true;
2670	else
2671	{
2672	fShadow = fHost = fGuest = false;
2673	if (strstr(pszArgs, "guest"))
2674	fGuest = true;
2675	if (strstr(pszArgs, "shadow"))
2676	fShadow = true;
2677	if (strstr(pszArgs, "host"))
2678	fHost = true;
2679	}
2680
2681	/** @todo SMP support! */
2682	/* print info. */
2683	if (fGuest)
2684	pHlp->pfnPrintf(pHlp, "Guest paging mode: %s (changed %RU64 times), A20 %s (changed %RU64 times)\n",
2685	PGMGetModeName(pVM->aCpus[0].pgm.s.enmGuestMode), pVM->aCpus[0].pgm.s.cGuestModeChanges.c,
2686	pVM->aCpus[0].pgm.s.fA20Enabled ? "enabled" : "disabled", pVM->aCpus[0].pgm.s.cA20Changes.c);
2687	if (fShadow)
2688	pHlp->pfnPrintf(pHlp, "Shadow paging mode: %s\n", PGMGetModeName(pVM->aCpus[0].pgm.s.enmShadowMode));
2689	if (fHost)
2690	{
2691	const char *psz;
2692	switch (pVM->pgm.s.enmHostMode)
2693	{
2694	case SUPPAGINGMODE_INVALID: psz = "invalid"; break;
2695	case SUPPAGINGMODE_32_BIT: psz = "32-bit"; break;
2696	case SUPPAGINGMODE_32_BIT_GLOBAL: psz = "32-bit+G"; break;
2697	case SUPPAGINGMODE_PAE: psz = "PAE"; break;
2698	case SUPPAGINGMODE_PAE_GLOBAL: psz = "PAE+G"; break;
2699	case SUPPAGINGMODE_PAE_NX: psz = "PAE+NX"; break;
2700	case SUPPAGINGMODE_PAE_GLOBAL_NX: psz = "PAE+G+NX"; break;
2701	case SUPPAGINGMODE_AMD64: psz = "AMD64"; break;
2702	case SUPPAGINGMODE_AMD64_GLOBAL: psz = "AMD64+G"; break;
2703	case SUPPAGINGMODE_AMD64_NX: psz = "AMD64+NX"; break;
2704	case SUPPAGINGMODE_AMD64_GLOBAL_NX: psz = "AMD64+G+NX"; break;
2705	default: psz = "unknown"; break;
2706	}
2707	pHlp->pfnPrintf(pHlp, "Host paging mode: %s\n", psz);
2708	}
2709	}
2710
2711
2712	/**
2713	* Dump registered MMIO ranges to the log.
2714	*
2715	* @param pVM Pointer to the VM.
2716	* @param pHlp The info helpers.
2717	* @param pszArgs Arguments, ignored.
2718	*/
2719	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2720	{
2721	NOREF(pszArgs);
2722	pHlp->pfnPrintf(pHlp,
2723	"RAM ranges (pVM=%p)\n"
2724	"%.s %.s\n",
2725	pVM,
2726	sizeof(RTGCPHYS) * 4 + 1, "GC Phys Range ",
2727	sizeof(RTHCPTR) * 2, "pvHC ");
2728
2729	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesXR3; pCur; pCur = pCur->pNextR3)
2730	pHlp->pfnPrintf(pHlp,
2731	"%RGp-%RGp %RHv %s\n",
2732	pCur->GCPhys,
2733	pCur->GCPhysLast,
2734	pCur->pvR3,
2735	pCur->pszDesc);
2736	}
2737
2738
2739	/**
2740	* Dump the page directory to the log.
2741	*
2742	* @param pVM Pointer to the VM.
2743	* @param pHlp The info helpers.
2744	* @param pszArgs Arguments, ignored.
2745	*/
2746	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2747	{
2748	/** @todo SMP support!! */
2749	PVMCPU pVCpu = &pVM->aCpus[0];
2750
2751	/** @todo fix this! Convert the PGMR3DumpHierarchyHC functions to do guest stuff. */
2752	/* Big pages supported? */
2753	const bool fPSE = !!(CPUMGetGuestCR4(pVCpu) & X86_CR4_PSE);
2754
2755	/* Global pages supported? */
2756	const bool fPGE = !!(CPUMGetGuestCR4(pVCpu) & X86_CR4_PGE);
2757
2758	NOREF(pszArgs);
2759
2760	/*
2761	* Get page directory addresses.
2762	*/
2763	pgmLock(pVM);
2764	PX86PD pPDSrc = pgmGstGet32bitPDPtr(pVCpu);
2765	Assert(pPDSrc);
2766
2767	/*
2768	* Iterate the page directory.
2769	*/
2770	for (unsigned iPD = 0; iPD < RT_ELEMENTS(pPDSrc->a); iPD++)
2771	{
2772	X86PDE PdeSrc = pPDSrc->a[iPD];
2773	if (PdeSrc.n.u1Present)
2774	{
2775	if (PdeSrc.b.u1Size && fPSE)
2776	pHlp->pfnPrintf(pHlp,
2777	"%04X - %RGp P=%d U=%d RW=%d G=%d - BIG\n",
2778	iPD,
2779	pgmGstGet4MBPhysPage(pVM, PdeSrc),
2780	PdeSrc.b.u1Present, PdeSrc.b.u1User, PdeSrc.b.u1Write, PdeSrc.b.u1Global && fPGE);
2781	else
2782	pHlp->pfnPrintf(pHlp,
2783	"%04X - %RGp P=%d U=%d RW=%d [G=%d]\n",
2784	iPD,
2785	(RTGCPHYS)(PdeSrc.u & X86_PDE_PG_MASK),
2786	PdeSrc.n.u1Present, PdeSrc.n.u1User, PdeSrc.n.u1Write, PdeSrc.b.u1Global && fPGE);
2787	}
2788	}
2789	pgmUnlock(pVM);
2790	}
2791
2792
2793	/**
2794	* Service a VMMCALLRING3_PGM_LOCK call.
2795	*
2796	* @returns VBox status code.
2797	* @param pVM Pointer to the VM.
2798	*/
2799	VMMR3DECL(int) PGMR3LockCall(PVM pVM)
2800	{
2801	int rc = PDMR3CritSectEnterEx(&pVM->pgm.s.CritSectX, true /* fHostCall */);
2802	AssertRC(rc);
2803	return rc;
2804	}
2805
2806
2807	/**
2808	* Converts a PGMMODE value to a PGM_TYPE_* \#define.
2809	*
2810	* @returns PGM_TYPE_*.
2811	* @param pgmMode The mode value to convert.
2812	*/
2813	DECLINLINE(unsigned) pgmModeToType(PGMMODE pgmMode)
2814	{
2815	switch (pgmMode)
2816	{
2817	case PGMMODE_REAL: return PGM_TYPE_REAL;
2818	case PGMMODE_PROTECTED: return PGM_TYPE_PROT;
2819	case PGMMODE_32_BIT: return PGM_TYPE_32BIT;
2820	case PGMMODE_PAE:
2821	case PGMMODE_PAE_NX: return PGM_TYPE_PAE;
2822	case PGMMODE_AMD64:
2823	case PGMMODE_AMD64_NX: return PGM_TYPE_AMD64;
2824	case PGMMODE_NESTED: return PGM_TYPE_NESTED;
2825	case PGMMODE_EPT: return PGM_TYPE_EPT;
2826	default:
2827	AssertFatalMsgFailed(("pgmMode=%d\n", pgmMode));
2828	}
2829	}
2830
2831
2832	/**
2833	* Gets the index into the paging mode data array of a SHW+GST mode.
2834	*
2835	* @returns PGM::paPagingData index.
2836	* @param uShwType The shadow paging mode type.
2837	* @param uGstType The guest paging mode type.
2838	*/
2839	DECLINLINE(unsigned) pgmModeDataIndex(unsigned uShwType, unsigned uGstType)
2840	{
2841	Assert(uShwType >= PGM_TYPE_32BIT && uShwType <= PGM_TYPE_MAX);
2842	Assert(uGstType >= PGM_TYPE_REAL && uGstType <= PGM_TYPE_AMD64);
2843	return (uShwType - PGM_TYPE_32BIT) * (PGM_TYPE_AMD64 - PGM_TYPE_REAL + 1)
2844	+ (uGstType - PGM_TYPE_REAL);
2845	}
2846
2847
2848	/**
2849	* Gets the index into the paging mode data array of a SHW+GST mode.
2850	*
2851	* @returns PGM::paPagingData index.
2852	* @param enmShw The shadow paging mode.
2853	* @param enmGst The guest paging mode.
2854	*/
2855	DECLINLINE(unsigned) pgmModeDataIndexByMode(PGMMODE enmShw, PGMMODE enmGst)
2856	{
2857	Assert(enmShw >= PGMMODE_32_BIT && enmShw <= PGMMODE_MAX);
2858	Assert(enmGst > PGMMODE_INVALID && enmGst < PGMMODE_MAX);
2859	return pgmModeDataIndex(pgmModeToType(enmShw), pgmModeToType(enmGst));
2860	}
2861
2862
2863	/**
2864	* Calculates the max data index.
2865	* @returns The number of entries in the paging data array.
2866	*/
2867	DECLINLINE(unsigned) pgmModeDataMaxIndex(void)
2868	{
2869	return pgmModeDataIndex(PGM_TYPE_MAX, PGM_TYPE_AMD64) + 1;
2870	}
2871
2872
2873	/**
2874	* Initializes the paging mode data kept in PGM::paModeData.
2875	*
2876	* @param pVM Pointer to the VM.
2877	* @param fResolveGCAndR0 Indicate whether or not GC and Ring-0 symbols can be resolved now.
2878	* This is used early in the init process to avoid trouble with PDM
2879	* not being initialized yet.
2880	*/
2881	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0)
2882	{
2883	PPGMMODEDATA pModeData;
2884	int rc;
2885
2886	/*
2887	* Allocate the array on the first call.
2888	*/
2889	if (!pVM->pgm.s.paModeData)
2890	{
2891	pVM->pgm.s.paModeData = (PPGMMODEDATA)MMR3HeapAllocZ(pVM, MM_TAG_PGM, sizeof(PGMMODEDATA) * pgmModeDataMaxIndex());
2892	AssertReturn(pVM->pgm.s.paModeData, VERR_NO_MEMORY);
2893	}
2894
2895	/*
2896	* Initialize the array entries.
2897	*/
2898	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_REAL)];
2899	pModeData->uShwType = PGM_TYPE_32BIT;
2900	pModeData->uGstType = PGM_TYPE_REAL;
2901	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2902	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2903	rc = PGM_BTH_NAME_32BIT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2904
2905	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGMMODE_PROTECTED)];
2906	pModeData->uShwType = PGM_TYPE_32BIT;
2907	pModeData->uGstType = PGM_TYPE_PROT;
2908	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2909	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2910	rc = PGM_BTH_NAME_32BIT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2911
2912	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_32BIT)];
2913	pModeData->uShwType = PGM_TYPE_32BIT;
2914	pModeData->uGstType = PGM_TYPE_32BIT;
2915	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2916	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2917	rc = PGM_BTH_NAME_32BIT_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2918
2919	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_REAL)];
2920	pModeData->uShwType = PGM_TYPE_PAE;
2921	pModeData->uGstType = PGM_TYPE_REAL;
2922	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2923	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2924	rc = PGM_BTH_NAME_PAE_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2925
2926	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PROT)];
2927	pModeData->uShwType = PGM_TYPE_PAE;
2928	pModeData->uGstType = PGM_TYPE_PROT;
2929	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2930	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2931	rc = PGM_BTH_NAME_PAE_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2932
2933	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_32BIT)];
2934	pModeData->uShwType = PGM_TYPE_PAE;
2935	pModeData->uGstType = PGM_TYPE_32BIT;
2936	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2937	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2938	rc = PGM_BTH_NAME_PAE_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2939
2940	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PAE)];
2941	pModeData->uShwType = PGM_TYPE_PAE;
2942	pModeData->uGstType = PGM_TYPE_PAE;
2943	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2944	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2945	rc = PGM_BTH_NAME_PAE_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2946
2947	#ifdef VBOX_WITH_64_BITS_GUESTS
2948	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_AMD64, PGM_TYPE_AMD64)];
2949	pModeData->uShwType = PGM_TYPE_AMD64;
2950	pModeData->uGstType = PGM_TYPE_AMD64;
2951	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2952	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2953	rc = PGM_BTH_NAME_AMD64_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2954	#endif
2955
2956	/* The nested paging mode. */
2957	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_REAL)];
2958	pModeData->uShwType = PGM_TYPE_NESTED;
2959	pModeData->uGstType = PGM_TYPE_REAL;
2960	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2961	rc = PGM_BTH_NAME_NESTED_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2962
2963	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGMMODE_PROTECTED)];
2964	pModeData->uShwType = PGM_TYPE_NESTED;
2965	pModeData->uGstType = PGM_TYPE_PROT;
2966	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2967	rc = PGM_BTH_NAME_NESTED_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2968
2969	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_32BIT)];
2970	pModeData->uShwType = PGM_TYPE_NESTED;
2971	pModeData->uGstType = PGM_TYPE_32BIT;
2972	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2973	rc = PGM_BTH_NAME_NESTED_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2974
2975	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_PAE)];
2976	pModeData->uShwType = PGM_TYPE_NESTED;
2977	pModeData->uGstType = PGM_TYPE_PAE;
2978	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2979	rc = PGM_BTH_NAME_NESTED_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2980
2981	#ifdef VBOX_WITH_64_BITS_GUESTS
2982	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2983	pModeData->uShwType = PGM_TYPE_NESTED;
2984	pModeData->uGstType = PGM_TYPE_AMD64;
2985	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2986	rc = PGM_BTH_NAME_NESTED_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2987	#endif
2988
2989	/* The shadow part of the nested callback mode depends on the host paging mode (AMD-V only). */
2990	switch (pVM->pgm.s.enmHostMode)
2991	{
2992	#if HC_ARCH_BITS == 32
2993	case SUPPAGINGMODE_32_BIT:
2994	case SUPPAGINGMODE_32_BIT_GLOBAL:
2995	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2996	{
2997	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2998	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2999	}
3000	# ifdef VBOX_WITH_64_BITS_GUESTS
3001	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
3002	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3003	# endif
3004	break;
3005
3006	case SUPPAGINGMODE_PAE:
3007	case SUPPAGINGMODE_PAE_NX:
3008	case SUPPAGINGMODE_PAE_GLOBAL:
3009	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3010	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
3011	{
3012	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
3013	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3014	}
3015	# ifdef VBOX_WITH_64_BITS_GUESTS
3016	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
3017	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3018	# endif
3019	break;
3020	#endif /* HC_ARCH_BITS == 32 */
3021
3022	#if HC_ARCH_BITS == 64 \|\| defined(RT_OS_DARWIN)
3023	case SUPPAGINGMODE_AMD64:
3024	case SUPPAGINGMODE_AMD64_GLOBAL:
3025	case SUPPAGINGMODE_AMD64_NX:
3026	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3027	# ifdef VBOX_WITH_64_BITS_GUESTS
3028	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_AMD64; i++)
3029	# else
3030	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
3031	# endif
3032	{
3033	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
3034	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3035	}
3036	break;
3037	#endif /* HC_ARCH_BITS == 64 \|\| RT_OS_DARWIN */
3038
3039	default:
3040	AssertFailed();
3041	break;
3042	}
3043
3044	/* Extended paging (EPT) / Intel VT-x */
3045	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_REAL)];
3046	pModeData->uShwType = PGM_TYPE_EPT;
3047	pModeData->uGstType = PGM_TYPE_REAL;
3048	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3049	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3050	rc = PGM_BTH_NAME_EPT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3051
3052	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_PROT)];
3053	pModeData->uShwType = PGM_TYPE_EPT;
3054	pModeData->uGstType = PGM_TYPE_PROT;
3055	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3056	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3057	rc = PGM_BTH_NAME_EPT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3058
3059	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_32BIT)];
3060	pModeData->uShwType = PGM_TYPE_EPT;
3061	pModeData->uGstType = PGM_TYPE_32BIT;
3062	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3063	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3064	rc = PGM_BTH_NAME_EPT_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3065
3066	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_PAE)];
3067	pModeData->uShwType = PGM_TYPE_EPT;
3068	pModeData->uGstType = PGM_TYPE_PAE;
3069	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3070	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3071	rc = PGM_BTH_NAME_EPT_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3072
3073	#ifdef VBOX_WITH_64_BITS_GUESTS
3074	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_AMD64)];
3075	pModeData->uShwType = PGM_TYPE_EPT;
3076	pModeData->uGstType = PGM_TYPE_AMD64;
3077	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3078	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3079	rc = PGM_BTH_NAME_EPT_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3080	#endif
3081	return VINF_SUCCESS;
3082	}
3083
3084
3085	/**
3086	* Switch to different (or relocated in the relocate case) mode data.
3087	*
3088	* @param pVM Pointer to the VM.
3089	* @param pVCpu Pointer to the VMCPU.
3090	* @param enmShw The shadow paging mode.
3091	* @param enmGst The guest paging mode.
3092	*/
3093	static void pgmR3ModeDataSwitch(PVM pVM, PVMCPU pVCpu, PGMMODE enmShw, PGMMODE enmGst)
3094	{
3095	PPGMMODEDATA pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndexByMode(enmShw, enmGst)];
3096
3097	Assert(pModeData->uGstType == pgmModeToType(enmGst));
3098	Assert(pModeData->uShwType == pgmModeToType(enmShw));
3099
3100	/* shadow */
3101	pVCpu->pgm.s.pfnR3ShwRelocate = pModeData->pfnR3ShwRelocate;
3102	pVCpu->pgm.s.pfnR3ShwExit = pModeData->pfnR3ShwExit;
3103	pVCpu->pgm.s.pfnR3ShwGetPage = pModeData->pfnR3ShwGetPage;
3104	Assert(pVCpu->pgm.s.pfnR3ShwGetPage);
3105	pVCpu->pgm.s.pfnR3ShwModifyPage = pModeData->pfnR3ShwModifyPage;
3106
3107	pVCpu->pgm.s.pfnRCShwGetPage = pModeData->pfnRCShwGetPage;
3108	pVCpu->pgm.s.pfnRCShwModifyPage = pModeData->pfnRCShwModifyPage;
3109
3110	pVCpu->pgm.s.pfnR0ShwGetPage = pModeData->pfnR0ShwGetPage;
3111	pVCpu->pgm.s.pfnR0ShwModifyPage = pModeData->pfnR0ShwModifyPage;
3112
3113
3114	/* guest */
3115	pVCpu->pgm.s.pfnR3GstRelocate = pModeData->pfnR3GstRelocate;
3116	pVCpu->pgm.s.pfnR3GstExit = pModeData->pfnR3GstExit;
3117	pVCpu->pgm.s.pfnR3GstGetPage = pModeData->pfnR3GstGetPage;
3118	Assert(pVCpu->pgm.s.pfnR3GstGetPage);
3119	pVCpu->pgm.s.pfnR3GstModifyPage = pModeData->pfnR3GstModifyPage;
3120	pVCpu->pgm.s.pfnR3GstGetPDE = pModeData->pfnR3GstGetPDE;
3121	pVCpu->pgm.s.pfnRCGstGetPage = pModeData->pfnRCGstGetPage;
3122	pVCpu->pgm.s.pfnRCGstModifyPage = pModeData->pfnRCGstModifyPage;
3123	pVCpu->pgm.s.pfnRCGstGetPDE = pModeData->pfnRCGstGetPDE;
3124	pVCpu->pgm.s.pfnR0GstGetPage = pModeData->pfnR0GstGetPage;
3125	pVCpu->pgm.s.pfnR0GstModifyPage = pModeData->pfnR0GstModifyPage;
3126	pVCpu->pgm.s.pfnR0GstGetPDE = pModeData->pfnR0GstGetPDE;
3127
3128	/* both */
3129	pVCpu->pgm.s.pfnR3BthRelocate = pModeData->pfnR3BthRelocate;
3130	pVCpu->pgm.s.pfnR3BthInvalidatePage = pModeData->pfnR3BthInvalidatePage;
3131	pVCpu->pgm.s.pfnR3BthSyncCR3 = pModeData->pfnR3BthSyncCR3;
3132	Assert(pVCpu->pgm.s.pfnR3BthSyncCR3);
3133	pVCpu->pgm.s.pfnR3BthPrefetchPage = pModeData->pfnR3BthPrefetchPage;
3134	pVCpu->pgm.s.pfnR3BthVerifyAccessSyncPage = pModeData->pfnR3BthVerifyAccessSyncPage;
3135	#ifdef VBOX_STRICT
3136	pVCpu->pgm.s.pfnR3BthAssertCR3 = pModeData->pfnR3BthAssertCR3;
3137	#endif
3138	pVCpu->pgm.s.pfnR3BthMapCR3 = pModeData->pfnR3BthMapCR3;
3139	pVCpu->pgm.s.pfnR3BthUnmapCR3 = pModeData->pfnR3BthUnmapCR3;
3140
3141	pVCpu->pgm.s.pfnRCBthTrap0eHandler = pModeData->pfnRCBthTrap0eHandler;
3142	pVCpu->pgm.s.pfnRCBthInvalidatePage = pModeData->pfnRCBthInvalidatePage;
3143	pVCpu->pgm.s.pfnRCBthSyncCR3 = pModeData->pfnRCBthSyncCR3;
3144	pVCpu->pgm.s.pfnRCBthPrefetchPage = pModeData->pfnRCBthPrefetchPage;
3145	pVCpu->pgm.s.pfnRCBthVerifyAccessSyncPage = pModeData->pfnRCBthVerifyAccessSyncPage;
3146	#ifdef VBOX_STRICT
3147	pVCpu->pgm.s.pfnRCBthAssertCR3 = pModeData->pfnRCBthAssertCR3;
3148	#endif
3149	pVCpu->pgm.s.pfnRCBthMapCR3 = pModeData->pfnRCBthMapCR3;
3150	pVCpu->pgm.s.pfnRCBthUnmapCR3 = pModeData->pfnRCBthUnmapCR3;
3151
3152	pVCpu->pgm.s.pfnR0BthTrap0eHandler = pModeData->pfnR0BthTrap0eHandler;
3153	pVCpu->pgm.s.pfnR0BthInvalidatePage = pModeData->pfnR0BthInvalidatePage;
3154	pVCpu->pgm.s.pfnR0BthSyncCR3 = pModeData->pfnR0BthSyncCR3;
3155	pVCpu->pgm.s.pfnR0BthPrefetchPage = pModeData->pfnR0BthPrefetchPage;
3156	pVCpu->pgm.s.pfnR0BthVerifyAccessSyncPage = pModeData->pfnR0BthVerifyAccessSyncPage;
3157	#ifdef VBOX_STRICT
3158	pVCpu->pgm.s.pfnR0BthAssertCR3 = pModeData->pfnR0BthAssertCR3;
3159	#endif
3160	pVCpu->pgm.s.pfnR0BthMapCR3 = pModeData->pfnR0BthMapCR3;
3161	pVCpu->pgm.s.pfnR0BthUnmapCR3 = pModeData->pfnR0BthUnmapCR3;
3162	}
3163
3164
3165	/**
3166	* Calculates the shadow paging mode.
3167	*
3168	* @returns The shadow paging mode.
3169	* @param pVM Pointer to the VM.
3170	* @param enmGuestMode The guest mode.
3171	* @param enmHostMode The host mode.
3172	* @param enmShadowMode The current shadow mode.
3173	* @param penmSwitcher Where to store the switcher to use.
3174	* VMMSWITCHER_INVALID means no change.
3175	*/
3176	static PGMMODE pgmR3CalcShadowMode(PVM pVM, PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher)
3177	{
3178	VMMSWITCHER enmSwitcher = VMMSWITCHER_INVALID;
3179	switch (enmGuestMode)
3180	{
3181	/*
3182	* When switching to real or protected mode we don't change
3183	* anything since it's likely that we'll switch back pretty soon.
3184	*
3185	* During pgmR3InitPaging we'll end up here with PGMMODE_INVALID
3186	* and is supposed to determine which shadow paging and switcher to
3187	* use during init.
3188	*/
3189	case PGMMODE_REAL:
3190	case PGMMODE_PROTECTED:
3191	if ( enmShadowMode != PGMMODE_INVALID
3192	&& !HMIsEnabled(pVM) /* always switch in hm mode! */)
3193	break; /* (no change) */
3194
3195	switch (enmHostMode)
3196	{
3197	case SUPPAGINGMODE_32_BIT:
3198	case SUPPAGINGMODE_32_BIT_GLOBAL:
3199	enmShadowMode = PGMMODE_32_BIT;
3200	enmSwitcher = VMMSWITCHER_32_TO_32;
3201	break;
3202
3203	case SUPPAGINGMODE_PAE:
3204	case SUPPAGINGMODE_PAE_NX:
3205	case SUPPAGINGMODE_PAE_GLOBAL:
3206	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3207	enmShadowMode = PGMMODE_PAE;
3208	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3209	#ifdef DEBUG_bird
3210	if (RTEnvExist("VBOX_32BIT"))
3211	{
3212	enmShadowMode = PGMMODE_32_BIT;
3213	enmSwitcher = VMMSWITCHER_PAE_TO_32;
3214	}
3215	#endif
3216	break;
3217
3218	case SUPPAGINGMODE_AMD64:
3219	case SUPPAGINGMODE_AMD64_GLOBAL:
3220	case SUPPAGINGMODE_AMD64_NX:
3221	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3222	enmShadowMode = PGMMODE_PAE;
3223	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3224	#ifdef DEBUG_bird
3225	if (RTEnvExist("VBOX_32BIT"))
3226	{
3227	enmShadowMode = PGMMODE_32_BIT;
3228	enmSwitcher = VMMSWITCHER_AMD64_TO_32;
3229	}
3230	#endif
3231	break;
3232
3233	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3234	}
3235	break;
3236
3237	case PGMMODE_32_BIT:
3238	switch (enmHostMode)
3239	{
3240	case SUPPAGINGMODE_32_BIT:
3241	case SUPPAGINGMODE_32_BIT_GLOBAL:
3242	enmShadowMode = PGMMODE_32_BIT;
3243	enmSwitcher = VMMSWITCHER_32_TO_32;
3244	break;
3245
3246	case SUPPAGINGMODE_PAE:
3247	case SUPPAGINGMODE_PAE_NX:
3248	case SUPPAGINGMODE_PAE_GLOBAL:
3249	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3250	enmShadowMode = PGMMODE_PAE;
3251	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3252	#ifdef DEBUG_bird
3253	if (RTEnvExist("VBOX_32BIT"))
3254	{
3255	enmShadowMode = PGMMODE_32_BIT;
3256	enmSwitcher = VMMSWITCHER_PAE_TO_32;
3257	}
3258	#endif
3259	break;
3260
3261	case SUPPAGINGMODE_AMD64:
3262	case SUPPAGINGMODE_AMD64_GLOBAL:
3263	case SUPPAGINGMODE_AMD64_NX:
3264	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3265	enmShadowMode = PGMMODE_PAE;
3266	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3267	#ifdef DEBUG_bird
3268	if (RTEnvExist("VBOX_32BIT"))
3269	{
3270	enmShadowMode = PGMMODE_32_BIT;
3271	enmSwitcher = VMMSWITCHER_AMD64_TO_32;
3272	}
3273	#endif
3274	break;
3275
3276	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3277	}
3278	break;
3279
3280	case PGMMODE_PAE:
3281	case PGMMODE_PAE_NX: /** @todo This might require more switchers and guest+both modes. */
3282	switch (enmHostMode)
3283	{
3284	case SUPPAGINGMODE_32_BIT:
3285	case SUPPAGINGMODE_32_BIT_GLOBAL:
3286	enmShadowMode = PGMMODE_PAE;
3287	enmSwitcher = VMMSWITCHER_32_TO_PAE;
3288	break;
3289
3290	case SUPPAGINGMODE_PAE:
3291	case SUPPAGINGMODE_PAE_NX:
3292	case SUPPAGINGMODE_PAE_GLOBAL:
3293	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3294	enmShadowMode = PGMMODE_PAE;
3295	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3296	break;
3297
3298	case SUPPAGINGMODE_AMD64:
3299	case SUPPAGINGMODE_AMD64_GLOBAL:
3300	case SUPPAGINGMODE_AMD64_NX:
3301	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3302	enmShadowMode = PGMMODE_PAE;
3303	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3304	break;
3305
3306	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3307	}
3308	break;
3309
3310	case PGMMODE_AMD64:
3311	case PGMMODE_AMD64_NX:
3312	switch (enmHostMode)
3313	{
3314	case SUPPAGINGMODE_32_BIT:
3315	case SUPPAGINGMODE_32_BIT_GLOBAL:
3316	enmShadowMode = PGMMODE_AMD64;
3317	enmSwitcher = VMMSWITCHER_32_TO_AMD64;
3318	break;
3319
3320	case SUPPAGINGMODE_PAE:
3321	case SUPPAGINGMODE_PAE_NX:
3322	case SUPPAGINGMODE_PAE_GLOBAL:
3323	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3324	enmShadowMode = PGMMODE_AMD64;
3325	enmSwitcher = VMMSWITCHER_PAE_TO_AMD64;
3326	break;
3327
3328	case SUPPAGINGMODE_AMD64:
3329	case SUPPAGINGMODE_AMD64_GLOBAL:
3330	case SUPPAGINGMODE_AMD64_NX:
3331	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3332	enmShadowMode = PGMMODE_AMD64;
3333	enmSwitcher = VMMSWITCHER_AMD64_TO_AMD64;
3334	break;
3335
3336	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3337	}
3338	break;
3339
3340
3341	default:
3342	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
3343	*penmSwitcher = VMMSWITCHER_INVALID;
3344	return PGMMODE_INVALID;
3345	}
3346	/* Override the shadow mode is nested paging is active. */
3347	pVM->pgm.s.fNestedPaging = HMIsNestedPagingActive(pVM);
3348	if (pVM->pgm.s.fNestedPaging)
3349	enmShadowMode = HMGetShwPagingMode(pVM);
3350
3351	*penmSwitcher = enmSwitcher;
3352	return enmShadowMode;
3353	}
3354
3355
3356	/**
3357	* Performs the actual mode change.
3358	* This is called by PGMChangeMode and pgmR3InitPaging().
3359	*
3360	* @returns VBox status code. May suspend or power off the VM on error, but this
3361	* will trigger using FFs and not status codes.
3362	*
3363	* @param pVM Pointer to the VM.
3364	* @param pVCpu Pointer to the VMCPU.
3365	* @param enmGuestMode The new guest mode. This is assumed to be different from
3366	* the current mode.
3367	*/
3368	VMMR3DECL(int) PGMR3ChangeMode(PVM pVM, PVMCPU pVCpu, PGMMODE enmGuestMode)
3369	{
3370	#if HC_ARCH_BITS == 32
3371	bool fIsOldGuestPagingMode64Bits = (pVCpu->pgm.s.enmGuestMode >= PGMMODE_AMD64);
3372	#endif
3373	bool fIsNewGuestPagingMode64Bits = (enmGuestMode >= PGMMODE_AMD64);
3374
3375	Log(("PGMR3ChangeMode: Guest mode: %s -> %s\n", PGMGetModeName(pVCpu->pgm.s.enmGuestMode), PGMGetModeName(enmGuestMode)));
3376	STAM_REL_COUNTER_INC(&pVCpu->pgm.s.cGuestModeChanges);
3377
3378	/*
3379	* Calc the shadow mode and switcher.
3380	*/
3381	VMMSWITCHER enmSwitcher;
3382	PGMMODE enmShadowMode;
3383	enmShadowMode = pgmR3CalcShadowMode(pVM, enmGuestMode, pVM->pgm.s.enmHostMode, pVCpu->pgm.s.enmShadowMode, &enmSwitcher);
3384
3385	#ifdef VBOX_WITH_RAW_MODE
3386	if ( enmSwitcher != VMMSWITCHER_INVALID
3387	&& !HMIsEnabled(pVM))
3388	{
3389	/*
3390	* Select new switcher.
3391	*/
3392	int rc = VMMR3SelectSwitcher(pVM, enmSwitcher);
3393	if (RT_FAILURE(rc))
3394	{
3395	AssertReleaseMsgFailed(("VMMR3SelectSwitcher(%d) -> %Rrc\n", enmSwitcher, rc));
3396	return rc;
3397	}
3398	}
3399	#endif
3400
3401	/*
3402	* Exit old mode(s).
3403	*/
3404	#if HC_ARCH_BITS == 32
3405	/* The nested shadow paging mode for AMD-V does change when running 64 bits guests on 32 bits hosts; typically PAE <-> AMD64 */
3406	const bool fForceShwEnterExit = ( fIsOldGuestPagingMode64Bits != fIsNewGuestPagingMode64Bits
3407	&& enmShadowMode == PGMMODE_NESTED);
3408	#else
3409	const bool fForceShwEnterExit = false;
3410	#endif
3411	/* shadow */
3412	if ( enmShadowMode != pVCpu->pgm.s.enmShadowMode
3413	\|\| fForceShwEnterExit)
3414	{
3415	LogFlow(("PGMR3ChangeMode: Shadow mode: %s -> %s\n", PGMGetModeName(pVCpu->pgm.s.enmShadowMode), PGMGetModeName(enmShadowMode)));
3416	if (PGM_SHW_PFN(Exit, pVCpu))
3417	{
3418	int rc = PGM_SHW_PFN(Exit, pVCpu)(pVCpu);
3419	if (RT_FAILURE(rc))
3420	{
3421	AssertMsgFailed(("Exit failed for shadow mode %d: %Rrc\n", pVCpu->pgm.s.enmShadowMode, rc));
3422	return rc;
3423	}
3424	}
3425
3426	}
3427	else
3428	LogFlow(("PGMR3ChangeMode: Shadow mode remains: %s\n", PGMGetModeName(pVCpu->pgm.s.enmShadowMode)));
3429
3430	/* guest */
3431	if (PGM_GST_PFN(Exit, pVCpu))
3432	{
3433	int rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
3434	if (RT_FAILURE(rc))
3435	{
3436	AssertMsgFailed(("Exit failed for guest mode %d: %Rrc\n", pVCpu->pgm.s.enmGuestMode, rc));
3437	return rc;
3438	}
3439	}
3440
3441	/*
3442	* Load new paging mode data.
3443	*/
3444	pgmR3ModeDataSwitch(pVM, pVCpu, enmShadowMode, enmGuestMode);
3445
3446	/*
3447	* Enter new shadow mode (if changed).
3448	*/
3449	if ( enmShadowMode != pVCpu->pgm.s.enmShadowMode
3450	\|\| fForceShwEnterExit)
3451	{
3452	int rc;
3453	pVCpu->pgm.s.enmShadowMode = enmShadowMode;
3454	switch (enmShadowMode)
3455	{
3456	case PGMMODE_32_BIT:
3457	rc = PGM_SHW_NAME_32BIT(Enter)(pVCpu, false);
3458	break;
3459	case PGMMODE_PAE:
3460	case PGMMODE_PAE_NX:
3461	rc = PGM_SHW_NAME_PAE(Enter)(pVCpu, false);
3462	break;
3463	case PGMMODE_AMD64:
3464	case PGMMODE_AMD64_NX:
3465	rc = PGM_SHW_NAME_AMD64(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3466	break;
3467	case PGMMODE_NESTED:
3468	rc = PGM_SHW_NAME_NESTED(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3469	break;
3470	case PGMMODE_EPT:
3471	rc = PGM_SHW_NAME_EPT(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3472	break;
3473	case PGMMODE_REAL:
3474	case PGMMODE_PROTECTED:
3475	default:
3476	AssertReleaseMsgFailed(("enmShadowMode=%d\n", enmShadowMode));
3477	return VERR_INTERNAL_ERROR;
3478	}
3479	if (RT_FAILURE(rc))
3480	{
3481	AssertReleaseMsgFailed(("Entering enmShadowMode=%d failed: %Rrc\n", enmShadowMode, rc));
3482	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
3483	return rc;
3484	}
3485	}
3486
3487	/*
3488	* Always flag the necessary updates
3489	*/
3490	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3491
3492	/*
3493	* Enter the new guest and shadow+guest modes.
3494	*/
3495	int rc = -1;
3496	int rc2 = -1;
3497	RTGCPHYS GCPhysCR3 = NIL_RTGCPHYS;
3498	pVCpu->pgm.s.enmGuestMode = enmGuestMode;
3499	switch (enmGuestMode)
3500	{
3501	case PGMMODE_REAL:
3502	rc = PGM_GST_NAME_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3503	switch (pVCpu->pgm.s.enmShadowMode)
3504	{
3505	case PGMMODE_32_BIT:
3506	rc2 = PGM_BTH_NAME_32BIT_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3507	break;
3508	case PGMMODE_PAE:
3509	case PGMMODE_PAE_NX:
3510	rc2 = PGM_BTH_NAME_PAE_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3511	break;
3512	case PGMMODE_NESTED:
3513	rc2 = PGM_BTH_NAME_NESTED_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3514	break;
3515	case PGMMODE_EPT:
3516	rc2 = PGM_BTH_NAME_EPT_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3517	break;
3518	case PGMMODE_AMD64:
3519	case PGMMODE_AMD64_NX:
3520	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3521	default: AssertFailed(); break;
3522	}
3523	break;
3524
3525	case PGMMODE_PROTECTED:
3526	rc = PGM_GST_NAME_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3527	switch (pVCpu->pgm.s.enmShadowMode)
3528	{
3529	case PGMMODE_32_BIT:
3530	rc2 = PGM_BTH_NAME_32BIT_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3531	break;
3532	case PGMMODE_PAE:
3533	case PGMMODE_PAE_NX:
3534	rc2 = PGM_BTH_NAME_PAE_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3535	break;
3536	case PGMMODE_NESTED:
3537	rc2 = PGM_BTH_NAME_NESTED_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3538	break;
3539	case PGMMODE_EPT:
3540	rc2 = PGM_BTH_NAME_EPT_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3541	break;
3542	case PGMMODE_AMD64:
3543	case PGMMODE_AMD64_NX:
3544	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3545	default: AssertFailed(); break;
3546	}
3547	break;
3548
3549	case PGMMODE_32_BIT:
3550	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & X86_CR3_PAGE_MASK;
3551	rc = PGM_GST_NAME_32BIT(Enter)(pVCpu, GCPhysCR3);
3552	switch (pVCpu->pgm.s.enmShadowMode)
3553	{
3554	case PGMMODE_32_BIT:
3555	rc2 = PGM_BTH_NAME_32BIT_32BIT(Enter)(pVCpu, GCPhysCR3);
3556	break;
3557	case PGMMODE_PAE:
3558	case PGMMODE_PAE_NX:
3559	rc2 = PGM_BTH_NAME_PAE_32BIT(Enter)(pVCpu, GCPhysCR3);
3560	break;
3561	case PGMMODE_NESTED:
3562	rc2 = PGM_BTH_NAME_NESTED_32BIT(Enter)(pVCpu, GCPhysCR3);
3563	break;
3564	case PGMMODE_EPT:
3565	rc2 = PGM_BTH_NAME_EPT_32BIT(Enter)(pVCpu, GCPhysCR3);
3566	break;
3567	case PGMMODE_AMD64:
3568	case PGMMODE_AMD64_NX:
3569	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3570	default: AssertFailed(); break;
3571	}
3572	break;
3573
3574	case PGMMODE_PAE_NX:
3575	case PGMMODE_PAE:
3576	{
3577	uint32_t u32Dummy, u32Features;
3578
3579	CPUMGetGuestCpuId(pVCpu, 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
3580	if (!(u32Features & X86_CPUID_FEATURE_EDX_PAE))
3581	return VMSetRuntimeError(pVM, VMSETRTERR_FLAGS_FATAL, "PAEmode",
3582	N_("The guest is trying to switch to the PAE mode which is currently disabled by default in VirtualBox. PAE support can be enabled using the VM settings (General/Advanced)"));
3583
3584	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & X86_CR3_PAE_PAGE_MASK;
3585	rc = PGM_GST_NAME_PAE(Enter)(pVCpu, GCPhysCR3);
3586	switch (pVCpu->pgm.s.enmShadowMode)
3587	{
3588	case PGMMODE_PAE:
3589	case PGMMODE_PAE_NX:
3590	rc2 = PGM_BTH_NAME_PAE_PAE(Enter)(pVCpu, GCPhysCR3);
3591	break;
3592	case PGMMODE_NESTED:
3593	rc2 = PGM_BTH_NAME_NESTED_PAE(Enter)(pVCpu, GCPhysCR3);
3594	break;
3595	case PGMMODE_EPT:
3596	rc2 = PGM_BTH_NAME_EPT_PAE(Enter)(pVCpu, GCPhysCR3);
3597	break;
3598	case PGMMODE_32_BIT:
3599	case PGMMODE_AMD64:
3600	case PGMMODE_AMD64_NX:
3601	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3602	default: AssertFailed(); break;
3603	}
3604	break;
3605	}
3606
3607	#ifdef VBOX_WITH_64_BITS_GUESTS
3608	case PGMMODE_AMD64_NX:
3609	case PGMMODE_AMD64:
3610	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & UINT64_C(0xfffffffffffff000); /** @todo define this mask! */
3611	rc = PGM_GST_NAME_AMD64(Enter)(pVCpu, GCPhysCR3);
3612	switch (pVCpu->pgm.s.enmShadowMode)
3613	{
3614	case PGMMODE_AMD64:
3615	case PGMMODE_AMD64_NX:
3616	rc2 = PGM_BTH_NAME_AMD64_AMD64(Enter)(pVCpu, GCPhysCR3);
3617	break;
3618	case PGMMODE_NESTED:
3619	rc2 = PGM_BTH_NAME_NESTED_AMD64(Enter)(pVCpu, GCPhysCR3);
3620	break;
3621	case PGMMODE_EPT:
3622	rc2 = PGM_BTH_NAME_EPT_AMD64(Enter)(pVCpu, GCPhysCR3);
3623	break;
3624	case PGMMODE_32_BIT:
3625	case PGMMODE_PAE:
3626	case PGMMODE_PAE_NX:
3627	AssertMsgFailed(("Should use AMD64 shadow mode!\n"));
3628	default: AssertFailed(); break;
3629	}
3630	break;
3631	#endif
3632
3633	default:
3634	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
3635	rc = VERR_NOT_IMPLEMENTED;
3636	break;
3637	}
3638
3639	/* status codes. */
3640	AssertRC(rc);
3641	AssertRC(rc2);
3642	if (RT_SUCCESS(rc))
3643	{
3644	rc = rc2;
3645	if (RT_SUCCESS(rc)) /* no informational status codes. */
3646	rc = VINF_SUCCESS;
3647	}
3648
3649	/* Notify HM as well. */
3650	HMR3PagingModeChanged(pVM, pVCpu, pVCpu->pgm.s.enmShadowMode, pVCpu->pgm.s.enmGuestMode);
3651	return rc;
3652	}
3653
3654
3655	/**
3656	* Called by pgmPoolFlushAllInt prior to flushing the pool.
3657	*
3658	* @returns VBox status code, fully asserted.
3659	* @param pVCpu Pointer to the VMCPU.
3660	*/
3661	int pgmR3ExitShadowModeBeforePoolFlush(PVMCPU pVCpu)
3662	{
3663	/* Unmap the old CR3 value before flushing everything. */
3664	int rc = PGM_BTH_PFN(UnmapCR3, pVCpu)(pVCpu);
3665	AssertRC(rc);
3666
3667	/* Exit the current shadow paging mode as well; nested paging and EPT use a root CR3 which will get flushed here. */
3668	rc = PGM_SHW_PFN(Exit, pVCpu)(pVCpu);
3669	AssertRC(rc);
3670	Assert(pVCpu->pgm.s.pShwPageCR3R3 == NULL);
3671	return rc;
3672	}
3673
3674
3675	/**
3676	* Called by pgmPoolFlushAllInt after flushing the pool.
3677	*
3678	* @returns VBox status code, fully asserted.
3679	* @param pVM Pointer to the VM.
3680	* @param pVCpu Pointer to the VMCPU.
3681	*/
3682	int pgmR3ReEnterShadowModeAfterPoolFlush(PVM pVM, PVMCPU pVCpu)
3683	{
3684	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
3685	int rc = PGMR3ChangeMode(pVM, pVCpu, PGMGetGuestMode(pVCpu));
3686	Assert(VMCPU_FF_IS_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3));
3687	AssertRCReturn(rc, rc);
3688	AssertRCSuccessReturn(rc, VERR_IPE_UNEXPECTED_INFO_STATUS);
3689
3690	Assert(pVCpu->pgm.s.pShwPageCR3R3 != NULL);
3691	AssertMsg( pVCpu->pgm.s.enmShadowMode >= PGMMODE_NESTED
3692	\|\| CPUMGetHyperCR3(pVCpu) == PGMGetHyperCR3(pVCpu),
3693	("%RHp != %RHp %s\n", (RTHCPHYS)CPUMGetHyperCR3(pVCpu), PGMGetHyperCR3(pVCpu), PGMGetModeName(pVCpu->pgm.s.enmShadowMode)));
3694	return rc;
3695	}
3696
3697
3698	/**
3699	* Called by PGMR3PhysSetA20 after changing the A20 state.
3700	*
3701	* @param pVCpu Pointer to the VMCPU.
3702	*/
3703	void pgmR3RefreshShadowModeAfterA20Change(PVMCPU pVCpu)
3704	{
3705	/** @todo Probably doing a bit too much here. */
3706	int rc = pgmR3ExitShadowModeBeforePoolFlush(pVCpu);
3707	AssertReleaseRC(rc);
3708	rc = pgmR3ReEnterShadowModeAfterPoolFlush(pVCpu->CTX_SUFF(pVM), pVCpu);
3709	AssertReleaseRC(rc);
3710	}
3711
3712
3713	#ifdef VBOX_WITH_DEBUGGER
3714
3715	/**
3716	* @callback_method_impl{FNDBGCCMD, The '.pgmerror' and '.pgmerroroff' commands.}
3717	*/
3718	static DECLCALLBACK(int) pgmR3CmdError(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PUVM pUVM, PCDBGCVAR paArgs, unsigned cArgs)
3719	{
3720	/*
3721	* Validate input.
3722	*/
3723	DBGC_CMDHLP_REQ_UVM_RET(pCmdHlp, pCmd, pUVM);
3724	PVM pVM = pUVM->pVM;
3725	DBGC_CMDHLP_ASSERT_PARSER_RET(pCmdHlp, pCmd, 0, cArgs == 0 \|\| (cArgs == 1 && paArgs[0].enmType == DBGCVAR_TYPE_STRING));
3726
3727	if (!cArgs)
3728	{
3729	/*
3730	* Print the list of error injection locations with status.
3731	*/
3732	DBGCCmdHlpPrintf(pCmdHlp, "PGM error inject locations:\n");
3733	DBGCCmdHlpPrintf(pCmdHlp, " handy - %RTbool\n", pVM->pgm.s.fErrInjHandyPages);
3734	}
3735	else
3736	{
3737	/*
3738	* String switch on where to inject the error.
3739	*/
3740	bool const fNewState = !strcmp(pCmd->pszCmd, "pgmerror");
3741	const char *pszWhere = paArgs[0].u.pszString;
3742	if (!strcmp(pszWhere, "handy"))
3743	ASMAtomicWriteBool(&pVM->pgm.s.fErrInjHandyPages, fNewState);
3744	else
3745	return DBGCCmdHlpPrintf(pCmdHlp, "error: Invalid 'where' value: %s.\n", pszWhere);
3746	DBGCCmdHlpPrintf(pCmdHlp, "done\n");
3747	}
3748	return VINF_SUCCESS;
3749	}
3750
3751
3752	/**
3753	* @callback_method_impl{FNDBGCCMD, The '.pgmsync' command.}
3754	*/
3755	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PUVM pUVM, PCDBGCVAR paArgs, unsigned cArgs)
3756	{
3757	/*
3758	* Validate input.
3759	*/
3760	NOREF(pCmd); NOREF(paArgs); NOREF(cArgs);
3761	DBGC_CMDHLP_REQ_UVM_RET(pCmdHlp, pCmd, pUVM);
3762	PVMCPU pVCpu = VMMR3GetCpuByIdU(pUVM, DBGCCmdHlpGetCurrentCpu(pCmdHlp));
3763	if (!pVCpu)
3764	return DBGCCmdHlpFail(pCmdHlp, pCmd, "Invalid CPU ID");
3765
3766	/*
3767	* Force page directory sync.
3768	*/
3769	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3770
3771	int rc = DBGCCmdHlpPrintf(pCmdHlp, "Forcing page directory sync.\n");
3772	if (RT_FAILURE(rc))
3773	return rc;
3774
3775	return VINF_SUCCESS;
3776	}
3777
3778	#ifdef VBOX_STRICT
3779
3780	/**
3781	* EMT callback for pgmR3CmdAssertCR3.
3782	*
3783	* @returns VBox status code.
3784	* @param pUVM The user mode VM handle.
3785	* @param pcErrors Where to return the error count.
3786	*/
3787	static DECLCALLBACK(int) pgmR3CmdAssertCR3EmtWorker(PUVM pUVM, unsigned *pcErrors)
3788	{
3789	PVM pVM = pUVM->pVM;
3790	VM_ASSERT_VALID_EXT_RETURN(pVM, VERR_INVALID_VM_HANDLE);
3791	PVMCPU pVCpu = VMMGetCpu(pVM);
3792
3793	*pcErrors = PGMAssertCR3(pVM, pVCpu, CPUMGetGuestCR3(pVCpu), CPUMGetGuestCR4(pVCpu));
3794
3795	return VINF_SUCCESS;
3796	}
3797
3798
3799	/**
3800	* @callback_method_impl{FNDBGCCMD, The '.pgmassertcr3' command.}
3801	*/
3802	static DECLCALLBACK(int) pgmR3CmdAssertCR3(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PUVM pUVM, PCDBGCVAR paArgs, unsigned cArgs)
3803	{
3804	/*
3805	* Validate input.
3806	*/
3807	NOREF(pCmd); NOREF(paArgs); NOREF(cArgs);
3808	DBGC_CMDHLP_REQ_UVM_RET(pCmdHlp, pCmd, pUVM);
3809
3810	int rc = DBGCCmdHlpPrintf(pCmdHlp, "Checking shadow CR3 page tables for consistency.\n");
3811	if (RT_FAILURE(rc))
3812	return rc;
3813
3814	unsigned cErrors = 0;
3815	rc = VMR3ReqCallWaitU(pUVM, DBGCCmdHlpGetCurrentCpu(pCmdHlp), (PFNRT)pgmR3CmdAssertCR3EmtWorker, 2, pUVM, &cErrors);
3816	if (RT_FAILURE(rc))
3817	return DBGCCmdHlpFail(pCmdHlp, pCmd, "VMR3ReqCallWaitU failed: %Rrc", rc);
3818	if (cErrors > 0)
3819	return DBGCCmdHlpFail(pCmdHlp, pCmd, "PGMAssertCR3: %u error(s)", cErrors);
3820	return DBGCCmdHlpPrintf(pCmdHlp, "PGMAssertCR3: OK\n");
3821	}
3822
3823	#endif /* VBOX_STRICT */
3824
3825	/**
3826	* @callback_method_impl{FNDBGCCMD, The '.pgmsyncalways' command.}
3827	*/
3828	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PUVM pUVM, PCDBGCVAR paArgs, unsigned cArgs)
3829	{
3830	/*
3831	* Validate input.
3832	*/
3833	NOREF(pCmd); NOREF(paArgs); NOREF(cArgs);
3834	DBGC_CMDHLP_REQ_UVM_RET(pCmdHlp, pCmd, pUVM);
3835	PVMCPU pVCpu = VMMR3GetCpuByIdU(pUVM, DBGCCmdHlpGetCurrentCpu(pCmdHlp));
3836	if (!pVCpu)
3837	return DBGCCmdHlpFail(pCmdHlp, pCmd, "Invalid CPU ID");
3838
3839	/*
3840	* Force page directory sync.
3841	*/
3842	int rc;
3843	if (pVCpu->pgm.s.fSyncFlags & PGM_SYNC_ALWAYS)
3844	{
3845	ASMAtomicAndU32(&pVCpu->pgm.s.fSyncFlags, ~PGM_SYNC_ALWAYS);
3846	rc = DBGCCmdHlpPrintf(pCmdHlp, "Disabled permanent forced page directory syncing.\n");
3847	}
3848	else
3849	{
3850	ASMAtomicOrU32(&pVCpu->pgm.s.fSyncFlags, PGM_SYNC_ALWAYS);
3851	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3852	rc = DBGCCmdHlpPrintf(pCmdHlp, "Enabled permanent forced page directory syncing.\n");
3853	}
3854	return rc;
3855	}
3856
3857
3858	/**
3859	* @callback_method_impl{FNDBGCCMD, The '.pgmphystofile' command.}
3860	*/
3861	static DECLCALLBACK(int) pgmR3CmdPhysToFile(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PUVM pUVM, PCDBGCVAR paArgs, unsigned cArgs)
3862	{
3863	/*
3864	* Validate input.
3865	*/
3866	NOREF(pCmd);
3867	DBGC_CMDHLP_REQ_UVM_RET(pCmdHlp, pCmd, pUVM);
3868	PVM pVM = pUVM->pVM;
3869	DBGC_CMDHLP_ASSERT_PARSER_RET(pCmdHlp, pCmd, 0, cArgs == 1 \|\| cArgs == 2);
3870	DBGC_CMDHLP_ASSERT_PARSER_RET(pCmdHlp, pCmd, 0, paArgs[0].enmType != DBGCVAR_TYPE_STRING);
3871	if (cArgs == 2)
3872	{
3873	DBGC_CMDHLP_ASSERT_PARSER_RET(pCmdHlp, pCmd, 1, paArgs[2].enmType != DBGCVAR_TYPE_STRING);
3874	if (strcmp(paArgs[1].u.pszString, "nozero"))
3875	return DBGCCmdHlpFail(pCmdHlp, pCmd, "Invalid 2nd argument '%s', must be 'nozero'.\n", paArgs[1].u.pszString);
3876	}
3877	bool fIncZeroPgs = cArgs < 2;
3878
3879	/*
3880	* Open the output file and get the ram parameters.
3881	*/
3882	RTFILE hFile;
3883	int rc = RTFileOpen(&hFile, paArgs[0].u.pszString, RTFILE_O_WRITE \| RTFILE_O_CREATE_REPLACE \| RTFILE_O_DENY_WRITE);
3884	if (RT_FAILURE(rc))
3885	return DBGCCmdHlpPrintf(pCmdHlp, "error: RTFileOpen(,'%s',) -> %Rrc.\n", paArgs[0].u.pszString, rc);
3886
3887	uint32_t cbRamHole = 0;
3888	CFGMR3QueryU32Def(CFGMR3GetRootU(pUVM), "RamHoleSize", &cbRamHole, MM_RAM_HOLE_SIZE_DEFAULT);
3889	uint64_t cbRam = 0;
3890	CFGMR3QueryU64Def(CFGMR3GetRootU(pUVM), "RamSize", &cbRam, 0);
3891	RTGCPHYS GCPhysEnd = cbRam + cbRamHole;
3892
3893	/*
3894	* Dump the physical memory, page by page.
3895	*/
3896	RTGCPHYS GCPhys = 0;
3897	char abZeroPg[PAGE_SIZE];
3898	RT_ZERO(abZeroPg);
3899
3900	pgmLock(pVM);
3901	for (PPGMRAMRANGE pRam = pVM->pgm.s.pRamRangesXR3;
3902	pRam && pRam->GCPhys < GCPhysEnd && RT_SUCCESS(rc);
3903	pRam = pRam->pNextR3)
3904	{
3905	/* fill the gap */
3906	if (pRam->GCPhys > GCPhys && fIncZeroPgs)
3907	{
3908	while (pRam->GCPhys > GCPhys && RT_SUCCESS(rc))
3909	{
3910	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
3911	GCPhys += PAGE_SIZE;
3912	}
3913	}
3914
3915	PCPGMPAGE pPage = &pRam->aPages[0];
3916	while (GCPhys < pRam->GCPhysLast && RT_SUCCESS(rc))
3917	{
3918	if ( PGM_PAGE_IS_ZERO(pPage)
3919	\|\| PGM_PAGE_IS_BALLOONED(pPage))
3920	{
3921	if (fIncZeroPgs)
3922	{
3923	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
3924	if (RT_FAILURE(rc))
3925	DBGCCmdHlpPrintf(pCmdHlp, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3926	}
3927	}
3928	else
3929	{
3930	switch (PGM_PAGE_GET_TYPE(pPage))
3931	{
3932	case PGMPAGETYPE_RAM:
3933	case PGMPAGETYPE_ROM_SHADOW: /* trouble?? */
3934	case PGMPAGETYPE_ROM:
3935	case PGMPAGETYPE_MMIO2:
3936	{
3937	void const *pvPage;
3938	PGMPAGEMAPLOCK Lock;
3939	rc = PGMPhysGCPhys2CCPtrReadOnly(pVM, GCPhys, &pvPage, &Lock);
3940	if (RT_SUCCESS(rc))
3941	{
3942	rc = RTFileWrite(hFile, pvPage, PAGE_SIZE, NULL);
3943	PGMPhysReleasePageMappingLock(pVM, &Lock);
3944	if (RT_FAILURE(rc))
3945	DBGCCmdHlpPrintf(pCmdHlp, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3946	}
3947	else
3948	DBGCCmdHlpPrintf(pCmdHlp, "error: PGMPhysGCPhys2CCPtrReadOnly -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3949	break;
3950	}
3951
3952	default:
3953	AssertFailed();
3954	case PGMPAGETYPE_MMIO:
3955	case PGMPAGETYPE_MMIO2_ALIAS_MMIO:
3956	case PGMPAGETYPE_SPECIAL_ALIAS_MMIO:
3957	if (fIncZeroPgs)
3958	{
3959	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
3960	if (RT_FAILURE(rc))
3961	DBGCCmdHlpPrintf(pCmdHlp, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3962	}
3963	break;
3964	}
3965	}
3966
3967
3968	/* advance */
3969	GCPhys += PAGE_SIZE;
3970	pPage++;
3971	}
3972	}
3973	pgmUnlock(pVM);
3974
3975	RTFileClose(hFile);
3976	if (RT_SUCCESS(rc))
3977	return DBGCCmdHlpPrintf(pCmdHlp, "Successfully saved physical memory to '%s'.\n", paArgs[0].u.pszString);
3978	return VINF_SUCCESS;
3979	}
3980
3981	#endif /* VBOX_WITH_DEBUGGER */
3982
3983	/**
3984	* pvUser argument of the pgmR3CheckIntegrity*Node callbacks.
3985	*/
3986	typedef struct PGMCHECKINTARGS
3987	{
3988	bool fLeftToRight; /*< true: left-to-right; false: right-to-left. /
3989	PPGMPHYSHANDLER pPrevPhys;
3990	PPGMVIRTHANDLER pPrevVirt;
3991	PPGMPHYS2VIRTHANDLER pPrevPhys2Virt;
3992	PVM pVM;
3993	} PGMCHECKINTARGS, *PPGMCHECKINTARGS;
3994
3995	/**
3996	* Validate a node in the physical handler tree.
3997	*
3998	* @returns 0 on if ok, other wise 1.
3999	* @param pNode The handler node.
4000	* @param pvUser pVM.
4001	*/
4002	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
4003	{
4004	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
4005	PPGMPHYSHANDLER pCur = (PPGMPHYSHANDLER)pNode;
4006	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
4007	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGp-%RGp %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
4008	AssertReleaseMsg( !pArgs->pPrevPhys
4009	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys->Core.KeyLast > pCur->Core.Key),
4010	("pPrevPhys=%p %RGp-%RGp %s\n"
4011	" pCur=%p %RGp-%RGp %s\n",
4012	pArgs->pPrevPhys, pArgs->pPrevPhys->Core.Key, pArgs->pPrevPhys->Core.KeyLast, pArgs->pPrevPhys->pszDesc,
4013	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
4014	pArgs->pPrevPhys = pCur;
4015	return 0;
4016	}
4017
4018
4019	/**
4020	* Validate a node in the virtual handler tree.
4021	*
4022	* @returns 0 on if ok, other wise 1.
4023	* @param pNode The handler node.
4024	* @param pvUser pVM.
4025	*/
4026	static DECLCALLBACK(int) pgmR3CheckIntegrityVirtHandlerNode(PAVLROGCPTRNODECORE pNode, void *pvUser)
4027	{
4028	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
4029	PPGMVIRTHANDLER pCur = (PPGMVIRTHANDLER)pNode;
4030	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
4031	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGv-%RGv %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
4032	AssertReleaseMsg( !pArgs->pPrevVirt
4033	\|\| (pArgs->fLeftToRight ? pArgs->pPrevVirt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevVirt->Core.KeyLast > pCur->Core.Key),
4034	("pPrevVirt=%p %RGv-%RGv %s\n"
4035	" pCur=%p %RGv-%RGv %s\n",
4036	pArgs->pPrevVirt, pArgs->pPrevVirt->Core.Key, pArgs->pPrevVirt->Core.KeyLast, pArgs->pPrevVirt->pszDesc,
4037	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
4038	for (unsigned iPage = 0; iPage < pCur->cPages; iPage++)
4039	{
4040	AssertReleaseMsg(pCur->aPhysToVirt[iPage].offVirtHandler == -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage]),
4041	("pCur=%p %RGv-%RGv %s\n"
4042	"iPage=%d offVirtHandle=%#x expected %#x\n",
4043	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc,
4044	iPage, pCur->aPhysToVirt[iPage].offVirtHandler, -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage])));
4045	}
4046	pArgs->pPrevVirt = pCur;
4047	return 0;
4048	}
4049
4050
4051	/**
4052	* Validate a node in the virtual handler tree.
4053	*
4054	* @returns 0 on if ok, other wise 1.
4055	* @param pNode The handler node.
4056	* @param pvUser pVM.
4057	*/
4058	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysToVirtHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
4059	{
4060	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
4061	PPGMPHYS2VIRTHANDLER pCur = (PPGMPHYS2VIRTHANDLER)pNode;
4062	AssertReleaseMsgReturn(!((uintptr_t)pCur & 3), ("\n"), 1);
4063	AssertReleaseMsgReturn(!(pCur->offVirtHandler & 3), ("\n"), 1);
4064	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGp-%RGp\n", pCur, pCur->Core.Key, pCur->Core.KeyLast));
4065	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
4066	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
4067	("pPrevPhys2Virt=%p %RGp-%RGp\n"
4068	" pCur=%p %RGp-%RGp\n",
4069	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
4070	pCur, pCur->Core.Key, pCur->Core.KeyLast));
4071	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
4072	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
4073	("pPrevPhys2Virt=%p %RGp-%RGp\n"
4074	" pCur=%p %RGp-%RGp\n",
4075	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
4076	pCur, pCur->Core.Key, pCur->Core.KeyLast));
4077	AssertReleaseMsg((pCur->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD),
4078	("pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4079	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
4080	if (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK)
4081	{
4082	PPGMPHYS2VIRTHANDLER pCur2 = pCur;
4083	for (;;)
4084	{
4085	pCur2 = (PPGMPHYS2VIRTHANDLER)((intptr_t)pCur + (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK));
4086	AssertReleaseMsg(pCur2 != pCur,
4087	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4088	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
4089	AssertReleaseMsg((pCur2->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == PGMPHYS2VIRTHANDLER_IN_TREE,
4090	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4091	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4092	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4093	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4094	AssertReleaseMsg((pCur2->Core.Key ^ pCur->Core.Key) < PAGE_SIZE,
4095	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4096	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4097	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4098	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4099	AssertReleaseMsg((pCur2->Core.KeyLast ^ pCur->Core.KeyLast) < PAGE_SIZE,
4100	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4101	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4102	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4103	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4104	if (!(pCur2->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK))
4105	break;
4106	}
4107	}
4108
4109	pArgs->pPrevPhys2Virt = pCur;
4110	return 0;
4111	}
4112
4113
4114	/**
4115	* Perform an integrity check on the PGM component.
4116	*
4117	* @returns VINF_SUCCESS if everything is fine.
4118	* @returns VBox error status after asserting on integrity breach.
4119	* @param pVM Pointer to the VM.
4120	*/
4121	VMMR3DECL(int) PGMR3CheckIntegrity(PVM pVM)
4122	{
4123	AssertReleaseReturn(pVM->pgm.s.offVM, VERR_INTERNAL_ERROR);
4124
4125	/*
4126	* Check the trees.
4127	*/
4128	int cErrors = 0;
4129	const static PGMCHECKINTARGS s_LeftToRight = { true, NULL, NULL, NULL, pVM };
4130	const static PGMCHECKINTARGS s_RightToLeft = { false, NULL, NULL, NULL, pVM };
4131	PGMCHECKINTARGS Args = s_LeftToRight;
4132	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, true, pgmR3CheckIntegrityPhysHandlerNode, &Args);
4133	Args = s_RightToLeft;
4134	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, false, pgmR3CheckIntegrityPhysHandlerNode, &Args);
4135	Args = s_LeftToRight;
4136	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->VirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4137	Args = s_RightToLeft;
4138	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->VirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4139	Args = s_LeftToRight;
4140	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->HyperVirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4141	Args = s_RightToLeft;
4142	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->HyperVirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4143	Args = s_LeftToRight;
4144	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysToVirtHandlers, true, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
4145	Args = s_RightToLeft;
4146	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysToVirtHandlers, false, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
4147
4148	return !cErrors ? VINF_SUCCESS : VERR_INTERNAL_ERROR;
4149	}
4150

Note: See TracBrowser for help on using the repository browser.

source: vbox/trunk/src/VBox/VMM/VMMR3/PGM.cpp@ 48568

Download in other formats: