PGM.cpp@ 39025

Last change on this file since 39025 was 38953, checked in by vboxsync, 14 years ago
PGM: Attempt at fixing the VERR_MAP_FAILED during state save problem on 32-bit hosts when assigning lots of memory to the guest. PGM should lock down guest RAM pages before use and release them afterwards like everyone else. Still quite some stuff left to do there, so I've deviced a little hack for tracking unlocked mappings and using this as input when deciding to do async or sync chunk unmapping at save/load time. See xtracker #5912 and public ticket 7929.
Property svn:eol-style set to `native` Property svn:keywords set to `Id`
File size: 201.6 KB

Line
1	/* $Id: PGM.cpp 38953 2011-10-06 08:49:36Z vboxsync $ */
2	/** @file
3	* PGM - Page Manager and Monitor. (Mixing stuff here, not good?)
4	*/
5
6	/*
7	* Copyright (C) 2006-2011 Oracle Corporation
8	*
9	* This file is part of VirtualBox Open Source Edition (OSE), as
10	* available from http://www.virtualbox.org. This file is free software;
11	* you can redistribute it and/or modify it under the terms of the GNU
12	* General Public License (GPL) as published by the Free Software
13	* Foundation, in version 2 as it comes in the "COPYING" file of the
14	* VirtualBox OSE distribution. VirtualBox OSE is distributed in the
15	* hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
16	*/
17
18
19	/** @page pg_pgm PGM - The Page Manager and Monitor
20	*
21	* @see grp_pgm,
22	* @ref pg_pgm_pool,
23	* @ref pg_pgm_phys.
24	*
25	*
26	* @section sec_pgm_modes Paging Modes
27	*
28	* There are three memory contexts: Host Context (HC), Guest Context (GC)
29	* and intermediate context. When talking about paging HC can also be referred to
30	* as "host paging", and GC referred to as "shadow paging".
31	*
32	* We define three basic paging modes: 32-bit, PAE and AMD64. The host paging mode
33	* is defined by the host operating system. The mode used in the shadow paging mode
34	* depends on the host paging mode and what the mode the guest is currently in. The
35	* following relation between the two is defined:
36	*
37	* @verbatim
38	Host > 32-bit \| PAE \| AMD64 \|
39	Guest \| \| \| \|
40	==v================================
41	32-bit 32-bit PAE PAE
42	-------\|--------\|--------\|--------\|
43	PAE PAE PAE PAE
44	-------\|--------\|--------\|--------\|
45	AMD64 AMD64 AMD64 AMD64
46	-------\|--------\|--------\|--------\| @endverbatim
47	*
48	* All configuration except those in the diagonal (upper left) are expected to
49	* require special effort from the switcher (i.e. a bit slower).
50	*
51	*
52	*
53	*
54	* @section sec_pgm_shw The Shadow Memory Context
55	*
56	*
57	* [..]
58	*
59	* Because of guest context mappings requires PDPT and PML4 entries to allow
60	* writing on AMD64, the two upper levels will have fixed flags whatever the
61	* guest is thinking of using there. So, when shadowing the PD level we will
62	* calculate the effective flags of PD and all the higher levels. In legacy
63	* PAE mode this only applies to the PWT and PCD bits (the rest are
64	* ignored/reserved/MBZ). We will ignore those bits for the present.
65	*
66	*
67	*
68	* @section sec_pgm_int The Intermediate Memory Context
69	*
70	* The world switch goes thru an intermediate memory context which purpose it is
71	* to provide different mappings of the switcher code. All guest mappings are also
72	* present in this context.
73	*
74	* The switcher code is mapped at the same location as on the host, at an
75	* identity mapped location (physical equals virtual address), and at the
76	* hypervisor location. The identity mapped location is for when the world
77	* switches that involves disabling paging.
78	*
79	* PGM maintain page tables for 32-bit, PAE and AMD64 paging modes. This
80	* simplifies switching guest CPU mode and consistency at the cost of more
81	* code to do the work. All memory use for those page tables is located below
82	* 4GB (this includes page tables for guest context mappings).
83	*
84	*
85	* @subsection subsec_pgm_int_gc Guest Context Mappings
86	*
87	* During assignment and relocation of a guest context mapping the intermediate
88	* memory context is used to verify the new location.
89	*
90	* Guest context mappings are currently restricted to below 4GB, for reasons
91	* of simplicity. This may change when we implement AMD64 support.
92	*
93	*
94	*
95	*
96	* @section sec_pgm_misc Misc
97	*
98	* @subsection subsec_pgm_misc_diff Differences Between Legacy PAE and Long Mode PAE
99	*
100	* The differences between legacy PAE and long mode PAE are:
101	* -# PDPE bits 1, 2, 5 and 6 are defined differently. In leagcy mode they are
102	* all marked down as must-be-zero, while in long mode 1, 2 and 5 have the
103	* usual meanings while 6 is ignored (AMD). This means that upon switching to
104	* legacy PAE mode we'll have to clear these bits and when going to long mode
105	* they must be set. This applies to both intermediate and shadow contexts,
106	* however we don't need to do it for the intermediate one since we're
107	* executing with CR0.WP at that time.
108	* -# CR3 allows a 32-byte aligned address in legacy mode, while in long mode
109	* a page aligned one is required.
110	*
111	*
112	* @section sec_pgm_handlers Access Handlers
113	*
114	* Placeholder.
115	*
116	*
117	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
118	*
119	* Placeholder.
120	*
121	*
122	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
123	*
124	* We currently implement three types of virtual access handlers: ALL, WRITE
125	* and HYPERVISOR (WRITE). See PGMVIRTHANDLERTYPE for some more details.
126	*
127	* The HYPERVISOR access handlers is kept in a separate tree since it doesn't apply
128	* to physical pages (PGMTREES::HyperVirtHandlers) and only needs to be consulted in
129	* a special \#PF case. The ALL and WRITE are in the PGMTREES::VirtHandlers tree, the
130	* rest of this section is going to be about these handlers.
131	*
132	* We'll go thru the life cycle of a handler and try make sense of it all, don't know
133	* how successful this is gonna be...
134	*
135	* 1. A handler is registered thru the PGMR3HandlerVirtualRegister and
136	* PGMHandlerVirtualRegisterEx APIs. We check for conflicting virtual handlers
137	* and create a new node that is inserted into the AVL tree (range key). Then
138	* a full PGM resync is flagged (clear pool, sync cr3, update virtual bit of PGMPAGE).
139	*
140	* 2. The following PGMSyncCR3/SyncCR3 operation will first make invoke HandlerVirtualUpdate.
141	*
142	* 2a. HandlerVirtualUpdate will will lookup all the pages covered by virtual handlers
143	* via the current guest CR3 and update the physical page -> virtual handler
144	* translation. Needless to say, this doesn't exactly scale very well. If any changes
145	* are detected, it will flag a virtual bit update just like we did on registration.
146	* PGMPHYS pages with changes will have their virtual handler state reset to NONE.
147	*
148	* 2b. The virtual bit update process will iterate all the pages covered by all the
149	* virtual handlers and update the PGMPAGE virtual handler state to the max of all
150	* virtual handlers on that page.
151	*
152	* 2c. Back in SyncCR3 we will now flush the entire shadow page cache to make sure
153	* we don't miss any alias mappings of the monitored pages.
154	*
155	* 2d. SyncCR3 will then proceed with syncing the CR3 table.
156	*
157	* 3. \#PF(np,read) on a page in the range. This will cause it to be synced
158	* read-only and resumed if it's a WRITE handler. If it's an ALL handler we
159	* will call the handlers like in the next step. If the physical mapping has
160	* changed we will - some time in the future - perform a handler callback
161	* (optional) and update the physical -> virtual handler cache.
162	*
163	* 4. \#PF(,write) on a page in the range. This will cause the handler to
164	* be invoked.
165	*
166	* 5. The guest invalidates the page and changes the physical backing or
167	* unmaps it. This should cause the invalidation callback to be invoked
168	* (it might not yet be 100% perfect). Exactly what happens next... is
169	* this where we mess up and end up out of sync for a while?
170	*
171	* 6. The handler is deregistered by the client via PGMHandlerVirtualDeregister.
172	* We will then set all PGMPAGEs in the physical -> virtual handler cache for
173	* this handler to NONE and trigger a full PGM resync (basically the same
174	* as int step 1). Which means 2 is executed again.
175	*
176	*
177	* @subsubsection sub_sec_pgm_handler_virt_todo TODOs
178	*
179	* There is a bunch of things that needs to be done to make the virtual handlers
180	* work 100% correctly and work more efficiently.
181	*
182	* The first bit hasn't been implemented yet because it's going to slow the
183	* whole mess down even more, and besides it seems to be working reliably for
184	* our current uses. OTOH, some of the optimizations might end up more or less
185	* implementing the missing bits, so we'll see.
186	*
187	* On the optimization side, the first thing to do is to try avoid unnecessary
188	* cache flushing. Then try team up with the shadowing code to track changes
189	* in mappings by means of access to them (shadow in), updates to shadows pages,
190	* invlpg, and shadow PT discarding (perhaps).
191	*
192	* Some idea that have popped up for optimization for current and new features:
193	* - bitmap indicating where there are virtual handlers installed.
194	* (4KB => 220 pages, page 212 => covers 32-bit address space 1:1!)
195	* - Further optimize this by min/max (needs min/max avl getters).
196	* - Shadow page table entry bit (if any left)?
197	*
198	*/
199
200
201	/** @page pg_pgm_phys PGM Physical Guest Memory Management
202	*
203	*
204	* Objectives:
205	* - Guest RAM over-commitment using memory ballooning,
206	* zero pages and general page sharing.
207	* - Moving or mirroring a VM onto a different physical machine.
208	*
209	*
210	* @subsection subsec_pgmPhys_Definitions Definitions
211	*
212	* Allocation chunk - A RTR0MemObjAllocPhysNC object and the tracking
213	* machinery associated with it.
214	*
215	*
216	*
217	*
218	* @subsection subsec_pgmPhys_AllocPage Allocating a page.
219	*
220	* Initially we map all guest memory to the (per VM) zero page, which
221	* means that none of the read functions will cause pages to be allocated.
222	*
223	* Exception, access bit in page tables that have been shared. This must
224	* be handled, but we must also make sure PGMGst*Modify doesn't make
225	* unnecessary modifications.
226	*
227	* Allocation points:
228	* - PGMPhysSimpleWriteGCPhys and PGMPhysWrite.
229	* - Replacing a zero page mapping at \#PF.
230	* - Replacing a shared page mapping at \#PF.
231	* - ROM registration (currently MMR3RomRegister).
232	* - VM restore (pgmR3Load).
233	*
234	* For the first three it would make sense to keep a few pages handy
235	* until we've reached the max memory commitment for the VM.
236	*
237	* For the ROM registration, we know exactly how many pages we need
238	* and will request these from ring-0. For restore, we will save
239	* the number of non-zero pages in the saved state and allocate
240	* them up front. This would allow the ring-0 component to refuse
241	* the request if the isn't sufficient memory available for VM use.
242	*
243	* Btw. for both ROM and restore allocations we won't be requiring
244	* zeroed pages as they are going to be filled instantly.
245	*
246	*
247	* @subsection subsec_pgmPhys_FreePage Freeing a page
248	*
249	* There are a few points where a page can be freed:
250	* - After being replaced by the zero page.
251	* - After being replaced by a shared page.
252	* - After being ballooned by the guest additions.
253	* - At reset.
254	* - At restore.
255	*
256	* When freeing one or more pages they will be returned to the ring-0
257	* component and replaced by the zero page.
258	*
259	* The reasoning for clearing out all the pages on reset is that it will
260	* return us to the exact same state as on power on, and may thereby help
261	* us reduce the memory load on the system. Further it might have a
262	* (temporary) positive influence on memory fragmentation (@see subsec_pgmPhys_Fragmentation).
263	*
264	* On restore, as mention under the allocation topic, pages should be
265	* freed / allocated depending on how many is actually required by the
266	* new VM state. The simplest approach is to do like on reset, and free
267	* all non-ROM pages and then allocate what we need.
268	*
269	* A measure to prevent some fragmentation, would be to let each allocation
270	* chunk have some affinity towards the VM having allocated the most pages
271	* from it. Also, try make sure to allocate from allocation chunks that
272	* are almost full. Admittedly, both these measures might work counter to
273	* our intentions and its probably not worth putting a lot of effort,
274	* cpu time or memory into this.
275	*
276	*
277	* @subsection subsec_pgmPhys_SharePage Sharing a page
278	*
279	* The basic idea is that there there will be a idle priority kernel
280	* thread walking the non-shared VM pages hashing them and looking for
281	* pages with the same checksum. If such pages are found, it will compare
282	* them byte-by-byte to see if they actually are identical. If found to be
283	* identical it will allocate a shared page, copy the content, check that
284	* the page didn't change while doing this, and finally request both the
285	* VMs to use the shared page instead. If the page is all zeros (special
286	* checksum and byte-by-byte check) it will request the VM that owns it
287	* to replace it with the zero page.
288	*
289	* To make this efficient, we will have to make sure not to try share a page
290	* that will change its contents soon. This part requires the most work.
291	* A simple idea would be to request the VM to write monitor the page for
292	* a while to make sure it isn't modified any time soon. Also, it may
293	* make sense to skip pages that are being write monitored since this
294	* information is readily available to the thread if it works on the
295	* per-VM guest memory structures (presently called PGMRAMRANGE).
296	*
297	*
298	* @subsection subsec_pgmPhys_Fragmentation Fragmentation Concerns and Counter Measures
299	*
300	* The pages are organized in allocation chunks in ring-0, this is a necessity
301	* if we wish to have an OS agnostic approach to this whole thing. (On Linux we
302	* could easily work on a page-by-page basis if we liked. Whether this is possible
303	* or efficient on NT I don't quite know.) Fragmentation within these chunks may
304	* become a problem as part of the idea here is that we wish to return memory to
305	* the host system.
306	*
307	* For instance, starting two VMs at the same time, they will both allocate the
308	* guest memory on-demand and if permitted their page allocations will be
309	* intermixed. Shut down one of the two VMs and it will be difficult to return
310	* any memory to the host system because the page allocation for the two VMs are
311	* mixed up in the same allocation chunks.
312	*
313	* To further complicate matters, when pages are freed because they have been
314	* ballooned or become shared/zero the whole idea is that the page is supposed
315	* to be reused by another VM or returned to the host system. This will cause
316	* allocation chunks to contain pages belonging to different VMs and prevent
317	* returning memory to the host when one of those VM shuts down.
318	*
319	* The only way to really deal with this problem is to move pages. This can
320	* either be done at VM shutdown and or by the idle priority worker thread
321	* that will be responsible for finding sharable/zero pages. The mechanisms
322	* involved for coercing a VM to move a page (or to do it for it) will be
323	* the same as when telling it to share/zero a page.
324	*
325	*
326	* @subsection subsec_pgmPhys_Tracking Tracking Structures And Their Cost
327	*
328	* There's a difficult balance between keeping the per-page tracking structures
329	* (global and guest page) easy to use and keeping them from eating too much
330	* memory. We have limited virtual memory resources available when operating in
331	* 32-bit kernel space (on 64-bit there'll it's quite a different story). The
332	* tracking structures will be attempted designed such that we can deal with up
333	* to 32GB of memory on a 32-bit system and essentially unlimited on 64-bit ones.
334	*
335	*
336	* @subsubsection subsubsec_pgmPhys_Tracking_Kernel Kernel Space
337	*
338	* @see pg_GMM
339	*
340	* @subsubsection subsubsec_pgmPhys_Tracking_PerVM Per-VM
341	*
342	* Fixed info is the physical address of the page (HCPhys) and the page id
343	* (described above). Theoretically we'll need 48(-12) bits for the HCPhys part.
344	* Today we've restricting ourselves to 40(-12) bits because this is the current
345	* restrictions of all AMD64 implementations (I think Barcelona will up this
346	* to 48(-12) bits, not that it really matters) and I needed the bits for
347	* tracking mappings of a page. 48-12 = 36. That leaves 28 bits, which means a
348	* decent range for the page id: 2^(28+12) = 1024TB.
349	*
350	* In additions to these, we'll have to keep maintaining the page flags as we
351	* currently do. Although it wouldn't harm to optimize these quite a bit, like
352	* for instance the ROM shouldn't depend on having a write handler installed
353	* in order for it to become read-only. A RO/RW bit should be considered so
354	* that the page syncing code doesn't have to mess about checking multiple
355	* flag combinations (ROM \|\| RW handler \|\| write monitored) in order to
356	* figure out how to setup a shadow PTE. But this of course, is second
357	* priority at present. Current this requires 12 bits, but could probably
358	* be optimized to ~8.
359	*
360	* Then there's the 24 bits used to track which shadow page tables are
361	* currently mapping a page for the purpose of speeding up physical
362	* access handlers, and thereby the page pool cache. More bit for this
363	* purpose wouldn't hurt IIRC.
364	*
365	* Then there is a new bit in which we need to record what kind of page
366	* this is, shared, zero, normal or write-monitored-normal. This'll
367	* require 2 bits. One bit might be needed for indicating whether a
368	* write monitored page has been written to. And yet another one or
369	* two for tracking migration status. 3-4 bits total then.
370	*
371	* Whatever is left will can be used to record the sharabilitiy of a
372	* page. The page checksum will not be stored in the per-VM table as
373	* the idle thread will not be permitted to do modifications to it.
374	* It will instead have to keep its own working set of potentially
375	* shareable pages and their check sums and stuff.
376	*
377	* For the present we'll keep the current packing of the
378	* PGMRAMRANGE::aHCPhys to keep the changes simple, only of course,
379	* we'll have to change it to a struct with a total of 128-bits at
380	* our disposal.
381	*
382	* The initial layout will be like this:
383	* @verbatim
384	RTHCPHYS HCPhys; The current stuff.
385	63:40 Current shadow PT tracking stuff.
386	39:12 The physical page frame number.
387	11:0 The current flags.
388	uint32_t u28PageId : 28; The page id.
389	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
390	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
391	uint32_t u1Reserved : 1; Reserved for later.
392	uint32_t u32Reserved; Reserved for later, mostly sharing stats.
393	@endverbatim
394	*
395	* The final layout will be something like this:
396	* @verbatim
397	RTHCPHYS HCPhys; The current stuff.
398	63:48 High page id (12+).
399	47:12 The physical page frame number.
400	11:0 Low page id.
401	uint32_t fReadOnly : 1; Whether it's readonly page (rom or monitored in some way).
402	uint32_t u3Type : 3; The page type {RESERVED, MMIO, MMIO2, ROM, shadowed ROM, RAM}.
403	uint32_t u2PhysMon : 2; Physical access handler type {none, read, write, all}.
404	uint32_t u2VirtMon : 2; Virtual access handler type {none, read, write, all}..
405	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
406	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
407	uint32_t u20Reserved : 20; Reserved for later, mostly sharing stats.
408	uint32_t u32Tracking; The shadow PT tracking stuff, roughly.
409	@endverbatim
410	*
411	* Cost wise, this means we'll double the cost for guest memory. There isn't anyway
412	* around that I'm afraid. It means that the cost of dealing out 32GB of memory
413	* to one or more VMs is: (32GB >> PAGE_SHIFT) * 16 bytes, or 128MBs. Or another
414	* example, the VM heap cost when assigning 1GB to a VM will be: 4MB.
415	*
416	* A couple of cost examples for the total cost per-VM + kernel.
417	* 32-bit Windows and 32-bit linux:
418	* 1GB guest ram, 256K pages: 4MB + 2MB(+) = 6MB
419	* 4GB guest ram, 1M pages: 16MB + 8MB(+) = 24MB
420	* 32GB guest ram, 8M pages: 128MB + 64MB(+) = 192MB
421	* 64-bit Windows and 64-bit linux:
422	* 1GB guest ram, 256K pages: 4MB + 3MB(+) = 7MB
423	* 4GB guest ram, 1M pages: 16MB + 12MB(+) = 28MB
424	* 32GB guest ram, 8M pages: 128MB + 96MB(+) = 224MB
425	*
426	* UPDATE - 2007-09-27:
427	* Will need a ballooned flag/state too because we cannot
428	* trust the guest 100% and reporting the same page as ballooned more
429	* than once will put the GMM off balance.
430	*
431	*
432	* @subsection subsec_pgmPhys_Serializing Serializing Access
433	*
434	* Initially, we'll try a simple scheme:
435	*
436	* - The per-VM RAM tracking structures (PGMRAMRANGE) is only modified
437	* by the EMT thread of that VM while in the pgm critsect.
438	* - Other threads in the VM process that needs to make reliable use of
439	* the per-VM RAM tracking structures will enter the critsect.
440	* - No process external thread or kernel thread will ever try enter
441	* the pgm critical section, as that just won't work.
442	* - The idle thread (and similar threads) doesn't not need 100% reliable
443	* data when performing it tasks as the EMT thread will be the one to
444	* do the actual changes later anyway. So, as long as it only accesses
445	* the main ram range, it can do so by somehow preventing the VM from
446	* being destroyed while it works on it...
447	*
448	* - The over-commitment management, including the allocating/freeing
449	* chunks, is serialized by a ring-0 mutex lock (a fast one since the
450	* more mundane mutex implementation is broken on Linux).
451	* - A separate mutex is protecting the set of allocation chunks so
452	* that pages can be shared or/and freed up while some other VM is
453	* allocating more chunks. This mutex can be take from under the other
454	* one, but not the other way around.
455	*
456	*
457	* @subsection subsec_pgmPhys_Request VM Request interface
458	*
459	* When in ring-0 it will become necessary to send requests to a VM so it can
460	* for instance move a page while defragmenting during VM destroy. The idle
461	* thread will make use of this interface to request VMs to setup shared
462	* pages and to perform write monitoring of pages.
463	*
464	* I would propose an interface similar to the current VMReq interface, similar
465	* in that it doesn't require locking and that the one sending the request may
466	* wait for completion if it wishes to. This shouldn't be very difficult to
467	* realize.
468	*
469	* The requests themselves are also pretty simple. They are basically:
470	* -# Check that some precondition is still true.
471	* -# Do the update.
472	* -# Update all shadow page tables involved with the page.
473	*
474	* The 3rd step is identical to what we're already doing when updating a
475	* physical handler, see pgmHandlerPhysicalSetRamFlagsAndFlushShadowPTs.
476	*
477	*
478	*
479	* @section sec_pgmPhys_MappingCaches Mapping Caches
480	*
481	* In order to be able to map in and out memory and to be able to support
482	* guest with more RAM than we've got virtual address space, we'll employing
483	* a mapping cache. Normally ring-0 and ring-3 can share the same cache,
484	* however on 32-bit darwin the ring-0 code is running in a different memory
485	* context and therefore needs a separate cache. In raw-mode context we also
486	* need a separate cache. The 32-bit darwin mapping cache and the one for
487	* raw-mode context share a lot of code, see PGMRZDYNMAP.
488	*
489	*
490	* @subsection subsec_pgmPhys_MappingCaches_R3 Ring-3
491	*
492	* We've considered implementing the ring-3 mapping cache page based but found
493	* that this was bother some when one had to take into account TLBs+SMP and
494	* portability (missing the necessary APIs on several platforms). There were
495	* also some performance concerns with this approach which hadn't quite been
496	* worked out.
497	*
498	* Instead, we'll be mapping allocation chunks into the VM process. This simplifies
499	* matters greatly quite a bit since we don't need to invent any new ring-0 stuff,
500	* only some minor RTR0MEMOBJ mapping stuff. The main concern here is that mapping
501	* compared to the previous idea is that mapping or unmapping a 1MB chunk is more
502	* costly than a single page, although how much more costly is uncertain. We'll
503	* try address this by using a very big cache, preferably bigger than the actual
504	* VM RAM size if possible. The current VM RAM sizes should give some idea for
505	* 32-bit boxes, while on 64-bit we can probably get away with employing an
506	* unlimited cache.
507	*
508	* The cache have to parts, as already indicated, the ring-3 side and the
509	* ring-0 side.
510	*
511	* The ring-0 will be tied to the page allocator since it will operate on the
512	* memory objects it contains. It will therefore require the first ring-0 mutex
513	* discussed in @ref subsec_pgmPhys_Serializing. We
514	* some double house keeping wrt to who has mapped what I think, since both
515	* VMMR0.r0 and RTR0MemObj will keep track of mapping relations
516	*
517	* The ring-3 part will be protected by the pgm critsect. For simplicity, we'll
518	* require anyone that desires to do changes to the mapping cache to do that
519	* from within this critsect. Alternatively, we could employ a separate critsect
520	* for serializing changes to the mapping cache as this would reduce potential
521	* contention with other threads accessing mappings unrelated to the changes
522	* that are in process. We can see about this later, contention will show
523	* up in the statistics anyway, so it'll be simple to tell.
524	*
525	* The organization of the ring-3 part will be very much like how the allocation
526	* chunks are organized in ring-0, that is in an AVL tree by chunk id. To avoid
527	* having to walk the tree all the time, we'll have a couple of lookaside entries
528	* like in we do for I/O ports and MMIO in IOM.
529	*
530	* The simplified flow of a PGMPhysRead/Write function:
531	* -# Enter the PGM critsect.
532	* -# Lookup GCPhys in the ram ranges and get the Page ID.
533	* -# Calc the Allocation Chunk ID from the Page ID.
534	* -# Check the lookaside entries and then the AVL tree for the Chunk ID.
535	* If not found in cache:
536	* -# Call ring-0 and request it to be mapped and supply
537	* a chunk to be unmapped if the cache is maxed out already.
538	* -# Insert the new mapping into the AVL tree (id + R3 address).
539	* -# Update the relevant lookaside entry and return the mapping address.
540	* -# Do the read/write according to monitoring flags and everything.
541	* -# Leave the critsect.
542	*
543	*
544	* @section sec_pgmPhys_Fallback Fallback
545	*
546	* Current all the "second tier" hosts will not support the RTR0MemObjAllocPhysNC
547	* API and thus require a fallback.
548	*
549	* So, when RTR0MemObjAllocPhysNC returns VERR_NOT_SUPPORTED the page allocator
550	* will return to the ring-3 caller (and later ring-0) and asking it to seed
551	* the page allocator with some fresh pages (VERR_GMM_SEED_ME). Ring-3 will
552	* then perform an SUPR3PageAlloc(cbChunk >> PAGE_SHIFT) call and make a
553	* "SeededAllocPages" call to ring-0.
554	*
555	* The first time ring-0 sees the VERR_NOT_SUPPORTED failure it will disable
556	* all page sharing (zero page detection will continue). It will also force
557	* all allocations to come from the VM which seeded the page. Both these
558	* measures are taken to make sure that there will never be any need for
559	* mapping anything into ring-3 - everything will be mapped already.
560	*
561	* Whether we'll continue to use the current MM locked memory management
562	* for this I don't quite know (I'd prefer not to and just ditch that all
563	* together), we'll see what's simplest to do.
564	*
565	*
566	*
567	* @section sec_pgmPhys_Changes Changes
568	*
569	* Breakdown of the changes involved?
570	*/
571
572	/*******************************************************************************
573	* Header Files *
574	*******************************************************************************/
575	#define LOG_GROUP LOG_GROUP_PGM
576	#include <VBox/vmm/dbgf.h>
577	#include <VBox/vmm/pgm.h>
578	#include <VBox/vmm/cpum.h>
579	#include <VBox/vmm/iom.h>
580	#include <VBox/sup.h>
581	#include <VBox/vmm/mm.h>
582	#include <VBox/vmm/em.h>
583	#include <VBox/vmm/stam.h>
584	#include <VBox/vmm/rem.h>
585	#include <VBox/vmm/selm.h>
586	#include <VBox/vmm/ssm.h>
587	#include <VBox/vmm/hwaccm.h>
588	#include "PGMInternal.h"
589	#include <VBox/vmm/vm.h>
590	#include "PGMInline.h"
591
592	#include <VBox/dbg.h>
593	#include <VBox/param.h>
594	#include <VBox/err.h>
595
596	#include <iprt/asm.h>
597	#include <iprt/asm-amd64-x86.h>
598	#include <iprt/assert.h>
599	#include <iprt/env.h>
600	#include <iprt/mem.h>
601	#include <iprt/file.h>
602	#include <iprt/string.h>
603	#include <iprt/thread.h>
604
605
606	/*******************************************************************************
607	* Internal Functions *
608	*******************************************************************************/
609	static int pgmR3InitPaging(PVM pVM);
610	static int pgmR3InitStats(PVM pVM);
611	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
612	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
613	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
614	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser);
615	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
616	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
617	#ifdef VBOX_STRICT
618	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser);
619	#endif
620	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0);
621	static void pgmR3ModeDataSwitch(PVM pVM, PVMCPU pVCpu, PGMMODE enmShw, PGMMODE enmGst);
622	static PGMMODE pgmR3CalcShadowMode(PVM pVM, PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher);
623
624	#ifdef VBOX_WITH_DEBUGGER
625	/** @todo Convert the first two commands to 'info' items. */
626	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
627	static DECLCALLBACK(int) pgmR3CmdError(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
628	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
629	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
630	# ifdef VBOX_STRICT
631	static DECLCALLBACK(int) pgmR3CmdAssertCR3(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
632	# endif
633	static DECLCALLBACK(int) pgmR3CmdPhysToFile(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
634	#endif
635
636
637	/*******************************************************************************
638	* Global Variables *
639	*******************************************************************************/
640	#ifdef VBOX_WITH_DEBUGGER
641	/** Argument descriptors for '.pgmerror' and '.pgmerroroff'. */
642	static const DBGCVARDESC g_aPgmErrorArgs[] =
643	{
644	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
645	{ 0, 1, DBGCVAR_CAT_STRING, 0, "where", "Error injection location." },
646	};
647
648	static const DBGCVARDESC g_aPgmPhysToFileArgs[] =
649	{
650	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
651	{ 1, 1, DBGCVAR_CAT_STRING, 0, "file", "The file name." },
652	{ 0, 1, DBGCVAR_CAT_STRING, 0, "nozero", "If present, zero pages are skipped." },
653	};
654
655	# ifdef DEBUG_sandervl
656	static const DBGCVARDESC g_aPgmCountPhysWritesArgs[] =
657	{
658	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
659	{ 1, 1, DBGCVAR_CAT_STRING, 0, "enabled", "on/off." },
660	{ 1, 1, DBGCVAR_CAT_NUMBER_NO_RANGE, 0, "interval", "Interval in ms." },
661	};
662	# endif
663
664	/** Command descriptors. */
665	static const DBGCCMD g_aCmds[] =
666	{
667	/* pszCmd, cArgsMin, cArgsMax, paArgDesc, cArgDescs, fFlags, pfnHandler pszSyntax, ....pszDescription */
668	{ "pgmram", 0, 0, NULL, 0, 0, pgmR3CmdRam, "", "Display the ram ranges." },
669	{ "pgmsync", 0, 0, NULL, 0, 0, pgmR3CmdSync, "", "Sync the CR3 page." },
670	{ "pgmerror", 0, 1, &g_aPgmErrorArgs[0], 1, 0, pgmR3CmdError, "", "Enables inject runtime of errors into parts of PGM." },
671	{ "pgmerroroff", 0, 1, &g_aPgmErrorArgs[0], 1, 0, pgmR3CmdError, "", "Disables inject runtime errors into parts of PGM." },
672	# ifdef VBOX_STRICT
673	{ "pgmassertcr3", 0, 0, NULL, 0, 0, pgmR3CmdAssertCR3, "", "Check the shadow CR3 mapping." },
674	# if HC_ARCH_BITS == 64
675	{ "pgmcheckduppages", 0, 0, NULL, 0, 0, pgmR3CmdCheckDuplicatePages, "", "Check for duplicate pages in all running VMs." },
676	{ "pgmsharedmodules", 0, 0, NULL, 0, 0, pgmR3CmdShowSharedModules, "", "Print shared modules info." },
677	# endif
678	# endif
679	{ "pgmsyncalways", 0, 0, NULL, 0, 0, pgmR3CmdSyncAlways, "", "Toggle permanent CR3 syncing." },
680	{ "pgmphystofile", 1, 2, &g_aPgmPhysToFileArgs[0], 2, 0, pgmR3CmdPhysToFile, "", "Save the physical memory to file." },
681	};
682	#endif
683
684
685
686
687	/*
688	* Shadow - 32-bit mode
689	*/
690	#define PGM_SHW_TYPE PGM_TYPE_32BIT
691	#define PGM_SHW_NAME(name) PGM_SHW_NAME_32BIT(name)
692	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_32BIT_STR(name)
693	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_32BIT_STR(name)
694	#include "PGMShw.h"
695
696	/* Guest - real mode */
697	#define PGM_GST_TYPE PGM_TYPE_REAL
698	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
699	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
700	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
701	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_REAL(name)
702	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_REAL_STR(name)
703	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_REAL_STR(name)
704	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
705	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD_PHYS
706	#include "PGMBth.h"
707	#include "PGMGstDefs.h"
708	#include "PGMGst.h"
709	#undef BTH_PGMPOOLKIND_PT_FOR_PT
710	#undef BTH_PGMPOOLKIND_ROOT
711	#undef PGM_BTH_NAME
712	#undef PGM_BTH_NAME_RC_STR
713	#undef PGM_BTH_NAME_R0_STR
714	#undef PGM_GST_TYPE
715	#undef PGM_GST_NAME
716	#undef PGM_GST_NAME_RC_STR
717	#undef PGM_GST_NAME_R0_STR
718
719	/* Guest - protected mode */
720	#define PGM_GST_TYPE PGM_TYPE_PROT
721	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
722	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
723	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
724	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_PROT(name)
725	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_PROT_STR(name)
726	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_PROT_STR(name)
727	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
728	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD_PHYS
729	#include "PGMBth.h"
730	#include "PGMGstDefs.h"
731	#include "PGMGst.h"
732	#undef BTH_PGMPOOLKIND_PT_FOR_PT
733	#undef BTH_PGMPOOLKIND_ROOT
734	#undef PGM_BTH_NAME
735	#undef PGM_BTH_NAME_RC_STR
736	#undef PGM_BTH_NAME_R0_STR
737	#undef PGM_GST_TYPE
738	#undef PGM_GST_NAME
739	#undef PGM_GST_NAME_RC_STR
740	#undef PGM_GST_NAME_R0_STR
741
742	/* Guest - 32-bit mode */
743	#define PGM_GST_TYPE PGM_TYPE_32BIT
744	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
745	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
746	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
747	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_32BIT(name)
748	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_32BIT_STR(name)
749	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_32BIT_STR(name)
750	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_32BIT_PT
751	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_32BIT_PT_FOR_32BIT_4MB
752	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD
753	#include "PGMBth.h"
754	#include "PGMGstDefs.h"
755	#include "PGMGst.h"
756	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
757	#undef BTH_PGMPOOLKIND_PT_FOR_PT
758	#undef BTH_PGMPOOLKIND_ROOT
759	#undef PGM_BTH_NAME
760	#undef PGM_BTH_NAME_RC_STR
761	#undef PGM_BTH_NAME_R0_STR
762	#undef PGM_GST_TYPE
763	#undef PGM_GST_NAME
764	#undef PGM_GST_NAME_RC_STR
765	#undef PGM_GST_NAME_R0_STR
766
767	#undef PGM_SHW_TYPE
768	#undef PGM_SHW_NAME
769	#undef PGM_SHW_NAME_RC_STR
770	#undef PGM_SHW_NAME_R0_STR
771
772
773	/*
774	* Shadow - PAE mode
775	*/
776	#define PGM_SHW_TYPE PGM_TYPE_PAE
777	#define PGM_SHW_NAME(name) PGM_SHW_NAME_PAE(name)
778	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_PAE_STR(name)
779	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_PAE_STR(name)
780	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
781	#include "PGMShw.h"
782
783	/* Guest - real mode */
784	#define PGM_GST_TYPE PGM_TYPE_REAL
785	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
786	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
787	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
788	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
789	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_REAL_STR(name)
790	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_REAL_STR(name)
791	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
792	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_PHYS
793	#include "PGMGstDefs.h"
794	#include "PGMBth.h"
795	#undef BTH_PGMPOOLKIND_PT_FOR_PT
796	#undef BTH_PGMPOOLKIND_ROOT
797	#undef PGM_BTH_NAME
798	#undef PGM_BTH_NAME_RC_STR
799	#undef PGM_BTH_NAME_R0_STR
800	#undef PGM_GST_TYPE
801	#undef PGM_GST_NAME
802	#undef PGM_GST_NAME_RC_STR
803	#undef PGM_GST_NAME_R0_STR
804
805	/* Guest - protected mode */
806	#define PGM_GST_TYPE PGM_TYPE_PROT
807	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
808	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
809	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
810	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PROT(name)
811	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_PROT_STR(name)
812	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PROT_STR(name)
813	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
814	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_PHYS
815	#include "PGMGstDefs.h"
816	#include "PGMBth.h"
817	#undef BTH_PGMPOOLKIND_PT_FOR_PT
818	#undef BTH_PGMPOOLKIND_ROOT
819	#undef PGM_BTH_NAME
820	#undef PGM_BTH_NAME_RC_STR
821	#undef PGM_BTH_NAME_R0_STR
822	#undef PGM_GST_TYPE
823	#undef PGM_GST_NAME
824	#undef PGM_GST_NAME_RC_STR
825	#undef PGM_GST_NAME_R0_STR
826
827	/* Guest - 32-bit mode */
828	#define PGM_GST_TYPE PGM_TYPE_32BIT
829	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
830	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
831	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
832	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_32BIT(name)
833	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_32BIT_STR(name)
834	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_32BIT_STR(name)
835	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
836	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
837	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_FOR_32BIT
838	#include "PGMGstDefs.h"
839	#include "PGMBth.h"
840	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
841	#undef BTH_PGMPOOLKIND_PT_FOR_PT
842	#undef BTH_PGMPOOLKIND_ROOT
843	#undef PGM_BTH_NAME
844	#undef PGM_BTH_NAME_RC_STR
845	#undef PGM_BTH_NAME_R0_STR
846	#undef PGM_GST_TYPE
847	#undef PGM_GST_NAME
848	#undef PGM_GST_NAME_RC_STR
849	#undef PGM_GST_NAME_R0_STR
850
851	/* Guest - PAE mode */
852	#define PGM_GST_TYPE PGM_TYPE_PAE
853	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
854	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
855	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
856	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PAE(name)
857	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_PAE_STR(name)
858	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PAE_STR(name)
859	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
860	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
861	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT
862	#include "PGMBth.h"
863	#include "PGMGstDefs.h"
864	#include "PGMGst.h"
865	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
866	#undef BTH_PGMPOOLKIND_PT_FOR_PT
867	#undef BTH_PGMPOOLKIND_ROOT
868	#undef PGM_BTH_NAME
869	#undef PGM_BTH_NAME_RC_STR
870	#undef PGM_BTH_NAME_R0_STR
871	#undef PGM_GST_TYPE
872	#undef PGM_GST_NAME
873	#undef PGM_GST_NAME_RC_STR
874	#undef PGM_GST_NAME_R0_STR
875
876	#undef PGM_SHW_TYPE
877	#undef PGM_SHW_NAME
878	#undef PGM_SHW_NAME_RC_STR
879	#undef PGM_SHW_NAME_R0_STR
880
881
882	/*
883	* Shadow - AMD64 mode
884	*/
885	#define PGM_SHW_TYPE PGM_TYPE_AMD64
886	#define PGM_SHW_NAME(name) PGM_SHW_NAME_AMD64(name)
887	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_AMD64_STR(name)
888	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_AMD64_STR(name)
889	#include "PGMShw.h"
890
891	#ifdef VBOX_WITH_64_BITS_GUESTS
892	/* Guest - AMD64 mode */
893	# define PGM_GST_TYPE PGM_TYPE_AMD64
894	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
895	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
896	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
897	# define PGM_BTH_NAME(name) PGM_BTH_NAME_AMD64_AMD64(name)
898	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_AMD64_AMD64_STR(name)
899	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_AMD64_AMD64_STR(name)
900	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
901	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
902	# define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_64BIT_PML4
903	# include "PGMBth.h"
904	# include "PGMGstDefs.h"
905	# include "PGMGst.h"
906	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
907	# undef BTH_PGMPOOLKIND_PT_FOR_PT
908	# undef BTH_PGMPOOLKIND_ROOT
909	# undef PGM_BTH_NAME
910	# undef PGM_BTH_NAME_RC_STR
911	# undef PGM_BTH_NAME_R0_STR
912	# undef PGM_GST_TYPE
913	# undef PGM_GST_NAME
914	# undef PGM_GST_NAME_RC_STR
915	# undef PGM_GST_NAME_R0_STR
916	#endif /* VBOX_WITH_64_BITS_GUESTS */
917
918	#undef PGM_SHW_TYPE
919	#undef PGM_SHW_NAME
920	#undef PGM_SHW_NAME_RC_STR
921	#undef PGM_SHW_NAME_R0_STR
922
923
924	/*
925	* Shadow - Nested paging mode
926	*/
927	#define PGM_SHW_TYPE PGM_TYPE_NESTED
928	#define PGM_SHW_NAME(name) PGM_SHW_NAME_NESTED(name)
929	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_NESTED_STR(name)
930	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_NESTED_STR(name)
931	#include "PGMShw.h"
932
933	/* Guest - real mode */
934	#define PGM_GST_TYPE PGM_TYPE_REAL
935	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
936	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
937	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
938	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_REAL(name)
939	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_REAL_STR(name)
940	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_REAL_STR(name)
941	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
942	#include "PGMGstDefs.h"
943	#include "PGMBth.h"
944	#undef BTH_PGMPOOLKIND_PT_FOR_PT
945	#undef PGM_BTH_NAME
946	#undef PGM_BTH_NAME_RC_STR
947	#undef PGM_BTH_NAME_R0_STR
948	#undef PGM_GST_TYPE
949	#undef PGM_GST_NAME
950	#undef PGM_GST_NAME_RC_STR
951	#undef PGM_GST_NAME_R0_STR
952
953	/* Guest - protected mode */
954	#define PGM_GST_TYPE PGM_TYPE_PROT
955	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
956	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
957	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
958	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PROT(name)
959	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_PROT_STR(name)
960	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PROT_STR(name)
961	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
962	#include "PGMGstDefs.h"
963	#include "PGMBth.h"
964	#undef BTH_PGMPOOLKIND_PT_FOR_PT
965	#undef PGM_BTH_NAME
966	#undef PGM_BTH_NAME_RC_STR
967	#undef PGM_BTH_NAME_R0_STR
968	#undef PGM_GST_TYPE
969	#undef PGM_GST_NAME
970	#undef PGM_GST_NAME_RC_STR
971	#undef PGM_GST_NAME_R0_STR
972
973	/* Guest - 32-bit mode */
974	#define PGM_GST_TYPE PGM_TYPE_32BIT
975	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
976	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
977	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
978	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_32BIT(name)
979	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_32BIT_STR(name)
980	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_32BIT_STR(name)
981	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
982	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
983	#include "PGMGstDefs.h"
984	#include "PGMBth.h"
985	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
986	#undef BTH_PGMPOOLKIND_PT_FOR_PT
987	#undef PGM_BTH_NAME
988	#undef PGM_BTH_NAME_RC_STR
989	#undef PGM_BTH_NAME_R0_STR
990	#undef PGM_GST_TYPE
991	#undef PGM_GST_NAME
992	#undef PGM_GST_NAME_RC_STR
993	#undef PGM_GST_NAME_R0_STR
994
995	/* Guest - PAE mode */
996	#define PGM_GST_TYPE PGM_TYPE_PAE
997	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
998	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
999	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
1000	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PAE(name)
1001	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_PAE_STR(name)
1002	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PAE_STR(name)
1003	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1004	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1005	#include "PGMGstDefs.h"
1006	#include "PGMBth.h"
1007	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1008	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1009	#undef PGM_BTH_NAME
1010	#undef PGM_BTH_NAME_RC_STR
1011	#undef PGM_BTH_NAME_R0_STR
1012	#undef PGM_GST_TYPE
1013	#undef PGM_GST_NAME
1014	#undef PGM_GST_NAME_RC_STR
1015	#undef PGM_GST_NAME_R0_STR
1016
1017	#ifdef VBOX_WITH_64_BITS_GUESTS
1018	/* Guest - AMD64 mode */
1019	# define PGM_GST_TYPE PGM_TYPE_AMD64
1020	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
1021	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
1022	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
1023	# define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_AMD64(name)
1024	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_AMD64_STR(name)
1025	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_AMD64_STR(name)
1026	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1027	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1028	# include "PGMGstDefs.h"
1029	# include "PGMBth.h"
1030	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
1031	# undef BTH_PGMPOOLKIND_PT_FOR_PT
1032	# undef PGM_BTH_NAME
1033	# undef PGM_BTH_NAME_RC_STR
1034	# undef PGM_BTH_NAME_R0_STR
1035	# undef PGM_GST_TYPE
1036	# undef PGM_GST_NAME
1037	# undef PGM_GST_NAME_RC_STR
1038	# undef PGM_GST_NAME_R0_STR
1039	#endif /* VBOX_WITH_64_BITS_GUESTS */
1040
1041	#undef PGM_SHW_TYPE
1042	#undef PGM_SHW_NAME
1043	#undef PGM_SHW_NAME_RC_STR
1044	#undef PGM_SHW_NAME_R0_STR
1045
1046
1047	/*
1048	* Shadow - EPT
1049	*/
1050	#define PGM_SHW_TYPE PGM_TYPE_EPT
1051	#define PGM_SHW_NAME(name) PGM_SHW_NAME_EPT(name)
1052	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_EPT_STR(name)
1053	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_EPT_STR(name)
1054	#include "PGMShw.h"
1055
1056	/* Guest - real mode */
1057	#define PGM_GST_TYPE PGM_TYPE_REAL
1058	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
1059	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
1060	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
1061	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_REAL(name)
1062	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_REAL_STR(name)
1063	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_REAL_STR(name)
1064	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
1065	#include "PGMGstDefs.h"
1066	#include "PGMBth.h"
1067	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1068	#undef PGM_BTH_NAME
1069	#undef PGM_BTH_NAME_RC_STR
1070	#undef PGM_BTH_NAME_R0_STR
1071	#undef PGM_GST_TYPE
1072	#undef PGM_GST_NAME
1073	#undef PGM_GST_NAME_RC_STR
1074	#undef PGM_GST_NAME_R0_STR
1075
1076	/* Guest - protected mode */
1077	#define PGM_GST_TYPE PGM_TYPE_PROT
1078	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
1079	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
1080	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
1081	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_PROT(name)
1082	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_PROT_STR(name)
1083	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_PROT_STR(name)
1084	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
1085	#include "PGMGstDefs.h"
1086	#include "PGMBth.h"
1087	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1088	#undef PGM_BTH_NAME
1089	#undef PGM_BTH_NAME_RC_STR
1090	#undef PGM_BTH_NAME_R0_STR
1091	#undef PGM_GST_TYPE
1092	#undef PGM_GST_NAME
1093	#undef PGM_GST_NAME_RC_STR
1094	#undef PGM_GST_NAME_R0_STR
1095
1096	/* Guest - 32-bit mode */
1097	#define PGM_GST_TYPE PGM_TYPE_32BIT
1098	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
1099	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
1100	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
1101	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_32BIT(name)
1102	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_32BIT_STR(name)
1103	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_32BIT_STR(name)
1104	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
1105	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
1106	#include "PGMGstDefs.h"
1107	#include "PGMBth.h"
1108	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1109	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1110	#undef PGM_BTH_NAME
1111	#undef PGM_BTH_NAME_RC_STR
1112	#undef PGM_BTH_NAME_R0_STR
1113	#undef PGM_GST_TYPE
1114	#undef PGM_GST_NAME
1115	#undef PGM_GST_NAME_RC_STR
1116	#undef PGM_GST_NAME_R0_STR
1117
1118	/* Guest - PAE mode */
1119	#define PGM_GST_TYPE PGM_TYPE_PAE
1120	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
1121	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
1122	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
1123	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_PAE(name)
1124	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_PAE_STR(name)
1125	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_PAE_STR(name)
1126	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1127	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1128	#include "PGMGstDefs.h"
1129	#include "PGMBth.h"
1130	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1131	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1132	#undef PGM_BTH_NAME
1133	#undef PGM_BTH_NAME_RC_STR
1134	#undef PGM_BTH_NAME_R0_STR
1135	#undef PGM_GST_TYPE
1136	#undef PGM_GST_NAME
1137	#undef PGM_GST_NAME_RC_STR
1138	#undef PGM_GST_NAME_R0_STR
1139
1140	#ifdef VBOX_WITH_64_BITS_GUESTS
1141	/* Guest - AMD64 mode */
1142	# define PGM_GST_TYPE PGM_TYPE_AMD64
1143	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
1144	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
1145	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
1146	# define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_AMD64(name)
1147	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_AMD64_STR(name)
1148	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_AMD64_STR(name)
1149	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1150	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1151	# include "PGMGstDefs.h"
1152	# include "PGMBth.h"
1153	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
1154	# undef BTH_PGMPOOLKIND_PT_FOR_PT
1155	# undef PGM_BTH_NAME
1156	# undef PGM_BTH_NAME_RC_STR
1157	# undef PGM_BTH_NAME_R0_STR
1158	# undef PGM_GST_TYPE
1159	# undef PGM_GST_NAME
1160	# undef PGM_GST_NAME_RC_STR
1161	# undef PGM_GST_NAME_R0_STR
1162	#endif /* VBOX_WITH_64_BITS_GUESTS */
1163
1164	#undef PGM_SHW_TYPE
1165	#undef PGM_SHW_NAME
1166	#undef PGM_SHW_NAME_RC_STR
1167	#undef PGM_SHW_NAME_R0_STR
1168
1169
1170
1171	/**
1172	* Initiates the paging of VM.
1173	*
1174	* @returns VBox status code.
1175	* @param pVM Pointer to VM structure.
1176	*/
1177	VMMR3DECL(int) PGMR3Init(PVM pVM)
1178	{
1179	LogFlow(("PGMR3Init:\n"));
1180	PCFGMNODE pCfgPGM = CFGMR3GetChild(CFGMR3GetRoot(pVM), "/PGM");
1181	int rc;
1182
1183	/*
1184	* Assert alignment and sizes.
1185	*/
1186	AssertCompile(sizeof(pVM->pgm.s) <= sizeof(pVM->pgm.padding));
1187	AssertCompile(sizeof(pVM->aCpus[0].pgm.s) <= sizeof(pVM->aCpus[0].pgm.padding));
1188	AssertCompileMemberAlignment(PGM, CritSectX, sizeof(uintptr_t));
1189
1190	/*
1191	* Init the structure.
1192	*/
1193	#ifdef PGM_WITHOUT_MAPPINGS
1194	pVM->pgm.s.fMappingsDisabled = true;
1195	#endif
1196	pVM->pgm.s.offVM = RT_OFFSETOF(VM, pgm.s);
1197	pVM->pgm.s.offVCpuPGM = RT_OFFSETOF(VMCPU, pgm.s);
1198
1199	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.aHandyPages); i++)
1200	{
1201	pVM->pgm.s.aHandyPages[i].HCPhysGCPhys = NIL_RTHCPHYS;
1202	pVM->pgm.s.aHandyPages[i].idPage = NIL_GMM_PAGEID;
1203	pVM->pgm.s.aHandyPages[i].idSharedPage = NIL_GMM_PAGEID;
1204	}
1205
1206	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.aLargeHandyPage); i++)
1207	{
1208	pVM->pgm.s.aLargeHandyPage[i].HCPhysGCPhys = NIL_RTHCPHYS;
1209	pVM->pgm.s.aLargeHandyPage[i].idPage = NIL_GMM_PAGEID;
1210	pVM->pgm.s.aLargeHandyPage[i].idSharedPage = NIL_GMM_PAGEID;
1211	}
1212
1213	/* Init the per-CPU part. */
1214	for (VMCPUID idCpu = 0; idCpu < pVM->cCpus; idCpu++)
1215	{
1216	PVMCPU pVCpu = &pVM->aCpus[idCpu];
1217	PPGMCPU pPGM = &pVCpu->pgm.s;
1218
1219	pPGM->offVM = (uintptr_t)&pVCpu->pgm.s - (uintptr_t)pVM;
1220	pPGM->offVCpu = RT_OFFSETOF(VMCPU, pgm.s);
1221	pPGM->offPGM = (uintptr_t)&pVCpu->pgm.s - (uintptr_t)&pVM->pgm.s;
1222
1223	pPGM->enmShadowMode = PGMMODE_INVALID;
1224	pPGM->enmGuestMode = PGMMODE_INVALID;
1225
1226	pPGM->GCPhysCR3 = NIL_RTGCPHYS;
1227
1228	pPGM->pGst32BitPdR3 = NULL;
1229	pPGM->pGstPaePdptR3 = NULL;
1230	pPGM->pGstAmd64Pml4R3 = NULL;
1231	#ifndef VBOX_WITH_2X_4GB_ADDR_SPACE
1232	pPGM->pGst32BitPdR0 = NIL_RTR0PTR;
1233	pPGM->pGstPaePdptR0 = NIL_RTR0PTR;
1234	pPGM->pGstAmd64Pml4R0 = NIL_RTR0PTR;
1235	#endif
1236	pPGM->pGst32BitPdRC = NIL_RTRCPTR;
1237	pPGM->pGstPaePdptRC = NIL_RTRCPTR;
1238	for (unsigned i = 0; i < RT_ELEMENTS(pVCpu->pgm.s.apGstPaePDsR3); i++)
1239	{
1240	pPGM->apGstPaePDsR3[i] = NULL;
1241	#ifndef VBOX_WITH_2X_4GB_ADDR_SPACE
1242	pPGM->apGstPaePDsR0[i] = NIL_RTR0PTR;
1243	#endif
1244	pPGM->apGstPaePDsRC[i] = NIL_RTRCPTR;
1245	pPGM->aGCPhysGstPaePDs[i] = NIL_RTGCPHYS;
1246	pPGM->aGstPaePdpeRegs[i].u = UINT64_MAX;
1247	pPGM->aGCPhysGstPaePDsMonitored[i] = NIL_RTGCPHYS;
1248	}
1249
1250	pPGM->fA20Enabled = true;
1251	}
1252
1253	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1254	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(32) - 1; /* default; checked later */
1255	pVM->pgm.s.GCPtrPrevRamRangeMapping = MM_HYPER_AREA_ADDRESS;
1256
1257	rc = CFGMR3QueryBoolDef(CFGMR3GetRoot(pVM), "RamPreAlloc", &pVM->pgm.s.fRamPreAlloc,
1258	#ifdef VBOX_WITH_PREALLOC_RAM_BY_DEFAULT
1259	true
1260	#else
1261	false
1262	#endif
1263	);
1264	AssertLogRelRCReturn(rc, rc);
1265
1266	#if HC_ARCH_BITS == 32
1267	# ifdef RT_OS_DARWIN
1268	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, _1G / GMM_CHUNK_SIZE * 3);
1269	# else
1270	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, _1G / GMM_CHUNK_SIZE);
1271	# endif
1272	#else
1273	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, UINT32_MAX);
1274	#endif
1275	AssertLogRelRCReturn(rc, rc);
1276	for (uint32_t i = 0; i < RT_ELEMENTS(pVM->pgm.s.ChunkR3Map.Tlb.aEntries); i++)
1277	pVM->pgm.s.ChunkR3Map.Tlb.aEntries[i].idChunk = NIL_GMM_CHUNKID;
1278
1279	/*
1280	* Get the configured RAM size - to estimate saved state size.
1281	*/
1282	uint64_t cbRam;
1283	rc = CFGMR3QueryU64(CFGMR3GetRoot(pVM), "RamSize", &cbRam);
1284	if (rc == VERR_CFGM_VALUE_NOT_FOUND)
1285	cbRam = 0;
1286	else if (RT_SUCCESS(rc))
1287	{
1288	if (cbRam < PAGE_SIZE)
1289	cbRam = 0;
1290	cbRam = RT_ALIGN_64(cbRam, PAGE_SIZE);
1291	}
1292	else
1293	{
1294	AssertMsgFailed(("Configuration error: Failed to query integer \"RamSize\", rc=%Rrc.\n", rc));
1295	return rc;
1296	}
1297
1298	/*
1299	* Check for PCI pass-through.
1300	*/
1301	rc = CFGMR3QueryBoolDef(pCfgPGM, "PciPassThrough", &pVM->pgm.s.fPciPassthrough, false);
1302	AssertMsgRCReturn(rc, ("Configuration error: Failed to query integer \"PciPassThrough\", rc=%Rrc.\n", rc), rc);
1303	AssertLogRelReturn(!pVM->pgm.s.fPciPassthrough \|\| pVM->pgm.s.fRamPreAlloc, VERR_INVALID_PARAMETER);
1304
1305	#ifdef VBOX_WITH_STATISTICS
1306	/*
1307	* Allocate memory for the statistics before someone tries to use them.
1308	*/
1309	size_t cbTotalStats = RT_ALIGN_Z(sizeof(PGMSTATS), 64) + RT_ALIGN_Z(sizeof(PGMCPUSTATS), 64) * pVM->cCpus;
1310	void *pv;
1311	rc = MMHyperAlloc(pVM, RT_ALIGN_Z(cbTotalStats, PAGE_SIZE), PAGE_SIZE, MM_TAG_PGM, &pv);
1312	AssertRCReturn(rc, rc);
1313
1314	pVM->pgm.s.pStatsR3 = (PGMSTATS *)pv;
1315	pVM->pgm.s.pStatsR0 = MMHyperCCToR0(pVM, pv);
1316	pVM->pgm.s.pStatsRC = MMHyperCCToRC(pVM, pv);
1317	pv = (uint8_t *)pv + RT_ALIGN_Z(sizeof(PGMSTATS), 64);
1318
1319	for (VMCPUID iCpu = 0; iCpu < pVM->cCpus; iCpu++)
1320	{
1321	pVM->aCpus[iCpu].pgm.s.pStatsR3 = (PGMCPUSTATS *)pv;
1322	pVM->aCpus[iCpu].pgm.s.pStatsR0 = MMHyperCCToR0(pVM, pv);
1323	pVM->aCpus[iCpu].pgm.s.pStatsRC = MMHyperCCToRC(pVM, pv);
1324
1325	pv = (uint8_t *)pv + RT_ALIGN_Z(sizeof(PGMCPUSTATS), 64);
1326	}
1327	#endif /* VBOX_WITH_STATISTICS */
1328
1329	/*
1330	* Register callbacks, string formatters and the saved state data unit.
1331	*/
1332	#ifdef VBOX_STRICT
1333	VMR3AtStateRegister(pVM, pgmR3ResetNoMorePhysWritesFlag, NULL);
1334	#endif
1335	PGMRegisterStringFormatTypes();
1336
1337	rc = pgmR3InitSavedState(pVM, cbRam);
1338	if (RT_FAILURE(rc))
1339	return rc;
1340
1341	/*
1342	* Initialize the PGM critical section and flush the phys TLBs
1343	*/
1344	rc = PDMR3CritSectInit(pVM, &pVM->pgm.s.CritSectX, RT_SRC_POS, "PGM");
1345	AssertRCReturn(rc, rc);
1346
1347	PGMR3PhysChunkInvalidateTLB(pVM);
1348	pgmPhysInvalidatePageMapTLB(pVM);
1349
1350	/*
1351	* For the time being we sport a full set of handy pages in addition to the base
1352	* memory to simplify things.
1353	*/
1354	rc = MMR3ReserveHandyPages(pVM, RT_ELEMENTS(pVM->pgm.s.aHandyPages)); /** @todo this should be changed to PGM_HANDY_PAGES_MIN but this needs proper testing... */
1355	AssertRCReturn(rc, rc);
1356
1357	/*
1358	* Trees
1359	*/
1360	rc = MMHyperAlloc(pVM, sizeof(PGMTREES), 0, MM_TAG_PGM, (void **)&pVM->pgm.s.pTreesR3);
1361	if (RT_SUCCESS(rc))
1362	{
1363	pVM->pgm.s.pTreesR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pTreesR3);
1364	pVM->pgm.s.pTreesRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pTreesR3);
1365
1366	/*
1367	* Allocate the zero page.
1368	*/
1369	rc = MMHyperAlloc(pVM, PAGE_SIZE, PAGE_SIZE, MM_TAG_PGM, &pVM->pgm.s.pvZeroPgR3);
1370	}
1371	if (RT_SUCCESS(rc))
1372	{
1373	pVM->pgm.s.pvZeroPgRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pvZeroPgR3);
1374	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
1375	pVM->pgm.s.HCPhysZeroPg = MMR3HyperHCVirt2HCPhys(pVM, pVM->pgm.s.pvZeroPgR3);
1376	AssertRelease(pVM->pgm.s.HCPhysZeroPg != NIL_RTHCPHYS);
1377
1378	/*
1379	* Allocate the invalid MMIO page.
1380	* (The invalid bits in HCPhysInvMmioPg are set later on init complete.)
1381	*/
1382	rc = MMHyperAlloc(pVM, PAGE_SIZE, PAGE_SIZE, MM_TAG_PGM, &pVM->pgm.s.pvMmioPgR3);
1383	}
1384	if (RT_SUCCESS(rc))
1385	{
1386	ASMMemFill32(pVM->pgm.s.pvMmioPgR3, PAGE_SIZE, 0xfeedface);
1387	pVM->pgm.s.HCPhysMmioPg = MMR3HyperHCVirt2HCPhys(pVM, pVM->pgm.s.pvMmioPgR3);
1388	AssertRelease(pVM->pgm.s.HCPhysMmioPg != NIL_RTHCPHYS);
1389	pVM->pgm.s.HCPhysInvMmioPg = pVM->pgm.s.HCPhysMmioPg;
1390
1391	/*
1392	* Init the paging.
1393	*/
1394	rc = pgmR3InitPaging(pVM);
1395	}
1396	if (RT_SUCCESS(rc))
1397	{
1398	/*
1399	* Init the page pool.
1400	*/
1401	rc = pgmR3PoolInit(pVM);
1402	}
1403	if (RT_SUCCESS(rc))
1404	{
1405	for (VMCPUID i = 0; i < pVM->cCpus; i++)
1406	{
1407	PVMCPU pVCpu = &pVM->aCpus[i];
1408	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
1409	if (RT_FAILURE(rc))
1410	break;
1411	}
1412	}
1413
1414	if (RT_SUCCESS(rc))
1415	{
1416	/*
1417	* Info & statistics
1418	*/
1419	DBGFR3InfoRegisterInternal(pVM, "mode",
1420	"Shows the current paging mode. "
1421	"Recognizes 'all', 'guest', 'shadow' and 'host' as arguments, defaulting to 'all' if nothing is given.",
1422	pgmR3InfoMode);
1423	DBGFR3InfoRegisterInternal(pVM, "pgmcr3",
1424	"Dumps all the entries in the top level paging table. No arguments.",
1425	pgmR3InfoCr3);
1426	DBGFR3InfoRegisterInternal(pVM, "phys",
1427	"Dumps all the physical address ranges. No arguments.",
1428	pgmR3PhysInfo);
1429	DBGFR3InfoRegisterInternal(pVM, "handlers",
1430	"Dumps physical, virtual and hyper virtual handlers. "
1431	"Pass 'phys', 'virt', 'hyper' as argument if only one kind is wanted."
1432	"Add 'nost' if the statistics are unwanted, use together with 'all' or explicit selection.",
1433	pgmR3InfoHandlers);
1434	DBGFR3InfoRegisterInternal(pVM, "mappings",
1435	"Dumps guest mappings.",
1436	pgmR3MapInfo);
1437
1438	pgmR3InitStats(pVM);
1439
1440	#ifdef VBOX_WITH_DEBUGGER
1441	/*
1442	* Debugger commands.
1443	*/
1444	static bool s_fRegisteredCmds = false;
1445	if (!s_fRegisteredCmds)
1446	{
1447	int rc2 = DBGCRegisterCommands(&g_aCmds[0], RT_ELEMENTS(g_aCmds));
1448	if (RT_SUCCESS(rc2))
1449	s_fRegisteredCmds = true;
1450	}
1451	#endif
1452	return VINF_SUCCESS;
1453	}
1454
1455	/* Almost no cleanup necessary, MM frees all memory. */
1456	PDMR3CritSectDelete(&pVM->pgm.s.CritSectX);
1457
1458	return rc;
1459	}
1460
1461
1462	/**
1463	* Init paging.
1464	*
1465	* Since we need to check what mode the host is operating in before we can choose
1466	* the right paging functions for the host we have to delay this until R0 has
1467	* been initialized.
1468	*
1469	* @returns VBox status code.
1470	* @param pVM VM handle.
1471	*/
1472	static int pgmR3InitPaging(PVM pVM)
1473	{
1474	/*
1475	* Force a recalculation of modes and switcher so everyone gets notified.
1476	*/
1477	for (VMCPUID i = 0; i < pVM->cCpus; i++)
1478	{
1479	PVMCPU pVCpu = &pVM->aCpus[i];
1480
1481	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
1482	pVCpu->pgm.s.enmGuestMode = PGMMODE_INVALID;
1483	}
1484
1485	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1486
1487	/*
1488	* Allocate static mapping space for whatever the cr3 register
1489	* points to and in the case of PAE mode to the 4 PDs.
1490	*/
1491	int rc = MMR3HyperReserve(pVM, PAGE_SIZE * 5, "CR3 mapping", &pVM->pgm.s.GCPtrCR3Mapping);
1492	if (RT_FAILURE(rc))
1493	{
1494	AssertMsgFailed(("Failed to reserve two pages for cr mapping in HMA, rc=%Rrc\n", rc));
1495	return rc;
1496	}
1497	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
1498
1499	/*
1500	* Allocate pages for the three possible intermediate contexts
1501	* (AMD64, PAE and plain 32-Bit). We maintain all three contexts
1502	* for the sake of simplicity. The AMD64 uses the PAE for the
1503	* lower levels, making the total number of pages 11 (3 + 7 + 1).
1504	*
1505	* We assume that two page tables will be enought for the core code
1506	* mappings (HC virtual and identity).
1507	*/
1508	pVM->pgm.s.pInterPD = (PX86PD)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPD, VERR_NO_PAGE_MEMORY);
1509	pVM->pgm.s.apInterPTs[0] = (PX86PT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.apInterPTs[0], VERR_NO_PAGE_MEMORY);
1510	pVM->pgm.s.apInterPTs[1] = (PX86PT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.apInterPTs[1], VERR_NO_PAGE_MEMORY);
1511	pVM->pgm.s.apInterPaePTs[0] = (PX86PTPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePTs[0], VERR_NO_PAGE_MEMORY);
1512	pVM->pgm.s.apInterPaePTs[1] = (PX86PTPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePTs[1], VERR_NO_PAGE_MEMORY);
1513	pVM->pgm.s.apInterPaePDs[0] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[0], VERR_NO_PAGE_MEMORY);
1514	pVM->pgm.s.apInterPaePDs[1] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[1], VERR_NO_PAGE_MEMORY);
1515	pVM->pgm.s.apInterPaePDs[2] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[2], VERR_NO_PAGE_MEMORY);
1516	pVM->pgm.s.apInterPaePDs[3] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[3], VERR_NO_PAGE_MEMORY);
1517	pVM->pgm.s.pInterPaePDPT = (PX86PDPT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePDPT, VERR_NO_PAGE_MEMORY);
1518	pVM->pgm.s.pInterPaePDPT64 = (PX86PDPT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePDPT64, VERR_NO_PAGE_MEMORY);
1519	pVM->pgm.s.pInterPaePML4 = (PX86PML4)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePML4, VERR_NO_PAGE_MEMORY);
1520
1521	pVM->pgm.s.HCPhysInterPD = MMPage2Phys(pVM, pVM->pgm.s.pInterPD);
1522	AssertRelease(pVM->pgm.s.HCPhysInterPD != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPD & PAGE_OFFSET_MASK));
1523	pVM->pgm.s.HCPhysInterPaePDPT = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT);
1524	AssertRelease(pVM->pgm.s.HCPhysInterPaePDPT != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePDPT & PAGE_OFFSET_MASK));
1525	pVM->pgm.s.HCPhysInterPaePML4 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePML4);
1526	AssertRelease(pVM->pgm.s.HCPhysInterPaePML4 != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePML4 & PAGE_OFFSET_MASK) && pVM->pgm.s.HCPhysInterPaePML4 < 0xffffffff);
1527
1528	/*
1529	* Initialize the pages, setting up the PML4 and PDPT for repetitive 4GB action.
1530	*/
1531	ASMMemZeroPage(pVM->pgm.s.pInterPD);
1532	ASMMemZeroPage(pVM->pgm.s.apInterPTs[0]);
1533	ASMMemZeroPage(pVM->pgm.s.apInterPTs[1]);
1534
1535	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[0]);
1536	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[1]);
1537
1538	ASMMemZeroPage(pVM->pgm.s.pInterPaePDPT);
1539	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.apInterPaePDs); i++)
1540	{
1541	ASMMemZeroPage(pVM->pgm.s.apInterPaePDs[i]);
1542	pVM->pgm.s.pInterPaePDPT->a[i].u = X86_PDPE_P \| PGM_PLXFLAGS_PERMANENT
1543	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[i]);
1544	}
1545
1546	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.pInterPaePDPT64->a); i++)
1547	{
1548	const unsigned iPD = i % RT_ELEMENTS(pVM->pgm.s.apInterPaePDs);
1549	pVM->pgm.s.pInterPaePDPT64->a[i].u = X86_PDPE_P \| X86_PDPE_RW \| X86_PDPE_US \| X86_PDPE_A \| PGM_PLXFLAGS_PERMANENT
1550	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[iPD]);
1551	}
1552
1553	RTHCPHYS HCPhysInterPaePDPT64 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64);
1554	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.pInterPaePML4->a); i++)
1555	pVM->pgm.s.pInterPaePML4->a[i].u = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_US \| X86_PML4E_A \| PGM_PLXFLAGS_PERMANENT
1556	\| HCPhysInterPaePDPT64;
1557
1558	/*
1559	* Initialize paging workers and mode from current host mode
1560	* and the guest running in real mode.
1561	*/
1562	pVM->pgm.s.enmHostMode = SUPR3GetPagingMode();
1563	switch (pVM->pgm.s.enmHostMode)
1564	{
1565	case SUPPAGINGMODE_32_BIT:
1566	case SUPPAGINGMODE_32_BIT_GLOBAL:
1567	case SUPPAGINGMODE_PAE:
1568	case SUPPAGINGMODE_PAE_GLOBAL:
1569	case SUPPAGINGMODE_PAE_NX:
1570	case SUPPAGINGMODE_PAE_GLOBAL_NX:
1571	break;
1572
1573	case SUPPAGINGMODE_AMD64:
1574	case SUPPAGINGMODE_AMD64_GLOBAL:
1575	case SUPPAGINGMODE_AMD64_NX:
1576	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
1577	#ifndef VBOX_WITH_HYBRID_32BIT_KERNEL
1578	if (ARCH_BITS != 64)
1579	{
1580	AssertMsgFailed(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1581	LogRel(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1582	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1583	}
1584	#endif
1585	break;
1586	default:
1587	AssertMsgFailed(("Host mode %d is not supported\n", pVM->pgm.s.enmHostMode));
1588	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1589	}
1590	rc = pgmR3ModeDataInit(pVM, false /* don't resolve GC and R0 syms yet */);
1591	if (RT_SUCCESS(rc))
1592	{
1593	LogFlow(("pgmR3InitPaging: returns successfully\n"));
1594	#if HC_ARCH_BITS == 64
1595	LogRel(("Debug: HCPhysInterPD=%RHp HCPhysInterPaePDPT=%RHp HCPhysInterPaePML4=%RHp\n",
1596	pVM->pgm.s.HCPhysInterPD, pVM->pgm.s.HCPhysInterPaePDPT, pVM->pgm.s.HCPhysInterPaePML4));
1597	LogRel(("Debug: apInterPTs={%RHp,%RHp} apInterPaePTs={%RHp,%RHp} apInterPaePDs={%RHp,%RHp,%RHp,%RHp} pInterPaePDPT64=%RHp\n",
1598	MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[1]),
1599	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[1]),
1600	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[1]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[2]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[3]),
1601	MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64)));
1602	#endif
1603
1604	/*
1605	* Log the host paging mode. It may come in handy.
1606	*/
1607	const char *pszHostMode;
1608	switch (pVM->pgm.s.enmHostMode)
1609	{
1610	case SUPPAGINGMODE_32_BIT: pszHostMode = "32-bit"; break;
1611	case SUPPAGINGMODE_32_BIT_GLOBAL: pszHostMode = "32-bit+PGE"; break;
1612	case SUPPAGINGMODE_PAE: pszHostMode = "PAE"; break;
1613	case SUPPAGINGMODE_PAE_GLOBAL: pszHostMode = "PAE+PGE"; break;
1614	case SUPPAGINGMODE_PAE_NX: pszHostMode = "PAE+NXE"; break;
1615	case SUPPAGINGMODE_PAE_GLOBAL_NX: pszHostMode = "PAE+PGE+NXE"; break;
1616	case SUPPAGINGMODE_AMD64: pszHostMode = "AMD64"; break;
1617	case SUPPAGINGMODE_AMD64_GLOBAL: pszHostMode = "AMD64+PGE"; break;
1618	case SUPPAGINGMODE_AMD64_NX: pszHostMode = "AMD64+NX"; break;
1619	case SUPPAGINGMODE_AMD64_GLOBAL_NX: pszHostMode = "AMD64+PGE+NX"; break;
1620	default: pszHostMode = "???"; break;
1621	}
1622	LogRel(("Host paging mode: %s\n", pszHostMode));
1623
1624	return VINF_SUCCESS;
1625	}
1626
1627	LogFlow(("pgmR3InitPaging: returns %Rrc\n", rc));
1628	return rc;
1629	}
1630
1631
1632	/**
1633	* Init statistics
1634	* @returns VBox status code.
1635	*/
1636	static int pgmR3InitStats(PVM pVM)
1637	{
1638	PPGM pPGM = &pVM->pgm.s;
1639	int rc;
1640
1641	/*
1642	* Release statistics.
1643	*/
1644	/* Common - misc variables */
1645	STAM_REL_REG(pVM, &pPGM->cAllPages, STAMTYPE_U32, "/PGM/Page/cAllPages", STAMUNIT_COUNT, "The total number of pages.");
1646	STAM_REL_REG(pVM, &pPGM->cPrivatePages, STAMTYPE_U32, "/PGM/Page/cPrivatePages", STAMUNIT_COUNT, "The number of private pages.");
1647	STAM_REL_REG(pVM, &pPGM->cSharedPages, STAMTYPE_U32, "/PGM/Page/cSharedPages", STAMUNIT_COUNT, "The number of shared pages.");
1648	STAM_REL_REG(pVM, &pPGM->cReusedSharedPages, STAMTYPE_U32, "/PGM/Page/cReusedSharedPages", STAMUNIT_COUNT, "The number of reused shared pages.");
1649	STAM_REL_REG(pVM, &pPGM->cZeroPages, STAMTYPE_U32, "/PGM/Page/cZeroPages", STAMUNIT_COUNT, "The number of zero backed pages.");
1650	STAM_REL_REG(pVM, &pPGM->cPureMmioPages, STAMTYPE_U32, "/PGM/Page/cPureMmioPages", STAMUNIT_COUNT, "The number of pure MMIO pages.");
1651	STAM_REL_REG(pVM, &pPGM->cMonitoredPages, STAMTYPE_U32, "/PGM/Page/cMonitoredPages", STAMUNIT_COUNT, "The number of write monitored pages.");
1652	STAM_REL_REG(pVM, &pPGM->cWrittenToPages, STAMTYPE_U32, "/PGM/Page/cWrittenToPages", STAMUNIT_COUNT, "The number of previously write monitored pages that have been written to.");
1653	STAM_REL_REG(pVM, &pPGM->cWriteLockedPages, STAMTYPE_U32, "/PGM/Page/cWriteLockedPages", STAMUNIT_COUNT, "The number of write(/read) locked pages.");
1654	STAM_REL_REG(pVM, &pPGM->cReadLockedPages, STAMTYPE_U32, "/PGM/Page/cReadLockedPages", STAMUNIT_COUNT, "The number of read (only) locked pages.");
1655	STAM_REL_REG(pVM, &pPGM->cBalloonedPages, STAMTYPE_U32, "/PGM/Page/cBalloonedPages", STAMUNIT_COUNT, "The number of ballooned pages.");
1656	STAM_REL_REG(pVM, &pPGM->cHandyPages, STAMTYPE_U32, "/PGM/Page/cHandyPages", STAMUNIT_COUNT, "The number of handy pages (not included in cAllPages).");
1657	STAM_REL_REG(pVM, &pPGM->cLargePages, STAMTYPE_U32, "/PGM/Page/cLargePages", STAMUNIT_COUNT, "The number of large pages allocated (includes disabled).");
1658	STAM_REL_REG(pVM, &pPGM->cLargePagesDisabled, STAMTYPE_U32, "/PGM/Page/cLargePagesDisabled", STAMUNIT_COUNT, "The number of disabled large pages.");
1659	STAM_REL_REG(pVM, &pPGM->cRelocations, STAMTYPE_COUNTER, "/PGM/cRelocations", STAMUNIT_OCCURENCES,"Number of hypervisor relocations.");
1660	STAM_REL_REG(pVM, &pPGM->ChunkR3Map.c, STAMTYPE_U32, "/PGM/ChunkR3Map/c", STAMUNIT_COUNT, "Number of mapped chunks.");
1661	STAM_REL_REG(pVM, &pPGM->ChunkR3Map.cMax, STAMTYPE_U32, "/PGM/ChunkR3Map/cMax", STAMUNIT_COUNT, "Maximum number of mapped chunks.");
1662	STAM_REL_REG(pVM, &pPGM->cMappedChunks, STAMTYPE_U32, "/PGM/ChunkR3Map/Mapped", STAMUNIT_COUNT, "Number of times we mapped a chunk.");
1663	STAM_REL_REG(pVM, &pPGM->cUnmappedChunks, STAMTYPE_U32, "/PGM/ChunkR3Map/Unmapped", STAMUNIT_COUNT, "Number of times we unmapped a chunk.");
1664
1665	STAM_REL_REG(pVM, &pPGM->StatLargePageReused, STAMTYPE_COUNTER, "/PGM/LargePage/Reused", STAMUNIT_OCCURENCES, "The number of times we've reused a large page.");
1666	STAM_REL_REG(pVM, &pPGM->StatLargePageRefused, STAMTYPE_COUNTER, "/PGM/LargePage/Refused", STAMUNIT_OCCURENCES, "The number of times we couldn't use a large page.");
1667	STAM_REL_REG(pVM, &pPGM->StatLargePageRecheck, STAMTYPE_COUNTER, "/PGM/LargePage/Recheck", STAMUNIT_OCCURENCES, "The number of times we've rechecked a disabled large page.");
1668
1669	/* Live save */
1670	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.fActive, STAMTYPE_U8, "/PGM/LiveSave/fActive", STAMUNIT_COUNT, "Active or not.");
1671	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cIgnoredPages, STAMTYPE_U32, "/PGM/LiveSave/cIgnoredPages", STAMUNIT_COUNT, "The number of ignored pages in the RAM ranges (i.e. MMIO, MMIO2 and ROM).");
1672	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cDirtyPagesLong, STAMTYPE_U32, "/PGM/LiveSave/cDirtyPagesLong", STAMUNIT_COUNT, "Longer term dirty page average.");
1673	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cDirtyPagesShort, STAMTYPE_U32, "/PGM/LiveSave/cDirtyPagesShort", STAMUNIT_COUNT, "Short term dirty page average.");
1674	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cPagesPerSecond, STAMTYPE_U32, "/PGM/LiveSave/cPagesPerSecond", STAMUNIT_COUNT, "Pages per second.");
1675	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cSavedPages, STAMTYPE_U64, "/PGM/LiveSave/cSavedPages", STAMUNIT_COUNT, "The total number of saved pages.");
1676	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cReadPages", STAMUNIT_COUNT, "RAM: Ready pages.");
1677	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cDirtyPages", STAMUNIT_COUNT, "RAM: Dirty pages.");
1678	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cZeroPages", STAMUNIT_COUNT, "RAM: Ready zero pages.");
1679	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cMonitoredPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cMonitoredPages", STAMUNIT_COUNT, "RAM: Write monitored pages.");
1680	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cReadPages", STAMUNIT_COUNT, "ROM: Ready pages.");
1681	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cDirtyPages", STAMUNIT_COUNT, "ROM: Dirty pages.");
1682	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cZeroPages", STAMUNIT_COUNT, "ROM: Ready zero pages.");
1683	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cMonitoredPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cMonitoredPages", STAMUNIT_COUNT, "ROM: Write monitored pages.");
1684	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cReadPages", STAMUNIT_COUNT, "MMIO2: Ready pages.");
1685	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cDirtyPages", STAMUNIT_COUNT, "MMIO2: Dirty pages.");
1686	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cZeroPages", STAMUNIT_COUNT, "MMIO2: Ready zero pages.");
1687	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cMonitoredPages,STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cMonitoredPages",STAMUNIT_COUNT, "MMIO2: Write monitored pages.");
1688
1689	#ifdef VBOX_WITH_STATISTICS
1690
1691	# define PGM_REG_COUNTER(a, b, c) \
1692	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_OCCURENCES, c, b); \
1693	AssertRC(rc);
1694
1695	# define PGM_REG_COUNTER_BYTES(a, b, c) \
1696	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_BYTES, c, b); \
1697	AssertRC(rc);
1698
1699	# define PGM_REG_PROFILE(a, b, c) \
1700	rc = STAMR3RegisterF(pVM, a, STAMTYPE_PROFILE, STAMVISIBILITY_ALWAYS, STAMUNIT_TICKS_PER_CALL, c, b); \
1701	AssertRC(rc);
1702
1703	PGMSTATS *pStats = pVM->pgm.s.pStatsR3;
1704
1705	PGM_REG_PROFILE(&pStats->StatAllocLargePage, "/PGM/LargePage/Prof/Alloc", "Time spent by the host OS for large page allocation.");
1706	PGM_REG_PROFILE(&pStats->StatClearLargePage, "/PGM/LargePage/Prof/Clear", "Time spent clearing the newly allocated large pages.");
1707	PGM_REG_COUNTER(&pStats->StatLargePageOverflow, "/PGM/LargePage/Overflow", "The number of times allocating a large page took too long.");
1708	PGM_REG_PROFILE(&pStats->StatR3IsValidLargePage, "/PGM/LargePage/Prof/R3/IsValid", "pgmPhysIsValidLargePage profiling - R3.");
1709	PGM_REG_PROFILE(&pStats->StatRZIsValidLargePage, "/PGM/LargePage/Prof/RZ/IsValid", "pgmPhysIsValidLargePage profiling - RZ.");
1710
1711	PGM_REG_COUNTER(&pStats->StatR3DetectedConflicts, "/PGM/R3/DetectedConflicts", "The number of times PGMR3CheckMappingConflicts() detected a conflict.");
1712	PGM_REG_PROFILE(&pStats->StatR3ResolveConflict, "/PGM/R3/ResolveConflict", "pgmR3SyncPTResolveConflict() profiling (includes the entire relocation).");
1713	PGM_REG_COUNTER(&pStats->StatR3PhysRead, "/PGM/R3/Phys/Read", "The number of times PGMPhysRead was called.");
1714	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysReadBytes, "/PGM/R3/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1715	PGM_REG_COUNTER(&pStats->StatR3PhysWrite, "/PGM/R3/Phys/Write", "The number of times PGMPhysWrite was called.");
1716	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysWriteBytes, "/PGM/R3/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1717	PGM_REG_COUNTER(&pStats->StatR3PhysSimpleRead, "/PGM/R3/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1718	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysSimpleReadBytes, "/PGM/R3/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1719	PGM_REG_COUNTER(&pStats->StatR3PhysSimpleWrite, "/PGM/R3/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1720	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysSimpleWriteBytes, "/PGM/R3/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1721
1722	PGM_REG_COUNTER(&pStats->StatRZChunkR3MapTlbHits, "/PGM/ChunkR3Map/TlbHitsRZ", "TLB hits.");
1723	PGM_REG_COUNTER(&pStats->StatRZChunkR3MapTlbMisses, "/PGM/ChunkR3Map/TlbMissesRZ", "TLB misses.");
1724	PGM_REG_PROFILE(&pStats->StatChunkAging, "/PGM/ChunkR3Map/Map/Aging", "Chunk aging profiling.");
1725	PGM_REG_PROFILE(&pStats->StatChunkFindCandidate, "/PGM/ChunkR3Map/Map/Find", "Chunk unmap find profiling.");
1726	PGM_REG_PROFILE(&pStats->StatChunkUnmap, "/PGM/ChunkR3Map/Map/Unmap", "Chunk unmap of address space profiling.");
1727	PGM_REG_PROFILE(&pStats->StatChunkMap, "/PGM/ChunkR3Map/Map/Map", "Chunk map of address space profiling.");
1728
1729	PGM_REG_COUNTER(&pStats->StatRZPageMapTlbHits, "/PGM/RZ/Page/MapTlbHits", "TLB hits.");
1730	PGM_REG_COUNTER(&pStats->StatRZPageMapTlbMisses, "/PGM/RZ/Page/MapTlbMisses", "TLB misses.");
1731	PGM_REG_COUNTER(&pStats->StatR3ChunkR3MapTlbHits, "/PGM/ChunkR3Map/TlbHitsR3", "TLB hits.");
1732	PGM_REG_COUNTER(&pStats->StatR3ChunkR3MapTlbMisses, "/PGM/ChunkR3Map/TlbMissesR3", "TLB misses.");
1733	PGM_REG_COUNTER(&pStats->StatR3PageMapTlbHits, "/PGM/R3/Page/MapTlbHits", "TLB hits.");
1734	PGM_REG_COUNTER(&pStats->StatR3PageMapTlbMisses, "/PGM/R3/Page/MapTlbMisses", "TLB misses.");
1735	PGM_REG_COUNTER(&pStats->StatPageMapTlbFlushes, "/PGM/R3/Page/MapTlbFlushes", "TLB flushes (all contexts).");
1736	PGM_REG_COUNTER(&pStats->StatPageMapTlbFlushEntry, "/PGM/R3/Page/MapTlbFlushEntry", "TLB entry flushes (all contexts).");
1737
1738	PGM_REG_COUNTER(&pStats->StatRZRamRangeTlbHits, "/PGM/RZ/RamRange/TlbHits", "TLB hits.");
1739	PGM_REG_COUNTER(&pStats->StatRZRamRangeTlbMisses, "/PGM/RZ/RamRange/TlbMisses", "TLB misses.");
1740	PGM_REG_COUNTER(&pStats->StatR3RamRangeTlbHits, "/PGM/R3/RamRange/TlbHits", "TLB hits.");
1741	PGM_REG_COUNTER(&pStats->StatR3RamRangeTlbMisses, "/PGM/R3/RamRange/TlbMisses", "TLB misses.");
1742
1743	PGM_REG_PROFILE(&pStats->StatRZSyncCR3HandlerVirtualUpdate, "/PGM/RZ/SyncCR3/Handlers/VirtualUpdate", "Profiling of the virtual handler updates.");
1744	PGM_REG_PROFILE(&pStats->StatRZSyncCR3HandlerVirtualReset, "/PGM/RZ/SyncCR3/Handlers/VirtualReset", "Profiling of the virtual handler resets.");
1745	PGM_REG_PROFILE(&pStats->StatR3SyncCR3HandlerVirtualUpdate, "/PGM/R3/SyncCR3/Handlers/VirtualUpdate", "Profiling of the virtual handler updates.");
1746	PGM_REG_PROFILE(&pStats->StatR3SyncCR3HandlerVirtualReset, "/PGM/R3/SyncCR3/Handlers/VirtualReset", "Profiling of the virtual handler resets.");
1747
1748	PGM_REG_COUNTER(&pStats->StatRZPhysHandlerReset, "/PGM/RZ/PhysHandlerReset", "The number of times PGMHandlerPhysicalReset is called.");
1749	PGM_REG_COUNTER(&pStats->StatR3PhysHandlerReset, "/PGM/R3/PhysHandlerReset", "The number of times PGMHandlerPhysicalReset is called.");
1750	PGM_REG_COUNTER(&pStats->StatRZPhysHandlerLookupHits, "/PGM/RZ/PhysHandlerLookupHits", "The number of cache hits when looking up physical handlers.");
1751	PGM_REG_COUNTER(&pStats->StatR3PhysHandlerLookupHits, "/PGM/R3/PhysHandlerLookupHits", "The number of cache hits when looking up physical handlers.");
1752	PGM_REG_COUNTER(&pStats->StatRZPhysHandlerLookupMisses, "/PGM/RZ/PhysHandlerLookupMisses", "The number of cache misses when looking up physical handlers.");
1753	PGM_REG_COUNTER(&pStats->StatR3PhysHandlerLookupMisses, "/PGM/R3/PhysHandlerLookupMisses", "The number of cache misses when looking up physical handlers.");
1754	PGM_REG_PROFILE(&pStats->StatRZVirtHandlerSearchByPhys, "/PGM/RZ/VirtHandlerSearchByPhys", "Profiling of pgmHandlerVirtualFindByPhysAddr.");
1755	PGM_REG_PROFILE(&pStats->StatR3VirtHandlerSearchByPhys, "/PGM/R3/VirtHandlerSearchByPhys", "Profiling of pgmHandlerVirtualFindByPhysAddr.");
1756
1757	PGM_REG_COUNTER(&pStats->StatRZPageReplaceShared, "/PGM/RZ/Page/ReplacedShared", "Times a shared page was replaced.");
1758	PGM_REG_COUNTER(&pStats->StatRZPageReplaceZero, "/PGM/RZ/Page/ReplacedZero", "Times the zero page was replaced.");
1759	/// @todo PGM_REG_COUNTER(&pStats->StatRZPageHandyAllocs, "/PGM/RZ/Page/HandyAllocs", "Number of times we've allocated more handy pages.");
1760	PGM_REG_COUNTER(&pStats->StatR3PageReplaceShared, "/PGM/R3/Page/ReplacedShared", "Times a shared page was replaced.");
1761	PGM_REG_COUNTER(&pStats->StatR3PageReplaceZero, "/PGM/R3/Page/ReplacedZero", "Times the zero page was replaced.");
1762	/// @todo PGM_REG_COUNTER(&pStats->StatR3PageHandyAllocs, "/PGM/R3/Page/HandyAllocs", "Number of times we've allocated more handy pages.");
1763
1764	PGM_REG_COUNTER(&pStats->StatRZPhysRead, "/PGM/RZ/Phys/Read", "The number of times PGMPhysRead was called.");
1765	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysReadBytes, "/PGM/RZ/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1766	PGM_REG_COUNTER(&pStats->StatRZPhysWrite, "/PGM/RZ/Phys/Write", "The number of times PGMPhysWrite was called.");
1767	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysWriteBytes, "/PGM/RZ/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1768	PGM_REG_COUNTER(&pStats->StatRZPhysSimpleRead, "/PGM/RZ/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1769	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysSimpleReadBytes, "/PGM/RZ/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1770	PGM_REG_COUNTER(&pStats->StatRZPhysSimpleWrite, "/PGM/RZ/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1771	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysSimpleWriteBytes, "/PGM/RZ/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1772
1773	/* GC only: */
1774	PGM_REG_COUNTER(&pStats->StatRCInvlPgConflict, "/PGM/RC/InvlPgConflict", "Number of times PGMInvalidatePage() detected a mapping conflict.");
1775	PGM_REG_COUNTER(&pStats->StatRCInvlPgSyncMonCR3, "/PGM/RC/InvlPgSyncMonitorCR3", "Number of times PGMInvalidatePage() ran into PGM_SYNC_MONITOR_CR3.");
1776
1777	PGM_REG_COUNTER(&pStats->StatRCPhysRead, "/PGM/RC/Phys/Read", "The number of times PGMPhysRead was called.");
1778	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysReadBytes, "/PGM/RC/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1779	PGM_REG_COUNTER(&pStats->StatRCPhysWrite, "/PGM/RC/Phys/Write", "The number of times PGMPhysWrite was called.");
1780	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysWriteBytes, "/PGM/RC/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1781	PGM_REG_COUNTER(&pStats->StatRCPhysSimpleRead, "/PGM/RC/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1782	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysSimpleReadBytes, "/PGM/RC/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1783	PGM_REG_COUNTER(&pStats->StatRCPhysSimpleWrite, "/PGM/RC/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1784	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysSimpleWriteBytes, "/PGM/RC/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1785
1786	PGM_REG_COUNTER(&pStats->StatTrackVirgin, "/PGM/Track/Virgin", "The number of first time shadowings");
1787	PGM_REG_COUNTER(&pStats->StatTrackAliased, "/PGM/Track/Aliased", "The number of times switching to cRef2, i.e. the page is being shadowed by two PTs.");
1788	PGM_REG_COUNTER(&pStats->StatTrackAliasedMany, "/PGM/Track/AliasedMany", "The number of times we're tracking using cRef2.");
1789	PGM_REG_COUNTER(&pStats->StatTrackAliasedLots, "/PGM/Track/AliasedLots", "The number of times we're hitting pages which has overflowed cRef2");
1790	PGM_REG_COUNTER(&pStats->StatTrackOverflows, "/PGM/Track/Overflows", "The number of times the extent list grows too long.");
1791	PGM_REG_COUNTER(&pStats->StatTrackNoExtentsLeft, "/PGM/Track/NoExtentLeft", "The number of times the extent list was exhausted.");
1792	PGM_REG_PROFILE(&pStats->StatTrackDeref, "/PGM/Track/Deref", "Profiling of SyncPageWorkerTrackDeref (expensive).");
1793
1794	# undef PGM_REG_COUNTER
1795	# undef PGM_REG_PROFILE
1796	#endif
1797
1798	/*
1799	* Note! The layout below matches the member layout exactly!
1800	*/
1801
1802	/*
1803	* Common - stats
1804	*/
1805	for (VMCPUID idCpu = 0; idCpu < pVM->cCpus; idCpu++)
1806	{
1807	PPGMCPU pPgmCpu = &pVM->aCpus[idCpu].pgm.s;
1808
1809	#define PGM_REG_COUNTER(a, b, c) \
1810	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_OCCURENCES, c, b, idCpu); \
1811	AssertRC(rc);
1812	#define PGM_REG_PROFILE(a, b, c) \
1813	rc = STAMR3RegisterF(pVM, a, STAMTYPE_PROFILE, STAMVISIBILITY_ALWAYS, STAMUNIT_TICKS_PER_CALL, c, b, idCpu); \
1814	AssertRC(rc);
1815
1816	PGM_REG_COUNTER(&pPgmCpu->cGuestModeChanges, "/PGM/CPU%u/cGuestModeChanges", "Number of guest mode changes.");
1817
1818	#ifdef VBOX_WITH_STATISTICS
1819	PGMCPUSTATS *pCpuStats = pVM->aCpus[idCpu].pgm.s.pStatsR3;
1820
1821	# if 0 /* rarely useful; leave for debugging. */
1822	for (unsigned j = 0; j < RT_ELEMENTS(pPgmCpu->StatSyncPtPD); j++)
1823	STAMR3RegisterF(pVM, &pCpuStats->StatSyncPtPD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1824	"The number of SyncPT per PD n.", "/PGM/CPU%u/PDSyncPT/%04X", i, j);
1825	for (unsigned j = 0; j < RT_ELEMENTS(pCpuStats->StatSyncPagePD); j++)
1826	STAMR3RegisterF(pVM, &pCpuStats->StatSyncPagePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1827	"The number of SyncPage per PD n.", "/PGM/CPU%u/PDSyncPage/%04X", i, j);
1828	# endif
1829	/* R0 only: */
1830	PGM_REG_PROFILE(&pCpuStats->StatR0NpMiscfg, "/PGM/CPU%u/R0/NpMiscfg", "PGMR0Trap0eHandlerNPMisconfig() profiling.");
1831	PGM_REG_COUNTER(&pCpuStats->StatR0NpMiscfgSyncPage, "/PGM/CPU%u/R0/NpMiscfgSyncPage", "SyncPage calls from PGMR0Trap0eHandlerNPMisconfig().");
1832
1833	/* RZ only: */
1834	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0e, "/PGM/CPU%u/RZ/Trap0e", "Profiling of the PGMTrap0eHandler() body.");
1835	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Ballooned, "/PGM/CPU%u/RZ/Trap0e/Time2/Ballooned", "Profiling of the Trap0eHandler body when the cause is read access to a ballooned page.");
1836	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2CSAM, "/PGM/CPU%u/RZ/Trap0e/Time2/CSAM", "Profiling of the Trap0eHandler body when the cause is CSAM.");
1837	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2DirtyAndAccessed, "/PGM/CPU%u/RZ/Trap0e/Time2/DirtyAndAccessedBits", "Profiling of the Trap0eHandler body when the cause is dirty and/or accessed bit emulation.");
1838	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2GuestTrap, "/PGM/CPU%u/RZ/Trap0e/Time2/GuestTrap", "Profiling of the Trap0eHandler body when the cause is a guest trap.");
1839	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2HndPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerPhysical", "Profiling of the Trap0eHandler body when the cause is a physical handler.");
1840	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2HndVirt, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerVirtual", "Profiling of the Trap0eHandler body when the cause is a virtual handler.");
1841	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2HndUnhandled, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerUnhandled", "Profiling of the Trap0eHandler body when the cause is access outside the monitored areas of a monitored page.");
1842	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2InvalidPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/InvalidPhys", "Profiling of the Trap0eHandler body when the cause is access to an invalid physical guest address.");
1843	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2MakeWritable, "/PGM/CPU%u/RZ/Trap0e/Time2/MakeWritable", "Profiling of the Trap0eHandler body when the cause is that a page needed to be made writeable.");
1844	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Mapping, "/PGM/CPU%u/RZ/Trap0e/Time2/Mapping", "Profiling of the Trap0eHandler body when the cause is related to the guest mappings.");
1845	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Misc, "/PGM/CPU%u/RZ/Trap0e/Time2/Misc", "Profiling of the Trap0eHandler body when the cause is not known.");
1846	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSync, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSync", "Profiling of the Trap0eHandler body when the cause is an out-of-sync page.");
1847	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSyncHndPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncHndPhys", "Profiling of the Trap0eHandler body when the cause is an out-of-sync physical handler page.");
1848	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSyncHndVirt, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncHndVirt", "Profiling of the Trap0eHandler body when the cause is an out-of-sync virtual handler page.");
1849	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSyncHndObs, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncObsHnd", "Profiling of the Trap0eHandler body when the cause is an obsolete handler page.");
1850	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2SyncPT, "/PGM/CPU%u/RZ/Trap0e/Time2/SyncPT", "Profiling of the Trap0eHandler body when the cause is lazy syncing of a PT.");
1851	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2WPEmulation, "/PGM/CPU%u/RZ/Trap0e/Time2/WPEmulation", "Profiling of the Trap0eHandler body when the cause is CR0.WP emulation.");
1852	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eConflicts, "/PGM/CPU%u/RZ/Trap0e/Conflicts", "The number of times #PF was caused by an undetected conflict.");
1853	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersMapping, "/PGM/CPU%u/RZ/Trap0e/Handlers/Mapping", "Number of traps due to access handlers in mappings.");
1854	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersOutOfSync, "/PGM/CPU%u/RZ/Trap0e/Handlers/OutOfSync", "Number of traps due to out-of-sync handled pages.");
1855	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersPhysAll, "/PGM/CPU%u/RZ/Trap0e/Handlers/PhysAll", "Number of traps due to physical all-access handlers.");
1856	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersPhysAllOpt, "/PGM/CPU%u/RZ/Trap0e/Handlers/PhysAllOpt", "Number of the physical all-access handler traps using the optimization.");
1857	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersPhysWrite, "/PGM/CPU%u/RZ/Trap0e/Handlers/PhysWrite", "Number of traps due to physical write-access handlers.");
1858	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersVirtual, "/PGM/CPU%u/RZ/Trap0e/Handlers/Virtual", "Number of traps due to virtual access handlers.");
1859	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersVirtualByPhys, "/PGM/CPU%u/RZ/Trap0e/Handlers/VirtualByPhys", "Number of traps due to virtual access handlers by physical address.");
1860	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersVirtualUnmarked,"/PGM/CPU%u/RZ/Trap0e/Handlers/VirtualUnmarked","Number of traps due to virtual access handlers by virtual address (without proper physical flags).");
1861	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersUnhandled, "/PGM/CPU%u/RZ/Trap0e/Handlers/Unhandled", "Number of traps due to access outside range of monitored page(s).");
1862	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersInvalid, "/PGM/CPU%u/RZ/Trap0e/Handlers/Invalid", "Number of traps due to access to invalid physical memory.");
1863	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSNotPresentRead, "/PGM/CPU%u/RZ/Trap0e/Err/User/NPRead", "Number of user mode not present read page faults.");
1864	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSNotPresentWrite, "/PGM/CPU%u/RZ/Trap0e/Err/User/NPWrite", "Number of user mode not present write page faults.");
1865	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSWrite, "/PGM/CPU%u/RZ/Trap0e/Err/User/Write", "Number of user mode write page faults.");
1866	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSReserved, "/PGM/CPU%u/RZ/Trap0e/Err/User/Reserved", "Number of user mode reserved bit page faults.");
1867	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSNXE, "/PGM/CPU%u/RZ/Trap0e/Err/User/NXE", "Number of user mode NXE page faults.");
1868	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSRead, "/PGM/CPU%u/RZ/Trap0e/Err/User/Read", "Number of user mode read page faults.");
1869	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVNotPresentRead, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NPRead", "Number of supervisor mode not present read page faults.");
1870	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVNotPresentWrite, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NPWrite", "Number of supervisor mode not present write page faults.");
1871	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVWrite, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/Write", "Number of supervisor mode write page faults.");
1872	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVReserved, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/Reserved", "Number of supervisor mode reserved bit page faults.");
1873	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSNXE, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NXE", "Number of supervisor mode NXE page faults.");
1874	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eGuestPF, "/PGM/CPU%u/RZ/Trap0e/GuestPF", "Number of real guest page faults.");
1875	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eGuestPFMapping, "/PGM/CPU%u/RZ/Trap0e/GuestPF/InMapping", "Number of real guest page faults in a mapping.");
1876	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eWPEmulInRZ, "/PGM/CPU%u/RZ/Trap0e/WP/InRZ", "Number of guest page faults due to X86_CR0_WP emulation.");
1877	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eWPEmulToR3, "/PGM/CPU%u/RZ/Trap0e/WP/ToR3", "Number of guest page faults due to X86_CR0_WP emulation (forward to R3 for emulation).");
1878	#if 0 /* rarely useful; leave for debugging. */
1879	for (unsigned j = 0; j < RT_ELEMENTS(pCpuStats->StatRZTrap0ePD); j++)
1880	STAMR3RegisterF(pVM, &pCpuStats->StatRZTrap0ePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1881	"The number of traps in page directory n.", "/PGM/CPU%u/RZ/Trap0e/PD/%04X", i, j);
1882	#endif
1883	PGM_REG_COUNTER(&pCpuStats->StatRZGuestCR3WriteHandled, "/PGM/CPU%u/RZ/CR3WriteHandled", "The number of times the Guest CR3 change was successfully handled.");
1884	PGM_REG_COUNTER(&pCpuStats->StatRZGuestCR3WriteUnhandled, "/PGM/CPU%u/RZ/CR3WriteUnhandled", "The number of times the Guest CR3 change was passed back to the recompiler.");
1885	PGM_REG_COUNTER(&pCpuStats->StatRZGuestCR3WriteConflict, "/PGM/CPU%u/RZ/CR3WriteConflict", "The number of times the Guest CR3 monitoring detected a conflict.");
1886	PGM_REG_COUNTER(&pCpuStats->StatRZGuestROMWriteHandled, "/PGM/CPU%u/RZ/ROMWriteHandled", "The number of times the Guest ROM change was successfully handled.");
1887	PGM_REG_COUNTER(&pCpuStats->StatRZGuestROMWriteUnhandled, "/PGM/CPU%u/RZ/ROMWriteUnhandled", "The number of times the Guest ROM change was passed back to the recompiler.");
1888
1889	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapMigrateInvlPg, "/PGM/CPU%u/RZ/DynMap/MigrateInvlPg", "invlpg count in PGMR0DynMapMigrateAutoSet.");
1890	PGM_REG_PROFILE(&pCpuStats->StatRZDynMapGCPageInl, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl", "Calls to pgmR0DynMapGCPageInlined.");
1891	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlHits, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/Hits", "Hash table lookup hits.");
1892	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlMisses, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/Misses", "Misses that falls back to the code common.");
1893	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlRamHits, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/RamHits", "1st ram range hits.");
1894	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlRamMisses, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/RamMisses", "1st ram range misses, takes slow path.");
1895	PGM_REG_PROFILE(&pCpuStats->StatRZDynMapHCPageInl, "/PGM/CPU%u/RZ/DynMap/PageHCPageInl", "Calls to pgmRZDynMapHCPageInlined.");
1896	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapHCPageInlHits, "/PGM/CPU%u/RZ/DynMap/PageHCPageInl/Hits", "Hash table lookup hits.");
1897	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapHCPageInlMisses, "/PGM/CPU%u/RZ/DynMap/PageHCPageInl/Misses", "Misses that falls back to the code common.");
1898	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPage, "/PGM/CPU%u/RZ/DynMap/Page", "Calls to pgmR0DynMapPage");
1899	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetOptimize, "/PGM/CPU%u/RZ/DynMap/Page/SetOptimize", "Calls to pgmRZDynMapOptimizeAutoSet.");
1900	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetSearchFlushes, "/PGM/CPU%u/RZ/DynMap/Page/SetSearchFlushes", "Set search restoring to subset flushes.");
1901	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetSearchHits, "/PGM/CPU%u/RZ/DynMap/Page/SetSearchHits", "Set search hits.");
1902	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetSearchMisses, "/PGM/CPU%u/RZ/DynMap/Page/SetSearchMisses", "Set search misses.");
1903	PGM_REG_PROFILE(&pCpuStats->StatRZDynMapHCPage, "/PGM/CPU%u/RZ/DynMap/Page/HCPage", "Calls to pgmRZDynMapHCPageCommon (ring-0).");
1904	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageHits0, "/PGM/CPU%u/RZ/DynMap/Page/Hits0", "Hits at iPage+0");
1905	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageHits1, "/PGM/CPU%u/RZ/DynMap/Page/Hits1", "Hits at iPage+1");
1906	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageHits2, "/PGM/CPU%u/RZ/DynMap/Page/Hits2", "Hits at iPage+2");
1907	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageInvlPg, "/PGM/CPU%u/RZ/DynMap/Page/InvlPg", "invlpg count in pgmR0DynMapPageSlow.");
1908	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlow, "/PGM/CPU%u/RZ/DynMap/Page/Slow", "Calls to pgmR0DynMapPageSlow - subtract this from pgmR0DynMapPage to get 1st level hits.");
1909	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlowLoopHits, "/PGM/CPU%u/RZ/DynMap/Page/SlowLoopHits" , "Hits in the loop path.");
1910	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlowLoopMisses, "/PGM/CPU%u/RZ/DynMap/Page/SlowLoopMisses", "Misses in the loop path. NonLoopMisses = Slow - SlowLoopHit - SlowLoopMisses");
1911	//PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlowLostHits, "/PGM/CPU%u/R0/DynMap/Page/SlowLostHits", "Lost hits.");
1912	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSubsets, "/PGM/CPU%u/RZ/DynMap/Subsets", "Times PGMRZDynMapPushAutoSubset was called.");
1913	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPopFlushes, "/PGM/CPU%u/RZ/DynMap/SubsetPopFlushes", "Times PGMRZDynMapPopAutoSubset flushes the subset.");
1914	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[0], "/PGM/CPU%u/RZ/DynMap/SetFilledPct000..09", "00-09% filled (RC: min(set-size, dynmap-size))");
1915	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[1], "/PGM/CPU%u/RZ/DynMap/SetFilledPct010..19", "10-19% filled (RC: min(set-size, dynmap-size))");
1916	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[2], "/PGM/CPU%u/RZ/DynMap/SetFilledPct020..29", "20-29% filled (RC: min(set-size, dynmap-size))");
1917	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[3], "/PGM/CPU%u/RZ/DynMap/SetFilledPct030..39", "30-39% filled (RC: min(set-size, dynmap-size))");
1918	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[4], "/PGM/CPU%u/RZ/DynMap/SetFilledPct040..49", "40-49% filled (RC: min(set-size, dynmap-size))");
1919	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[5], "/PGM/CPU%u/RZ/DynMap/SetFilledPct050..59", "50-59% filled (RC: min(set-size, dynmap-size))");
1920	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[6], "/PGM/CPU%u/RZ/DynMap/SetFilledPct060..69", "60-69% filled (RC: min(set-size, dynmap-size))");
1921	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[7], "/PGM/CPU%u/RZ/DynMap/SetFilledPct070..79", "70-79% filled (RC: min(set-size, dynmap-size))");
1922	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[8], "/PGM/CPU%u/RZ/DynMap/SetFilledPct080..89", "80-89% filled (RC: min(set-size, dynmap-size))");
1923	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[9], "/PGM/CPU%u/RZ/DynMap/SetFilledPct090..99", "90-99% filled (RC: min(set-size, dynmap-size))");
1924	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[10], "/PGM/CPU%u/RZ/DynMap/SetFilledPct100", "100% filled (RC: min(set-size, dynmap-size))");
1925
1926	/* HC only: */
1927
1928	/* RZ & R3: */
1929	PGM_REG_PROFILE(&pCpuStats->StatRZSyncCR3, "/PGM/CPU%u/RZ/SyncCR3", "Profiling of the PGMSyncCR3() body.");
1930	PGM_REG_PROFILE(&pCpuStats->StatRZSyncCR3Handlers, "/PGM/CPU%u/RZ/SyncCR3/Handlers", "Profiling of the PGMSyncCR3() update handler section.");
1931	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3Global, "/PGM/CPU%u/RZ/SyncCR3/Global", "The number of global CR3 syncs.");
1932	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3NotGlobal, "/PGM/CPU%u/RZ/SyncCR3/NotGlobal", "The number of non-global CR3 syncs.");
1933	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstCacheHit, "/PGM/CPU%u/RZ/SyncCR3/DstChacheHit", "The number of times we got some kind of a cache hit.");
1934	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstFreed, "/PGM/CPU%u/RZ/SyncCR3/DstFreed", "The number of times we've had to free a shadow entry.");
1935	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstFreedSrcNP, "/PGM/CPU%u/RZ/SyncCR3/DstFreedSrcNP", "The number of times we've had to free a shadow entry for which the source entry was not present.");
1936	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstNotPresent, "/PGM/CPU%u/RZ/SyncCR3/DstNotPresent", "The number of times we've encountered a not present shadow entry for a present guest entry.");
1937	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstSkippedGlobalPD, "/PGM/CPU%u/RZ/SyncCR3/DstSkippedGlobalPD", "The number of times a global page directory wasn't flushed.");
1938	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstSkippedGlobalPT, "/PGM/CPU%u/RZ/SyncCR3/DstSkippedGlobalPT", "The number of times a page table with only global entries wasn't flushed.");
1939	PGM_REG_PROFILE(&pCpuStats->StatRZSyncPT, "/PGM/CPU%u/RZ/SyncPT", "Profiling of the pfnSyncPT() body.");
1940	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPTFailed, "/PGM/CPU%u/RZ/SyncPT/Failed", "The number of times pfnSyncPT() failed.");
1941	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPT4K, "/PGM/CPU%u/RZ/SyncPT/4K", "Nr of 4K PT syncs");
1942	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPT4M, "/PGM/CPU%u/RZ/SyncPT/4M", "Nr of 4M PT syncs");
1943	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPagePDNAs, "/PGM/CPU%u/RZ/SyncPagePDNAs", "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1944	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPagePDOutOfSync, "/PGM/CPU%u/RZ/SyncPagePDOutOfSync", "The number of time we've encountered an out-of-sync PD in SyncPage.");
1945	PGM_REG_COUNTER(&pCpuStats->StatRZAccessedPage, "/PGM/CPU%u/RZ/AccessedPage", "The number of pages marked not present for accessed bit emulation.");
1946	PGM_REG_PROFILE(&pCpuStats->StatRZDirtyBitTracking, "/PGM/CPU%u/RZ/DirtyPage", "Profiling the dirty bit tracking in CheckPageFault().");
1947	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPage, "/PGM/CPU%u/RZ/DirtyPage/Mark", "The number of pages marked read-only for dirty bit tracking.");
1948	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageBig, "/PGM/CPU%u/RZ/DirtyPage/MarkBig", "The number of 4MB pages marked read-only for dirty bit tracking.");
1949	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageSkipped, "/PGM/CPU%u/RZ/DirtyPage/Skipped", "The number of pages already dirty or readonly.");
1950	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageTrap, "/PGM/CPU%u/RZ/DirtyPage/Trap", "The number of traps generated for dirty bit tracking.");
1951	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageStale, "/PGM/CPU%u/RZ/DirtyPage/Stale", "The number of traps generated for dirty bit tracking (stale tlb entries).");
1952	PGM_REG_COUNTER(&pCpuStats->StatRZDirtiedPage, "/PGM/CPU%u/RZ/DirtyPage/SetDirty", "The number of pages marked dirty because of write accesses.");
1953	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyTrackRealPF, "/PGM/CPU%u/RZ/DirtyPage/RealPF", "The number of real pages faults during dirty bit tracking.");
1954	PGM_REG_COUNTER(&pCpuStats->StatRZPageAlreadyDirty, "/PGM/CPU%u/RZ/DirtyPage/AlreadySet", "The number of pages already marked dirty because of write accesses.");
1955	PGM_REG_PROFILE(&pCpuStats->StatRZInvalidatePage, "/PGM/CPU%u/RZ/InvalidatePage", "PGMInvalidatePage() profiling.");
1956	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePage4KBPages, "/PGM/CPU%u/RZ/InvalidatePage/4KBPages", "The number of times PGMInvalidatePage() was called for a 4KB page.");
1957	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePage4MBPages, "/PGM/CPU%u/RZ/InvalidatePage/4MBPages", "The number of times PGMInvalidatePage() was called for a 4MB page.");
1958	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePage4MBPagesSkip, "/PGM/CPU%u/RZ/InvalidatePage/4MBPagesSkip","The number of times PGMInvalidatePage() skipped a 4MB page.");
1959	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDMappings, "/PGM/CPU%u/RZ/InvalidatePage/PDMappings", "The number of times PGMInvalidatePage() was called for a page directory containing mappings (no conflict).");
1960	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDNAs, "/PGM/CPU%u/RZ/InvalidatePage/PDNAs", "The number of times PGMInvalidatePage() was called for a not accessed page directory.");
1961	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDNPs, "/PGM/CPU%u/RZ/InvalidatePage/PDNPs", "The number of times PGMInvalidatePage() was called for a not present page directory.");
1962	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDOutOfSync, "/PGM/CPU%u/RZ/InvalidatePage/PDOutOfSync", "The number of times PGMInvalidatePage() was called for an out of sync page directory.");
1963	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePageSkipped, "/PGM/CPU%u/RZ/InvalidatePage/Skipped", "The number of times PGMInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
1964	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncSupervisor, "/PGM/CPU%u/RZ/OutOfSync/SuperVisor", "Number of traps due to pages out of sync (P) and times VerifyAccessSyncPage calls SyncPage.");
1965	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncUser, "/PGM/CPU%u/RZ/OutOfSync/User", "Number of traps due to pages out of sync (P) and times VerifyAccessSyncPage calls SyncPage.");
1966	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncSupervisorWrite,"/PGM/CPU%u/RZ/OutOfSync/SuperVisorWrite", "Number of traps due to pages out of sync (RW) and times VerifyAccessSyncPage calls SyncPage.");
1967	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncUserWrite, "/PGM/CPU%u/RZ/OutOfSync/UserWrite", "Number of traps due to pages out of sync (RW) and times VerifyAccessSyncPage calls SyncPage.");
1968	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncBallloon, "/PGM/CPU%u/RZ/OutOfSync/Balloon", "The number of times a ballooned page was accessed (read).");
1969	PGM_REG_PROFILE(&pCpuStats->StatRZPrefetch, "/PGM/CPU%u/RZ/Prefetch", "PGMPrefetchPage profiling.");
1970	PGM_REG_PROFILE(&pCpuStats->StatRZFlushTLB, "/PGM/CPU%u/RZ/FlushTLB", "Profiling of the PGMFlushTLB() body.");
1971	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBNewCR3, "/PGM/CPU%u/RZ/FlushTLB/NewCR3", "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
1972	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBNewCR3Global, "/PGM/CPU%u/RZ/FlushTLB/NewCR3Global", "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
1973	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBSameCR3, "/PGM/CPU%u/RZ/FlushTLB/SameCR3", "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
1974	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBSameCR3Global, "/PGM/CPU%u/RZ/FlushTLB/SameCR3Global", "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
1975	PGM_REG_PROFILE(&pCpuStats->StatRZGstModifyPage, "/PGM/CPU%u/RZ/GstModifyPage", "Profiling of the PGMGstModifyPage() body.");
1976
1977	PGM_REG_PROFILE(&pCpuStats->StatR3SyncCR3, "/PGM/CPU%u/R3/SyncCR3", "Profiling of the PGMSyncCR3() body.");
1978	PGM_REG_PROFILE(&pCpuStats->StatR3SyncCR3Handlers, "/PGM/CPU%u/R3/SyncCR3/Handlers", "Profiling of the PGMSyncCR3() update handler section.");
1979	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3Global, "/PGM/CPU%u/R3/SyncCR3/Global", "The number of global CR3 syncs.");
1980	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3NotGlobal, "/PGM/CPU%u/R3/SyncCR3/NotGlobal", "The number of non-global CR3 syncs.");
1981	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstCacheHit, "/PGM/CPU%u/R3/SyncCR3/DstChacheHit", "The number of times we got some kind of a cache hit.");
1982	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstFreed, "/PGM/CPU%u/R3/SyncCR3/DstFreed", "The number of times we've had to free a shadow entry.");
1983	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstFreedSrcNP, "/PGM/CPU%u/R3/SyncCR3/DstFreedSrcNP", "The number of times we've had to free a shadow entry for which the source entry was not present.");
1984	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstNotPresent, "/PGM/CPU%u/R3/SyncCR3/DstNotPresent", "The number of times we've encountered a not present shadow entry for a present guest entry.");
1985	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstSkippedGlobalPD, "/PGM/CPU%u/R3/SyncCR3/DstSkippedGlobalPD", "The number of times a global page directory wasn't flushed.");
1986	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstSkippedGlobalPT, "/PGM/CPU%u/R3/SyncCR3/DstSkippedGlobalPT", "The number of times a page table with only global entries wasn't flushed.");
1987	PGM_REG_PROFILE(&pCpuStats->StatR3SyncPT, "/PGM/CPU%u/R3/SyncPT", "Profiling of the pfnSyncPT() body.");
1988	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPTFailed, "/PGM/CPU%u/R3/SyncPT/Failed", "The number of times pfnSyncPT() failed.");
1989	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPT4K, "/PGM/CPU%u/R3/SyncPT/4K", "Nr of 4K PT syncs");
1990	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPT4M, "/PGM/CPU%u/R3/SyncPT/4M", "Nr of 4M PT syncs");
1991	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPagePDNAs, "/PGM/CPU%u/R3/SyncPagePDNAs", "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1992	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPagePDOutOfSync, "/PGM/CPU%u/R3/SyncPagePDOutOfSync", "The number of time we've encountered an out-of-sync PD in SyncPage.");
1993	PGM_REG_COUNTER(&pCpuStats->StatR3AccessedPage, "/PGM/CPU%u/R3/AccessedPage", "The number of pages marked not present for accessed bit emulation.");
1994	PGM_REG_PROFILE(&pCpuStats->StatR3DirtyBitTracking, "/PGM/CPU%u/R3/DirtyPage", "Profiling the dirty bit tracking in CheckPageFault().");
1995	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPage, "/PGM/CPU%u/R3/DirtyPage/Mark", "The number of pages marked read-only for dirty bit tracking.");
1996	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPageBig, "/PGM/CPU%u/R3/DirtyPage/MarkBig", "The number of 4MB pages marked read-only for dirty bit tracking.");
1997	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPageSkipped, "/PGM/CPU%u/R3/DirtyPage/Skipped", "The number of pages already dirty or readonly.");
1998	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPageTrap, "/PGM/CPU%u/R3/DirtyPage/Trap", "The number of traps generated for dirty bit tracking.");
1999	PGM_REG_COUNTER(&pCpuStats->StatR3DirtiedPage, "/PGM/CPU%u/R3/DirtyPage/SetDirty", "The number of pages marked dirty because of write accesses.");
2000	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyTrackRealPF, "/PGM/CPU%u/R3/DirtyPage/RealPF", "The number of real pages faults during dirty bit tracking.");
2001	PGM_REG_COUNTER(&pCpuStats->StatR3PageAlreadyDirty, "/PGM/CPU%u/R3/DirtyPage/AlreadySet", "The number of pages already marked dirty because of write accesses.");
2002	PGM_REG_PROFILE(&pCpuStats->StatR3InvalidatePage, "/PGM/CPU%u/R3/InvalidatePage", "PGMInvalidatePage() profiling.");
2003	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePage4KBPages, "/PGM/CPU%u/R3/InvalidatePage/4KBPages", "The number of times PGMInvalidatePage() was called for a 4KB page.");
2004	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePage4MBPages, "/PGM/CPU%u/R3/InvalidatePage/4MBPages", "The number of times PGMInvalidatePage() was called for a 4MB page.");
2005	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePage4MBPagesSkip, "/PGM/CPU%u/R3/InvalidatePage/4MBPagesSkip","The number of times PGMInvalidatePage() skipped a 4MB page.");
2006	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDMappings, "/PGM/CPU%u/R3/InvalidatePage/PDMappings", "The number of times PGMInvalidatePage() was called for a page directory containing mappings (no conflict).");
2007	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDNAs, "/PGM/CPU%u/R3/InvalidatePage/PDNAs", "The number of times PGMInvalidatePage() was called for a not accessed page directory.");
2008	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDNPs, "/PGM/CPU%u/R3/InvalidatePage/PDNPs", "The number of times PGMInvalidatePage() was called for a not present page directory.");
2009	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDOutOfSync, "/PGM/CPU%u/R3/InvalidatePage/PDOutOfSync", "The number of times PGMInvalidatePage() was called for an out of sync page directory.");
2010	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePageSkipped, "/PGM/CPU%u/R3/InvalidatePage/Skipped", "The number of times PGMInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
2011	PGM_REG_COUNTER(&pCpuStats->StatR3PageOutOfSyncSupervisor, "/PGM/CPU%u/R3/OutOfSync/SuperVisor", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
2012	PGM_REG_COUNTER(&pCpuStats->StatR3PageOutOfSyncUser, "/PGM/CPU%u/R3/OutOfSync/User", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
2013	PGM_REG_COUNTER(&pCpuStats->StatR3PageOutOfSyncBallloon, "/PGM/CPU%u/R3/OutOfSync/Balloon", "The number of times a ballooned page was accessed (read).");
2014	PGM_REG_PROFILE(&pCpuStats->StatR3Prefetch, "/PGM/CPU%u/R3/Prefetch", "PGMPrefetchPage profiling.");
2015	PGM_REG_PROFILE(&pCpuStats->StatR3FlushTLB, "/PGM/CPU%u/R3/FlushTLB", "Profiling of the PGMFlushTLB() body.");
2016	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBNewCR3, "/PGM/CPU%u/R3/FlushTLB/NewCR3", "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
2017	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBNewCR3Global, "/PGM/CPU%u/R3/FlushTLB/NewCR3Global", "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
2018	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBSameCR3, "/PGM/CPU%u/R3/FlushTLB/SameCR3", "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
2019	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBSameCR3Global, "/PGM/CPU%u/R3/FlushTLB/SameCR3Global", "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
2020	PGM_REG_PROFILE(&pCpuStats->StatR3GstModifyPage, "/PGM/CPU%u/R3/GstModifyPage", "Profiling of the PGMGstModifyPage() body.");
2021	#endif /* VBOX_WITH_STATISTICS */
2022
2023	#undef PGM_REG_PROFILE
2024	#undef PGM_REG_COUNTER
2025
2026	}
2027
2028	return VINF_SUCCESS;
2029	}
2030
2031
2032	/**
2033	* Init the PGM bits that rely on VMMR0 and MM to be fully initialized.
2034	*
2035	* The dynamic mapping area will also be allocated and initialized at this
2036	* time. We could allocate it during PGMR3Init of course, but the mapping
2037	* wouldn't be allocated at that time preventing us from setting up the
2038	* page table entries with the dummy page.
2039	*
2040	* @returns VBox status code.
2041	* @param pVM VM handle.
2042	*/
2043	VMMR3DECL(int) PGMR3InitDynMap(PVM pVM)
2044	{
2045	RTGCPTR GCPtr;
2046	int rc;
2047
2048	/*
2049	* Reserve space for the dynamic mappings.
2050	*/
2051	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping", &GCPtr);
2052	if (RT_SUCCESS(rc))
2053	pVM->pgm.s.pbDynPageMapBaseGC = GCPtr;
2054
2055	if ( RT_SUCCESS(rc)
2056	&& (pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_PAE_SHIFT) != ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_PAE_SHIFT))
2057	{
2058	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping not crossing", &GCPtr);
2059	if (RT_SUCCESS(rc))
2060	pVM->pgm.s.pbDynPageMapBaseGC = GCPtr;
2061	}
2062	if (RT_SUCCESS(rc))
2063	{
2064	AssertRelease((pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_PAE_SHIFT) == ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_PAE_SHIFT));
2065	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
2066	}
2067	return rc;
2068	}
2069
2070
2071	/**
2072	* Ring-3 init finalizing.
2073	*
2074	* @returns VBox status code.
2075	* @param pVM The VM handle.
2076	*/
2077	VMMR3DECL(int) PGMR3InitFinalize(PVM pVM)
2078	{
2079	int rc;
2080
2081	/*
2082	* Reserve space for the dynamic mappings.
2083	* Initialize the dynamic mapping pages with dummy pages to simply the cache.
2084	*/
2085	/* get the pointer to the page table entries. */
2086	PPGMMAPPING pMapping = pgmGetMapping(pVM, pVM->pgm.s.pbDynPageMapBaseGC);
2087	AssertRelease(pMapping);
2088	const uintptr_t off = pVM->pgm.s.pbDynPageMapBaseGC - pMapping->GCPtr;
2089	const unsigned iPT = off >> X86_PD_SHIFT;
2090	const unsigned iPG = (off >> X86_PT_SHIFT) & X86_PT_MASK;
2091	pVM->pgm.s.paDynPageMap32BitPTEsGC = pMapping->aPTs[iPT].pPTRC + iPG * sizeof(pMapping->aPTs[0].pPTR3->a[0]);
2092	pVM->pgm.s.paDynPageMapPaePTEsGC = pMapping->aPTs[iPT].paPaePTsRC + iPG * sizeof(pMapping->aPTs[0].paPaePTsR3->a[0]);
2093
2094	/* init cache area */
2095	RTHCPHYS HCPhysDummy = MMR3PageDummyHCPhys(pVM);
2096	for (uint32_t offDynMap = 0; offDynMap < MM_HYPER_DYNAMIC_SIZE; offDynMap += PAGE_SIZE)
2097	{
2098	rc = PGMMap(pVM, pVM->pgm.s.pbDynPageMapBaseGC + offDynMap, HCPhysDummy, PAGE_SIZE, 0);
2099	AssertRCReturn(rc, rc);
2100	}
2101
2102	/*
2103	* Determine the max physical address width (MAXPHYADDR) and apply it to
2104	* all the mask members and stuff.
2105	*/
2106	uint32_t cMaxPhysAddrWidth;
2107	uint32_t uMaxExtLeaf = ASMCpuId_EAX(0x80000000);
2108	if ( uMaxExtLeaf >= 0x80000008
2109	&& uMaxExtLeaf <= 0x80000fff)
2110	{
2111	cMaxPhysAddrWidth = ASMCpuId_EAX(0x80000008) & 0xff;
2112	LogRel(("PGM: The CPU physical address width is %u bits\n", cMaxPhysAddrWidth));
2113	cMaxPhysAddrWidth = RT_MIN(52, cMaxPhysAddrWidth);
2114	pVM->pgm.s.fLessThan52PhysicalAddressBits = cMaxPhysAddrWidth < 52;
2115	for (uint32_t iBit = cMaxPhysAddrWidth; iBit < 52; iBit++)
2116	pVM->pgm.s.HCPhysInvMmioPg \|= RT_BIT_64(iBit);
2117	}
2118	else
2119	{
2120	LogRel(("PGM: ASSUMING CPU physical address width of 48 bits (uMaxExtLeaf=%#x)\n", uMaxExtLeaf));
2121	cMaxPhysAddrWidth = 48;
2122	pVM->pgm.s.fLessThan52PhysicalAddressBits = true;
2123	pVM->pgm.s.HCPhysInvMmioPg \|= UINT64_C(0x000f0000000000);
2124	}
2125
2126	pVM->pgm.s.GCPhysInvAddrMask = 0;
2127	for (uint32_t iBit = cMaxPhysAddrWidth; iBit < 64; iBit++)
2128	pVM->pgm.s.GCPhysInvAddrMask \|= RT_BIT_64(iBit);
2129
2130	/*
2131	* Initialize the invalid paging entry masks, assuming NX is disabled.
2132	*/
2133	uint64_t fMbzPageFrameMask = pVM->pgm.s.GCPhysInvAddrMask & UINT64_C(0x000ffffffffff000);
2134	for (VMCPUID iCpu = 0; iCpu < pVM->cCpus; iCpu++)
2135	{
2136	PVMCPU pVCpu = &pVM->aCpus[iCpu];
2137
2138	/** @todo The manuals are not entirely clear whether the physical
2139	* address width is relevant. See table 5-9 in the intel
2140	* manual vs the PDE4M descriptions. Write testcase (NP). */
2141	pVCpu->pgm.s.fGst32BitMbzBigPdeMask = ((uint32_t)(fMbzPageFrameMask >> (32 - 13)) & X86_PDE4M_PG_HIGH_MASK)
2142	\| X86_PDE4M_MBZ_MASK;
2143
2144	pVCpu->pgm.s.fGstPaeMbzPteMask = fMbzPageFrameMask \| X86_PTE_PAE_MBZ_MASK_NO_NX;
2145	pVCpu->pgm.s.fGstPaeMbzPdeMask = fMbzPageFrameMask \| X86_PDE_PAE_MBZ_MASK_NO_NX;
2146	pVCpu->pgm.s.fGstPaeMbzBigPdeMask = fMbzPageFrameMask \| X86_PDE2M_PAE_MBZ_MASK_NO_NX;
2147	pVCpu->pgm.s.fGstPaeMbzPdpeMask = fMbzPageFrameMask \| X86_PDPE_PAE_MBZ_MASK;
2148
2149	pVCpu->pgm.s.fGstAmd64MbzPteMask = fMbzPageFrameMask \| X86_PTE_LM_MBZ_MASK_NO_NX;
2150	pVCpu->pgm.s.fGstAmd64MbzPdeMask = fMbzPageFrameMask \| X86_PDE_LM_MBZ_MASK_NX;
2151	pVCpu->pgm.s.fGstAmd64MbzBigPdeMask = fMbzPageFrameMask \| X86_PDE2M_LM_MBZ_MASK_NX;
2152	pVCpu->pgm.s.fGstAmd64MbzPdpeMask = fMbzPageFrameMask \| X86_PDPE_LM_MBZ_MASK_NO_NX;
2153	pVCpu->pgm.s.fGstAmd64MbzBigPdpeMask = fMbzPageFrameMask \| X86_PDPE1G_LM_MBZ_MASK_NO_NX;
2154	pVCpu->pgm.s.fGstAmd64MbzPml4eMask = fMbzPageFrameMask \| X86_PML4E_MBZ_MASK_NO_NX;
2155
2156	pVCpu->pgm.s.fGst64ShadowedPteMask = X86_PTE_P \| X86_PTE_RW \| X86_PTE_US \| X86_PTE_G \| X86_PTE_A \| X86_PTE_D;
2157	pVCpu->pgm.s.fGst64ShadowedPdeMask = X86_PDE_P \| X86_PDE_RW \| X86_PDE_US \| X86_PDE_A;
2158	pVCpu->pgm.s.fGst64ShadowedBigPdeMask = X86_PDE4M_P \| X86_PDE4M_RW \| X86_PDE4M_US \| X86_PDE4M_A;
2159	pVCpu->pgm.s.fGst64ShadowedBigPde4PteMask =
2160	X86_PDE4M_P \| X86_PDE4M_RW \| X86_PDE4M_US \| X86_PDE4M_G \| X86_PDE4M_A \| X86_PDE4M_D;
2161	pVCpu->pgm.s.fGstAmd64ShadowedPdpeMask = X86_PDPE_P \| X86_PDPE_RW \| X86_PDPE_US \| X86_PDPE_A;
2162	pVCpu->pgm.s.fGstAmd64ShadowedPml4eMask = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_US \| X86_PML4E_A;
2163	}
2164
2165	/*
2166	* Note that AMD uses all the 8 reserved bits for the address (so 40 bits in total);
2167	* Intel only goes up to 36 bits, so we stick to 36 as well.
2168	* Update: More recent intel manuals specifies 40 bits just like AMD.
2169	*/
2170	uint32_t u32Dummy, u32Features;
2171	CPUMGetGuestCpuId(VMMGetCpu(pVM), 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
2172	if (u32Features & X86_CPUID_FEATURE_EDX_PSE36)
2173	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(RT_MAX(36, cMaxPhysAddrWidth)) - 1;
2174	else
2175	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(32) - 1;
2176
2177	/*
2178	* Allocate memory if we're supposed to do that.
2179	*/
2180	if (pVM->pgm.s.fRamPreAlloc)
2181	rc = pgmR3PhysRamPreAllocate(pVM);
2182
2183	LogRel(("PGMR3InitFinalize: 4 MB PSE mask %RGp\n", pVM->pgm.s.GCPhys4MBPSEMask));
2184	return rc;
2185	}
2186
2187
2188	/**
2189	* Init phase completed callback.
2190	*
2191	* @returns VBox status code.
2192	* @param pVM The VM handle.
2193	* @param enmWhat What has been completed.
2194	* @thread EMT(0)
2195	*/
2196	VMMR3_INT_DECL(int) PGMR3InitCompleted(PVM pVM, VMINITCOMPLETED enmWhat)
2197	{
2198	switch (enmWhat)
2199	{
2200	case VMINITCOMPLETED_HWACCM:
2201	#ifdef VBOX_WITH_PCI_PASSTHROUGH
2202	if (pVM->pgm.s.fPciPassthrough)
2203	{
2204	AssertLogRelReturn(pVM->pgm.s.fRamPreAlloc, VERR_PCI_PASSTHROUGH_NO_RAM_PREALLOC);
2205	AssertLogRelReturn(HWACCMIsEnabled(pVM), VERR_PCI_PASSTHROUGH_NO_HWACCM);
2206	AssertLogRelReturn(HWACCMIsNestedPagingActive(pVM), VERR_PCI_PASSTHROUGH_NO_NESTED_PAGING);
2207
2208	/*
2209	* Report assignments to the IOMMU (hope that's good enough for now).
2210	*/
2211	if (pVM->pgm.s.fPciPassthrough)
2212	{
2213	int rc = VMMR3CallR0(pVM, VMMR0_DO_PGM_PHYS_SETUP_IOMMU, 0, NULL);
2214	AssertRCReturn(rc, rc);
2215	}
2216	}
2217	#else
2218	AssertLogRelReturn(!pVM->pgm.s.fPciPassthrough, VERR_INTERNAL_ERROR_5);
2219	#endif
2220	break;
2221
2222	default:
2223	/* shut up gcc */
2224	break;
2225	}
2226
2227	return VINF_SUCCESS;
2228	}
2229
2230
2231	/**
2232	* Applies relocations to data and code managed by this component.
2233	*
2234	* This function will be called at init and whenever the VMM need to relocate it
2235	* self inside the GC.
2236	*
2237	* @param pVM The VM.
2238	* @param offDelta Relocation delta relative to old location.
2239	*/
2240	VMMR3DECL(void) PGMR3Relocate(PVM pVM, RTGCINTPTR offDelta)
2241	{
2242	LogFlow(("PGMR3Relocate %RGv to %RGv\n", pVM->pgm.s.GCPtrCR3Mapping, pVM->pgm.s.GCPtrCR3Mapping + offDelta));
2243
2244	/*
2245	* Paging stuff.
2246	*/
2247	pVM->pgm.s.GCPtrCR3Mapping += offDelta;
2248
2249	pgmR3ModeDataInit(pVM, true /* resolve GC/R0 symbols */);
2250
2251	/* Shadow, guest and both mode switch & relocation for each VCPU. */
2252	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2253	{
2254	PVMCPU pVCpu = &pVM->aCpus[i];
2255
2256	pgmR3ModeDataSwitch(pVM, pVCpu, pVCpu->pgm.s.enmShadowMode, pVCpu->pgm.s.enmGuestMode);
2257
2258	PGM_SHW_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2259	PGM_GST_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2260	PGM_BTH_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2261	}
2262
2263	/*
2264	* Trees.
2265	*/
2266	pVM->pgm.s.pTreesRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pTreesR3);
2267
2268	/*
2269	* Ram ranges.
2270	*/
2271	if (pVM->pgm.s.pRamRangesXR3)
2272	{
2273	/* Update the pSelfRC pointers and relink them. */
2274	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesXR3; pCur; pCur = pCur->pNextR3)
2275	if (!(pCur->fFlags & PGM_RAM_RANGE_FLAGS_FLOATING))
2276	pCur->pSelfRC = MMHyperCCToRC(pVM, pCur);
2277	pgmR3PhysRelinkRamRanges(pVM);
2278
2279	/* Flush the RC TLB. */
2280	for (unsigned i = 0; i < PGM_RAMRANGE_TLB_ENTRIES; i++)
2281	pVM->pgm.s.apRamRangesTlbRC[i] = NIL_RTRCPTR;
2282	}
2283
2284	/*
2285	* Update the pSelfRC pointer of the MMIO2 ram ranges since they might not
2286	* be mapped and thus not included in the above exercise.
2287	*/
2288	for (PPGMMMIO2RANGE pCur = pVM->pgm.s.pMmio2RangesR3; pCur; pCur = pCur->pNextR3)
2289	if (!(pCur->RamRange.fFlags & PGM_RAM_RANGE_FLAGS_FLOATING))
2290	pCur->RamRange.pSelfRC = MMHyperCCToRC(pVM, &pCur->RamRange);
2291
2292	/*
2293	* Update the two page directories with all page table mappings.
2294	* (One or more of them have changed, that's why we're here.)
2295	*/
2296	pVM->pgm.s.pMappingsRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pMappingsR3);
2297	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur->pNextR3; pCur = pCur->pNextR3)
2298	pCur->pNextRC = MMHyperR3ToRC(pVM, pCur->pNextR3);
2299
2300	/* Relocate GC addresses of Page Tables. */
2301	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur; pCur = pCur->pNextR3)
2302	{
2303	for (RTHCUINT i = 0; i < pCur->cPTs; i++)
2304	{
2305	pCur->aPTs[i].pPTRC = MMHyperR3ToRC(pVM, pCur->aPTs[i].pPTR3);
2306	pCur->aPTs[i].paPaePTsRC = MMHyperR3ToRC(pVM, pCur->aPTs[i].paPaePTsR3);
2307	}
2308	}
2309
2310	/*
2311	* Dynamic page mapping area.
2312	*/
2313	pVM->pgm.s.paDynPageMap32BitPTEsGC += offDelta;
2314	pVM->pgm.s.paDynPageMapPaePTEsGC += offDelta;
2315	pVM->pgm.s.pbDynPageMapBaseGC += offDelta;
2316
2317	if (pVM->pgm.s.pRCDynMap)
2318	{
2319	pVM->pgm.s.pRCDynMap += offDelta;
2320	PPGMRCDYNMAP pDynMap = (PPGMRCDYNMAP)MMHyperRCToCC(pVM, pVM->pgm.s.pRCDynMap);
2321
2322	pDynMap->paPages += offDelta;
2323	PPGMRCDYNMAPENTRY paPages = (PPGMRCDYNMAPENTRY)MMHyperRCToCC(pVM, pDynMap->paPages);
2324
2325	for (uint32_t iPage = 0; iPage < pDynMap->cPages; iPage++)
2326	{
2327	paPages[iPage].pvPage += offDelta;
2328	paPages[iPage].uPte.pLegacy += offDelta;
2329	paPages[iPage].uPte.pPae += offDelta;
2330	}
2331	}
2332
2333	/*
2334	* The Zero page.
2335	*/
2336	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
2337	#ifdef VBOX_WITH_2X_4GB_ADDR_SPACE
2338	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTR0PTR \|\| !VMMIsHwVirtExtForced(pVM));
2339	#else
2340	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTR0PTR);
2341	#endif
2342
2343	/*
2344	* Physical and virtual handlers.
2345	*/
2346	RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, true, pgmR3RelocatePhysHandler, &offDelta);
2347	pVM->pgm.s.pLastPhysHandlerRC = NIL_RTRCPTR;
2348	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesR3->VirtHandlers, true, pgmR3RelocateVirtHandler, &offDelta);
2349	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesR3->HyperVirtHandlers, true, pgmR3RelocateHyperVirtHandler, &offDelta);
2350
2351	/*
2352	* The page pool.
2353	*/
2354	pgmR3PoolRelocate(pVM);
2355
2356	#ifdef VBOX_WITH_STATISTICS
2357	/*
2358	* Statistics.
2359	*/
2360	pVM->pgm.s.pStatsRC = MMHyperCCToRC(pVM, pVM->pgm.s.pStatsR3);
2361	for (VMCPUID iCpu = 0; iCpu < pVM->cCpus; iCpu++)
2362	pVM->aCpus[iCpu].pgm.s.pStatsRC = MMHyperCCToRC(pVM, pVM->aCpus[iCpu].pgm.s.pStatsR3);
2363	#endif
2364	}
2365
2366
2367	/**
2368	* Callback function for relocating a physical access handler.
2369	*
2370	* @returns 0 (continue enum)
2371	* @param pNode Pointer to a PGMPHYSHANDLER node.
2372	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2373	* not certain the delta will fit in a void pointer for all possible configs.
2374	*/
2375	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser)
2376	{
2377	PPGMPHYSHANDLER pHandler = (PPGMPHYSHANDLER)pNode;
2378	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2379	if (pHandler->pfnHandlerRC)
2380	pHandler->pfnHandlerRC += offDelta;
2381	if (pHandler->pvUserRC >= 0x10000)
2382	pHandler->pvUserRC += offDelta;
2383	return 0;
2384	}
2385
2386
2387	/**
2388	* Callback function for relocating a virtual access handler.
2389	*
2390	* @returns 0 (continue enum)
2391	* @param pNode Pointer to a PGMVIRTHANDLER node.
2392	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2393	* not certain the delta will fit in a void pointer for all possible configs.
2394	*/
2395	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
2396	{
2397	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
2398	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2399	Assert( pHandler->enmType == PGMVIRTHANDLERTYPE_ALL
2400	\|\| pHandler->enmType == PGMVIRTHANDLERTYPE_WRITE);
2401	Assert(pHandler->pfnHandlerRC);
2402	pHandler->pfnHandlerRC += offDelta;
2403	return 0;
2404	}
2405
2406
2407	/**
2408	* Callback function for relocating a virtual access handler for the hypervisor mapping.
2409	*
2410	* @returns 0 (continue enum)
2411	* @param pNode Pointer to a PGMVIRTHANDLER node.
2412	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2413	* not certain the delta will fit in a void pointer for all possible configs.
2414	*/
2415	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
2416	{
2417	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
2418	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2419	Assert(pHandler->enmType == PGMVIRTHANDLERTYPE_HYPERVISOR);
2420	Assert(pHandler->pfnHandlerRC);
2421	pHandler->pfnHandlerRC += offDelta;
2422	return 0;
2423	}
2424
2425
2426	/**
2427	* Resets a virtual CPU when unplugged.
2428	*
2429	* @param pVM The VM handle.
2430	* @param pVCpu The virtual CPU handle.
2431	*/
2432	VMMR3DECL(void) PGMR3ResetUnpluggedCpu(PVM pVM, PVMCPU pVCpu)
2433	{
2434	int rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
2435	AssertRC(rc);
2436
2437	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
2438	AssertRC(rc);
2439
2440	STAM_REL_COUNTER_RESET(&pVCpu->pgm.s.cGuestModeChanges);
2441
2442	pgmR3PoolResetUnpluggedCpu(pVM, pVCpu);
2443
2444	/*
2445	* Re-init other members.
2446	*/
2447	pVCpu->pgm.s.fA20Enabled = true;
2448
2449	/*
2450	* Clear the FFs PGM owns.
2451	*/
2452	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
2453	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3_NON_GLOBAL);
2454	}
2455
2456
2457	/**
2458	* The VM is being reset.
2459	*
2460	* For the PGM component this means that any PD write monitors
2461	* needs to be removed.
2462	*
2463	* @param pVM VM handle.
2464	*/
2465	VMMR3DECL(void) PGMR3Reset(PVM pVM)
2466	{
2467	int rc;
2468
2469	LogFlow(("PGMR3Reset:\n"));
2470	VM_ASSERT_EMT(pVM);
2471
2472	pgmLock(pVM);
2473
2474	/*
2475	* Unfix any fixed mappings and disable CR3 monitoring.
2476	*/
2477	pVM->pgm.s.fMappingsFixed = false;
2478	pVM->pgm.s.fMappingsFixedRestored = false;
2479	pVM->pgm.s.GCPtrMappingFixed = NIL_RTGCPTR;
2480	pVM->pgm.s.cbMappingFixed = 0;
2481
2482	/*
2483	* Exit the guest paging mode before the pgm pool gets reset.
2484	* Important to clean up the amd64 case.
2485	*/
2486	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2487	{
2488	PVMCPU pVCpu = &pVM->aCpus[i];
2489	rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
2490	AssertRC(rc);
2491	}
2492
2493	#ifdef DEBUG
2494	DBGFR3InfoLog(pVM, "mappings", NULL);
2495	DBGFR3InfoLog(pVM, "handlers", "all nostat");
2496	#endif
2497
2498	/*
2499	* Switch mode back to real mode. (before resetting the pgm pool!)
2500	*/
2501	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2502	{
2503	PVMCPU pVCpu = &pVM->aCpus[i];
2504
2505	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
2506	AssertRC(rc);
2507
2508	STAM_REL_COUNTER_RESET(&pVCpu->pgm.s.cGuestModeChanges);
2509	}
2510
2511	/*
2512	* Reset the shadow page pool.
2513	*/
2514	pgmR3PoolReset(pVM);
2515
2516	/*
2517	* Re-init various other members and clear the FFs that PGM owns.
2518	*/
2519	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2520	{
2521	PVMCPU pVCpu = &pVM->aCpus[i];
2522
2523	pVCpu->pgm.s.fA20Enabled = true;
2524	pVCpu->pgm.s.fGst32BitPageSizeExtension = false;
2525	PGMNotifyNxeChanged(pVCpu, false);
2526
2527	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
2528	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3_NON_GLOBAL);
2529	}
2530
2531	/*
2532	* Reset (zero) RAM and shadow ROM pages.
2533	*/
2534	rc = pgmR3PhysRamReset(pVM);
2535	if (RT_SUCCESS(rc))
2536	rc = pgmR3PhysRomReset(pVM);
2537
2538
2539	pgmUnlock(pVM);
2540	AssertReleaseRC(rc);
2541	}
2542
2543
2544	#ifdef VBOX_STRICT
2545	/**
2546	* VM state change callback for clearing fNoMorePhysWrites after
2547	* a snapshot has been created.
2548	*/
2549	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser)
2550	{
2551	if ( enmState == VMSTATE_RUNNING
2552	\|\| enmState == VMSTATE_RESUMING)
2553	pVM->pgm.s.fNoMorePhysWrites = false;
2554	}
2555	#endif
2556
2557	/**
2558	* Private API to reset fNoMorePhysWrites.
2559	*/
2560	VMMR3DECL(void) PGMR3ResetNoMorePhysWritesFlag(PVM pVM)
2561	{
2562	pVM->pgm.s.fNoMorePhysWrites = false;
2563	}
2564
2565	/**
2566	* Terminates the PGM.
2567	*
2568	* @returns VBox status code.
2569	* @param pVM Pointer to VM structure.
2570	*/
2571	VMMR3DECL(int) PGMR3Term(PVM pVM)
2572	{
2573	/* Must free shared pages here. */
2574	pgmLock(pVM);
2575	pgmR3PhysRamTerm(pVM);
2576	pgmR3PhysRomTerm(pVM);
2577	pgmUnlock(pVM);
2578
2579	PGMDeregisterStringFormatTypes();
2580	return PDMR3CritSectDelete(&pVM->pgm.s.CritSectX);
2581	}
2582
2583
2584	/**
2585	* Show paging mode.
2586	*
2587	* @param pVM VM Handle.
2588	* @param pHlp The info helpers.
2589	* @param pszArgs "all" (default), "guest", "shadow" or "host".
2590	*/
2591	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2592	{
2593	/* digest argument. */
2594	bool fGuest, fShadow, fHost;
2595	if (pszArgs)
2596	pszArgs = RTStrStripL(pszArgs);
2597	if (!pszArgs \|\| !*pszArgs \|\| strstr(pszArgs, "all"))
2598	fShadow = fHost = fGuest = true;
2599	else
2600	{
2601	fShadow = fHost = fGuest = false;
2602	if (strstr(pszArgs, "guest"))
2603	fGuest = true;
2604	if (strstr(pszArgs, "shadow"))
2605	fShadow = true;
2606	if (strstr(pszArgs, "host"))
2607	fHost = true;
2608	}
2609
2610	/** @todo SMP support! */
2611	/* print info. */
2612	if (fGuest)
2613	pHlp->pfnPrintf(pHlp, "Guest paging mode: %s, changed %RU64 times, A20 %s\n",
2614	PGMGetModeName(pVM->aCpus[0].pgm.s.enmGuestMode), pVM->aCpus[0].pgm.s.cGuestModeChanges.c,
2615	pVM->aCpus[0].pgm.s.fA20Enabled ? "enabled" : "disabled");
2616	if (fShadow)
2617	pHlp->pfnPrintf(pHlp, "Shadow paging mode: %s\n", PGMGetModeName(pVM->aCpus[0].pgm.s.enmShadowMode));
2618	if (fHost)
2619	{
2620	const char *psz;
2621	switch (pVM->pgm.s.enmHostMode)
2622	{
2623	case SUPPAGINGMODE_INVALID: psz = "invalid"; break;
2624	case SUPPAGINGMODE_32_BIT: psz = "32-bit"; break;
2625	case SUPPAGINGMODE_32_BIT_GLOBAL: psz = "32-bit+G"; break;
2626	case SUPPAGINGMODE_PAE: psz = "PAE"; break;
2627	case SUPPAGINGMODE_PAE_GLOBAL: psz = "PAE+G"; break;
2628	case SUPPAGINGMODE_PAE_NX: psz = "PAE+NX"; break;
2629	case SUPPAGINGMODE_PAE_GLOBAL_NX: psz = "PAE+G+NX"; break;
2630	case SUPPAGINGMODE_AMD64: psz = "AMD64"; break;
2631	case SUPPAGINGMODE_AMD64_GLOBAL: psz = "AMD64+G"; break;
2632	case SUPPAGINGMODE_AMD64_NX: psz = "AMD64+NX"; break;
2633	case SUPPAGINGMODE_AMD64_GLOBAL_NX: psz = "AMD64+G+NX"; break;
2634	default: psz = "unknown"; break;
2635	}
2636	pHlp->pfnPrintf(pHlp, "Host paging mode: %s\n", psz);
2637	}
2638	}
2639
2640
2641	/**
2642	* Dump registered MMIO ranges to the log.
2643	*
2644	* @param pVM VM Handle.
2645	* @param pHlp The info helpers.
2646	* @param pszArgs Arguments, ignored.
2647	*/
2648	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2649	{
2650	NOREF(pszArgs);
2651	pHlp->pfnPrintf(pHlp,
2652	"RAM ranges (pVM=%p)\n"
2653	"%.s %.s\n",
2654	pVM,
2655	sizeof(RTGCPHYS) * 4 + 1, "GC Phys Range ",
2656	sizeof(RTHCPTR) * 2, "pvHC ");
2657
2658	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesXR3; pCur; pCur = pCur->pNextR3)
2659	pHlp->pfnPrintf(pHlp,
2660	"%RGp-%RGp %RHv %s\n",
2661	pCur->GCPhys,
2662	pCur->GCPhysLast,
2663	pCur->pvR3,
2664	pCur->pszDesc);
2665	}
2666
2667	/**
2668	* Dump the page directory to the log.
2669	*
2670	* @param pVM VM Handle.
2671	* @param pHlp The info helpers.
2672	* @param pszArgs Arguments, ignored.
2673	*/
2674	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2675	{
2676	/** @todo SMP support!! */
2677	PVMCPU pVCpu = &pVM->aCpus[0];
2678
2679	/** @todo fix this! Convert the PGMR3DumpHierarchyHC functions to do guest stuff. */
2680	/* Big pages supported? */
2681	const bool fPSE = !!(CPUMGetGuestCR4(pVCpu) & X86_CR4_PSE);
2682
2683	/* Global pages supported? */
2684	const bool fPGE = !!(CPUMGetGuestCR4(pVCpu) & X86_CR4_PGE);
2685
2686	NOREF(pszArgs);
2687
2688	/*
2689	* Get page directory addresses.
2690	*/
2691	pgmLock(pVM);
2692	PX86PD pPDSrc = pgmGstGet32bitPDPtr(pVCpu);
2693	Assert(pPDSrc);
2694
2695	/*
2696	* Iterate the page directory.
2697	*/
2698	for (unsigned iPD = 0; iPD < RT_ELEMENTS(pPDSrc->a); iPD++)
2699	{
2700	X86PDE PdeSrc = pPDSrc->a[iPD];
2701	if (PdeSrc.n.u1Present)
2702	{
2703	if (PdeSrc.b.u1Size && fPSE)
2704	pHlp->pfnPrintf(pHlp,
2705	"%04X - %RGp P=%d U=%d RW=%d G=%d - BIG\n",
2706	iPD,
2707	pgmGstGet4MBPhysPage(pVM, PdeSrc),
2708	PdeSrc.b.u1Present, PdeSrc.b.u1User, PdeSrc.b.u1Write, PdeSrc.b.u1Global && fPGE);
2709	else
2710	pHlp->pfnPrintf(pHlp,
2711	"%04X - %RGp P=%d U=%d RW=%d [G=%d]\n",
2712	iPD,
2713	(RTGCPHYS)(PdeSrc.u & X86_PDE_PG_MASK),
2714	PdeSrc.n.u1Present, PdeSrc.n.u1User, PdeSrc.n.u1Write, PdeSrc.b.u1Global && fPGE);
2715	}
2716	}
2717	pgmUnlock(pVM);
2718	}
2719
2720
2721	/**
2722	* Service a VMMCALLRING3_PGM_LOCK call.
2723	*
2724	* @returns VBox status code.
2725	* @param pVM The VM handle.
2726	*/
2727	VMMR3DECL(int) PGMR3LockCall(PVM pVM)
2728	{
2729	int rc = PDMR3CritSectEnterEx(&pVM->pgm.s.CritSectX, true /* fHostCall */);
2730	AssertRC(rc);
2731	return rc;
2732	}
2733
2734
2735	/**
2736	* Converts a PGMMODE value to a PGM_TYPE_* \#define.
2737	*
2738	* @returns PGM_TYPE_*.
2739	* @param pgmMode The mode value to convert.
2740	*/
2741	DECLINLINE(unsigned) pgmModeToType(PGMMODE pgmMode)
2742	{
2743	switch (pgmMode)
2744	{
2745	case PGMMODE_REAL: return PGM_TYPE_REAL;
2746	case PGMMODE_PROTECTED: return PGM_TYPE_PROT;
2747	case PGMMODE_32_BIT: return PGM_TYPE_32BIT;
2748	case PGMMODE_PAE:
2749	case PGMMODE_PAE_NX: return PGM_TYPE_PAE;
2750	case PGMMODE_AMD64:
2751	case PGMMODE_AMD64_NX: return PGM_TYPE_AMD64;
2752	case PGMMODE_NESTED: return PGM_TYPE_NESTED;
2753	case PGMMODE_EPT: return PGM_TYPE_EPT;
2754	default:
2755	AssertFatalMsgFailed(("pgmMode=%d\n", pgmMode));
2756	}
2757	}
2758
2759
2760	/**
2761	* Gets the index into the paging mode data array of a SHW+GST mode.
2762	*
2763	* @returns PGM::paPagingData index.
2764	* @param uShwType The shadow paging mode type.
2765	* @param uGstType The guest paging mode type.
2766	*/
2767	DECLINLINE(unsigned) pgmModeDataIndex(unsigned uShwType, unsigned uGstType)
2768	{
2769	Assert(uShwType >= PGM_TYPE_32BIT && uShwType <= PGM_TYPE_MAX);
2770	Assert(uGstType >= PGM_TYPE_REAL && uGstType <= PGM_TYPE_AMD64);
2771	return (uShwType - PGM_TYPE_32BIT) * (PGM_TYPE_AMD64 - PGM_TYPE_REAL + 1)
2772	+ (uGstType - PGM_TYPE_REAL);
2773	}
2774
2775
2776	/**
2777	* Gets the index into the paging mode data array of a SHW+GST mode.
2778	*
2779	* @returns PGM::paPagingData index.
2780	* @param enmShw The shadow paging mode.
2781	* @param enmGst The guest paging mode.
2782	*/
2783	DECLINLINE(unsigned) pgmModeDataIndexByMode(PGMMODE enmShw, PGMMODE enmGst)
2784	{
2785	Assert(enmShw >= PGMMODE_32_BIT && enmShw <= PGMMODE_MAX);
2786	Assert(enmGst > PGMMODE_INVALID && enmGst < PGMMODE_MAX);
2787	return pgmModeDataIndex(pgmModeToType(enmShw), pgmModeToType(enmGst));
2788	}
2789
2790
2791	/**
2792	* Calculates the max data index.
2793	* @returns The number of entries in the paging data array.
2794	*/
2795	DECLINLINE(unsigned) pgmModeDataMaxIndex(void)
2796	{
2797	return pgmModeDataIndex(PGM_TYPE_MAX, PGM_TYPE_AMD64) + 1;
2798	}
2799
2800
2801	/**
2802	* Initializes the paging mode data kept in PGM::paModeData.
2803	*
2804	* @param pVM The VM handle.
2805	* @param fResolveGCAndR0 Indicate whether or not GC and Ring-0 symbols can be resolved now.
2806	* This is used early in the init process to avoid trouble with PDM
2807	* not being initialized yet.
2808	*/
2809	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0)
2810	{
2811	PPGMMODEDATA pModeData;
2812	int rc;
2813
2814	/*
2815	* Allocate the array on the first call.
2816	*/
2817	if (!pVM->pgm.s.paModeData)
2818	{
2819	pVM->pgm.s.paModeData = (PPGMMODEDATA)MMR3HeapAllocZ(pVM, MM_TAG_PGM, sizeof(PGMMODEDATA) * pgmModeDataMaxIndex());
2820	AssertReturn(pVM->pgm.s.paModeData, VERR_NO_MEMORY);
2821	}
2822
2823	/*
2824	* Initialize the array entries.
2825	*/
2826	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_REAL)];
2827	pModeData->uShwType = PGM_TYPE_32BIT;
2828	pModeData->uGstType = PGM_TYPE_REAL;
2829	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2830	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2831	rc = PGM_BTH_NAME_32BIT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2832
2833	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGMMODE_PROTECTED)];
2834	pModeData->uShwType = PGM_TYPE_32BIT;
2835	pModeData->uGstType = PGM_TYPE_PROT;
2836	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2837	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2838	rc = PGM_BTH_NAME_32BIT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2839
2840	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_32BIT)];
2841	pModeData->uShwType = PGM_TYPE_32BIT;
2842	pModeData->uGstType = PGM_TYPE_32BIT;
2843	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2844	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2845	rc = PGM_BTH_NAME_32BIT_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2846
2847	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_REAL)];
2848	pModeData->uShwType = PGM_TYPE_PAE;
2849	pModeData->uGstType = PGM_TYPE_REAL;
2850	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2851	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2852	rc = PGM_BTH_NAME_PAE_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2853
2854	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PROT)];
2855	pModeData->uShwType = PGM_TYPE_PAE;
2856	pModeData->uGstType = PGM_TYPE_PROT;
2857	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2858	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2859	rc = PGM_BTH_NAME_PAE_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2860
2861	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_32BIT)];
2862	pModeData->uShwType = PGM_TYPE_PAE;
2863	pModeData->uGstType = PGM_TYPE_32BIT;
2864	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2865	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2866	rc = PGM_BTH_NAME_PAE_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2867
2868	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PAE)];
2869	pModeData->uShwType = PGM_TYPE_PAE;
2870	pModeData->uGstType = PGM_TYPE_PAE;
2871	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2872	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2873	rc = PGM_BTH_NAME_PAE_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2874
2875	#ifdef VBOX_WITH_64_BITS_GUESTS
2876	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_AMD64, PGM_TYPE_AMD64)];
2877	pModeData->uShwType = PGM_TYPE_AMD64;
2878	pModeData->uGstType = PGM_TYPE_AMD64;
2879	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2880	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2881	rc = PGM_BTH_NAME_AMD64_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2882	#endif
2883
2884	/* The nested paging mode. */
2885	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_REAL)];
2886	pModeData->uShwType = PGM_TYPE_NESTED;
2887	pModeData->uGstType = PGM_TYPE_REAL;
2888	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2889	rc = PGM_BTH_NAME_NESTED_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2890
2891	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGMMODE_PROTECTED)];
2892	pModeData->uShwType = PGM_TYPE_NESTED;
2893	pModeData->uGstType = PGM_TYPE_PROT;
2894	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2895	rc = PGM_BTH_NAME_NESTED_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2896
2897	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_32BIT)];
2898	pModeData->uShwType = PGM_TYPE_NESTED;
2899	pModeData->uGstType = PGM_TYPE_32BIT;
2900	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2901	rc = PGM_BTH_NAME_NESTED_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2902
2903	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_PAE)];
2904	pModeData->uShwType = PGM_TYPE_NESTED;
2905	pModeData->uGstType = PGM_TYPE_PAE;
2906	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2907	rc = PGM_BTH_NAME_NESTED_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2908
2909	#ifdef VBOX_WITH_64_BITS_GUESTS
2910	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2911	pModeData->uShwType = PGM_TYPE_NESTED;
2912	pModeData->uGstType = PGM_TYPE_AMD64;
2913	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2914	rc = PGM_BTH_NAME_NESTED_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2915	#endif
2916
2917	/* The shadow part of the nested callback mode depends on the host paging mode (AMD-V only). */
2918	switch (pVM->pgm.s.enmHostMode)
2919	{
2920	#if HC_ARCH_BITS == 32
2921	case SUPPAGINGMODE_32_BIT:
2922	case SUPPAGINGMODE_32_BIT_GLOBAL:
2923	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2924	{
2925	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2926	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2927	}
2928	# ifdef VBOX_WITH_64_BITS_GUESTS
2929	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2930	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2931	# endif
2932	break;
2933
2934	case SUPPAGINGMODE_PAE:
2935	case SUPPAGINGMODE_PAE_NX:
2936	case SUPPAGINGMODE_PAE_GLOBAL:
2937	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2938	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2939	{
2940	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2941	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2942	}
2943	# ifdef VBOX_WITH_64_BITS_GUESTS
2944	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2945	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2946	# endif
2947	break;
2948	#endif /* HC_ARCH_BITS == 32 */
2949
2950	#if HC_ARCH_BITS == 64 \|\| defined(RT_OS_DARWIN)
2951	case SUPPAGINGMODE_AMD64:
2952	case SUPPAGINGMODE_AMD64_GLOBAL:
2953	case SUPPAGINGMODE_AMD64_NX:
2954	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2955	# ifdef VBOX_WITH_64_BITS_GUESTS
2956	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_AMD64; i++)
2957	# else
2958	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2959	# endif
2960	{
2961	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2962	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2963	}
2964	break;
2965	#endif /* HC_ARCH_BITS == 64 \|\| RT_OS_DARWIN */
2966
2967	default:
2968	AssertFailed();
2969	break;
2970	}
2971
2972	/* Extended paging (EPT) / Intel VT-x */
2973	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_REAL)];
2974	pModeData->uShwType = PGM_TYPE_EPT;
2975	pModeData->uGstType = PGM_TYPE_REAL;
2976	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2977	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2978	rc = PGM_BTH_NAME_EPT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2979
2980	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_PROT)];
2981	pModeData->uShwType = PGM_TYPE_EPT;
2982	pModeData->uGstType = PGM_TYPE_PROT;
2983	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2984	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2985	rc = PGM_BTH_NAME_EPT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2986
2987	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_32BIT)];
2988	pModeData->uShwType = PGM_TYPE_EPT;
2989	pModeData->uGstType = PGM_TYPE_32BIT;
2990	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2991	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2992	rc = PGM_BTH_NAME_EPT_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2993
2994	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_PAE)];
2995	pModeData->uShwType = PGM_TYPE_EPT;
2996	pModeData->uGstType = PGM_TYPE_PAE;
2997	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2998	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2999	rc = PGM_BTH_NAME_EPT_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3000
3001	#ifdef VBOX_WITH_64_BITS_GUESTS
3002	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_AMD64)];
3003	pModeData->uShwType = PGM_TYPE_EPT;
3004	pModeData->uGstType = PGM_TYPE_AMD64;
3005	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3006	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3007	rc = PGM_BTH_NAME_EPT_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
3008	#endif
3009	return VINF_SUCCESS;
3010	}
3011
3012
3013	/**
3014	* Switch to different (or relocated in the relocate case) mode data.
3015	*
3016	* @param pVM The VM handle.
3017	* @param pVCpu The VMCPU to operate on.
3018	* @param enmShw The the shadow paging mode.
3019	* @param enmGst The the guest paging mode.
3020	*/
3021	static void pgmR3ModeDataSwitch(PVM pVM, PVMCPU pVCpu, PGMMODE enmShw, PGMMODE enmGst)
3022	{
3023	PPGMMODEDATA pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndexByMode(enmShw, enmGst)];
3024
3025	Assert(pModeData->uGstType == pgmModeToType(enmGst));
3026	Assert(pModeData->uShwType == pgmModeToType(enmShw));
3027
3028	/* shadow */
3029	pVCpu->pgm.s.pfnR3ShwRelocate = pModeData->pfnR3ShwRelocate;
3030	pVCpu->pgm.s.pfnR3ShwExit = pModeData->pfnR3ShwExit;
3031	pVCpu->pgm.s.pfnR3ShwGetPage = pModeData->pfnR3ShwGetPage;
3032	Assert(pVCpu->pgm.s.pfnR3ShwGetPage);
3033	pVCpu->pgm.s.pfnR3ShwModifyPage = pModeData->pfnR3ShwModifyPage;
3034
3035	pVCpu->pgm.s.pfnRCShwGetPage = pModeData->pfnRCShwGetPage;
3036	pVCpu->pgm.s.pfnRCShwModifyPage = pModeData->pfnRCShwModifyPage;
3037
3038	pVCpu->pgm.s.pfnR0ShwGetPage = pModeData->pfnR0ShwGetPage;
3039	pVCpu->pgm.s.pfnR0ShwModifyPage = pModeData->pfnR0ShwModifyPage;
3040
3041
3042	/* guest */
3043	pVCpu->pgm.s.pfnR3GstRelocate = pModeData->pfnR3GstRelocate;
3044	pVCpu->pgm.s.pfnR3GstExit = pModeData->pfnR3GstExit;
3045	pVCpu->pgm.s.pfnR3GstGetPage = pModeData->pfnR3GstGetPage;
3046	Assert(pVCpu->pgm.s.pfnR3GstGetPage);
3047	pVCpu->pgm.s.pfnR3GstModifyPage = pModeData->pfnR3GstModifyPage;
3048	pVCpu->pgm.s.pfnR3GstGetPDE = pModeData->pfnR3GstGetPDE;
3049	pVCpu->pgm.s.pfnRCGstGetPage = pModeData->pfnRCGstGetPage;
3050	pVCpu->pgm.s.pfnRCGstModifyPage = pModeData->pfnRCGstModifyPage;
3051	pVCpu->pgm.s.pfnRCGstGetPDE = pModeData->pfnRCGstGetPDE;
3052	pVCpu->pgm.s.pfnR0GstGetPage = pModeData->pfnR0GstGetPage;
3053	pVCpu->pgm.s.pfnR0GstModifyPage = pModeData->pfnR0GstModifyPage;
3054	pVCpu->pgm.s.pfnR0GstGetPDE = pModeData->pfnR0GstGetPDE;
3055
3056	/* both */
3057	pVCpu->pgm.s.pfnR3BthRelocate = pModeData->pfnR3BthRelocate;
3058	pVCpu->pgm.s.pfnR3BthInvalidatePage = pModeData->pfnR3BthInvalidatePage;
3059	pVCpu->pgm.s.pfnR3BthSyncCR3 = pModeData->pfnR3BthSyncCR3;
3060	Assert(pVCpu->pgm.s.pfnR3BthSyncCR3);
3061	pVCpu->pgm.s.pfnR3BthPrefetchPage = pModeData->pfnR3BthPrefetchPage;
3062	pVCpu->pgm.s.pfnR3BthVerifyAccessSyncPage = pModeData->pfnR3BthVerifyAccessSyncPage;
3063	#ifdef VBOX_STRICT
3064	pVCpu->pgm.s.pfnR3BthAssertCR3 = pModeData->pfnR3BthAssertCR3;
3065	#endif
3066	pVCpu->pgm.s.pfnR3BthMapCR3 = pModeData->pfnR3BthMapCR3;
3067	pVCpu->pgm.s.pfnR3BthUnmapCR3 = pModeData->pfnR3BthUnmapCR3;
3068
3069	pVCpu->pgm.s.pfnRCBthTrap0eHandler = pModeData->pfnRCBthTrap0eHandler;
3070	pVCpu->pgm.s.pfnRCBthInvalidatePage = pModeData->pfnRCBthInvalidatePage;
3071	pVCpu->pgm.s.pfnRCBthSyncCR3 = pModeData->pfnRCBthSyncCR3;
3072	pVCpu->pgm.s.pfnRCBthPrefetchPage = pModeData->pfnRCBthPrefetchPage;
3073	pVCpu->pgm.s.pfnRCBthVerifyAccessSyncPage = pModeData->pfnRCBthVerifyAccessSyncPage;
3074	#ifdef VBOX_STRICT
3075	pVCpu->pgm.s.pfnRCBthAssertCR3 = pModeData->pfnRCBthAssertCR3;
3076	#endif
3077	pVCpu->pgm.s.pfnRCBthMapCR3 = pModeData->pfnRCBthMapCR3;
3078	pVCpu->pgm.s.pfnRCBthUnmapCR3 = pModeData->pfnRCBthUnmapCR3;
3079
3080	pVCpu->pgm.s.pfnR0BthTrap0eHandler = pModeData->pfnR0BthTrap0eHandler;
3081	pVCpu->pgm.s.pfnR0BthInvalidatePage = pModeData->pfnR0BthInvalidatePage;
3082	pVCpu->pgm.s.pfnR0BthSyncCR3 = pModeData->pfnR0BthSyncCR3;
3083	pVCpu->pgm.s.pfnR0BthPrefetchPage = pModeData->pfnR0BthPrefetchPage;
3084	pVCpu->pgm.s.pfnR0BthVerifyAccessSyncPage = pModeData->pfnR0BthVerifyAccessSyncPage;
3085	#ifdef VBOX_STRICT
3086	pVCpu->pgm.s.pfnR0BthAssertCR3 = pModeData->pfnR0BthAssertCR3;
3087	#endif
3088	pVCpu->pgm.s.pfnR0BthMapCR3 = pModeData->pfnR0BthMapCR3;
3089	pVCpu->pgm.s.pfnR0BthUnmapCR3 = pModeData->pfnR0BthUnmapCR3;
3090	}
3091
3092
3093	/**
3094	* Calculates the shadow paging mode.
3095	*
3096	* @returns The shadow paging mode.
3097	* @param pVM VM handle.
3098	* @param enmGuestMode The guest mode.
3099	* @param enmHostMode The host mode.
3100	* @param enmShadowMode The current shadow mode.
3101	* @param penmSwitcher Where to store the switcher to use.
3102	* VMMSWITCHER_INVALID means no change.
3103	*/
3104	static PGMMODE pgmR3CalcShadowMode(PVM pVM, PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher)
3105	{
3106	VMMSWITCHER enmSwitcher = VMMSWITCHER_INVALID;
3107	switch (enmGuestMode)
3108	{
3109	/*
3110	* When switching to real or protected mode we don't change
3111	* anything since it's likely that we'll switch back pretty soon.
3112	*
3113	* During pgmR3InitPaging we'll end up here with PGMMODE_INVALID
3114	* and is supposed to determine which shadow paging and switcher to
3115	* use during init.
3116	*/
3117	case PGMMODE_REAL:
3118	case PGMMODE_PROTECTED:
3119	if ( enmShadowMode != PGMMODE_INVALID
3120	&& !HWACCMIsEnabled(pVM) /* always switch in hwaccm mode! */)
3121	break; /* (no change) */
3122
3123	switch (enmHostMode)
3124	{
3125	case SUPPAGINGMODE_32_BIT:
3126	case SUPPAGINGMODE_32_BIT_GLOBAL:
3127	enmShadowMode = PGMMODE_32_BIT;
3128	enmSwitcher = VMMSWITCHER_32_TO_32;
3129	break;
3130
3131	case SUPPAGINGMODE_PAE:
3132	case SUPPAGINGMODE_PAE_NX:
3133	case SUPPAGINGMODE_PAE_GLOBAL:
3134	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3135	enmShadowMode = PGMMODE_PAE;
3136	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3137	#ifdef DEBUG_bird
3138	if (RTEnvExist("VBOX_32BIT"))
3139	{
3140	enmShadowMode = PGMMODE_32_BIT;
3141	enmSwitcher = VMMSWITCHER_PAE_TO_32;
3142	}
3143	#endif
3144	break;
3145
3146	case SUPPAGINGMODE_AMD64:
3147	case SUPPAGINGMODE_AMD64_GLOBAL:
3148	case SUPPAGINGMODE_AMD64_NX:
3149	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3150	enmShadowMode = PGMMODE_PAE;
3151	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3152	#ifdef DEBUG_bird
3153	if (RTEnvExist("VBOX_32BIT"))
3154	{
3155	enmShadowMode = PGMMODE_32_BIT;
3156	enmSwitcher = VMMSWITCHER_AMD64_TO_32;
3157	}
3158	#endif
3159	break;
3160
3161	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3162	}
3163	break;
3164
3165	case PGMMODE_32_BIT:
3166	switch (enmHostMode)
3167	{
3168	case SUPPAGINGMODE_32_BIT:
3169	case SUPPAGINGMODE_32_BIT_GLOBAL:
3170	enmShadowMode = PGMMODE_32_BIT;
3171	enmSwitcher = VMMSWITCHER_32_TO_32;
3172	break;
3173
3174	case SUPPAGINGMODE_PAE:
3175	case SUPPAGINGMODE_PAE_NX:
3176	case SUPPAGINGMODE_PAE_GLOBAL:
3177	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3178	enmShadowMode = PGMMODE_PAE;
3179	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3180	#ifdef DEBUG_bird
3181	if (RTEnvExist("VBOX_32BIT"))
3182	{
3183	enmShadowMode = PGMMODE_32_BIT;
3184	enmSwitcher = VMMSWITCHER_PAE_TO_32;
3185	}
3186	#endif
3187	break;
3188
3189	case SUPPAGINGMODE_AMD64:
3190	case SUPPAGINGMODE_AMD64_GLOBAL:
3191	case SUPPAGINGMODE_AMD64_NX:
3192	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3193	enmShadowMode = PGMMODE_PAE;
3194	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3195	#ifdef DEBUG_bird
3196	if (RTEnvExist("VBOX_32BIT"))
3197	{
3198	enmShadowMode = PGMMODE_32_BIT;
3199	enmSwitcher = VMMSWITCHER_AMD64_TO_32;
3200	}
3201	#endif
3202	break;
3203
3204	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3205	}
3206	break;
3207
3208	case PGMMODE_PAE:
3209	case PGMMODE_PAE_NX: /** @todo This might require more switchers and guest+both modes. */
3210	switch (enmHostMode)
3211	{
3212	case SUPPAGINGMODE_32_BIT:
3213	case SUPPAGINGMODE_32_BIT_GLOBAL:
3214	enmShadowMode = PGMMODE_PAE;
3215	enmSwitcher = VMMSWITCHER_32_TO_PAE;
3216	break;
3217
3218	case SUPPAGINGMODE_PAE:
3219	case SUPPAGINGMODE_PAE_NX:
3220	case SUPPAGINGMODE_PAE_GLOBAL:
3221	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3222	enmShadowMode = PGMMODE_PAE;
3223	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3224	break;
3225
3226	case SUPPAGINGMODE_AMD64:
3227	case SUPPAGINGMODE_AMD64_GLOBAL:
3228	case SUPPAGINGMODE_AMD64_NX:
3229	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3230	enmShadowMode = PGMMODE_PAE;
3231	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3232	break;
3233
3234	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3235	}
3236	break;
3237
3238	case PGMMODE_AMD64:
3239	case PGMMODE_AMD64_NX:
3240	switch (enmHostMode)
3241	{
3242	case SUPPAGINGMODE_32_BIT:
3243	case SUPPAGINGMODE_32_BIT_GLOBAL:
3244	enmShadowMode = PGMMODE_AMD64;
3245	enmSwitcher = VMMSWITCHER_32_TO_AMD64;
3246	break;
3247
3248	case SUPPAGINGMODE_PAE:
3249	case SUPPAGINGMODE_PAE_NX:
3250	case SUPPAGINGMODE_PAE_GLOBAL:
3251	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3252	enmShadowMode = PGMMODE_AMD64;
3253	enmSwitcher = VMMSWITCHER_PAE_TO_AMD64;
3254	break;
3255
3256	case SUPPAGINGMODE_AMD64:
3257	case SUPPAGINGMODE_AMD64_GLOBAL:
3258	case SUPPAGINGMODE_AMD64_NX:
3259	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3260	enmShadowMode = PGMMODE_AMD64;
3261	enmSwitcher = VMMSWITCHER_AMD64_TO_AMD64;
3262	break;
3263
3264	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3265	}
3266	break;
3267
3268
3269	default:
3270	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
3271	*penmSwitcher = VMMSWITCHER_INVALID;
3272	return PGMMODE_INVALID;
3273	}
3274	/* Override the shadow mode is nested paging is active. */
3275	pVM->pgm.s.fNestedPaging = HWACCMIsNestedPagingActive(pVM);
3276	if (pVM->pgm.s.fNestedPaging)
3277	enmShadowMode = HWACCMGetShwPagingMode(pVM);
3278
3279	*penmSwitcher = enmSwitcher;
3280	return enmShadowMode;
3281	}
3282
3283
3284	/**
3285	* Performs the actual mode change.
3286	* This is called by PGMChangeMode and pgmR3InitPaging().
3287	*
3288	* @returns VBox status code. May suspend or power off the VM on error, but this
3289	* will trigger using FFs and not status codes.
3290	*
3291	* @param pVM VM handle.
3292	* @param pVCpu The VMCPU to operate on.
3293	* @param enmGuestMode The new guest mode. This is assumed to be different from
3294	* the current mode.
3295	*/
3296	VMMR3DECL(int) PGMR3ChangeMode(PVM pVM, PVMCPU pVCpu, PGMMODE enmGuestMode)
3297	{
3298	bool fIsOldGuestPagingMode64Bits = (pVCpu->pgm.s.enmGuestMode >= PGMMODE_AMD64);
3299	bool fIsNewGuestPagingMode64Bits = (enmGuestMode >= PGMMODE_AMD64);
3300
3301	Log(("PGMR3ChangeMode: Guest mode: %s -> %s\n", PGMGetModeName(pVCpu->pgm.s.enmGuestMode), PGMGetModeName(enmGuestMode)));
3302	STAM_REL_COUNTER_INC(&pVCpu->pgm.s.cGuestModeChanges);
3303
3304	/*
3305	* Calc the shadow mode and switcher.
3306	*/
3307	VMMSWITCHER enmSwitcher;
3308	PGMMODE enmShadowMode;
3309	enmShadowMode = pgmR3CalcShadowMode(pVM, enmGuestMode, pVM->pgm.s.enmHostMode, pVCpu->pgm.s.enmShadowMode, &enmSwitcher);
3310
3311	#ifdef VBOX_WITH_RAW_MODE
3312	if (enmSwitcher != VMMSWITCHER_INVALID)
3313	{
3314	/*
3315	* Select new switcher.
3316	*/
3317	int rc = VMMR3SelectSwitcher(pVM, enmSwitcher);
3318	if (RT_FAILURE(rc))
3319	{
3320	AssertReleaseMsgFailed(("VMMR3SelectSwitcher(%d) -> %Rrc\n", enmSwitcher, rc));
3321	return rc;
3322	}
3323	}
3324	#endif
3325
3326	/*
3327	* Exit old mode(s).
3328	*/
3329	#if HC_ARCH_BITS == 32
3330	/* The nested shadow paging mode for AMD-V does change when running 64 bits guests on 32 bits hosts; typically PAE <-> AMD64 */
3331	const bool fForceShwEnterExit = ( fIsOldGuestPagingMode64Bits != fIsNewGuestPagingMode64Bits
3332	&& enmShadowMode == PGMMODE_NESTED);
3333	#else
3334	const bool fForceShwEnterExit = false;
3335	#endif
3336	/* shadow */
3337	if ( enmShadowMode != pVCpu->pgm.s.enmShadowMode
3338	\|\| fForceShwEnterExit)
3339	{
3340	LogFlow(("PGMR3ChangeMode: Shadow mode: %s -> %s\n", PGMGetModeName(pVCpu->pgm.s.enmShadowMode), PGMGetModeName(enmShadowMode)));
3341	if (PGM_SHW_PFN(Exit, pVCpu))
3342	{
3343	int rc = PGM_SHW_PFN(Exit, pVCpu)(pVCpu);
3344	if (RT_FAILURE(rc))
3345	{
3346	AssertMsgFailed(("Exit failed for shadow mode %d: %Rrc\n", pVCpu->pgm.s.enmShadowMode, rc));
3347	return rc;
3348	}
3349	}
3350
3351	}
3352	else
3353	LogFlow(("PGMR3ChangeMode: Shadow mode remains: %s\n", PGMGetModeName(pVCpu->pgm.s.enmShadowMode)));
3354
3355	/* guest */
3356	if (PGM_GST_PFN(Exit, pVCpu))
3357	{
3358	int rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
3359	if (RT_FAILURE(rc))
3360	{
3361	AssertMsgFailed(("Exit failed for guest mode %d: %Rrc\n", pVCpu->pgm.s.enmGuestMode, rc));
3362	return rc;
3363	}
3364	}
3365
3366	/*
3367	* Load new paging mode data.
3368	*/
3369	pgmR3ModeDataSwitch(pVM, pVCpu, enmShadowMode, enmGuestMode);
3370
3371	/*
3372	* Enter new shadow mode (if changed).
3373	*/
3374	if ( enmShadowMode != pVCpu->pgm.s.enmShadowMode
3375	\|\| fForceShwEnterExit)
3376	{
3377	int rc;
3378	pVCpu->pgm.s.enmShadowMode = enmShadowMode;
3379	switch (enmShadowMode)
3380	{
3381	case PGMMODE_32_BIT:
3382	rc = PGM_SHW_NAME_32BIT(Enter)(pVCpu, false);
3383	break;
3384	case PGMMODE_PAE:
3385	case PGMMODE_PAE_NX:
3386	rc = PGM_SHW_NAME_PAE(Enter)(pVCpu, false);
3387	break;
3388	case PGMMODE_AMD64:
3389	case PGMMODE_AMD64_NX:
3390	rc = PGM_SHW_NAME_AMD64(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3391	break;
3392	case PGMMODE_NESTED:
3393	rc = PGM_SHW_NAME_NESTED(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3394	break;
3395	case PGMMODE_EPT:
3396	rc = PGM_SHW_NAME_EPT(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3397	break;
3398	case PGMMODE_REAL:
3399	case PGMMODE_PROTECTED:
3400	default:
3401	AssertReleaseMsgFailed(("enmShadowMode=%d\n", enmShadowMode));
3402	return VERR_INTERNAL_ERROR;
3403	}
3404	if (RT_FAILURE(rc))
3405	{
3406	AssertReleaseMsgFailed(("Entering enmShadowMode=%d failed: %Rrc\n", enmShadowMode, rc));
3407	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
3408	return rc;
3409	}
3410	}
3411
3412	/*
3413	* Always flag the necessary updates
3414	*/
3415	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3416
3417	/*
3418	* Enter the new guest and shadow+guest modes.
3419	*/
3420	int rc = -1;
3421	int rc2 = -1;
3422	RTGCPHYS GCPhysCR3 = NIL_RTGCPHYS;
3423	pVCpu->pgm.s.enmGuestMode = enmGuestMode;
3424	switch (enmGuestMode)
3425	{
3426	case PGMMODE_REAL:
3427	rc = PGM_GST_NAME_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3428	switch (pVCpu->pgm.s.enmShadowMode)
3429	{
3430	case PGMMODE_32_BIT:
3431	rc2 = PGM_BTH_NAME_32BIT_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3432	break;
3433	case PGMMODE_PAE:
3434	case PGMMODE_PAE_NX:
3435	rc2 = PGM_BTH_NAME_PAE_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3436	break;
3437	case PGMMODE_NESTED:
3438	rc2 = PGM_BTH_NAME_NESTED_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3439	break;
3440	case PGMMODE_EPT:
3441	rc2 = PGM_BTH_NAME_EPT_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3442	break;
3443	case PGMMODE_AMD64:
3444	case PGMMODE_AMD64_NX:
3445	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3446	default: AssertFailed(); break;
3447	}
3448	break;
3449
3450	case PGMMODE_PROTECTED:
3451	rc = PGM_GST_NAME_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3452	switch (pVCpu->pgm.s.enmShadowMode)
3453	{
3454	case PGMMODE_32_BIT:
3455	rc2 = PGM_BTH_NAME_32BIT_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3456	break;
3457	case PGMMODE_PAE:
3458	case PGMMODE_PAE_NX:
3459	rc2 = PGM_BTH_NAME_PAE_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3460	break;
3461	case PGMMODE_NESTED:
3462	rc2 = PGM_BTH_NAME_NESTED_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3463	break;
3464	case PGMMODE_EPT:
3465	rc2 = PGM_BTH_NAME_EPT_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3466	break;
3467	case PGMMODE_AMD64:
3468	case PGMMODE_AMD64_NX:
3469	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3470	default: AssertFailed(); break;
3471	}
3472	break;
3473
3474	case PGMMODE_32_BIT:
3475	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & X86_CR3_PAGE_MASK;
3476	rc = PGM_GST_NAME_32BIT(Enter)(pVCpu, GCPhysCR3);
3477	switch (pVCpu->pgm.s.enmShadowMode)
3478	{
3479	case PGMMODE_32_BIT:
3480	rc2 = PGM_BTH_NAME_32BIT_32BIT(Enter)(pVCpu, GCPhysCR3);
3481	break;
3482	case PGMMODE_PAE:
3483	case PGMMODE_PAE_NX:
3484	rc2 = PGM_BTH_NAME_PAE_32BIT(Enter)(pVCpu, GCPhysCR3);
3485	break;
3486	case PGMMODE_NESTED:
3487	rc2 = PGM_BTH_NAME_NESTED_32BIT(Enter)(pVCpu, GCPhysCR3);
3488	break;
3489	case PGMMODE_EPT:
3490	rc2 = PGM_BTH_NAME_EPT_32BIT(Enter)(pVCpu, GCPhysCR3);
3491	break;
3492	case PGMMODE_AMD64:
3493	case PGMMODE_AMD64_NX:
3494	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3495	default: AssertFailed(); break;
3496	}
3497	break;
3498
3499	case PGMMODE_PAE_NX:
3500	case PGMMODE_PAE:
3501	{
3502	uint32_t u32Dummy, u32Features;
3503
3504	CPUMGetGuestCpuId(pVCpu, 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
3505	if (!(u32Features & X86_CPUID_FEATURE_EDX_PAE))
3506	return VMSetRuntimeError(pVM, VMSETRTERR_FLAGS_FATAL, "PAEmode",
3507	N_("The guest is trying to switch to the PAE mode which is currently disabled by default in VirtualBox. PAE support can be enabled using the VM settings (General/Advanced)"));
3508
3509	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & X86_CR3_PAE_PAGE_MASK;
3510	rc = PGM_GST_NAME_PAE(Enter)(pVCpu, GCPhysCR3);
3511	switch (pVCpu->pgm.s.enmShadowMode)
3512	{
3513	case PGMMODE_PAE:
3514	case PGMMODE_PAE_NX:
3515	rc2 = PGM_BTH_NAME_PAE_PAE(Enter)(pVCpu, GCPhysCR3);
3516	break;
3517	case PGMMODE_NESTED:
3518	rc2 = PGM_BTH_NAME_NESTED_PAE(Enter)(pVCpu, GCPhysCR3);
3519	break;
3520	case PGMMODE_EPT:
3521	rc2 = PGM_BTH_NAME_EPT_PAE(Enter)(pVCpu, GCPhysCR3);
3522	break;
3523	case PGMMODE_32_BIT:
3524	case PGMMODE_AMD64:
3525	case PGMMODE_AMD64_NX:
3526	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3527	default: AssertFailed(); break;
3528	}
3529	break;
3530	}
3531
3532	#ifdef VBOX_WITH_64_BITS_GUESTS
3533	case PGMMODE_AMD64_NX:
3534	case PGMMODE_AMD64:
3535	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & UINT64_C(0xfffffffffffff000); /** @todo define this mask! */
3536	rc = PGM_GST_NAME_AMD64(Enter)(pVCpu, GCPhysCR3);
3537	switch (pVCpu->pgm.s.enmShadowMode)
3538	{
3539	case PGMMODE_AMD64:
3540	case PGMMODE_AMD64_NX:
3541	rc2 = PGM_BTH_NAME_AMD64_AMD64(Enter)(pVCpu, GCPhysCR3);
3542	break;
3543	case PGMMODE_NESTED:
3544	rc2 = PGM_BTH_NAME_NESTED_AMD64(Enter)(pVCpu, GCPhysCR3);
3545	break;
3546	case PGMMODE_EPT:
3547	rc2 = PGM_BTH_NAME_EPT_AMD64(Enter)(pVCpu, GCPhysCR3);
3548	break;
3549	case PGMMODE_32_BIT:
3550	case PGMMODE_PAE:
3551	case PGMMODE_PAE_NX:
3552	AssertMsgFailed(("Should use AMD64 shadow mode!\n"));
3553	default: AssertFailed(); break;
3554	}
3555	break;
3556	#endif
3557
3558	default:
3559	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
3560	rc = VERR_NOT_IMPLEMENTED;
3561	break;
3562	}
3563
3564	/* status codes. */
3565	AssertRC(rc);
3566	AssertRC(rc2);
3567	if (RT_SUCCESS(rc))
3568	{
3569	rc = rc2;
3570	if (RT_SUCCESS(rc)) /* no informational status codes. */
3571	rc = VINF_SUCCESS;
3572	}
3573
3574	/* Notify HWACCM as well. */
3575	HWACCMR3PagingModeChanged(pVM, pVCpu, pVCpu->pgm.s.enmShadowMode, pVCpu->pgm.s.enmGuestMode);
3576	return rc;
3577	}
3578
3579
3580	/**
3581	* Called by pgmPoolFlushAllInt prior to flushing the pool.
3582	*
3583	* @returns VBox status code, fully asserted.
3584	* @param pVM The VM handle.
3585	* @param pVCpu The VMCPU to operate on.
3586	*/
3587	int pgmR3ExitShadowModeBeforePoolFlush(PVM pVM, PVMCPU pVCpu)
3588	{
3589	/* Unmap the old CR3 value before flushing everything. */
3590	int rc = PGM_BTH_PFN(UnmapCR3, pVCpu)(pVCpu);
3591	AssertRC(rc);
3592
3593	/* Exit the current shadow paging mode as well; nested paging and EPT use a root CR3 which will get flushed here. */
3594	rc = PGM_SHW_PFN(Exit, pVCpu)(pVCpu);
3595	AssertRC(rc);
3596	Assert(pVCpu->pgm.s.pShwPageCR3R3 == NULL);
3597	return rc;
3598	}
3599
3600
3601	/**
3602	* Called by pgmPoolFlushAllInt after flushing the pool.
3603	*
3604	* @returns VBox status code, fully asserted.
3605	* @param pVM The VM handle.
3606	* @param pVCpu The VMCPU to operate on.
3607	*/
3608	int pgmR3ReEnterShadowModeAfterPoolFlush(PVM pVM, PVMCPU pVCpu)
3609	{
3610	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
3611	int rc = PGMR3ChangeMode(pVM, pVCpu, PGMGetGuestMode(pVCpu));
3612	Assert(VMCPU_FF_ISSET(pVCpu, VMCPU_FF_PGM_SYNC_CR3));
3613	AssertRCReturn(rc, rc);
3614	AssertRCSuccessReturn(rc, VERR_IPE_UNEXPECTED_INFO_STATUS);
3615
3616	Assert(pVCpu->pgm.s.pShwPageCR3R3 != NULL);
3617	AssertMsg( pVCpu->pgm.s.enmShadowMode >= PGMMODE_NESTED
3618	\|\| CPUMGetHyperCR3(pVCpu) == PGMGetHyperCR3(pVCpu),
3619	("%RHp != %RHp %s\n", (RTHCPHYS)CPUMGetHyperCR3(pVCpu), PGMGetHyperCR3(pVCpu), PGMGetModeName(pVCpu->pgm.s.enmShadowMode)));
3620	return rc;
3621	}
3622
3623	#ifdef VBOX_WITH_DEBUGGER
3624
3625	/**
3626	* The '.pgmram' command.
3627	*
3628	* @returns VBox status.
3629	* @param pCmd Pointer to the command descriptor (as registered).
3630	* @param pCmdHlp Pointer to command helper functions.
3631	* @param pVM Pointer to the current VM (if any).
3632	* @param paArgs Pointer to (readonly) array of arguments.
3633	* @param cArgs Number of arguments in the array.
3634	*/
3635	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3636	{
3637	/*
3638	* Validate input.
3639	*/
3640	if (!pVM)
3641	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3642	if (!pVM->pgm.s.pRamRangesXR3)
3643	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Sorry, no Ram is registered.\n");
3644
3645	/*
3646	* Dump the ranges.
3647	*/
3648	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "From - To (incl) pvHC\n");
3649	PPGMRAMRANGE pRam;
3650	for (pRam = pVM->pgm.s.pRamRangesXR3; pRam; pRam = pRam->pNextR3)
3651	{
3652	rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL,
3653	"%RGp - %RGp %p\n",
3654	pRam->GCPhys, pRam->GCPhysLast, pRam->pvR3);
3655	if (RT_FAILURE(rc))
3656	return rc;
3657	}
3658
3659	return VINF_SUCCESS;
3660	}
3661
3662
3663	/**
3664	* The '.pgmerror' and '.pgmerroroff' commands.
3665	*
3666	* @returns VBox status.
3667	* @param pCmd Pointer to the command descriptor (as registered).
3668	* @param pCmdHlp Pointer to command helper functions.
3669	* @param pVM Pointer to the current VM (if any).
3670	* @param paArgs Pointer to (readonly) array of arguments.
3671	* @param cArgs Number of arguments in the array.
3672	*/
3673	static DECLCALLBACK(int) pgmR3CmdError(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3674	{
3675	/*
3676	* Validate input.
3677	*/
3678	if (!pVM)
3679	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3680	AssertReturn(cArgs == 0 \|\| (cArgs == 1 && paArgs[0].enmType == DBGCVAR_TYPE_STRING),
3681	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Hit bug in the parser.\n"));
3682
3683	if (!cArgs)
3684	{
3685	/*
3686	* Print the list of error injection locations with status.
3687	*/
3688	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "PGM error inject locations:\n");
3689	pCmdHlp->pfnPrintf(pCmdHlp, NULL, " handy - %RTbool\n", pVM->pgm.s.fErrInjHandyPages);
3690	}
3691	else
3692	{
3693
3694	/*
3695	* String switch on where to inject the error.
3696	*/
3697	bool const fNewState = !strcmp(pCmd->pszCmd, "pgmerror");
3698	const char *pszWhere = paArgs[0].u.pszString;
3699	if (!strcmp(pszWhere, "handy"))
3700	ASMAtomicWriteBool(&pVM->pgm.s.fErrInjHandyPages, fNewState);
3701	else
3702	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Invalid 'where' value: %s.\n", pszWhere);
3703	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "done\n");
3704	}
3705	return VINF_SUCCESS;
3706	}
3707
3708
3709	/**
3710	* The '.pgmsync' command.
3711	*
3712	* @returns VBox status.
3713	* @param pCmd Pointer to the command descriptor (as registered).
3714	* @param pCmdHlp Pointer to command helper functions.
3715	* @param pVM Pointer to the current VM (if any).
3716	* @param paArgs Pointer to (readonly) array of arguments.
3717	* @param cArgs Number of arguments in the array.
3718	*/
3719	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3720	{
3721	/** @todo SMP support */
3722
3723	/*
3724	* Validate input.
3725	*/
3726	if (!pVM)
3727	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3728
3729	PVMCPU pVCpu = &pVM->aCpus[0];
3730
3731	/*
3732	* Force page directory sync.
3733	*/
3734	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3735
3736	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Forcing page directory sync.\n");
3737	if (RT_FAILURE(rc))
3738	return rc;
3739
3740	return VINF_SUCCESS;
3741	}
3742
3743
3744	#ifdef VBOX_STRICT
3745	/**
3746	* The '.pgmassertcr3' command.
3747	*
3748	* @returns VBox status.
3749	* @param pCmd Pointer to the command descriptor (as registered).
3750	* @param pCmdHlp Pointer to command helper functions.
3751	* @param pVM Pointer to the current VM (if any).
3752	* @param paArgs Pointer to (readonly) array of arguments.
3753	* @param cArgs Number of arguments in the array.
3754	*/
3755	static DECLCALLBACK(int) pgmR3CmdAssertCR3(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3756	{
3757	/** @todo SMP support!! */
3758
3759	/*
3760	* Validate input.
3761	*/
3762	if (!pVM)
3763	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3764
3765	PVMCPU pVCpu = &pVM->aCpus[0];
3766
3767	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Checking shadow CR3 page tables for consistency.\n");
3768	if (RT_FAILURE(rc))
3769	return rc;
3770
3771	PGMAssertCR3(pVM, pVCpu, CPUMGetGuestCR3(pVCpu), CPUMGetGuestCR4(pVCpu));
3772
3773	return VINF_SUCCESS;
3774	}
3775	#endif /* VBOX_STRICT */
3776
3777
3778	/**
3779	* The '.pgmsyncalways' command.
3780	*
3781	* @returns VBox status.
3782	* @param pCmd Pointer to the command descriptor (as registered).
3783	* @param pCmdHlp Pointer to command helper functions.
3784	* @param pVM Pointer to the current VM (if any).
3785	* @param paArgs Pointer to (readonly) array of arguments.
3786	* @param cArgs Number of arguments in the array.
3787	*/
3788	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3789	{
3790	/** @todo SMP support!! */
3791	PVMCPU pVCpu = &pVM->aCpus[0];
3792
3793	/*
3794	* Validate input.
3795	*/
3796	if (!pVM)
3797	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3798
3799	/*
3800	* Force page directory sync.
3801	*/
3802	if (pVCpu->pgm.s.fSyncFlags & PGM_SYNC_ALWAYS)
3803	{
3804	ASMAtomicAndU32(&pVCpu->pgm.s.fSyncFlags, ~PGM_SYNC_ALWAYS);
3805	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Disabled permanent forced page directory syncing.\n");
3806	}
3807	else
3808	{
3809	ASMAtomicOrU32(&pVCpu->pgm.s.fSyncFlags, PGM_SYNC_ALWAYS);
3810	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3811	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Enabled permanent forced page directory syncing.\n");
3812	}
3813	}
3814
3815
3816	/**
3817	* The '.pgmphystofile' command.
3818	*
3819	* @returns VBox status.
3820	* @param pCmd Pointer to the command descriptor (as registered).
3821	* @param pCmdHlp Pointer to command helper functions.
3822	* @param pVM Pointer to the current VM (if any).
3823	* @param paArgs Pointer to (readonly) array of arguments.
3824	* @param cArgs Number of arguments in the array.
3825	*/
3826	static DECLCALLBACK(int) pgmR3CmdPhysToFile(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3827	{
3828	/*
3829	* Validate input.
3830	*/
3831	if (!pVM)
3832	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3833	if ( cArgs < 1
3834	\|\| cArgs > 2
3835	\|\| paArgs[0].enmType != DBGCVAR_TYPE_STRING
3836	\|\| ( cArgs > 1
3837	&& paArgs[1].enmType != DBGCVAR_TYPE_STRING))
3838	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: parser error, invalid arguments.\n");
3839	if ( cArgs >= 2
3840	&& strcmp(paArgs[1].u.pszString, "nozero"))
3841	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Invalid 2nd argument '%s', must be 'nozero'.\n", paArgs[1].u.pszString);
3842	bool fIncZeroPgs = cArgs < 2;
3843
3844	/*
3845	* Open the output file and get the ram parameters.
3846	*/
3847	RTFILE hFile;
3848	int rc = RTFileOpen(&hFile, paArgs[0].u.pszString, RTFILE_O_WRITE \| RTFILE_O_CREATE_REPLACE \| RTFILE_O_DENY_WRITE);
3849	if (RT_FAILURE(rc))
3850	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileOpen(,'%s',) -> %Rrc.\n", paArgs[0].u.pszString, rc);
3851
3852	uint32_t cbRamHole = 0;
3853	CFGMR3QueryU32Def(CFGMR3GetRoot(pVM), "RamHoleSize", &cbRamHole, MM_RAM_HOLE_SIZE_DEFAULT);
3854	uint64_t cbRam = 0;
3855	CFGMR3QueryU64Def(CFGMR3GetRoot(pVM), "RamSize", &cbRam, 0);
3856	RTGCPHYS GCPhysEnd = cbRam + cbRamHole;
3857
3858	/*
3859	* Dump the physical memory, page by page.
3860	*/
3861	RTGCPHYS GCPhys = 0;
3862	char abZeroPg[PAGE_SIZE];
3863	RT_ZERO(abZeroPg);
3864
3865	pgmLock(pVM);
3866	for (PPGMRAMRANGE pRam = pVM->pgm.s.pRamRangesXR3;
3867	pRam && pRam->GCPhys < GCPhysEnd && RT_SUCCESS(rc);
3868	pRam = pRam->pNextR3)
3869	{
3870	/* fill the gap */
3871	if (pRam->GCPhys > GCPhys && fIncZeroPgs)
3872	{
3873	while (pRam->GCPhys > GCPhys && RT_SUCCESS(rc))
3874	{
3875	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
3876	GCPhys += PAGE_SIZE;
3877	}
3878	}
3879
3880	PCPGMPAGE pPage = &pRam->aPages[0];
3881	while (GCPhys < pRam->GCPhysLast && RT_SUCCESS(rc))
3882	{
3883	if ( PGM_PAGE_IS_ZERO(pPage)
3884	\|\| PGM_PAGE_IS_BALLOONED(pPage))
3885	{
3886	if (fIncZeroPgs)
3887	{
3888	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
3889	if (RT_FAILURE(rc))
3890	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3891	}
3892	}
3893	else
3894	{
3895	switch (PGM_PAGE_GET_TYPE(pPage))
3896	{
3897	case PGMPAGETYPE_RAM:
3898	case PGMPAGETYPE_ROM_SHADOW: /* trouble?? */
3899	case PGMPAGETYPE_ROM:
3900	case PGMPAGETYPE_MMIO2:
3901	{
3902	void const *pvPage;
3903	PGMPAGEMAPLOCK Lock;
3904	rc = PGMPhysGCPhys2CCPtrReadOnly(pVM, GCPhys, &pvPage, &Lock);
3905	if (RT_SUCCESS(rc))
3906	{
3907	rc = RTFileWrite(hFile, pvPage, PAGE_SIZE, NULL);
3908	PGMPhysReleasePageMappingLock(pVM, &Lock);
3909	if (RT_FAILURE(rc))
3910	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3911	}
3912	else
3913	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: PGMPhysGCPhys2CCPtrReadOnly -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3914	break;
3915	}
3916
3917	default:
3918	AssertFailed();
3919	case PGMPAGETYPE_MMIO2_ALIAS_MMIO:
3920	case PGMPAGETYPE_MMIO:
3921	if (fIncZeroPgs)
3922	{
3923	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
3924	if (RT_FAILURE(rc))
3925	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3926	}
3927	break;
3928	}
3929	}
3930
3931
3932	/* advance */
3933	GCPhys += PAGE_SIZE;
3934	pPage++;
3935	}
3936	}
3937	pgmUnlock(pVM);
3938
3939	RTFileClose(hFile);
3940	if (RT_SUCCESS(rc))
3941	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Successfully saved physical memory to '%s'.\n", paArgs[0].u.pszString);
3942	return VINF_SUCCESS;
3943	}
3944
3945	#endif /* VBOX_WITH_DEBUGGER */
3946
3947	/**
3948	* pvUser argument of the pgmR3CheckIntegrity*Node callbacks.
3949	*/
3950	typedef struct PGMCHECKINTARGS
3951	{
3952	bool fLeftToRight; /*< true: left-to-right; false: right-to-left. /
3953	PPGMPHYSHANDLER pPrevPhys;
3954	PPGMVIRTHANDLER pPrevVirt;
3955	PPGMPHYS2VIRTHANDLER pPrevPhys2Virt;
3956	PVM pVM;
3957	} PGMCHECKINTARGS, *PPGMCHECKINTARGS;
3958
3959	/**
3960	* Validate a node in the physical handler tree.
3961	*
3962	* @returns 0 on if ok, other wise 1.
3963	* @param pNode The handler node.
3964	* @param pvUser pVM.
3965	*/
3966	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
3967	{
3968	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3969	PPGMPHYSHANDLER pCur = (PPGMPHYSHANDLER)pNode;
3970	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
3971	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGp-%RGp %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3972	AssertReleaseMsg( !pArgs->pPrevPhys
3973	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys->Core.KeyLast > pCur->Core.Key),
3974	("pPrevPhys=%p %RGp-%RGp %s\n"
3975	" pCur=%p %RGp-%RGp %s\n",
3976	pArgs->pPrevPhys, pArgs->pPrevPhys->Core.Key, pArgs->pPrevPhys->Core.KeyLast, pArgs->pPrevPhys->pszDesc,
3977	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3978	pArgs->pPrevPhys = pCur;
3979	return 0;
3980	}
3981
3982
3983	/**
3984	* Validate a node in the virtual handler tree.
3985	*
3986	* @returns 0 on if ok, other wise 1.
3987	* @param pNode The handler node.
3988	* @param pvUser pVM.
3989	*/
3990	static DECLCALLBACK(int) pgmR3CheckIntegrityVirtHandlerNode(PAVLROGCPTRNODECORE pNode, void *pvUser)
3991	{
3992	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3993	PPGMVIRTHANDLER pCur = (PPGMVIRTHANDLER)pNode;
3994	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
3995	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGv-%RGv %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3996	AssertReleaseMsg( !pArgs->pPrevVirt
3997	\|\| (pArgs->fLeftToRight ? pArgs->pPrevVirt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevVirt->Core.KeyLast > pCur->Core.Key),
3998	("pPrevVirt=%p %RGv-%RGv %s\n"
3999	" pCur=%p %RGv-%RGv %s\n",
4000	pArgs->pPrevVirt, pArgs->pPrevVirt->Core.Key, pArgs->pPrevVirt->Core.KeyLast, pArgs->pPrevVirt->pszDesc,
4001	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
4002	for (unsigned iPage = 0; iPage < pCur->cPages; iPage++)
4003	{
4004	AssertReleaseMsg(pCur->aPhysToVirt[iPage].offVirtHandler == -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage]),
4005	("pCur=%p %RGv-%RGv %s\n"
4006	"iPage=%d offVirtHandle=%#x expected %#x\n",
4007	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc,
4008	iPage, pCur->aPhysToVirt[iPage].offVirtHandler, -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage])));
4009	}
4010	pArgs->pPrevVirt = pCur;
4011	return 0;
4012	}
4013
4014
4015	/**
4016	* Validate a node in the virtual handler tree.
4017	*
4018	* @returns 0 on if ok, other wise 1.
4019	* @param pNode The handler node.
4020	* @param pvUser pVM.
4021	*/
4022	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysToVirtHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
4023	{
4024	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
4025	PPGMPHYS2VIRTHANDLER pCur = (PPGMPHYS2VIRTHANDLER)pNode;
4026	AssertReleaseMsgReturn(!((uintptr_t)pCur & 3), ("\n"), 1);
4027	AssertReleaseMsgReturn(!(pCur->offVirtHandler & 3), ("\n"), 1);
4028	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGp-%RGp\n", pCur, pCur->Core.Key, pCur->Core.KeyLast));
4029	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
4030	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
4031	("pPrevPhys2Virt=%p %RGp-%RGp\n"
4032	" pCur=%p %RGp-%RGp\n",
4033	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
4034	pCur, pCur->Core.Key, pCur->Core.KeyLast));
4035	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
4036	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
4037	("pPrevPhys2Virt=%p %RGp-%RGp\n"
4038	" pCur=%p %RGp-%RGp\n",
4039	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
4040	pCur, pCur->Core.Key, pCur->Core.KeyLast));
4041	AssertReleaseMsg((pCur->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD),
4042	("pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4043	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
4044	if (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK)
4045	{
4046	PPGMPHYS2VIRTHANDLER pCur2 = pCur;
4047	for (;;)
4048	{
4049	pCur2 = (PPGMPHYS2VIRTHANDLER)((intptr_t)pCur + (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK));
4050	AssertReleaseMsg(pCur2 != pCur,
4051	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4052	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
4053	AssertReleaseMsg((pCur2->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == PGMPHYS2VIRTHANDLER_IN_TREE,
4054	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4055	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4056	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4057	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4058	AssertReleaseMsg((pCur2->Core.Key ^ pCur->Core.Key) < PAGE_SIZE,
4059	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4060	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4061	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4062	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4063	AssertReleaseMsg((pCur2->Core.KeyLast ^ pCur->Core.KeyLast) < PAGE_SIZE,
4064	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4065	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4066	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4067	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4068	if (!(pCur2->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK))
4069	break;
4070	}
4071	}
4072
4073	pArgs->pPrevPhys2Virt = pCur;
4074	return 0;
4075	}
4076
4077
4078	/**
4079	* Perform an integrity check on the PGM component.
4080	*
4081	* @returns VINF_SUCCESS if everything is fine.
4082	* @returns VBox error status after asserting on integrity breach.
4083	* @param pVM The VM handle.
4084	*/
4085	VMMR3DECL(int) PGMR3CheckIntegrity(PVM pVM)
4086	{
4087	AssertReleaseReturn(pVM->pgm.s.offVM, VERR_INTERNAL_ERROR);
4088
4089	/*
4090	* Check the trees.
4091	*/
4092	int cErrors = 0;
4093	const static PGMCHECKINTARGS s_LeftToRight = { true, NULL, NULL, NULL, pVM };
4094	const static PGMCHECKINTARGS s_RightToLeft = { false, NULL, NULL, NULL, pVM };
4095	PGMCHECKINTARGS Args = s_LeftToRight;
4096	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, true, pgmR3CheckIntegrityPhysHandlerNode, &Args);
4097	Args = s_RightToLeft;
4098	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, false, pgmR3CheckIntegrityPhysHandlerNode, &Args);
4099	Args = s_LeftToRight;
4100	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->VirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4101	Args = s_RightToLeft;
4102	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->VirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4103	Args = s_LeftToRight;
4104	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->HyperVirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4105	Args = s_RightToLeft;
4106	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->HyperVirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4107	Args = s_LeftToRight;
4108	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysToVirtHandlers, true, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
4109	Args = s_RightToLeft;
4110	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysToVirtHandlers, false, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
4111
4112	return !cErrors ? VINF_SUCCESS : VERR_INTERNAL_ERROR;
4113	}
4114

Note: See TracBrowser for help on using the repository browser.

source: vbox/trunk/src/VBox/VMM/VMMR3/PGM.cpp@ 39025

Download in other formats: