PGM.cpp@ 34942

Last change on this file since 34942 was 34326, checked in by vboxsync, 14 years ago
VMM: Removed the XXXInitCPU and XXXTermCPU methods since all but the HWACCM ones where stubs and the XXXTermCPU bits was not called in all expected paths. The HWACCMR3InitCPU was hooked up as a VMINITCOMPLETED_RING3 hook, essentially leaving it's position in the order of things unchanged, while the HWACCMR3TermCPU call was made static without changing its position at the end of HWACCMR3Term.
Property svn:eol-style set to `native` Property svn:keywords set to `Id`
File size: 197.8 KB

Line
1	/* $Id: PGM.cpp 34326 2010-11-24 14:03:55Z vboxsync $ */
2	/** @file
3	* PGM - Page Manager and Monitor. (Mixing stuff here, not good?)
4	*/
5
6	/*
7	* Copyright (C) 2006-2010 Oracle Corporation
8	*
9	* This file is part of VirtualBox Open Source Edition (OSE), as
10	* available from http://www.virtualbox.org. This file is free software;
11	* you can redistribute it and/or modify it under the terms of the GNU
12	* General Public License (GPL) as published by the Free Software
13	* Foundation, in version 2 as it comes in the "COPYING" file of the
14	* VirtualBox OSE distribution. VirtualBox OSE is distributed in the
15	* hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
16	*/
17
18
19	/** @page pg_pgm PGM - The Page Manager and Monitor
20	*
21	* @see grp_pgm,
22	* @ref pg_pgm_pool,
23	* @ref pg_pgm_phys.
24	*
25	*
26	* @section sec_pgm_modes Paging Modes
27	*
28	* There are three memory contexts: Host Context (HC), Guest Context (GC)
29	* and intermediate context. When talking about paging HC can also be referred to
30	* as "host paging", and GC referred to as "shadow paging".
31	*
32	* We define three basic paging modes: 32-bit, PAE and AMD64. The host paging mode
33	* is defined by the host operating system. The mode used in the shadow paging mode
34	* depends on the host paging mode and what the mode the guest is currently in. The
35	* following relation between the two is defined:
36	*
37	* @verbatim
38	Host > 32-bit \| PAE \| AMD64 \|
39	Guest \| \| \| \|
40	==v================================
41	32-bit 32-bit PAE PAE
42	-------\|--------\|--------\|--------\|
43	PAE PAE PAE PAE
44	-------\|--------\|--------\|--------\|
45	AMD64 AMD64 AMD64 AMD64
46	-------\|--------\|--------\|--------\| @endverbatim
47	*
48	* All configuration except those in the diagonal (upper left) are expected to
49	* require special effort from the switcher (i.e. a bit slower).
50	*
51	*
52	*
53	*
54	* @section sec_pgm_shw The Shadow Memory Context
55	*
56	*
57	* [..]
58	*
59	* Because of guest context mappings requires PDPT and PML4 entries to allow
60	* writing on AMD64, the two upper levels will have fixed flags whatever the
61	* guest is thinking of using there. So, when shadowing the PD level we will
62	* calculate the effective flags of PD and all the higher levels. In legacy
63	* PAE mode this only applies to the PWT and PCD bits (the rest are
64	* ignored/reserved/MBZ). We will ignore those bits for the present.
65	*
66	*
67	*
68	* @section sec_pgm_int The Intermediate Memory Context
69	*
70	* The world switch goes thru an intermediate memory context which purpose it is
71	* to provide different mappings of the switcher code. All guest mappings are also
72	* present in this context.
73	*
74	* The switcher code is mapped at the same location as on the host, at an
75	* identity mapped location (physical equals virtual address), and at the
76	* hypervisor location. The identity mapped location is for when the world
77	* switches that involves disabling paging.
78	*
79	* PGM maintain page tables for 32-bit, PAE and AMD64 paging modes. This
80	* simplifies switching guest CPU mode and consistency at the cost of more
81	* code to do the work. All memory use for those page tables is located below
82	* 4GB (this includes page tables for guest context mappings).
83	*
84	*
85	* @subsection subsec_pgm_int_gc Guest Context Mappings
86	*
87	* During assignment and relocation of a guest context mapping the intermediate
88	* memory context is used to verify the new location.
89	*
90	* Guest context mappings are currently restricted to below 4GB, for reasons
91	* of simplicity. This may change when we implement AMD64 support.
92	*
93	*
94	*
95	*
96	* @section sec_pgm_misc Misc
97	*
98	* @subsection subsec_pgm_misc_diff Differences Between Legacy PAE and Long Mode PAE
99	*
100	* The differences between legacy PAE and long mode PAE are:
101	* -# PDPE bits 1, 2, 5 and 6 are defined differently. In leagcy mode they are
102	* all marked down as must-be-zero, while in long mode 1, 2 and 5 have the
103	* usual meanings while 6 is ignored (AMD). This means that upon switching to
104	* legacy PAE mode we'll have to clear these bits and when going to long mode
105	* they must be set. This applies to both intermediate and shadow contexts,
106	* however we don't need to do it for the intermediate one since we're
107	* executing with CR0.WP at that time.
108	* -# CR3 allows a 32-byte aligned address in legacy mode, while in long mode
109	* a page aligned one is required.
110	*
111	*
112	* @section sec_pgm_handlers Access Handlers
113	*
114	* Placeholder.
115	*
116	*
117	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
118	*
119	* Placeholder.
120	*
121	*
122	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
123	*
124	* We currently implement three types of virtual access handlers: ALL, WRITE
125	* and HYPERVISOR (WRITE). See PGMVIRTHANDLERTYPE for some more details.
126	*
127	* The HYPERVISOR access handlers is kept in a separate tree since it doesn't apply
128	* to physical pages (PGMTREES::HyperVirtHandlers) and only needs to be consulted in
129	* a special \#PF case. The ALL and WRITE are in the PGMTREES::VirtHandlers tree, the
130	* rest of this section is going to be about these handlers.
131	*
132	* We'll go thru the life cycle of a handler and try make sense of it all, don't know
133	* how successful this is gonna be...
134	*
135	* 1. A handler is registered thru the PGMR3HandlerVirtualRegister and
136	* PGMHandlerVirtualRegisterEx APIs. We check for conflicting virtual handlers
137	* and create a new node that is inserted into the AVL tree (range key). Then
138	* a full PGM resync is flagged (clear pool, sync cr3, update virtual bit of PGMPAGE).
139	*
140	* 2. The following PGMSyncCR3/SyncCR3 operation will first make invoke HandlerVirtualUpdate.
141	*
142	* 2a. HandlerVirtualUpdate will will lookup all the pages covered by virtual handlers
143	* via the current guest CR3 and update the physical page -> virtual handler
144	* translation. Needless to say, this doesn't exactly scale very well. If any changes
145	* are detected, it will flag a virtual bit update just like we did on registration.
146	* PGMPHYS pages with changes will have their virtual handler state reset to NONE.
147	*
148	* 2b. The virtual bit update process will iterate all the pages covered by all the
149	* virtual handlers and update the PGMPAGE virtual handler state to the max of all
150	* virtual handlers on that page.
151	*
152	* 2c. Back in SyncCR3 we will now flush the entire shadow page cache to make sure
153	* we don't miss any alias mappings of the monitored pages.
154	*
155	* 2d. SyncCR3 will then proceed with syncing the CR3 table.
156	*
157	* 3. \#PF(np,read) on a page in the range. This will cause it to be synced
158	* read-only and resumed if it's a WRITE handler. If it's an ALL handler we
159	* will call the handlers like in the next step. If the physical mapping has
160	* changed we will - some time in the future - perform a handler callback
161	* (optional) and update the physical -> virtual handler cache.
162	*
163	* 4. \#PF(,write) on a page in the range. This will cause the handler to
164	* be invoked.
165	*
166	* 5. The guest invalidates the page and changes the physical backing or
167	* unmaps it. This should cause the invalidation callback to be invoked
168	* (it might not yet be 100% perfect). Exactly what happens next... is
169	* this where we mess up and end up out of sync for a while?
170	*
171	* 6. The handler is deregistered by the client via PGMHandlerVirtualDeregister.
172	* We will then set all PGMPAGEs in the physical -> virtual handler cache for
173	* this handler to NONE and trigger a full PGM resync (basically the same
174	* as int step 1). Which means 2 is executed again.
175	*
176	*
177	* @subsubsection sub_sec_pgm_handler_virt_todo TODOs
178	*
179	* There is a bunch of things that needs to be done to make the virtual handlers
180	* work 100% correctly and work more efficiently.
181	*
182	* The first bit hasn't been implemented yet because it's going to slow the
183	* whole mess down even more, and besides it seems to be working reliably for
184	* our current uses. OTOH, some of the optimizations might end up more or less
185	* implementing the missing bits, so we'll see.
186	*
187	* On the optimization side, the first thing to do is to try avoid unnecessary
188	* cache flushing. Then try team up with the shadowing code to track changes
189	* in mappings by means of access to them (shadow in), updates to shadows pages,
190	* invlpg, and shadow PT discarding (perhaps).
191	*
192	* Some idea that have popped up for optimization for current and new features:
193	* - bitmap indicating where there are virtual handlers installed.
194	* (4KB => 220 pages, page 212 => covers 32-bit address space 1:1!)
195	* - Further optimize this by min/max (needs min/max avl getters).
196	* - Shadow page table entry bit (if any left)?
197	*
198	*/
199
200
201	/** @page pg_pgm_phys PGM Physical Guest Memory Management
202	*
203	*
204	* Objectives:
205	* - Guest RAM over-commitment using memory ballooning,
206	* zero pages and general page sharing.
207	* - Moving or mirroring a VM onto a different physical machine.
208	*
209	*
210	* @subsection subsec_pgmPhys_Definitions Definitions
211	*
212	* Allocation chunk - A RTR0MemObjAllocPhysNC object and the tracking
213	* machinery associated with it.
214	*
215	*
216	*
217	*
218	* @subsection subsec_pgmPhys_AllocPage Allocating a page.
219	*
220	* Initially we map all guest memory to the (per VM) zero page, which
221	* means that none of the read functions will cause pages to be allocated.
222	*
223	* Exception, access bit in page tables that have been shared. This must
224	* be handled, but we must also make sure PGMGst*Modify doesn't make
225	* unnecessary modifications.
226	*
227	* Allocation points:
228	* - PGMPhysSimpleWriteGCPhys and PGMPhysWrite.
229	* - Replacing a zero page mapping at \#PF.
230	* - Replacing a shared page mapping at \#PF.
231	* - ROM registration (currently MMR3RomRegister).
232	* - VM restore (pgmR3Load).
233	*
234	* For the first three it would make sense to keep a few pages handy
235	* until we've reached the max memory commitment for the VM.
236	*
237	* For the ROM registration, we know exactly how many pages we need
238	* and will request these from ring-0. For restore, we will save
239	* the number of non-zero pages in the saved state and allocate
240	* them up front. This would allow the ring-0 component to refuse
241	* the request if the isn't sufficient memory available for VM use.
242	*
243	* Btw. for both ROM and restore allocations we won't be requiring
244	* zeroed pages as they are going to be filled instantly.
245	*
246	*
247	* @subsection subsec_pgmPhys_FreePage Freeing a page
248	*
249	* There are a few points where a page can be freed:
250	* - After being replaced by the zero page.
251	* - After being replaced by a shared page.
252	* - After being ballooned by the guest additions.
253	* - At reset.
254	* - At restore.
255	*
256	* When freeing one or more pages they will be returned to the ring-0
257	* component and replaced by the zero page.
258	*
259	* The reasoning for clearing out all the pages on reset is that it will
260	* return us to the exact same state as on power on, and may thereby help
261	* us reduce the memory load on the system. Further it might have a
262	* (temporary) positive influence on memory fragmentation (@see subsec_pgmPhys_Fragmentation).
263	*
264	* On restore, as mention under the allocation topic, pages should be
265	* freed / allocated depending on how many is actually required by the
266	* new VM state. The simplest approach is to do like on reset, and free
267	* all non-ROM pages and then allocate what we need.
268	*
269	* A measure to prevent some fragmentation, would be to let each allocation
270	* chunk have some affinity towards the VM having allocated the most pages
271	* from it. Also, try make sure to allocate from allocation chunks that
272	* are almost full. Admittedly, both these measures might work counter to
273	* our intentions and its probably not worth putting a lot of effort,
274	* cpu time or memory into this.
275	*
276	*
277	* @subsection subsec_pgmPhys_SharePage Sharing a page
278	*
279	* The basic idea is that there there will be a idle priority kernel
280	* thread walking the non-shared VM pages hashing them and looking for
281	* pages with the same checksum. If such pages are found, it will compare
282	* them byte-by-byte to see if they actually are identical. If found to be
283	* identical it will allocate a shared page, copy the content, check that
284	* the page didn't change while doing this, and finally request both the
285	* VMs to use the shared page instead. If the page is all zeros (special
286	* checksum and byte-by-byte check) it will request the VM that owns it
287	* to replace it with the zero page.
288	*
289	* To make this efficient, we will have to make sure not to try share a page
290	* that will change its contents soon. This part requires the most work.
291	* A simple idea would be to request the VM to write monitor the page for
292	* a while to make sure it isn't modified any time soon. Also, it may
293	* make sense to skip pages that are being write monitored since this
294	* information is readily available to the thread if it works on the
295	* per-VM guest memory structures (presently called PGMRAMRANGE).
296	*
297	*
298	* @subsection subsec_pgmPhys_Fragmentation Fragmentation Concerns and Counter Measures
299	*
300	* The pages are organized in allocation chunks in ring-0, this is a necessity
301	* if we wish to have an OS agnostic approach to this whole thing. (On Linux we
302	* could easily work on a page-by-page basis if we liked. Whether this is possible
303	* or efficient on NT I don't quite know.) Fragmentation within these chunks may
304	* become a problem as part of the idea here is that we wish to return memory to
305	* the host system.
306	*
307	* For instance, starting two VMs at the same time, they will both allocate the
308	* guest memory on-demand and if permitted their page allocations will be
309	* intermixed. Shut down one of the two VMs and it will be difficult to return
310	* any memory to the host system because the page allocation for the two VMs are
311	* mixed up in the same allocation chunks.
312	*
313	* To further complicate matters, when pages are freed because they have been
314	* ballooned or become shared/zero the whole idea is that the page is supposed
315	* to be reused by another VM or returned to the host system. This will cause
316	* allocation chunks to contain pages belonging to different VMs and prevent
317	* returning memory to the host when one of those VM shuts down.
318	*
319	* The only way to really deal with this problem is to move pages. This can
320	* either be done at VM shutdown and or by the idle priority worker thread
321	* that will be responsible for finding sharable/zero pages. The mechanisms
322	* involved for coercing a VM to move a page (or to do it for it) will be
323	* the same as when telling it to share/zero a page.
324	*
325	*
326	* @subsection subsec_pgmPhys_Tracking Tracking Structures And Their Cost
327	*
328	* There's a difficult balance between keeping the per-page tracking structures
329	* (global and guest page) easy to use and keeping them from eating too much
330	* memory. We have limited virtual memory resources available when operating in
331	* 32-bit kernel space (on 64-bit there'll it's quite a different story). The
332	* tracking structures will be attempted designed such that we can deal with up
333	* to 32GB of memory on a 32-bit system and essentially unlimited on 64-bit ones.
334	*
335	*
336	* @subsubsection subsubsec_pgmPhys_Tracking_Kernel Kernel Space
337	*
338	* @see pg_GMM
339	*
340	* @subsubsection subsubsec_pgmPhys_Tracking_PerVM Per-VM
341	*
342	* Fixed info is the physical address of the page (HCPhys) and the page id
343	* (described above). Theoretically we'll need 48(-12) bits for the HCPhys part.
344	* Today we've restricting ourselves to 40(-12) bits because this is the current
345	* restrictions of all AMD64 implementations (I think Barcelona will up this
346	* to 48(-12) bits, not that it really matters) and I needed the bits for
347	* tracking mappings of a page. 48-12 = 36. That leaves 28 bits, which means a
348	* decent range for the page id: 2^(28+12) = 1024TB.
349	*
350	* In additions to these, we'll have to keep maintaining the page flags as we
351	* currently do. Although it wouldn't harm to optimize these quite a bit, like
352	* for instance the ROM shouldn't depend on having a write handler installed
353	* in order for it to become read-only. A RO/RW bit should be considered so
354	* that the page syncing code doesn't have to mess about checking multiple
355	* flag combinations (ROM \|\| RW handler \|\| write monitored) in order to
356	* figure out how to setup a shadow PTE. But this of course, is second
357	* priority at present. Current this requires 12 bits, but could probably
358	* be optimized to ~8.
359	*
360	* Then there's the 24 bits used to track which shadow page tables are
361	* currently mapping a page for the purpose of speeding up physical
362	* access handlers, and thereby the page pool cache. More bit for this
363	* purpose wouldn't hurt IIRC.
364	*
365	* Then there is a new bit in which we need to record what kind of page
366	* this is, shared, zero, normal or write-monitored-normal. This'll
367	* require 2 bits. One bit might be needed for indicating whether a
368	* write monitored page has been written to. And yet another one or
369	* two for tracking migration status. 3-4 bits total then.
370	*
371	* Whatever is left will can be used to record the sharabilitiy of a
372	* page. The page checksum will not be stored in the per-VM table as
373	* the idle thread will not be permitted to do modifications to it.
374	* It will instead have to keep its own working set of potentially
375	* shareable pages and their check sums and stuff.
376	*
377	* For the present we'll keep the current packing of the
378	* PGMRAMRANGE::aHCPhys to keep the changes simple, only of course,
379	* we'll have to change it to a struct with a total of 128-bits at
380	* our disposal.
381	*
382	* The initial layout will be like this:
383	* @verbatim
384	RTHCPHYS HCPhys; The current stuff.
385	63:40 Current shadow PT tracking stuff.
386	39:12 The physical page frame number.
387	11:0 The current flags.
388	uint32_t u28PageId : 28; The page id.
389	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
390	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
391	uint32_t u1Reserved : 1; Reserved for later.
392	uint32_t u32Reserved; Reserved for later, mostly sharing stats.
393	@endverbatim
394	*
395	* The final layout will be something like this:
396	* @verbatim
397	RTHCPHYS HCPhys; The current stuff.
398	63:48 High page id (12+).
399	47:12 The physical page frame number.
400	11:0 Low page id.
401	uint32_t fReadOnly : 1; Whether it's readonly page (rom or monitored in some way).
402	uint32_t u3Type : 3; The page type {RESERVED, MMIO, MMIO2, ROM, shadowed ROM, RAM}.
403	uint32_t u2PhysMon : 2; Physical access handler type {none, read, write, all}.
404	uint32_t u2VirtMon : 2; Virtual access handler type {none, read, write, all}..
405	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
406	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
407	uint32_t u20Reserved : 20; Reserved for later, mostly sharing stats.
408	uint32_t u32Tracking; The shadow PT tracking stuff, roughly.
409	@endverbatim
410	*
411	* Cost wise, this means we'll double the cost for guest memory. There isn't anyway
412	* around that I'm afraid. It means that the cost of dealing out 32GB of memory
413	* to one or more VMs is: (32GB >> PAGE_SHIFT) * 16 bytes, or 128MBs. Or another
414	* example, the VM heap cost when assigning 1GB to a VM will be: 4MB.
415	*
416	* A couple of cost examples for the total cost per-VM + kernel.
417	* 32-bit Windows and 32-bit linux:
418	* 1GB guest ram, 256K pages: 4MB + 2MB(+) = 6MB
419	* 4GB guest ram, 1M pages: 16MB + 8MB(+) = 24MB
420	* 32GB guest ram, 8M pages: 128MB + 64MB(+) = 192MB
421	* 64-bit Windows and 64-bit linux:
422	* 1GB guest ram, 256K pages: 4MB + 3MB(+) = 7MB
423	* 4GB guest ram, 1M pages: 16MB + 12MB(+) = 28MB
424	* 32GB guest ram, 8M pages: 128MB + 96MB(+) = 224MB
425	*
426	* UPDATE - 2007-09-27:
427	* Will need a ballooned flag/state too because we cannot
428	* trust the guest 100% and reporting the same page as ballooned more
429	* than once will put the GMM off balance.
430	*
431	*
432	* @subsection subsec_pgmPhys_Serializing Serializing Access
433	*
434	* Initially, we'll try a simple scheme:
435	*
436	* - The per-VM RAM tracking structures (PGMRAMRANGE) is only modified
437	* by the EMT thread of that VM while in the pgm critsect.
438	* - Other threads in the VM process that needs to make reliable use of
439	* the per-VM RAM tracking structures will enter the critsect.
440	* - No process external thread or kernel thread will ever try enter
441	* the pgm critical section, as that just won't work.
442	* - The idle thread (and similar threads) doesn't not need 100% reliable
443	* data when performing it tasks as the EMT thread will be the one to
444	* do the actual changes later anyway. So, as long as it only accesses
445	* the main ram range, it can do so by somehow preventing the VM from
446	* being destroyed while it works on it...
447	*
448	* - The over-commitment management, including the allocating/freeing
449	* chunks, is serialized by a ring-0 mutex lock (a fast one since the
450	* more mundane mutex implementation is broken on Linux).
451	* - A separate mutex is protecting the set of allocation chunks so
452	* that pages can be shared or/and freed up while some other VM is
453	* allocating more chunks. This mutex can be take from under the other
454	* one, but not the other way around.
455	*
456	*
457	* @subsection subsec_pgmPhys_Request VM Request interface
458	*
459	* When in ring-0 it will become necessary to send requests to a VM so it can
460	* for instance move a page while defragmenting during VM destroy. The idle
461	* thread will make use of this interface to request VMs to setup shared
462	* pages and to perform write monitoring of pages.
463	*
464	* I would propose an interface similar to the current VMReq interface, similar
465	* in that it doesn't require locking and that the one sending the request may
466	* wait for completion if it wishes to. This shouldn't be very difficult to
467	* realize.
468	*
469	* The requests themselves are also pretty simple. They are basically:
470	* -# Check that some precondition is still true.
471	* -# Do the update.
472	* -# Update all shadow page tables involved with the page.
473	*
474	* The 3rd step is identical to what we're already doing when updating a
475	* physical handler, see pgmHandlerPhysicalSetRamFlagsAndFlushShadowPTs.
476	*
477	*
478	*
479	* @section sec_pgmPhys_MappingCaches Mapping Caches
480	*
481	* In order to be able to map in and out memory and to be able to support
482	* guest with more RAM than we've got virtual address space, we'll employing
483	* a mapping cache. Normally ring-0 and ring-3 can share the same cache,
484	* however on 32-bit darwin the ring-0 code is running in a different memory
485	* context and therefore needs a separate cache. In raw-mode context we also
486	* need a separate cache. The 32-bit darwin mapping cache and the one for
487	* raw-mode context share a lot of code, see PGMRZDYNMAP.
488	*
489	*
490	* @subsection subsec_pgmPhys_MappingCaches_R3 Ring-3
491	*
492	* We've considered implementing the ring-3 mapping cache page based but found
493	* that this was bother some when one had to take into account TLBs+SMP and
494	* portability (missing the necessary APIs on several platforms). There were
495	* also some performance concerns with this approach which hadn't quite been
496	* worked out.
497	*
498	* Instead, we'll be mapping allocation chunks into the VM process. This simplifies
499	* matters greatly quite a bit since we don't need to invent any new ring-0 stuff,
500	* only some minor RTR0MEMOBJ mapping stuff. The main concern here is that mapping
501	* compared to the previous idea is that mapping or unmapping a 1MB chunk is more
502	* costly than a single page, although how much more costly is uncertain. We'll
503	* try address this by using a very big cache, preferably bigger than the actual
504	* VM RAM size if possible. The current VM RAM sizes should give some idea for
505	* 32-bit boxes, while on 64-bit we can probably get away with employing an
506	* unlimited cache.
507	*
508	* The cache have to parts, as already indicated, the ring-3 side and the
509	* ring-0 side.
510	*
511	* The ring-0 will be tied to the page allocator since it will operate on the
512	* memory objects it contains. It will therefore require the first ring-0 mutex
513	* discussed in @ref subsec_pgmPhys_Serializing. We
514	* some double house keeping wrt to who has mapped what I think, since both
515	* VMMR0.r0 and RTR0MemObj will keep track of mapping relations
516	*
517	* The ring-3 part will be protected by the pgm critsect. For simplicity, we'll
518	* require anyone that desires to do changes to the mapping cache to do that
519	* from within this critsect. Alternatively, we could employ a separate critsect
520	* for serializing changes to the mapping cache as this would reduce potential
521	* contention with other threads accessing mappings unrelated to the changes
522	* that are in process. We can see about this later, contention will show
523	* up in the statistics anyway, so it'll be simple to tell.
524	*
525	* The organization of the ring-3 part will be very much like how the allocation
526	* chunks are organized in ring-0, that is in an AVL tree by chunk id. To avoid
527	* having to walk the tree all the time, we'll have a couple of lookaside entries
528	* like in we do for I/O ports and MMIO in IOM.
529	*
530	* The simplified flow of a PGMPhysRead/Write function:
531	* -# Enter the PGM critsect.
532	* -# Lookup GCPhys in the ram ranges and get the Page ID.
533	* -# Calc the Allocation Chunk ID from the Page ID.
534	* -# Check the lookaside entries and then the AVL tree for the Chunk ID.
535	* If not found in cache:
536	* -# Call ring-0 and request it to be mapped and supply
537	* a chunk to be unmapped if the cache is maxed out already.
538	* -# Insert the new mapping into the AVL tree (id + R3 address).
539	* -# Update the relevant lookaside entry and return the mapping address.
540	* -# Do the read/write according to monitoring flags and everything.
541	* -# Leave the critsect.
542	*
543	*
544	* @section sec_pgmPhys_Fallback Fallback
545	*
546	* Current all the "second tier" hosts will not support the RTR0MemObjAllocPhysNC
547	* API and thus require a fallback.
548	*
549	* So, when RTR0MemObjAllocPhysNC returns VERR_NOT_SUPPORTED the page allocator
550	* will return to the ring-3 caller (and later ring-0) and asking it to seed
551	* the page allocator with some fresh pages (VERR_GMM_SEED_ME). Ring-3 will
552	* then perform an SUPR3PageAlloc(cbChunk >> PAGE_SHIFT) call and make a
553	* "SeededAllocPages" call to ring-0.
554	*
555	* The first time ring-0 sees the VERR_NOT_SUPPORTED failure it will disable
556	* all page sharing (zero page detection will continue). It will also force
557	* all allocations to come from the VM which seeded the page. Both these
558	* measures are taken to make sure that there will never be any need for
559	* mapping anything into ring-3 - everything will be mapped already.
560	*
561	* Whether we'll continue to use the current MM locked memory management
562	* for this I don't quite know (I'd prefer not to and just ditch that all
563	* together), we'll see what's simplest to do.
564	*
565	*
566	*
567	* @section sec_pgmPhys_Changes Changes
568	*
569	* Breakdown of the changes involved?
570	*/
571
572	/*******************************************************************************
573	* Header Files *
574	*******************************************************************************/
575	#define LOG_GROUP LOG_GROUP_PGM
576	#include <VBox/dbgf.h>
577	#include <VBox/pgm.h>
578	#include <VBox/cpum.h>
579	#include <VBox/iom.h>
580	#include <VBox/sup.h>
581	#include <VBox/mm.h>
582	#include <VBox/em.h>
583	#include <VBox/stam.h>
584	#include <VBox/rem.h>
585	#include <VBox/selm.h>
586	#include <VBox/ssm.h>
587	#include <VBox/hwaccm.h>
588	#include "PGMInternal.h"
589	#include <VBox/vm.h>
590	#include "PGMInline.h"
591
592	#include <VBox/dbg.h>
593	#include <VBox/param.h>
594	#include <VBox/err.h>
595
596	#include <iprt/asm.h>
597	#include <iprt/asm-amd64-x86.h>
598	#include <iprt/assert.h>
599	#include <iprt/env.h>
600	#include <iprt/mem.h>
601	#include <iprt/file.h>
602	#include <iprt/string.h>
603	#include <iprt/thread.h>
604
605
606	/*******************************************************************************
607	* Internal Functions *
608	*******************************************************************************/
609	static int pgmR3InitPaging(PVM pVM);
610	static int pgmR3InitStats(PVM pVM);
611	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
612	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
613	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
614	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser);
615	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
616	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
617	#ifdef VBOX_STRICT
618	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser);
619	#endif
620	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0);
621	static void pgmR3ModeDataSwitch(PVM pVM, PVMCPU pVCpu, PGMMODE enmShw, PGMMODE enmGst);
622	static PGMMODE pgmR3CalcShadowMode(PVM pVM, PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher);
623
624	#ifdef VBOX_WITH_DEBUGGER
625	/** @todo Convert the first two commands to 'info' items. */
626	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
627	static DECLCALLBACK(int) pgmR3CmdError(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
628	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
629	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
630	# ifdef VBOX_STRICT
631	static DECLCALLBACK(int) pgmR3CmdAssertCR3(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
632	# endif
633	static DECLCALLBACK(int) pgmR3CmdPhysToFile(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult);
634	#endif
635
636
637	/*******************************************************************************
638	* Global Variables *
639	*******************************************************************************/
640	#ifdef VBOX_WITH_DEBUGGER
641	/** Argument descriptors for '.pgmerror' and '.pgmerroroff'. */
642	static const DBGCVARDESC g_aPgmErrorArgs[] =
643	{
644	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
645	{ 0, 1, DBGCVAR_CAT_STRING, 0, "where", "Error injection location." },
646	};
647
648	static const DBGCVARDESC g_aPgmPhysToFileArgs[] =
649	{
650	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
651	{ 1, 1, DBGCVAR_CAT_STRING, 0, "file", "The file name." },
652	{ 0, 1, DBGCVAR_CAT_STRING, 0, "nozero", "If present, zero pages are skipped." },
653	};
654
655	# ifdef DEBUG_sandervl
656	static const DBGCVARDESC g_aPgmCountPhysWritesArgs[] =
657	{
658	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
659	{ 1, 1, DBGCVAR_CAT_STRING, 0, "enabled", "on/off." },
660	{ 1, 1, DBGCVAR_CAT_NUMBER_NO_RANGE, 0, "interval", "Interval in ms." },
661	};
662	# endif
663
664	/** Command descriptors. */
665	static const DBGCCMD g_aCmds[] =
666	{
667	/* pszCmd, cArgsMin, cArgsMax, paArgDesc, cArgDescs, pResultDesc, fFlags, pfnHandler pszSyntax, ....pszDescription */
668	{ "pgmram", 0, 0, NULL, 0, NULL, 0, pgmR3CmdRam, "", "Display the ram ranges." },
669	{ "pgmsync", 0, 0, NULL, 0, NULL, 0, pgmR3CmdSync, "", "Sync the CR3 page." },
670	{ "pgmerror", 0, 1, &g_aPgmErrorArgs[0], 1, NULL, 0, pgmR3CmdError, "", "Enables inject runtime of errors into parts of PGM." },
671	{ "pgmerroroff", 0, 1, &g_aPgmErrorArgs[0], 1, NULL, 0, pgmR3CmdError, "", "Disables inject runtime errors into parts of PGM." },
672	# ifdef VBOX_STRICT
673	{ "pgmassertcr3", 0, 0, NULL, 0, NULL, 0, pgmR3CmdAssertCR3, "", "Check the shadow CR3 mapping." },
674	# if HC_ARCH_BITS == 64
675	{ "pgmcheckduppages", 0, 0, NULL, 0, NULL, 0, pgmR3CmdCheckDuplicatePages, "", "Check for duplicate pages in all running VMs." },
676	{ "pgmsharedmodules", 0, 0, NULL, 0, NULL, 0, pgmR3CmdShowSharedModules, "", "Print shared modules info." },
677	# endif
678	# endif
679	{ "pgmsyncalways", 0, 0, NULL, 0, NULL, 0, pgmR3CmdSyncAlways, "", "Toggle permanent CR3 syncing." },
680	{ "pgmphystofile", 1, 2, &g_aPgmPhysToFileArgs[0], 2, NULL, 0, pgmR3CmdPhysToFile, "", "Save the physical memory to file." },
681	};
682	#endif
683
684
685
686
687	/*
688	* Shadow - 32-bit mode
689	*/
690	#define PGM_SHW_TYPE PGM_TYPE_32BIT
691	#define PGM_SHW_NAME(name) PGM_SHW_NAME_32BIT(name)
692	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_32BIT_STR(name)
693	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_32BIT_STR(name)
694	#include "PGMShw.h"
695
696	/* Guest - real mode */
697	#define PGM_GST_TYPE PGM_TYPE_REAL
698	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
699	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
700	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
701	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_REAL(name)
702	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_REAL_STR(name)
703	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_REAL_STR(name)
704	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
705	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD_PHYS
706	#include "PGMBth.h"
707	#include "PGMGstDefs.h"
708	#include "PGMGst.h"
709	#undef BTH_PGMPOOLKIND_PT_FOR_PT
710	#undef BTH_PGMPOOLKIND_ROOT
711	#undef PGM_BTH_NAME
712	#undef PGM_BTH_NAME_RC_STR
713	#undef PGM_BTH_NAME_R0_STR
714	#undef PGM_GST_TYPE
715	#undef PGM_GST_NAME
716	#undef PGM_GST_NAME_RC_STR
717	#undef PGM_GST_NAME_R0_STR
718
719	/* Guest - protected mode */
720	#define PGM_GST_TYPE PGM_TYPE_PROT
721	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
722	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
723	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
724	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_PROT(name)
725	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_PROT_STR(name)
726	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_PROT_STR(name)
727	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
728	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD_PHYS
729	#include "PGMBth.h"
730	#include "PGMGstDefs.h"
731	#include "PGMGst.h"
732	#undef BTH_PGMPOOLKIND_PT_FOR_PT
733	#undef BTH_PGMPOOLKIND_ROOT
734	#undef PGM_BTH_NAME
735	#undef PGM_BTH_NAME_RC_STR
736	#undef PGM_BTH_NAME_R0_STR
737	#undef PGM_GST_TYPE
738	#undef PGM_GST_NAME
739	#undef PGM_GST_NAME_RC_STR
740	#undef PGM_GST_NAME_R0_STR
741
742	/* Guest - 32-bit mode */
743	#define PGM_GST_TYPE PGM_TYPE_32BIT
744	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
745	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
746	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
747	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_32BIT(name)
748	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_32BIT_STR(name)
749	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_32BIT_STR(name)
750	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_32BIT_PT
751	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_32BIT_PT_FOR_32BIT_4MB
752	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD
753	#include "PGMBth.h"
754	#include "PGMGstDefs.h"
755	#include "PGMGst.h"
756	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
757	#undef BTH_PGMPOOLKIND_PT_FOR_PT
758	#undef BTH_PGMPOOLKIND_ROOT
759	#undef PGM_BTH_NAME
760	#undef PGM_BTH_NAME_RC_STR
761	#undef PGM_BTH_NAME_R0_STR
762	#undef PGM_GST_TYPE
763	#undef PGM_GST_NAME
764	#undef PGM_GST_NAME_RC_STR
765	#undef PGM_GST_NAME_R0_STR
766
767	#undef PGM_SHW_TYPE
768	#undef PGM_SHW_NAME
769	#undef PGM_SHW_NAME_RC_STR
770	#undef PGM_SHW_NAME_R0_STR
771
772
773	/*
774	* Shadow - PAE mode
775	*/
776	#define PGM_SHW_TYPE PGM_TYPE_PAE
777	#define PGM_SHW_NAME(name) PGM_SHW_NAME_PAE(name)
778	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_PAE_STR(name)
779	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_PAE_STR(name)
780	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
781	#include "PGMShw.h"
782
783	/* Guest - real mode */
784	#define PGM_GST_TYPE PGM_TYPE_REAL
785	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
786	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
787	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
788	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
789	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_REAL_STR(name)
790	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_REAL_STR(name)
791	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
792	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_PHYS
793	#include "PGMGstDefs.h"
794	#include "PGMBth.h"
795	#undef BTH_PGMPOOLKIND_PT_FOR_PT
796	#undef BTH_PGMPOOLKIND_ROOT
797	#undef PGM_BTH_NAME
798	#undef PGM_BTH_NAME_RC_STR
799	#undef PGM_BTH_NAME_R0_STR
800	#undef PGM_GST_TYPE
801	#undef PGM_GST_NAME
802	#undef PGM_GST_NAME_RC_STR
803	#undef PGM_GST_NAME_R0_STR
804
805	/* Guest - protected mode */
806	#define PGM_GST_TYPE PGM_TYPE_PROT
807	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
808	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
809	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
810	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PROT(name)
811	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_PROT_STR(name)
812	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PROT_STR(name)
813	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
814	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_PHYS
815	#include "PGMGstDefs.h"
816	#include "PGMBth.h"
817	#undef BTH_PGMPOOLKIND_PT_FOR_PT
818	#undef BTH_PGMPOOLKIND_ROOT
819	#undef PGM_BTH_NAME
820	#undef PGM_BTH_NAME_RC_STR
821	#undef PGM_BTH_NAME_R0_STR
822	#undef PGM_GST_TYPE
823	#undef PGM_GST_NAME
824	#undef PGM_GST_NAME_RC_STR
825	#undef PGM_GST_NAME_R0_STR
826
827	/* Guest - 32-bit mode */
828	#define PGM_GST_TYPE PGM_TYPE_32BIT
829	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
830	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
831	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
832	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_32BIT(name)
833	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_32BIT_STR(name)
834	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_32BIT_STR(name)
835	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
836	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
837	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_FOR_32BIT
838	#include "PGMGstDefs.h"
839	#include "PGMBth.h"
840	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
841	#undef BTH_PGMPOOLKIND_PT_FOR_PT
842	#undef BTH_PGMPOOLKIND_ROOT
843	#undef PGM_BTH_NAME
844	#undef PGM_BTH_NAME_RC_STR
845	#undef PGM_BTH_NAME_R0_STR
846	#undef PGM_GST_TYPE
847	#undef PGM_GST_NAME
848	#undef PGM_GST_NAME_RC_STR
849	#undef PGM_GST_NAME_R0_STR
850
851	/* Guest - PAE mode */
852	#define PGM_GST_TYPE PGM_TYPE_PAE
853	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
854	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
855	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
856	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PAE(name)
857	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_PAE_STR(name)
858	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PAE_STR(name)
859	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
860	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
861	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT
862	#include "PGMBth.h"
863	#include "PGMGstDefs.h"
864	#include "PGMGst.h"
865	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
866	#undef BTH_PGMPOOLKIND_PT_FOR_PT
867	#undef BTH_PGMPOOLKIND_ROOT
868	#undef PGM_BTH_NAME
869	#undef PGM_BTH_NAME_RC_STR
870	#undef PGM_BTH_NAME_R0_STR
871	#undef PGM_GST_TYPE
872	#undef PGM_GST_NAME
873	#undef PGM_GST_NAME_RC_STR
874	#undef PGM_GST_NAME_R0_STR
875
876	#undef PGM_SHW_TYPE
877	#undef PGM_SHW_NAME
878	#undef PGM_SHW_NAME_RC_STR
879	#undef PGM_SHW_NAME_R0_STR
880
881
882	/*
883	* Shadow - AMD64 mode
884	*/
885	#define PGM_SHW_TYPE PGM_TYPE_AMD64
886	#define PGM_SHW_NAME(name) PGM_SHW_NAME_AMD64(name)
887	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_AMD64_STR(name)
888	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_AMD64_STR(name)
889	#include "PGMShw.h"
890
891	#ifdef VBOX_WITH_64_BITS_GUESTS
892	/* Guest - AMD64 mode */
893	# define PGM_GST_TYPE PGM_TYPE_AMD64
894	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
895	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
896	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
897	# define PGM_BTH_NAME(name) PGM_BTH_NAME_AMD64_AMD64(name)
898	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_AMD64_AMD64_STR(name)
899	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_AMD64_AMD64_STR(name)
900	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
901	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
902	# define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_64BIT_PML4
903	# include "PGMBth.h"
904	# include "PGMGstDefs.h"
905	# include "PGMGst.h"
906	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
907	# undef BTH_PGMPOOLKIND_PT_FOR_PT
908	# undef BTH_PGMPOOLKIND_ROOT
909	# undef PGM_BTH_NAME
910	# undef PGM_BTH_NAME_RC_STR
911	# undef PGM_BTH_NAME_R0_STR
912	# undef PGM_GST_TYPE
913	# undef PGM_GST_NAME
914	# undef PGM_GST_NAME_RC_STR
915	# undef PGM_GST_NAME_R0_STR
916	#endif /* VBOX_WITH_64_BITS_GUESTS */
917
918	#undef PGM_SHW_TYPE
919	#undef PGM_SHW_NAME
920	#undef PGM_SHW_NAME_RC_STR
921	#undef PGM_SHW_NAME_R0_STR
922
923
924	/*
925	* Shadow - Nested paging mode
926	*/
927	#define PGM_SHW_TYPE PGM_TYPE_NESTED
928	#define PGM_SHW_NAME(name) PGM_SHW_NAME_NESTED(name)
929	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_NESTED_STR(name)
930	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_NESTED_STR(name)
931	#include "PGMShw.h"
932
933	/* Guest - real mode */
934	#define PGM_GST_TYPE PGM_TYPE_REAL
935	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
936	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
937	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
938	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_REAL(name)
939	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_REAL_STR(name)
940	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_REAL_STR(name)
941	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
942	#include "PGMGstDefs.h"
943	#include "PGMBth.h"
944	#undef BTH_PGMPOOLKIND_PT_FOR_PT
945	#undef PGM_BTH_NAME
946	#undef PGM_BTH_NAME_RC_STR
947	#undef PGM_BTH_NAME_R0_STR
948	#undef PGM_GST_TYPE
949	#undef PGM_GST_NAME
950	#undef PGM_GST_NAME_RC_STR
951	#undef PGM_GST_NAME_R0_STR
952
953	/* Guest - protected mode */
954	#define PGM_GST_TYPE PGM_TYPE_PROT
955	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
956	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
957	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
958	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PROT(name)
959	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_PROT_STR(name)
960	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PROT_STR(name)
961	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
962	#include "PGMGstDefs.h"
963	#include "PGMBth.h"
964	#undef BTH_PGMPOOLKIND_PT_FOR_PT
965	#undef PGM_BTH_NAME
966	#undef PGM_BTH_NAME_RC_STR
967	#undef PGM_BTH_NAME_R0_STR
968	#undef PGM_GST_TYPE
969	#undef PGM_GST_NAME
970	#undef PGM_GST_NAME_RC_STR
971	#undef PGM_GST_NAME_R0_STR
972
973	/* Guest - 32-bit mode */
974	#define PGM_GST_TYPE PGM_TYPE_32BIT
975	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
976	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
977	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
978	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_32BIT(name)
979	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_32BIT_STR(name)
980	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_32BIT_STR(name)
981	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
982	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
983	#include "PGMGstDefs.h"
984	#include "PGMBth.h"
985	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
986	#undef BTH_PGMPOOLKIND_PT_FOR_PT
987	#undef PGM_BTH_NAME
988	#undef PGM_BTH_NAME_RC_STR
989	#undef PGM_BTH_NAME_R0_STR
990	#undef PGM_GST_TYPE
991	#undef PGM_GST_NAME
992	#undef PGM_GST_NAME_RC_STR
993	#undef PGM_GST_NAME_R0_STR
994
995	/* Guest - PAE mode */
996	#define PGM_GST_TYPE PGM_TYPE_PAE
997	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
998	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
999	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
1000	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PAE(name)
1001	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_PAE_STR(name)
1002	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PAE_STR(name)
1003	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1004	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1005	#include "PGMGstDefs.h"
1006	#include "PGMBth.h"
1007	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1008	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1009	#undef PGM_BTH_NAME
1010	#undef PGM_BTH_NAME_RC_STR
1011	#undef PGM_BTH_NAME_R0_STR
1012	#undef PGM_GST_TYPE
1013	#undef PGM_GST_NAME
1014	#undef PGM_GST_NAME_RC_STR
1015	#undef PGM_GST_NAME_R0_STR
1016
1017	#ifdef VBOX_WITH_64_BITS_GUESTS
1018	/* Guest - AMD64 mode */
1019	# define PGM_GST_TYPE PGM_TYPE_AMD64
1020	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
1021	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
1022	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
1023	# define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_AMD64(name)
1024	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_AMD64_STR(name)
1025	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_AMD64_STR(name)
1026	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1027	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1028	# include "PGMGstDefs.h"
1029	# include "PGMBth.h"
1030	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
1031	# undef BTH_PGMPOOLKIND_PT_FOR_PT
1032	# undef PGM_BTH_NAME
1033	# undef PGM_BTH_NAME_RC_STR
1034	# undef PGM_BTH_NAME_R0_STR
1035	# undef PGM_GST_TYPE
1036	# undef PGM_GST_NAME
1037	# undef PGM_GST_NAME_RC_STR
1038	# undef PGM_GST_NAME_R0_STR
1039	#endif /* VBOX_WITH_64_BITS_GUESTS */
1040
1041	#undef PGM_SHW_TYPE
1042	#undef PGM_SHW_NAME
1043	#undef PGM_SHW_NAME_RC_STR
1044	#undef PGM_SHW_NAME_R0_STR
1045
1046
1047	/*
1048	* Shadow - EPT
1049	*/
1050	#define PGM_SHW_TYPE PGM_TYPE_EPT
1051	#define PGM_SHW_NAME(name) PGM_SHW_NAME_EPT(name)
1052	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_EPT_STR(name)
1053	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_EPT_STR(name)
1054	#include "PGMShw.h"
1055
1056	/* Guest - real mode */
1057	#define PGM_GST_TYPE PGM_TYPE_REAL
1058	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
1059	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
1060	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
1061	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_REAL(name)
1062	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_REAL_STR(name)
1063	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_REAL_STR(name)
1064	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
1065	#include "PGMGstDefs.h"
1066	#include "PGMBth.h"
1067	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1068	#undef PGM_BTH_NAME
1069	#undef PGM_BTH_NAME_RC_STR
1070	#undef PGM_BTH_NAME_R0_STR
1071	#undef PGM_GST_TYPE
1072	#undef PGM_GST_NAME
1073	#undef PGM_GST_NAME_RC_STR
1074	#undef PGM_GST_NAME_R0_STR
1075
1076	/* Guest - protected mode */
1077	#define PGM_GST_TYPE PGM_TYPE_PROT
1078	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
1079	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
1080	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
1081	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_PROT(name)
1082	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_PROT_STR(name)
1083	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_PROT_STR(name)
1084	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
1085	#include "PGMGstDefs.h"
1086	#include "PGMBth.h"
1087	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1088	#undef PGM_BTH_NAME
1089	#undef PGM_BTH_NAME_RC_STR
1090	#undef PGM_BTH_NAME_R0_STR
1091	#undef PGM_GST_TYPE
1092	#undef PGM_GST_NAME
1093	#undef PGM_GST_NAME_RC_STR
1094	#undef PGM_GST_NAME_R0_STR
1095
1096	/* Guest - 32-bit mode */
1097	#define PGM_GST_TYPE PGM_TYPE_32BIT
1098	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
1099	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
1100	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
1101	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_32BIT(name)
1102	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_32BIT_STR(name)
1103	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_32BIT_STR(name)
1104	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
1105	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
1106	#include "PGMGstDefs.h"
1107	#include "PGMBth.h"
1108	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1109	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1110	#undef PGM_BTH_NAME
1111	#undef PGM_BTH_NAME_RC_STR
1112	#undef PGM_BTH_NAME_R0_STR
1113	#undef PGM_GST_TYPE
1114	#undef PGM_GST_NAME
1115	#undef PGM_GST_NAME_RC_STR
1116	#undef PGM_GST_NAME_R0_STR
1117
1118	/* Guest - PAE mode */
1119	#define PGM_GST_TYPE PGM_TYPE_PAE
1120	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
1121	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
1122	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
1123	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_PAE(name)
1124	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_PAE_STR(name)
1125	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_PAE_STR(name)
1126	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1127	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1128	#include "PGMGstDefs.h"
1129	#include "PGMBth.h"
1130	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1131	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1132	#undef PGM_BTH_NAME
1133	#undef PGM_BTH_NAME_RC_STR
1134	#undef PGM_BTH_NAME_R0_STR
1135	#undef PGM_GST_TYPE
1136	#undef PGM_GST_NAME
1137	#undef PGM_GST_NAME_RC_STR
1138	#undef PGM_GST_NAME_R0_STR
1139
1140	#ifdef VBOX_WITH_64_BITS_GUESTS
1141	/* Guest - AMD64 mode */
1142	# define PGM_GST_TYPE PGM_TYPE_AMD64
1143	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
1144	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
1145	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
1146	# define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_AMD64(name)
1147	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_AMD64_STR(name)
1148	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_AMD64_STR(name)
1149	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1150	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1151	# include "PGMGstDefs.h"
1152	# include "PGMBth.h"
1153	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
1154	# undef BTH_PGMPOOLKIND_PT_FOR_PT
1155	# undef PGM_BTH_NAME
1156	# undef PGM_BTH_NAME_RC_STR
1157	# undef PGM_BTH_NAME_R0_STR
1158	# undef PGM_GST_TYPE
1159	# undef PGM_GST_NAME
1160	# undef PGM_GST_NAME_RC_STR
1161	# undef PGM_GST_NAME_R0_STR
1162	#endif /* VBOX_WITH_64_BITS_GUESTS */
1163
1164	#undef PGM_SHW_TYPE
1165	#undef PGM_SHW_NAME
1166	#undef PGM_SHW_NAME_RC_STR
1167	#undef PGM_SHW_NAME_R0_STR
1168
1169
1170
1171	/**
1172	* Initiates the paging of VM.
1173	*
1174	* @returns VBox status code.
1175	* @param pVM Pointer to VM structure.
1176	*/
1177	VMMR3DECL(int) PGMR3Init(PVM pVM)
1178	{
1179	LogFlow(("PGMR3Init:\n"));
1180	PCFGMNODE pCfgPGM = CFGMR3GetChild(CFGMR3GetRoot(pVM), "/PGM");
1181	int rc;
1182
1183	/*
1184	* Assert alignment and sizes.
1185	*/
1186	AssertCompile(sizeof(pVM->pgm.s) <= sizeof(pVM->pgm.padding));
1187	AssertCompile(sizeof(pVM->aCpus[0].pgm.s) <= sizeof(pVM->aCpus[0].pgm.padding));
1188	AssertCompileMemberAlignment(PGM, CritSect, sizeof(uintptr_t));
1189
1190	/*
1191	* Init the structure.
1192	*/
1193	#ifdef PGM_WITHOUT_MAPPINGS
1194	pVM->pgm.s.fMappingsDisabled = true;
1195	#endif
1196	pVM->pgm.s.offVM = RT_OFFSETOF(VM, pgm.s);
1197	pVM->pgm.s.offVCpuPGM = RT_OFFSETOF(VMCPU, pgm.s);
1198
1199	/* Init the per-CPU part. */
1200	for (VMCPUID idCpu = 0; idCpu < pVM->cCpus; idCpu++)
1201	{
1202	PVMCPU pVCpu = &pVM->aCpus[idCpu];
1203	PPGMCPU pPGM = &pVCpu->pgm.s;
1204
1205	pPGM->offVM = (uintptr_t)&pVCpu->pgm.s - (uintptr_t)pVM;
1206	pPGM->offVCpu = RT_OFFSETOF(VMCPU, pgm.s);
1207	pPGM->offPGM = (uintptr_t)&pVCpu->pgm.s - (uintptr_t)&pVM->pgm.s;
1208
1209	pPGM->enmShadowMode = PGMMODE_INVALID;
1210	pPGM->enmGuestMode = PGMMODE_INVALID;
1211
1212	pPGM->GCPhysCR3 = NIL_RTGCPHYS;
1213
1214	pPGM->pGst32BitPdR3 = NULL;
1215	pPGM->pGstPaePdptR3 = NULL;
1216	pPGM->pGstAmd64Pml4R3 = NULL;
1217	#ifndef VBOX_WITH_2X_4GB_ADDR_SPACE
1218	pPGM->pGst32BitPdR0 = NIL_RTR0PTR;
1219	pPGM->pGstPaePdptR0 = NIL_RTR0PTR;
1220	pPGM->pGstAmd64Pml4R0 = NIL_RTR0PTR;
1221	#endif
1222	pPGM->pGst32BitPdRC = NIL_RTRCPTR;
1223	pPGM->pGstPaePdptRC = NIL_RTRCPTR;
1224	for (unsigned i = 0; i < RT_ELEMENTS(pVCpu->pgm.s.apGstPaePDsR3); i++)
1225	{
1226	pPGM->apGstPaePDsR3[i] = NULL;
1227	#ifndef VBOX_WITH_2X_4GB_ADDR_SPACE
1228	pPGM->apGstPaePDsR0[i] = NIL_RTR0PTR;
1229	#endif
1230	pPGM->apGstPaePDsRC[i] = NIL_RTRCPTR;
1231	pPGM->aGCPhysGstPaePDs[i] = NIL_RTGCPHYS;
1232	pPGM->aGCPhysGstPaePDsMonitored[i] = NIL_RTGCPHYS;
1233	}
1234
1235	pPGM->fA20Enabled = true;
1236	}
1237
1238	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1239	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(32) - 1; /* default; checked later */
1240	pVM->pgm.s.GCPtrPrevRamRangeMapping = MM_HYPER_AREA_ADDRESS;
1241
1242	rc = CFGMR3QueryBoolDef(CFGMR3GetRoot(pVM), "RamPreAlloc", &pVM->pgm.s.fRamPreAlloc,
1243	#ifdef VBOX_WITH_PREALLOC_RAM_BY_DEFAULT
1244	true
1245	#else
1246	false
1247	#endif
1248	);
1249	AssertLogRelRCReturn(rc, rc);
1250
1251	#ifdef PGM_WITH_LARGE_ADDRESS_SPACE_ON_32_BIT_HOST
1252	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, _1G / GMM_CHUNK_SIZE);
1253	#else
1254	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, UINT32_MAX);
1255	#endif
1256	AssertLogRelRCReturn(rc, rc);
1257	for (uint32_t i = 0; i < RT_ELEMENTS(pVM->pgm.s.ChunkR3Map.Tlb.aEntries); i++)
1258	pVM->pgm.s.ChunkR3Map.Tlb.aEntries[i].idChunk = NIL_GMM_CHUNKID;
1259
1260	/*
1261	* Get the configured RAM size - to estimate saved state size.
1262	*/
1263	uint64_t cbRam;
1264	rc = CFGMR3QueryU64(CFGMR3GetRoot(pVM), "RamSize", &cbRam);
1265	if (rc == VERR_CFGM_VALUE_NOT_FOUND)
1266	cbRam = 0;
1267	else if (RT_SUCCESS(rc))
1268	{
1269	if (cbRam < PAGE_SIZE)
1270	cbRam = 0;
1271	cbRam = RT_ALIGN_64(cbRam, PAGE_SIZE);
1272	}
1273	else
1274	{
1275	AssertMsgFailed(("Configuration error: Failed to query integer \"RamSize\", rc=%Rrc.\n", rc));
1276	return rc;
1277	}
1278
1279	#ifdef VBOX_WITH_STATISTICS
1280	/*
1281	* Allocate memory for the statistics before someone tries to use them.
1282	*/
1283	size_t cbTotalStats = RT_ALIGN_Z(sizeof(PGMSTATS), 64) + RT_ALIGN_Z(sizeof(PGMCPUSTATS), 64) * pVM->cCpus;
1284	void *pv;
1285	rc = MMHyperAlloc(pVM, RT_ALIGN_Z(cbTotalStats, PAGE_SIZE), PAGE_SIZE, MM_TAG_PGM, &pv);
1286	AssertRCReturn(rc, rc);
1287
1288	pVM->pgm.s.pStatsR3 = (PGMSTATS *)pv;
1289	pVM->pgm.s.pStatsR0 = MMHyperCCToR0(pVM, pv);
1290	pVM->pgm.s.pStatsRC = MMHyperCCToRC(pVM, pv);
1291	pv = (uint8_t *)pv + RT_ALIGN_Z(sizeof(PGMSTATS), 64);
1292
1293	for (VMCPUID iCpu = 0; iCpu < pVM->cCpus; iCpu++)
1294	{
1295	pVM->aCpus[iCpu].pgm.s.pStatsR3 = (PGMCPUSTATS *)pv;
1296	pVM->aCpus[iCpu].pgm.s.pStatsR0 = MMHyperCCToR0(pVM, pv);
1297	pVM->aCpus[iCpu].pgm.s.pStatsRC = MMHyperCCToRC(pVM, pv);
1298
1299	pv = (uint8_t *)pv + RT_ALIGN_Z(sizeof(PGMCPUSTATS), 64);
1300	}
1301	#endif /* VBOX_WITH_STATISTICS */
1302
1303	/*
1304	* Register callbacks, string formatters and the saved state data unit.
1305	*/
1306	#ifdef VBOX_STRICT
1307	VMR3AtStateRegister(pVM, pgmR3ResetNoMorePhysWritesFlag, NULL);
1308	#endif
1309	PGMRegisterStringFormatTypes();
1310
1311	rc = pgmR3InitSavedState(pVM, cbRam);
1312	if (RT_FAILURE(rc))
1313	return rc;
1314
1315	/*
1316	* Initialize the PGM critical section and flush the phys TLBs
1317	*/
1318	rc = PDMR3CritSectInit(pVM, &pVM->pgm.s.CritSect, RT_SRC_POS, "PGM");
1319	AssertRCReturn(rc, rc);
1320
1321	PGMR3PhysChunkInvalidateTLB(pVM);
1322	PGMPhysInvalidatePageMapTLB(pVM);
1323
1324	/*
1325	* For the time being we sport a full set of handy pages in addition to the base
1326	* memory to simplify things.
1327	*/
1328	rc = MMR3ReserveHandyPages(pVM, RT_ELEMENTS(pVM->pgm.s.aHandyPages)); /** @todo this should be changed to PGM_HANDY_PAGES_MIN but this needs proper testing... */
1329	AssertRCReturn(rc, rc);
1330
1331	/*
1332	* Trees
1333	*/
1334	rc = MMHyperAlloc(pVM, sizeof(PGMTREES), 0, MM_TAG_PGM, (void **)&pVM->pgm.s.pTreesR3);
1335	if (RT_SUCCESS(rc))
1336	{
1337	pVM->pgm.s.pTreesR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pTreesR3);
1338	pVM->pgm.s.pTreesRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pTreesR3);
1339
1340	/*
1341	* Allocate the zero page.
1342	*/
1343	rc = MMHyperAlloc(pVM, PAGE_SIZE, PAGE_SIZE, MM_TAG_PGM, &pVM->pgm.s.pvZeroPgR3);
1344	}
1345	if (RT_SUCCESS(rc))
1346	{
1347	pVM->pgm.s.pvZeroPgRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pvZeroPgR3);
1348	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
1349	pVM->pgm.s.HCPhysZeroPg = MMR3HyperHCVirt2HCPhys(pVM, pVM->pgm.s.pvZeroPgR3);
1350	AssertRelease(pVM->pgm.s.HCPhysZeroPg != NIL_RTHCPHYS);
1351
1352	/*
1353	* Allocate the invalid MMIO page.
1354	* (The invalid bits in HCPhysInvMmioPg are set later on init complete.)
1355	*/
1356	rc = MMHyperAlloc(pVM, PAGE_SIZE, PAGE_SIZE, MM_TAG_PGM, &pVM->pgm.s.pvMmioPgR3);
1357	}
1358	if (RT_SUCCESS(rc))
1359	{
1360	ASMMemFill32(pVM->pgm.s.pvMmioPgR3, PAGE_SIZE, 0xfeedface);
1361	pVM->pgm.s.HCPhysMmioPg = MMR3HyperHCVirt2HCPhys(pVM, pVM->pgm.s.pvMmioPgR3);
1362	AssertRelease(pVM->pgm.s.HCPhysMmioPg != NIL_RTHCPHYS);
1363	pVM->pgm.s.HCPhysInvMmioPg = pVM->pgm.s.HCPhysMmioPg;
1364
1365	/*
1366	* Init the paging.
1367	*/
1368	rc = pgmR3InitPaging(pVM);
1369	}
1370	if (RT_SUCCESS(rc))
1371	{
1372	/*
1373	* Init the page pool.
1374	*/
1375	rc = pgmR3PoolInit(pVM);
1376	}
1377	if (RT_SUCCESS(rc))
1378	{
1379	for (VMCPUID i = 0; i < pVM->cCpus; i++)
1380	{
1381	PVMCPU pVCpu = &pVM->aCpus[i];
1382	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
1383	if (RT_FAILURE(rc))
1384	break;
1385	}
1386	}
1387
1388	if (RT_SUCCESS(rc))
1389	{
1390	/*
1391	* Info & statistics
1392	*/
1393	DBGFR3InfoRegisterInternal(pVM, "mode",
1394	"Shows the current paging mode. "
1395	"Recognizes 'all', 'guest', 'shadow' and 'host' as arguments, defaulting to 'all' if nothing is given.",
1396	pgmR3InfoMode);
1397	DBGFR3InfoRegisterInternal(pVM, "pgmcr3",
1398	"Dumps all the entries in the top level paging table. No arguments.",
1399	pgmR3InfoCr3);
1400	DBGFR3InfoRegisterInternal(pVM, "phys",
1401	"Dumps all the physical address ranges. No arguments.",
1402	pgmR3PhysInfo);
1403	DBGFR3InfoRegisterInternal(pVM, "handlers",
1404	"Dumps physical, virtual and hyper virtual handlers. "
1405	"Pass 'phys', 'virt', 'hyper' as argument if only one kind is wanted."
1406	"Add 'nost' if the statistics are unwanted, use together with 'all' or explicit selection.",
1407	pgmR3InfoHandlers);
1408	DBGFR3InfoRegisterInternal(pVM, "mappings",
1409	"Dumps guest mappings.",
1410	pgmR3MapInfo);
1411
1412	pgmR3InitStats(pVM);
1413
1414	#ifdef VBOX_WITH_DEBUGGER
1415	/*
1416	* Debugger commands.
1417	*/
1418	static bool s_fRegisteredCmds = false;
1419	if (!s_fRegisteredCmds)
1420	{
1421	int rc2 = DBGCRegisterCommands(&g_aCmds[0], RT_ELEMENTS(g_aCmds));
1422	if (RT_SUCCESS(rc2))
1423	s_fRegisteredCmds = true;
1424	}
1425	#endif
1426	return VINF_SUCCESS;
1427	}
1428
1429	/* Almost no cleanup necessary, MM frees all memory. */
1430	PDMR3CritSectDelete(&pVM->pgm.s.CritSect);
1431
1432	return rc;
1433	}
1434
1435
1436	/**
1437	* Init paging.
1438	*
1439	* Since we need to check what mode the host is operating in before we can choose
1440	* the right paging functions for the host we have to delay this until R0 has
1441	* been initialized.
1442	*
1443	* @returns VBox status code.
1444	* @param pVM VM handle.
1445	*/
1446	static int pgmR3InitPaging(PVM pVM)
1447	{
1448	/*
1449	* Force a recalculation of modes and switcher so everyone gets notified.
1450	*/
1451	for (VMCPUID i = 0; i < pVM->cCpus; i++)
1452	{
1453	PVMCPU pVCpu = &pVM->aCpus[i];
1454
1455	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
1456	pVCpu->pgm.s.enmGuestMode = PGMMODE_INVALID;
1457	}
1458
1459	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1460
1461	/*
1462	* Allocate static mapping space for whatever the cr3 register
1463	* points to and in the case of PAE mode to the 4 PDs.
1464	*/
1465	int rc = MMR3HyperReserve(pVM, PAGE_SIZE * 5, "CR3 mapping", &pVM->pgm.s.GCPtrCR3Mapping);
1466	if (RT_FAILURE(rc))
1467	{
1468	AssertMsgFailed(("Failed to reserve two pages for cr mapping in HMA, rc=%Rrc\n", rc));
1469	return rc;
1470	}
1471	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
1472
1473	/*
1474	* Allocate pages for the three possible intermediate contexts
1475	* (AMD64, PAE and plain 32-Bit). We maintain all three contexts
1476	* for the sake of simplicity. The AMD64 uses the PAE for the
1477	* lower levels, making the total number of pages 11 (3 + 7 + 1).
1478	*
1479	* We assume that two page tables will be enought for the core code
1480	* mappings (HC virtual and identity).
1481	*/
1482	pVM->pgm.s.pInterPD = (PX86PD)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPD, VERR_NO_PAGE_MEMORY);
1483	pVM->pgm.s.apInterPTs[0] = (PX86PT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.apInterPTs[0], VERR_NO_PAGE_MEMORY);
1484	pVM->pgm.s.apInterPTs[1] = (PX86PT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.apInterPTs[1], VERR_NO_PAGE_MEMORY);
1485	pVM->pgm.s.apInterPaePTs[0] = (PX86PTPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePTs[0], VERR_NO_PAGE_MEMORY);
1486	pVM->pgm.s.apInterPaePTs[1] = (PX86PTPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePTs[1], VERR_NO_PAGE_MEMORY);
1487	pVM->pgm.s.apInterPaePDs[0] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[0], VERR_NO_PAGE_MEMORY);
1488	pVM->pgm.s.apInterPaePDs[1] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[1], VERR_NO_PAGE_MEMORY);
1489	pVM->pgm.s.apInterPaePDs[2] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[2], VERR_NO_PAGE_MEMORY);
1490	pVM->pgm.s.apInterPaePDs[3] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[3], VERR_NO_PAGE_MEMORY);
1491	pVM->pgm.s.pInterPaePDPT = (PX86PDPT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePDPT, VERR_NO_PAGE_MEMORY);
1492	pVM->pgm.s.pInterPaePDPT64 = (PX86PDPT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePDPT64, VERR_NO_PAGE_MEMORY);
1493	pVM->pgm.s.pInterPaePML4 = (PX86PML4)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePML4, VERR_NO_PAGE_MEMORY);
1494
1495	pVM->pgm.s.HCPhysInterPD = MMPage2Phys(pVM, pVM->pgm.s.pInterPD);
1496	AssertRelease(pVM->pgm.s.HCPhysInterPD != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPD & PAGE_OFFSET_MASK));
1497	pVM->pgm.s.HCPhysInterPaePDPT = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT);
1498	AssertRelease(pVM->pgm.s.HCPhysInterPaePDPT != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePDPT & PAGE_OFFSET_MASK));
1499	pVM->pgm.s.HCPhysInterPaePML4 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePML4);
1500	AssertRelease(pVM->pgm.s.HCPhysInterPaePML4 != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePML4 & PAGE_OFFSET_MASK) && pVM->pgm.s.HCPhysInterPaePML4 < 0xffffffff);
1501
1502	/*
1503	* Initialize the pages, setting up the PML4 and PDPT for repetitive 4GB action.
1504	*/
1505	ASMMemZeroPage(pVM->pgm.s.pInterPD);
1506	ASMMemZeroPage(pVM->pgm.s.apInterPTs[0]);
1507	ASMMemZeroPage(pVM->pgm.s.apInterPTs[1]);
1508
1509	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[0]);
1510	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[1]);
1511
1512	ASMMemZeroPage(pVM->pgm.s.pInterPaePDPT);
1513	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.apInterPaePDs); i++)
1514	{
1515	ASMMemZeroPage(pVM->pgm.s.apInterPaePDs[i]);
1516	pVM->pgm.s.pInterPaePDPT->a[i].u = X86_PDPE_P \| PGM_PLXFLAGS_PERMANENT
1517	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[i]);
1518	}
1519
1520	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.pInterPaePDPT64->a); i++)
1521	{
1522	const unsigned iPD = i % RT_ELEMENTS(pVM->pgm.s.apInterPaePDs);
1523	pVM->pgm.s.pInterPaePDPT64->a[i].u = X86_PDPE_P \| X86_PDPE_RW \| X86_PDPE_US \| X86_PDPE_A \| PGM_PLXFLAGS_PERMANENT
1524	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[iPD]);
1525	}
1526
1527	RTHCPHYS HCPhysInterPaePDPT64 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64);
1528	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.pInterPaePML4->a); i++)
1529	pVM->pgm.s.pInterPaePML4->a[i].u = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_US \| X86_PML4E_A \| PGM_PLXFLAGS_PERMANENT
1530	\| HCPhysInterPaePDPT64;
1531
1532	/*
1533	* Initialize paging workers and mode from current host mode
1534	* and the guest running in real mode.
1535	*/
1536	pVM->pgm.s.enmHostMode = SUPR3GetPagingMode();
1537	switch (pVM->pgm.s.enmHostMode)
1538	{
1539	case SUPPAGINGMODE_32_BIT:
1540	case SUPPAGINGMODE_32_BIT_GLOBAL:
1541	case SUPPAGINGMODE_PAE:
1542	case SUPPAGINGMODE_PAE_GLOBAL:
1543	case SUPPAGINGMODE_PAE_NX:
1544	case SUPPAGINGMODE_PAE_GLOBAL_NX:
1545	break;
1546
1547	case SUPPAGINGMODE_AMD64:
1548	case SUPPAGINGMODE_AMD64_GLOBAL:
1549	case SUPPAGINGMODE_AMD64_NX:
1550	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
1551	#ifndef VBOX_WITH_HYBRID_32BIT_KERNEL
1552	if (ARCH_BITS != 64)
1553	{
1554	AssertMsgFailed(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1555	LogRel(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1556	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1557	}
1558	#endif
1559	break;
1560	default:
1561	AssertMsgFailed(("Host mode %d is not supported\n", pVM->pgm.s.enmHostMode));
1562	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1563	}
1564	rc = pgmR3ModeDataInit(pVM, false /* don't resolve GC and R0 syms yet */);
1565	if (RT_SUCCESS(rc))
1566	{
1567	LogFlow(("pgmR3InitPaging: returns successfully\n"));
1568	#if HC_ARCH_BITS == 64
1569	LogRel(("Debug: HCPhysInterPD=%RHp HCPhysInterPaePDPT=%RHp HCPhysInterPaePML4=%RHp\n",
1570	pVM->pgm.s.HCPhysInterPD, pVM->pgm.s.HCPhysInterPaePDPT, pVM->pgm.s.HCPhysInterPaePML4));
1571	LogRel(("Debug: apInterPTs={%RHp,%RHp} apInterPaePTs={%RHp,%RHp} apInterPaePDs={%RHp,%RHp,%RHp,%RHp} pInterPaePDPT64=%RHp\n",
1572	MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[1]),
1573	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[1]),
1574	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[1]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[2]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[3]),
1575	MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64)));
1576	#endif
1577	return VINF_SUCCESS;
1578	}
1579
1580	LogFlow(("pgmR3InitPaging: returns %Rrc\n", rc));
1581	return rc;
1582	}
1583
1584
1585	/**
1586	* Init statistics
1587	* @returns VBox status code.
1588	*/
1589	static int pgmR3InitStats(PVM pVM)
1590	{
1591	PPGM pPGM = &pVM->pgm.s;
1592	int rc;
1593
1594	/*
1595	* Release statistics.
1596	*/
1597	/* Common - misc variables */
1598	STAM_REL_REG(pVM, &pPGM->cAllPages, STAMTYPE_U32, "/PGM/Page/cAllPages", STAMUNIT_COUNT, "The total number of pages.");
1599	STAM_REL_REG(pVM, &pPGM->cPrivatePages, STAMTYPE_U32, "/PGM/Page/cPrivatePages", STAMUNIT_COUNT, "The number of private pages.");
1600	STAM_REL_REG(pVM, &pPGM->cSharedPages, STAMTYPE_U32, "/PGM/Page/cSharedPages", STAMUNIT_COUNT, "The number of shared pages.");
1601	STAM_REL_REG(pVM, &pPGM->cReusedSharedPages, STAMTYPE_U32, "/PGM/Page/cReusedSharedPages", STAMUNIT_COUNT, "The number of reused shared pages.");
1602	STAM_REL_REG(pVM, &pPGM->cZeroPages, STAMTYPE_U32, "/PGM/Page/cZeroPages", STAMUNIT_COUNT, "The number of zero backed pages.");
1603	STAM_REL_REG(pVM, &pPGM->cPureMmioPages, STAMTYPE_U32, "/PGM/Page/cPureMmioPages", STAMUNIT_COUNT, "The number of pure MMIO pages.");
1604	STAM_REL_REG(pVM, &pPGM->cMonitoredPages, STAMTYPE_U32, "/PGM/Page/cMonitoredPages", STAMUNIT_COUNT, "The number of write monitored pages.");
1605	STAM_REL_REG(pVM, &pPGM->cWrittenToPages, STAMTYPE_U32, "/PGM/Page/cWrittenToPages", STAMUNIT_COUNT, "The number of previously write monitored pages that have been written to.");
1606	STAM_REL_REG(pVM, &pPGM->cWriteLockedPages, STAMTYPE_U32, "/PGM/Page/cWriteLockedPages", STAMUNIT_COUNT, "The number of write(/read) locked pages.");
1607	STAM_REL_REG(pVM, &pPGM->cReadLockedPages, STAMTYPE_U32, "/PGM/Page/cReadLockedPages", STAMUNIT_COUNT, "The number of read (only) locked pages.");
1608	STAM_REL_REG(pVM, &pPGM->cBalloonedPages, STAMTYPE_U32, "/PGM/Page/cBalloonedPages", STAMUNIT_COUNT, "The number of ballooned pages.");
1609	STAM_REL_REG(pVM, &pPGM->cHandyPages, STAMTYPE_U32, "/PGM/Page/cHandyPages", STAMUNIT_COUNT, "The number of handy pages (not included in cAllPages).");
1610	STAM_REL_REG(pVM, &pPGM->cRelocations, STAMTYPE_COUNTER, "/PGM/cRelocations", STAMUNIT_OCCURENCES,"Number of hypervisor relocations.");
1611	STAM_REL_REG(pVM, &pPGM->ChunkR3Map.c, STAMTYPE_U32, "/PGM/ChunkR3Map/c", STAMUNIT_COUNT, "Number of mapped chunks.");
1612	STAM_REL_REG(pVM, &pPGM->ChunkR3Map.cMax, STAMTYPE_U32, "/PGM/ChunkR3Map/cMax", STAMUNIT_COUNT, "Maximum number of mapped chunks.");
1613	STAM_REL_REG(pVM, &pPGM->cMappedChunks, STAMTYPE_U32, "/PGM/ChunkR3Map/Mapped", STAMUNIT_COUNT, "Number of times we mapped a chunk.");
1614	STAM_REL_REG(pVM, &pPGM->cUnmappedChunks, STAMTYPE_U32, "/PGM/ChunkR3Map/Unmapped", STAMUNIT_COUNT, "Number of times we unmapped a chunk.");
1615
1616	STAM_REL_REG(pVM, &pPGM->StatLargePageAlloc, STAMTYPE_COUNTER, "/PGM/LargePage/Alloc", STAMUNIT_OCCURENCES, "The number of large pages we've used.");
1617	STAM_REL_REG(pVM, &pPGM->StatLargePageReused, STAMTYPE_COUNTER, "/PGM/LargePage/Reused", STAMUNIT_OCCURENCES, "The number of times we've reused a large page.");
1618	STAM_REL_REG(pVM, &pPGM->StatLargePageRefused, STAMTYPE_COUNTER, "/PGM/LargePage/Refused", STAMUNIT_OCCURENCES, "The number of times we couldn't use a large page.");
1619	STAM_REL_REG(pVM, &pPGM->StatLargePageRecheck, STAMTYPE_COUNTER, "/PGM/LargePage/Recheck", STAMUNIT_OCCURENCES, "The number of times we've rechecked a disabled large page.");
1620
1621	/* Live save */
1622	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.fActive, STAMTYPE_U8, "/PGM/LiveSave/fActive", STAMUNIT_COUNT, "Active or not.");
1623	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cIgnoredPages, STAMTYPE_U32, "/PGM/LiveSave/cIgnoredPages", STAMUNIT_COUNT, "The number of ignored pages in the RAM ranges (i.e. MMIO, MMIO2 and ROM).");
1624	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cDirtyPagesLong, STAMTYPE_U32, "/PGM/LiveSave/cDirtyPagesLong", STAMUNIT_COUNT, "Longer term dirty page average.");
1625	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cDirtyPagesShort, STAMTYPE_U32, "/PGM/LiveSave/cDirtyPagesShort", STAMUNIT_COUNT, "Short term dirty page average.");
1626	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cPagesPerSecond, STAMTYPE_U32, "/PGM/LiveSave/cPagesPerSecond", STAMUNIT_COUNT, "Pages per second.");
1627	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cSavedPages, STAMTYPE_U64, "/PGM/LiveSave/cSavedPages", STAMUNIT_COUNT, "The total number of saved pages.");
1628	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cReadPages", STAMUNIT_COUNT, "RAM: Ready pages.");
1629	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cDirtyPages", STAMUNIT_COUNT, "RAM: Dirty pages.");
1630	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cZeroPages", STAMUNIT_COUNT, "RAM: Ready zero pages.");
1631	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cMonitoredPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cMonitoredPages", STAMUNIT_COUNT, "RAM: Write monitored pages.");
1632	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cReadPages", STAMUNIT_COUNT, "ROM: Ready pages.");
1633	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cDirtyPages", STAMUNIT_COUNT, "ROM: Dirty pages.");
1634	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cZeroPages", STAMUNIT_COUNT, "ROM: Ready zero pages.");
1635	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cMonitoredPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cMonitoredPages", STAMUNIT_COUNT, "ROM: Write monitored pages.");
1636	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cReadPages", STAMUNIT_COUNT, "MMIO2: Ready pages.");
1637	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cDirtyPages", STAMUNIT_COUNT, "MMIO2: Dirty pages.");
1638	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cZeroPages", STAMUNIT_COUNT, "MMIO2: Ready zero pages.");
1639	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cMonitoredPages,STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cMonitoredPages",STAMUNIT_COUNT, "MMIO2: Write monitored pages.");
1640
1641	#ifdef VBOX_WITH_STATISTICS
1642
1643	# define PGM_REG_COUNTER(a, b, c) \
1644	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_OCCURENCES, c, b); \
1645	AssertRC(rc);
1646
1647	# define PGM_REG_COUNTER_BYTES(a, b, c) \
1648	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_BYTES, c, b); \
1649	AssertRC(rc);
1650
1651	# define PGM_REG_PROFILE(a, b, c) \
1652	rc = STAMR3RegisterF(pVM, a, STAMTYPE_PROFILE, STAMVISIBILITY_ALWAYS, STAMUNIT_TICKS_PER_CALL, c, b); \
1653	AssertRC(rc);
1654
1655	PGMSTATS *pStats = pVM->pgm.s.pStatsR3;
1656
1657	PGM_REG_PROFILE(&pStats->StatAllocLargePage, "/PGM/LargePage/Prof/Alloc", "Time spent by the host OS for large page allocation.");
1658	PGM_REG_PROFILE(&pStats->StatClearLargePage, "/PGM/LargePage/Prof/Clear", "Time spent clearing the newly allocated large pages.");
1659	PGM_REG_COUNTER(&pStats->StatLargePageOverflow, "/PGM/LargePage/Overflow", "The number of times allocating a large page took too long.");
1660	PGM_REG_PROFILE(&pStats->StatR3IsValidLargePage, "/PGM/LargePage/Prof/R3/IsValid", "pgmPhysIsValidLargePage profiling - R3.");
1661	PGM_REG_PROFILE(&pStats->StatRZIsValidLargePage, "/PGM/LargePage/Prof/RZ/IsValid", "pgmPhysIsValidLargePage profiling - RZ.");
1662
1663	PGM_REG_COUNTER(&pStats->StatR3DetectedConflicts, "/PGM/R3/DetectedConflicts", "The number of times PGMR3CheckMappingConflicts() detected a conflict.");
1664	PGM_REG_PROFILE(&pStats->StatR3ResolveConflict, "/PGM/R3/ResolveConflict", "pgmR3SyncPTResolveConflict() profiling (includes the entire relocation).");
1665	PGM_REG_COUNTER(&pStats->StatR3PhysRead, "/PGM/R3/Phys/Read", "The number of times PGMPhysRead was called.");
1666	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysReadBytes, "/PGM/R3/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1667	PGM_REG_COUNTER(&pStats->StatR3PhysWrite, "/PGM/R3/Phys/Write", "The number of times PGMPhysWrite was called.");
1668	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysWriteBytes, "/PGM/R3/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1669	PGM_REG_COUNTER(&pStats->StatR3PhysSimpleRead, "/PGM/R3/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1670	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysSimpleReadBytes, "/PGM/R3/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1671	PGM_REG_COUNTER(&pStats->StatR3PhysSimpleWrite, "/PGM/R3/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1672	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysSimpleWriteBytes, "/PGM/R3/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1673
1674	PGM_REG_COUNTER(&pStats->StatRZChunkR3MapTlbHits, "/PGM/ChunkR3Map/TlbHitsRZ", "TLB hits.");
1675	PGM_REG_COUNTER(&pStats->StatRZChunkR3MapTlbMisses, "/PGM/ChunkR3Map/TlbMissesRZ", "TLB misses.");
1676	PGM_REG_PROFILE(&pStats->StatChunkAging, "/PGM/ChunkR3Map/Map/Aging", "Chunk aging profiling.");
1677	PGM_REG_PROFILE(&pStats->StatChunkFindCandidate, "/PGM/ChunkR3Map/Map/Find", "Chunk unmap find profiling.");
1678	PGM_REG_PROFILE(&pStats->StatChunkUnmap, "/PGM/ChunkR3Map/Map/Unmap", "Chunk unmap of address space profiling.");
1679	PGM_REG_PROFILE(&pStats->StatChunkMap, "/PGM/ChunkR3Map/Map/Map", "Chunk map of address space profiling.");
1680
1681	PGM_REG_COUNTER(&pStats->StatRZPageMapTlbHits, "/PGM/RZ/Page/MapTlbHits", "TLB hits.");
1682	PGM_REG_COUNTER(&pStats->StatRZPageMapTlbMisses, "/PGM/RZ/Page/MapTlbMisses", "TLB misses.");
1683	PGM_REG_COUNTER(&pStats->StatR3ChunkR3MapTlbHits, "/PGM/ChunkR3Map/TlbHitsR3", "TLB hits.");
1684	PGM_REG_COUNTER(&pStats->StatR3ChunkR3MapTlbMisses, "/PGM/ChunkR3Map/TlbMissesR3", "TLB misses.");
1685	PGM_REG_COUNTER(&pStats->StatR3PageMapTlbHits, "/PGM/R3/Page/MapTlbHits", "TLB hits.");
1686	PGM_REG_COUNTER(&pStats->StatR3PageMapTlbMisses, "/PGM/R3/Page/MapTlbMisses", "TLB misses.");
1687	PGM_REG_COUNTER(&pStats->StatPageMapTlbFlushes, "/PGM/R3/Page/MapTlbFlushes", "TLB flushes (all contexts).");
1688	PGM_REG_COUNTER(&pStats->StatPageMapTlbFlushEntry, "/PGM/R3/Page/MapTlbFlushEntry", "TLB entry flushes (all contexts).");
1689
1690	PGM_REG_PROFILE(&pStats->StatRZSyncCR3HandlerVirtualUpdate, "/PGM/RZ/SyncCR3/Handlers/VirtualUpdate", "Profiling of the virtual handler updates.");
1691	PGM_REG_PROFILE(&pStats->StatRZSyncCR3HandlerVirtualReset, "/PGM/RZ/SyncCR3/Handlers/VirtualReset", "Profiling of the virtual handler resets.");
1692	PGM_REG_PROFILE(&pStats->StatR3SyncCR3HandlerVirtualUpdate, "/PGM/R3/SyncCR3/Handlers/VirtualUpdate", "Profiling of the virtual handler updates.");
1693	PGM_REG_PROFILE(&pStats->StatR3SyncCR3HandlerVirtualReset, "/PGM/R3/SyncCR3/Handlers/VirtualReset", "Profiling of the virtual handler resets.");
1694
1695	PGM_REG_COUNTER(&pStats->StatRZPhysHandlerReset, "/PGM/RZ/PhysHandlerReset", "The number of times PGMHandlerPhysicalReset is called.");
1696	PGM_REG_COUNTER(&pStats->StatR3PhysHandlerReset, "/PGM/R3/PhysHandlerReset", "The number of times PGMHandlerPhysicalReset is called.");
1697	PGM_REG_COUNTER(&pStats->StatRZPhysHandlerLookupHits, "/PGM/RZ/PhysHandlerLookupHits", "The number of cache hits when looking up physical handlers.");
1698	PGM_REG_COUNTER(&pStats->StatR3PhysHandlerLookupHits, "/PGM/R3/PhysHandlerLookupHits", "The number of cache hits when looking up physical handlers.");
1699	PGM_REG_COUNTER(&pStats->StatRZPhysHandlerLookupMisses, "/PGM/RZ/PhysHandlerLookupMisses", "The number of cache misses when looking up physical handlers.");
1700	PGM_REG_COUNTER(&pStats->StatR3PhysHandlerLookupMisses, "/PGM/R3/PhysHandlerLookupMisses", "The number of cache misses when looking up physical handlers.");
1701	PGM_REG_PROFILE(&pStats->StatRZVirtHandlerSearchByPhys, "/PGM/RZ/VirtHandlerSearchByPhys", "Profiling of pgmHandlerVirtualFindByPhysAddr.");
1702	PGM_REG_PROFILE(&pStats->StatR3VirtHandlerSearchByPhys, "/PGM/R3/VirtHandlerSearchByPhys", "Profiling of pgmHandlerVirtualFindByPhysAddr.");
1703
1704	PGM_REG_COUNTER(&pStats->StatRZPageReplaceShared, "/PGM/RZ/Page/ReplacedShared", "Times a shared page was replaced.");
1705	PGM_REG_COUNTER(&pStats->StatRZPageReplaceZero, "/PGM/RZ/Page/ReplacedZero", "Times the zero page was replaced.");
1706	/// @todo PGM_REG_COUNTER(&pStats->StatRZPageHandyAllocs, "/PGM/RZ/Page/HandyAllocs", "Number of times we've allocated more handy pages.");
1707	PGM_REG_COUNTER(&pStats->StatR3PageReplaceShared, "/PGM/R3/Page/ReplacedShared", "Times a shared page was replaced.");
1708	PGM_REG_COUNTER(&pStats->StatR3PageReplaceZero, "/PGM/R3/Page/ReplacedZero", "Times the zero page was replaced.");
1709	/// @todo PGM_REG_COUNTER(&pStats->StatR3PageHandyAllocs, "/PGM/R3/Page/HandyAllocs", "Number of times we've allocated more handy pages.");
1710
1711	PGM_REG_COUNTER(&pStats->StatRZPhysRead, "/PGM/RZ/Phys/Read", "The number of times PGMPhysRead was called.");
1712	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysReadBytes, "/PGM/RZ/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1713	PGM_REG_COUNTER(&pStats->StatRZPhysWrite, "/PGM/RZ/Phys/Write", "The number of times PGMPhysWrite was called.");
1714	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysWriteBytes, "/PGM/RZ/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1715	PGM_REG_COUNTER(&pStats->StatRZPhysSimpleRead, "/PGM/RZ/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1716	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysSimpleReadBytes, "/PGM/RZ/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1717	PGM_REG_COUNTER(&pStats->StatRZPhysSimpleWrite, "/PGM/RZ/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1718	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysSimpleWriteBytes, "/PGM/RZ/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1719
1720	/* GC only: */
1721	PGM_REG_COUNTER(&pStats->StatRCInvlPgConflict, "/PGM/RC/InvlPgConflict", "Number of times PGMInvalidatePage() detected a mapping conflict.");
1722	PGM_REG_COUNTER(&pStats->StatRCInvlPgSyncMonCR3, "/PGM/RC/InvlPgSyncMonitorCR3", "Number of times PGMInvalidatePage() ran into PGM_SYNC_MONITOR_CR3.");
1723
1724	PGM_REG_COUNTER(&pStats->StatRCPhysRead, "/PGM/RC/Phys/Read", "The number of times PGMPhysRead was called.");
1725	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysReadBytes, "/PGM/RC/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1726	PGM_REG_COUNTER(&pStats->StatRCPhysWrite, "/PGM/RC/Phys/Write", "The number of times PGMPhysWrite was called.");
1727	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysWriteBytes, "/PGM/RC/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1728	PGM_REG_COUNTER(&pStats->StatRCPhysSimpleRead, "/PGM/RC/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1729	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysSimpleReadBytes, "/PGM/RC/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1730	PGM_REG_COUNTER(&pStats->StatRCPhysSimpleWrite, "/PGM/RC/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1731	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysSimpleWriteBytes, "/PGM/RC/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1732
1733	PGM_REG_COUNTER(&pStats->StatTrackVirgin, "/PGM/Track/Virgin", "The number of first time shadowings");
1734	PGM_REG_COUNTER(&pStats->StatTrackAliased, "/PGM/Track/Aliased", "The number of times switching to cRef2, i.e. the page is being shadowed by two PTs.");
1735	PGM_REG_COUNTER(&pStats->StatTrackAliasedMany, "/PGM/Track/AliasedMany", "The number of times we're tracking using cRef2.");
1736	PGM_REG_COUNTER(&pStats->StatTrackAliasedLots, "/PGM/Track/AliasedLots", "The number of times we're hitting pages which has overflowed cRef2");
1737	PGM_REG_COUNTER(&pStats->StatTrackOverflows, "/PGM/Track/Overflows", "The number of times the extent list grows too long.");
1738	PGM_REG_COUNTER(&pStats->StatTrackNoExtentsLeft, "/PGM/Track/NoExtentLeft", "The number of times the extent list was exhausted.");
1739	PGM_REG_PROFILE(&pStats->StatTrackDeref, "/PGM/Track/Deref", "Profiling of SyncPageWorkerTrackDeref (expensive).");
1740
1741	# undef PGM_REG_COUNTER
1742	# undef PGM_REG_PROFILE
1743	#endif
1744
1745	/*
1746	* Note! The layout below matches the member layout exactly!
1747	*/
1748
1749	/*
1750	* Common - stats
1751	*/
1752	for (VMCPUID idCpu = 0; idCpu < pVM->cCpus; idCpu++)
1753	{
1754	PPGMCPU pPgmCpu = &pVM->aCpus[idCpu].pgm.s;
1755
1756	#define PGM_REG_COUNTER(a, b, c) \
1757	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_OCCURENCES, c, b, idCpu); \
1758	AssertRC(rc);
1759	#define PGM_REG_PROFILE(a, b, c) \
1760	rc = STAMR3RegisterF(pVM, a, STAMTYPE_PROFILE, STAMVISIBILITY_ALWAYS, STAMUNIT_TICKS_PER_CALL, c, b, idCpu); \
1761	AssertRC(rc);
1762
1763	PGM_REG_COUNTER(&pPgmCpu->cGuestModeChanges, "/PGM/CPU%u/cGuestModeChanges", "Number of guest mode changes.");
1764
1765	#ifdef VBOX_WITH_STATISTICS
1766	PGMCPUSTATS *pCpuStats = pVM->aCpus[idCpu].pgm.s.pStatsR3;
1767
1768	# if 0 /* rarely useful; leave for debugging. */
1769	for (unsigned j = 0; j < RT_ELEMENTS(pPgmCpu->StatSyncPtPD); j++)
1770	STAMR3RegisterF(pVM, &pCpuStats->StatSyncPtPD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1771	"The number of SyncPT per PD n.", "/PGM/CPU%u/PDSyncPT/%04X", i, j);
1772	for (unsigned j = 0; j < RT_ELEMENTS(pCpuStats->StatSyncPagePD); j++)
1773	STAMR3RegisterF(pVM, &pCpuStats->StatSyncPagePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1774	"The number of SyncPage per PD n.", "/PGM/CPU%u/PDSyncPage/%04X", i, j);
1775	# endif
1776	/* R0 only: */
1777	PGM_REG_PROFILE(&pCpuStats->StatR0NpMiscfg, "/PGM/CPU%u/R0/NpMiscfg", "PGMR0Trap0eHandlerNPMisconfig() profiling.");
1778	PGM_REG_COUNTER(&pCpuStats->StatR0NpMiscfgSyncPage, "/PGM/CPU%u/R0/NpMiscfgSyncPage", "SyncPage calls from PGMR0Trap0eHandlerNPMisconfig().");
1779
1780	/* RZ only: */
1781	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0e, "/PGM/CPU%u/RZ/Trap0e", "Profiling of the PGMTrap0eHandler() body.");
1782	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Ballooned, "/PGM/CPU%u/RZ/Trap0e/Time2/Ballooned", "Profiling of the Trap0eHandler body when the cause is read access to a ballooned page.");
1783	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2CSAM, "/PGM/CPU%u/RZ/Trap0e/Time2/CSAM", "Profiling of the Trap0eHandler body when the cause is CSAM.");
1784	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2DirtyAndAccessed, "/PGM/CPU%u/RZ/Trap0e/Time2/DirtyAndAccessedBits", "Profiling of the Trap0eHandler body when the cause is dirty and/or accessed bit emulation.");
1785	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2GuestTrap, "/PGM/CPU%u/RZ/Trap0e/Time2/GuestTrap", "Profiling of the Trap0eHandler body when the cause is a guest trap.");
1786	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2HndPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerPhysical", "Profiling of the Trap0eHandler body when the cause is a physical handler.");
1787	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2HndVirt, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerVirtual", "Profiling of the Trap0eHandler body when the cause is a virtual handler.");
1788	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2HndUnhandled, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerUnhandled", "Profiling of the Trap0eHandler body when the cause is access outside the monitored areas of a monitored page.");
1789	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2InvalidPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/InvalidPhys", "Profiling of the Trap0eHandler body when the cause is access to an invalid physical guest address.");
1790	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2MakeWritable, "/PGM/CPU%u/RZ/Trap0e/Time2/MakeWritable", "Profiling of the Trap0eHandler body when the cause is that a page needed to be made writeable.");
1791	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Mapping, "/PGM/CPU%u/RZ/Trap0e/Time2/Mapping", "Profiling of the Trap0eHandler body when the cause is related to the guest mappings.");
1792	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Misc, "/PGM/CPU%u/RZ/Trap0e/Time2/Misc", "Profiling of the Trap0eHandler body when the cause is not known.");
1793	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSync, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSync", "Profiling of the Trap0eHandler body when the cause is an out-of-sync page.");
1794	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSyncHndPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncHndPhys", "Profiling of the Trap0eHandler body when the cause is an out-of-sync physical handler page.");
1795	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSyncHndVirt, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncHndVirt", "Profiling of the Trap0eHandler body when the cause is an out-of-sync virtual handler page.");
1796	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSyncHndObs, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncObsHnd", "Profiling of the Trap0eHandler body when the cause is an obsolete handler page.");
1797	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2SyncPT, "/PGM/CPU%u/RZ/Trap0e/Time2/SyncPT", "Profiling of the Trap0eHandler body when the cause is lazy syncing of a PT.");
1798	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2WPEmulation, "/PGM/CPU%u/RZ/Trap0e/Time2/WPEmulation", "Profiling of the Trap0eHandler body when the cause is CR0.WP emulation.");
1799	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eConflicts, "/PGM/CPU%u/RZ/Trap0e/Conflicts", "The number of times #PF was caused by an undetected conflict.");
1800	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersMapping, "/PGM/CPU%u/RZ/Trap0e/Handlers/Mapping", "Number of traps due to access handlers in mappings.");
1801	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersOutOfSync, "/PGM/CPU%u/RZ/Trap0e/Handlers/OutOfSync", "Number of traps due to out-of-sync handled pages.");
1802	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersPhysAll, "/PGM/CPU%u/RZ/Trap0e/Handlers/PhysAll", "Number of traps due to physical all-access handlers.");
1803	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersPhysAllOpt, "/PGM/CPU%u/RZ/Trap0e/Handlers/PhysAllOpt", "Number of the physical all-access handler traps using the optimization.");
1804	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersPhysWrite, "/PGM/CPU%u/RZ/Trap0e/Handlers/PhysWrite", "Number of traps due to physical write-access handlers.");
1805	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersVirtual, "/PGM/CPU%u/RZ/Trap0e/Handlers/Virtual", "Number of traps due to virtual access handlers.");
1806	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersVirtualByPhys, "/PGM/CPU%u/RZ/Trap0e/Handlers/VirtualByPhys", "Number of traps due to virtual access handlers by physical address.");
1807	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersVirtualUnmarked,"/PGM/CPU%u/RZ/Trap0e/Handlers/VirtualUnmarked","Number of traps due to virtual access handlers by virtual address (without proper physical flags).");
1808	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersUnhandled, "/PGM/CPU%u/RZ/Trap0e/Handlers/Unhandled", "Number of traps due to access outside range of monitored page(s).");
1809	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersInvalid, "/PGM/CPU%u/RZ/Trap0e/Handlers/Invalid", "Number of traps due to access to invalid physical memory.");
1810	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSNotPresentRead, "/PGM/CPU%u/RZ/Trap0e/Err/User/NPRead", "Number of user mode not present read page faults.");
1811	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSNotPresentWrite, "/PGM/CPU%u/RZ/Trap0e/Err/User/NPWrite", "Number of user mode not present write page faults.");
1812	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSWrite, "/PGM/CPU%u/RZ/Trap0e/Err/User/Write", "Number of user mode write page faults.");
1813	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSReserved, "/PGM/CPU%u/RZ/Trap0e/Err/User/Reserved", "Number of user mode reserved bit page faults.");
1814	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSNXE, "/PGM/CPU%u/RZ/Trap0e/Err/User/NXE", "Number of user mode NXE page faults.");
1815	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSRead, "/PGM/CPU%u/RZ/Trap0e/Err/User/Read", "Number of user mode read page faults.");
1816	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVNotPresentRead, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NPRead", "Number of supervisor mode not present read page faults.");
1817	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVNotPresentWrite, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NPWrite", "Number of supervisor mode not present write page faults.");
1818	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVWrite, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/Write", "Number of supervisor mode write page faults.");
1819	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVReserved, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/Reserved", "Number of supervisor mode reserved bit page faults.");
1820	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSNXE, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NXE", "Number of supervisor mode NXE page faults.");
1821	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eGuestPF, "/PGM/CPU%u/RZ/Trap0e/GuestPF", "Number of real guest page faults.");
1822	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eGuestPFMapping, "/PGM/CPU%u/RZ/Trap0e/GuestPF/InMapping", "Number of real guest page faults in a mapping.");
1823	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eWPEmulInRZ, "/PGM/CPU%u/RZ/Trap0e/WP/InRZ", "Number of guest page faults due to X86_CR0_WP emulation.");
1824	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eWPEmulToR3, "/PGM/CPU%u/RZ/Trap0e/WP/ToR3", "Number of guest page faults due to X86_CR0_WP emulation (forward to R3 for emulation).");
1825	#if 0 /* rarely useful; leave for debugging. */
1826	for (unsigned j = 0; j < RT_ELEMENTS(pCpuStats->StatRZTrap0ePD); j++)
1827	STAMR3RegisterF(pVM, &pCpuStats->StatRZTrap0ePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1828	"The number of traps in page directory n.", "/PGM/CPU%u/RZ/Trap0e/PD/%04X", i, j);
1829	#endif
1830	PGM_REG_COUNTER(&pCpuStats->StatRZGuestCR3WriteHandled, "/PGM/CPU%u/RZ/CR3WriteHandled", "The number of times the Guest CR3 change was successfully handled.");
1831	PGM_REG_COUNTER(&pCpuStats->StatRZGuestCR3WriteUnhandled, "/PGM/CPU%u/RZ/CR3WriteUnhandled", "The number of times the Guest CR3 change was passed back to the recompiler.");
1832	PGM_REG_COUNTER(&pCpuStats->StatRZGuestCR3WriteConflict, "/PGM/CPU%u/RZ/CR3WriteConflict", "The number of times the Guest CR3 monitoring detected a conflict.");
1833	PGM_REG_COUNTER(&pCpuStats->StatRZGuestROMWriteHandled, "/PGM/CPU%u/RZ/ROMWriteHandled", "The number of times the Guest ROM change was successfully handled.");
1834	PGM_REG_COUNTER(&pCpuStats->StatRZGuestROMWriteUnhandled, "/PGM/CPU%u/RZ/ROMWriteUnhandled", "The number of times the Guest ROM change was passed back to the recompiler.");
1835
1836	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapMigrateInvlPg, "/PGM/CPU%u/RZ/DynMap/MigrateInvlPg", "invlpg count in PGMR0DynMapMigrateAutoSet.");
1837	PGM_REG_PROFILE(&pCpuStats->StatRZDynMapGCPageInl, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl", "Calls to pgmR0DynMapGCPageInlined.");
1838	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlHits, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/Hits", "Hash table lookup hits.");
1839	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlMisses, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/Misses", "Misses that falls back to the code common.");
1840	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlRamHits, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/RamHits", "1st ram range hits.");
1841	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlRamMisses, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/RamMisses", "1st ram range misses, takes slow path.");
1842	PGM_REG_PROFILE(&pCpuStats->StatRZDynMapHCPageInl, "/PGM/CPU%u/RZ/DynMap/PageHCPageInl", "Calls to pgmRZDynMapHCPageInlined.");
1843	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapHCPageInlHits, "/PGM/CPU%u/RZ/DynMap/PageHCPageInl/Hits", "Hash table lookup hits.");
1844	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapHCPageInlMisses, "/PGM/CPU%u/RZ/DynMap/PageHCPageInl/Misses", "Misses that falls back to the code common.");
1845	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPage, "/PGM/CPU%u/RZ/DynMap/Page", "Calls to pgmR0DynMapPage");
1846	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetOptimize, "/PGM/CPU%u/RZ/DynMap/Page/SetOptimize", "Calls to pgmRZDynMapOptimizeAutoSet.");
1847	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetSearchFlushes, "/PGM/CPU%u/RZ/DynMap/Page/SetSearchFlushes", "Set search restoring to subset flushes.");
1848	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetSearchHits, "/PGM/CPU%u/RZ/DynMap/Page/SetSearchHits", "Set search hits.");
1849	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetSearchMisses, "/PGM/CPU%u/RZ/DynMap/Page/SetSearchMisses", "Set search misses.");
1850	PGM_REG_PROFILE(&pCpuStats->StatRZDynMapHCPage, "/PGM/CPU%u/RZ/DynMap/Page/HCPage", "Calls to pgmRZDynMapHCPageCommon (ring-0).");
1851	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageHits0, "/PGM/CPU%u/RZ/DynMap/Page/Hits0", "Hits at iPage+0");
1852	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageHits1, "/PGM/CPU%u/RZ/DynMap/Page/Hits1", "Hits at iPage+1");
1853	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageHits2, "/PGM/CPU%u/RZ/DynMap/Page/Hits2", "Hits at iPage+2");
1854	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageInvlPg, "/PGM/CPU%u/RZ/DynMap/Page/InvlPg", "invlpg count in pgmR0DynMapPageSlow.");
1855	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlow, "/PGM/CPU%u/RZ/DynMap/Page/Slow", "Calls to pgmR0DynMapPageSlow - subtract this from pgmR0DynMapPage to get 1st level hits.");
1856	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlowLoopHits, "/PGM/CPU%u/RZ/DynMap/Page/SlowLoopHits" , "Hits in the loop path.");
1857	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlowLoopMisses, "/PGM/CPU%u/RZ/DynMap/Page/SlowLoopMisses", "Misses in the loop path. NonLoopMisses = Slow - SlowLoopHit - SlowLoopMisses");
1858	//PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlowLostHits, "/PGM/CPU%u/R0/DynMap/Page/SlowLostHits", "Lost hits.");
1859	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSubsets, "/PGM/CPU%u/RZ/DynMap/Subsets", "Times PGMRZDynMapPushAutoSubset was called.");
1860	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPopFlushes, "/PGM/CPU%u/RZ/DynMap/SubsetPopFlushes", "Times PGMRZDynMapPopAutoSubset flushes the subset.");
1861	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[0], "/PGM/CPU%u/RZ/DynMap/SetFilledPct000..09", "00-09% filled (RC: min(set-size, dynmap-size))");
1862	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[1], "/PGM/CPU%u/RZ/DynMap/SetFilledPct010..19", "10-19% filled (RC: min(set-size, dynmap-size))");
1863	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[2], "/PGM/CPU%u/RZ/DynMap/SetFilledPct020..29", "20-29% filled (RC: min(set-size, dynmap-size))");
1864	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[3], "/PGM/CPU%u/RZ/DynMap/SetFilledPct030..39", "30-39% filled (RC: min(set-size, dynmap-size))");
1865	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[4], "/PGM/CPU%u/RZ/DynMap/SetFilledPct040..49", "40-49% filled (RC: min(set-size, dynmap-size))");
1866	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[5], "/PGM/CPU%u/RZ/DynMap/SetFilledPct050..59", "50-59% filled (RC: min(set-size, dynmap-size))");
1867	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[6], "/PGM/CPU%u/RZ/DynMap/SetFilledPct060..69", "60-69% filled (RC: min(set-size, dynmap-size))");
1868	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[7], "/PGM/CPU%u/RZ/DynMap/SetFilledPct070..79", "70-79% filled (RC: min(set-size, dynmap-size))");
1869	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[8], "/PGM/CPU%u/RZ/DynMap/SetFilledPct080..89", "80-89% filled (RC: min(set-size, dynmap-size))");
1870	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[9], "/PGM/CPU%u/RZ/DynMap/SetFilledPct090..99", "90-99% filled (RC: min(set-size, dynmap-size))");
1871	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[10], "/PGM/CPU%u/RZ/DynMap/SetFilledPct100", "100% filled (RC: min(set-size, dynmap-size))");
1872
1873	/* HC only: */
1874
1875	/* RZ & R3: */
1876	PGM_REG_PROFILE(&pCpuStats->StatRZSyncCR3, "/PGM/CPU%u/RZ/SyncCR3", "Profiling of the PGMSyncCR3() body.");
1877	PGM_REG_PROFILE(&pCpuStats->StatRZSyncCR3Handlers, "/PGM/CPU%u/RZ/SyncCR3/Handlers", "Profiling of the PGMSyncCR3() update handler section.");
1878	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3Global, "/PGM/CPU%u/RZ/SyncCR3/Global", "The number of global CR3 syncs.");
1879	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3NotGlobal, "/PGM/CPU%u/RZ/SyncCR3/NotGlobal", "The number of non-global CR3 syncs.");
1880	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstCacheHit, "/PGM/CPU%u/RZ/SyncCR3/DstChacheHit", "The number of times we got some kind of a cache hit.");
1881	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstFreed, "/PGM/CPU%u/RZ/SyncCR3/DstFreed", "The number of times we've had to free a shadow entry.");
1882	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstFreedSrcNP, "/PGM/CPU%u/RZ/SyncCR3/DstFreedSrcNP", "The number of times we've had to free a shadow entry for which the source entry was not present.");
1883	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstNotPresent, "/PGM/CPU%u/RZ/SyncCR3/DstNotPresent", "The number of times we've encountered a not present shadow entry for a present guest entry.");
1884	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstSkippedGlobalPD, "/PGM/CPU%u/RZ/SyncCR3/DstSkippedGlobalPD", "The number of times a global page directory wasn't flushed.");
1885	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstSkippedGlobalPT, "/PGM/CPU%u/RZ/SyncCR3/DstSkippedGlobalPT", "The number of times a page table with only global entries wasn't flushed.");
1886	PGM_REG_PROFILE(&pCpuStats->StatRZSyncPT, "/PGM/CPU%u/RZ/SyncPT", "Profiling of the pfnSyncPT() body.");
1887	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPTFailed, "/PGM/CPU%u/RZ/SyncPT/Failed", "The number of times pfnSyncPT() failed.");
1888	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPT4K, "/PGM/CPU%u/RZ/SyncPT/4K", "Nr of 4K PT syncs");
1889	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPT4M, "/PGM/CPU%u/RZ/SyncPT/4M", "Nr of 4M PT syncs");
1890	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPagePDNAs, "/PGM/CPU%u/RZ/SyncPagePDNAs", "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1891	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPagePDOutOfSync, "/PGM/CPU%u/RZ/SyncPagePDOutOfSync", "The number of time we've encountered an out-of-sync PD in SyncPage.");
1892	PGM_REG_COUNTER(&pCpuStats->StatRZAccessedPage, "/PGM/CPU%u/RZ/AccessedPage", "The number of pages marked not present for accessed bit emulation.");
1893	PGM_REG_PROFILE(&pCpuStats->StatRZDirtyBitTracking, "/PGM/CPU%u/RZ/DirtyPage", "Profiling the dirty bit tracking in CheckPageFault().");
1894	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPage, "/PGM/CPU%u/RZ/DirtyPage/Mark", "The number of pages marked read-only for dirty bit tracking.");
1895	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageBig, "/PGM/CPU%u/RZ/DirtyPage/MarkBig", "The number of 4MB pages marked read-only for dirty bit tracking.");
1896	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageSkipped, "/PGM/CPU%u/RZ/DirtyPage/Skipped", "The number of pages already dirty or readonly.");
1897	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageTrap, "/PGM/CPU%u/RZ/DirtyPage/Trap", "The number of traps generated for dirty bit tracking.");
1898	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageStale, "/PGM/CPU%u/RZ/DirtyPage/Stale", "The number of traps generated for dirty bit tracking (stale tlb entries).");
1899	PGM_REG_COUNTER(&pCpuStats->StatRZDirtiedPage, "/PGM/CPU%u/RZ/DirtyPage/SetDirty", "The number of pages marked dirty because of write accesses.");
1900	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyTrackRealPF, "/PGM/CPU%u/RZ/DirtyPage/RealPF", "The number of real pages faults during dirty bit tracking.");
1901	PGM_REG_COUNTER(&pCpuStats->StatRZPageAlreadyDirty, "/PGM/CPU%u/RZ/DirtyPage/AlreadySet", "The number of pages already marked dirty because of write accesses.");
1902	PGM_REG_PROFILE(&pCpuStats->StatRZInvalidatePage, "/PGM/CPU%u/RZ/InvalidatePage", "PGMInvalidatePage() profiling.");
1903	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePage4KBPages, "/PGM/CPU%u/RZ/InvalidatePage/4KBPages", "The number of times PGMInvalidatePage() was called for a 4KB page.");
1904	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePage4MBPages, "/PGM/CPU%u/RZ/InvalidatePage/4MBPages", "The number of times PGMInvalidatePage() was called for a 4MB page.");
1905	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePage4MBPagesSkip, "/PGM/CPU%u/RZ/InvalidatePage/4MBPagesSkip","The number of times PGMInvalidatePage() skipped a 4MB page.");
1906	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDMappings, "/PGM/CPU%u/RZ/InvalidatePage/PDMappings", "The number of times PGMInvalidatePage() was called for a page directory containing mappings (no conflict).");
1907	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDNAs, "/PGM/CPU%u/RZ/InvalidatePage/PDNAs", "The number of times PGMInvalidatePage() was called for a not accessed page directory.");
1908	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDNPs, "/PGM/CPU%u/RZ/InvalidatePage/PDNPs", "The number of times PGMInvalidatePage() was called for a not present page directory.");
1909	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDOutOfSync, "/PGM/CPU%u/RZ/InvalidatePage/PDOutOfSync", "The number of times PGMInvalidatePage() was called for an out of sync page directory.");
1910	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePageSkipped, "/PGM/CPU%u/RZ/InvalidatePage/Skipped", "The number of times PGMInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
1911	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncSupervisor, "/PGM/CPU%u/RZ/OutOfSync/SuperVisor", "Number of traps due to pages out of sync (P) and times VerifyAccessSyncPage calls SyncPage.");
1912	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncUser, "/PGM/CPU%u/RZ/OutOfSync/User", "Number of traps due to pages out of sync (P) and times VerifyAccessSyncPage calls SyncPage.");
1913	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncSupervisorWrite,"/PGM/CPU%u/RZ/OutOfSync/SuperVisorWrite", "Number of traps due to pages out of sync (RW) and times VerifyAccessSyncPage calls SyncPage.");
1914	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncUserWrite, "/PGM/CPU%u/RZ/OutOfSync/UserWrite", "Number of traps due to pages out of sync (RW) and times VerifyAccessSyncPage calls SyncPage.");
1915	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncBallloon, "/PGM/CPU%u/RZ/OutOfSync/Balloon", "The number of times a ballooned page was accessed (read).");
1916	PGM_REG_PROFILE(&pCpuStats->StatRZPrefetch, "/PGM/CPU%u/RZ/Prefetch", "PGMPrefetchPage profiling.");
1917	PGM_REG_PROFILE(&pCpuStats->StatRZFlushTLB, "/PGM/CPU%u/RZ/FlushTLB", "Profiling of the PGMFlushTLB() body.");
1918	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBNewCR3, "/PGM/CPU%u/RZ/FlushTLB/NewCR3", "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
1919	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBNewCR3Global, "/PGM/CPU%u/RZ/FlushTLB/NewCR3Global", "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
1920	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBSameCR3, "/PGM/CPU%u/RZ/FlushTLB/SameCR3", "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
1921	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBSameCR3Global, "/PGM/CPU%u/RZ/FlushTLB/SameCR3Global", "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
1922	PGM_REG_PROFILE(&pCpuStats->StatRZGstModifyPage, "/PGM/CPU%u/RZ/GstModifyPage", "Profiling of the PGMGstModifyPage() body.");
1923
1924	PGM_REG_PROFILE(&pCpuStats->StatR3SyncCR3, "/PGM/CPU%u/R3/SyncCR3", "Profiling of the PGMSyncCR3() body.");
1925	PGM_REG_PROFILE(&pCpuStats->StatR3SyncCR3Handlers, "/PGM/CPU%u/R3/SyncCR3/Handlers", "Profiling of the PGMSyncCR3() update handler section.");
1926	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3Global, "/PGM/CPU%u/R3/SyncCR3/Global", "The number of global CR3 syncs.");
1927	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3NotGlobal, "/PGM/CPU%u/R3/SyncCR3/NotGlobal", "The number of non-global CR3 syncs.");
1928	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstCacheHit, "/PGM/CPU%u/R3/SyncCR3/DstChacheHit", "The number of times we got some kind of a cache hit.");
1929	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstFreed, "/PGM/CPU%u/R3/SyncCR3/DstFreed", "The number of times we've had to free a shadow entry.");
1930	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstFreedSrcNP, "/PGM/CPU%u/R3/SyncCR3/DstFreedSrcNP", "The number of times we've had to free a shadow entry for which the source entry was not present.");
1931	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstNotPresent, "/PGM/CPU%u/R3/SyncCR3/DstNotPresent", "The number of times we've encountered a not present shadow entry for a present guest entry.");
1932	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstSkippedGlobalPD, "/PGM/CPU%u/R3/SyncCR3/DstSkippedGlobalPD", "The number of times a global page directory wasn't flushed.");
1933	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstSkippedGlobalPT, "/PGM/CPU%u/R3/SyncCR3/DstSkippedGlobalPT", "The number of times a page table with only global entries wasn't flushed.");
1934	PGM_REG_PROFILE(&pCpuStats->StatR3SyncPT, "/PGM/CPU%u/R3/SyncPT", "Profiling of the pfnSyncPT() body.");
1935	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPTFailed, "/PGM/CPU%u/R3/SyncPT/Failed", "The number of times pfnSyncPT() failed.");
1936	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPT4K, "/PGM/CPU%u/R3/SyncPT/4K", "Nr of 4K PT syncs");
1937	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPT4M, "/PGM/CPU%u/R3/SyncPT/4M", "Nr of 4M PT syncs");
1938	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPagePDNAs, "/PGM/CPU%u/R3/SyncPagePDNAs", "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1939	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPagePDOutOfSync, "/PGM/CPU%u/R3/SyncPagePDOutOfSync", "The number of time we've encountered an out-of-sync PD in SyncPage.");
1940	PGM_REG_COUNTER(&pCpuStats->StatR3AccessedPage, "/PGM/CPU%u/R3/AccessedPage", "The number of pages marked not present for accessed bit emulation.");
1941	PGM_REG_PROFILE(&pCpuStats->StatR3DirtyBitTracking, "/PGM/CPU%u/R3/DirtyPage", "Profiling the dirty bit tracking in CheckPageFault().");
1942	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPage, "/PGM/CPU%u/R3/DirtyPage/Mark", "The number of pages marked read-only for dirty bit tracking.");
1943	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPageBig, "/PGM/CPU%u/R3/DirtyPage/MarkBig", "The number of 4MB pages marked read-only for dirty bit tracking.");
1944	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPageSkipped, "/PGM/CPU%u/R3/DirtyPage/Skipped", "The number of pages already dirty or readonly.");
1945	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPageTrap, "/PGM/CPU%u/R3/DirtyPage/Trap", "The number of traps generated for dirty bit tracking.");
1946	PGM_REG_COUNTER(&pCpuStats->StatR3DirtiedPage, "/PGM/CPU%u/R3/DirtyPage/SetDirty", "The number of pages marked dirty because of write accesses.");
1947	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyTrackRealPF, "/PGM/CPU%u/R3/DirtyPage/RealPF", "The number of real pages faults during dirty bit tracking.");
1948	PGM_REG_COUNTER(&pCpuStats->StatR3PageAlreadyDirty, "/PGM/CPU%u/R3/DirtyPage/AlreadySet", "The number of pages already marked dirty because of write accesses.");
1949	PGM_REG_PROFILE(&pCpuStats->StatR3InvalidatePage, "/PGM/CPU%u/R3/InvalidatePage", "PGMInvalidatePage() profiling.");
1950	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePage4KBPages, "/PGM/CPU%u/R3/InvalidatePage/4KBPages", "The number of times PGMInvalidatePage() was called for a 4KB page.");
1951	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePage4MBPages, "/PGM/CPU%u/R3/InvalidatePage/4MBPages", "The number of times PGMInvalidatePage() was called for a 4MB page.");
1952	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePage4MBPagesSkip, "/PGM/CPU%u/R3/InvalidatePage/4MBPagesSkip","The number of times PGMInvalidatePage() skipped a 4MB page.");
1953	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDMappings, "/PGM/CPU%u/R3/InvalidatePage/PDMappings", "The number of times PGMInvalidatePage() was called for a page directory containing mappings (no conflict).");
1954	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDNAs, "/PGM/CPU%u/R3/InvalidatePage/PDNAs", "The number of times PGMInvalidatePage() was called for a not accessed page directory.");
1955	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDNPs, "/PGM/CPU%u/R3/InvalidatePage/PDNPs", "The number of times PGMInvalidatePage() was called for a not present page directory.");
1956	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDOutOfSync, "/PGM/CPU%u/R3/InvalidatePage/PDOutOfSync", "The number of times PGMInvalidatePage() was called for an out of sync page directory.");
1957	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePageSkipped, "/PGM/CPU%u/R3/InvalidatePage/Skipped", "The number of times PGMInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
1958	PGM_REG_COUNTER(&pCpuStats->StatR3PageOutOfSyncSupervisor, "/PGM/CPU%u/R3/OutOfSync/SuperVisor", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
1959	PGM_REG_COUNTER(&pCpuStats->StatR3PageOutOfSyncUser, "/PGM/CPU%u/R3/OutOfSync/User", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
1960	PGM_REG_COUNTER(&pCpuStats->StatR3PageOutOfSyncBallloon, "/PGM/CPU%u/R3/OutOfSync/Balloon", "The number of times a ballooned page was accessed (read).");
1961	PGM_REG_PROFILE(&pCpuStats->StatR3Prefetch, "/PGM/CPU%u/R3/Prefetch", "PGMPrefetchPage profiling.");
1962	PGM_REG_PROFILE(&pCpuStats->StatR3FlushTLB, "/PGM/CPU%u/R3/FlushTLB", "Profiling of the PGMFlushTLB() body.");
1963	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBNewCR3, "/PGM/CPU%u/R3/FlushTLB/NewCR3", "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
1964	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBNewCR3Global, "/PGM/CPU%u/R3/FlushTLB/NewCR3Global", "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
1965	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBSameCR3, "/PGM/CPU%u/R3/FlushTLB/SameCR3", "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
1966	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBSameCR3Global, "/PGM/CPU%u/R3/FlushTLB/SameCR3Global", "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
1967	PGM_REG_PROFILE(&pCpuStats->StatR3GstModifyPage, "/PGM/CPU%u/R3/GstModifyPage", "Profiling of the PGMGstModifyPage() body.");
1968	#endif /* VBOX_WITH_STATISTICS */
1969
1970	#undef PGM_REG_PROFILE
1971	#undef PGM_REG_COUNTER
1972
1973	}
1974
1975	return VINF_SUCCESS;
1976	}
1977
1978
1979	/**
1980	* Init the PGM bits that rely on VMMR0 and MM to be fully initialized.
1981	*
1982	* The dynamic mapping area will also be allocated and initialized at this
1983	* time. We could allocate it during PGMR3Init of course, but the mapping
1984	* wouldn't be allocated at that time preventing us from setting up the
1985	* page table entries with the dummy page.
1986	*
1987	* @returns VBox status code.
1988	* @param pVM VM handle.
1989	*/
1990	VMMR3DECL(int) PGMR3InitDynMap(PVM pVM)
1991	{
1992	RTGCPTR GCPtr;
1993	int rc;
1994
1995	/*
1996	* Reserve space for the dynamic mappings.
1997	*/
1998	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping", &GCPtr);
1999	if (RT_SUCCESS(rc))
2000	pVM->pgm.s.pbDynPageMapBaseGC = GCPtr;
2001
2002	if ( RT_SUCCESS(rc)
2003	&& (pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_PAE_SHIFT) != ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_PAE_SHIFT))
2004	{
2005	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping not crossing", &GCPtr);
2006	if (RT_SUCCESS(rc))
2007	pVM->pgm.s.pbDynPageMapBaseGC = GCPtr;
2008	}
2009	if (RT_SUCCESS(rc))
2010	{
2011	AssertRelease((pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_PAE_SHIFT) == ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_PAE_SHIFT));
2012	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
2013	}
2014	return rc;
2015	}
2016
2017
2018	/**
2019	* Ring-3 init finalizing.
2020	*
2021	* @returns VBox status code.
2022	* @param pVM The VM handle.
2023	*/
2024	VMMR3DECL(int) PGMR3InitFinalize(PVM pVM)
2025	{
2026	int rc;
2027
2028	/*
2029	* Reserve space for the dynamic mappings.
2030	* Initialize the dynamic mapping pages with dummy pages to simply the cache.
2031	*/
2032	/* get the pointer to the page table entries. */
2033	PPGMMAPPING pMapping = pgmGetMapping(pVM, pVM->pgm.s.pbDynPageMapBaseGC);
2034	AssertRelease(pMapping);
2035	const uintptr_t off = pVM->pgm.s.pbDynPageMapBaseGC - pMapping->GCPtr;
2036	const unsigned iPT = off >> X86_PD_SHIFT;
2037	const unsigned iPG = (off >> X86_PT_SHIFT) & X86_PT_MASK;
2038	pVM->pgm.s.paDynPageMap32BitPTEsGC = pMapping->aPTs[iPT].pPTRC + iPG * sizeof(pMapping->aPTs[0].pPTR3->a[0]);
2039	pVM->pgm.s.paDynPageMapPaePTEsGC = pMapping->aPTs[iPT].paPaePTsRC + iPG * sizeof(pMapping->aPTs[0].paPaePTsR3->a[0]);
2040
2041	/* init cache area */
2042	RTHCPHYS HCPhysDummy = MMR3PageDummyHCPhys(pVM);
2043	for (uint32_t offDynMap = 0; offDynMap < MM_HYPER_DYNAMIC_SIZE; offDynMap += PAGE_SIZE)
2044	{
2045	rc = PGMMap(pVM, pVM->pgm.s.pbDynPageMapBaseGC + offDynMap, HCPhysDummy, PAGE_SIZE, 0);
2046	AssertRCReturn(rc, rc);
2047	}
2048
2049	/*
2050	* Determine the max physical address width (MAXPHYADDR) and apply it to
2051	* all the mask members and stuff.
2052	*/
2053	uint32_t cMaxPhysAddrWidth;
2054	uint32_t uMaxExtLeaf = ASMCpuId_EAX(0x80000000);
2055	if ( uMaxExtLeaf >= 0x80000008
2056	&& uMaxExtLeaf <= 0x80000fff)
2057	{
2058	cMaxPhysAddrWidth = ASMCpuId_EAX(0x80000008) & 0xff;
2059	LogRel(("PGM: The CPU physical address width is %u bits\n", cMaxPhysAddrWidth));
2060	cMaxPhysAddrWidth = RT_MIN(52, cMaxPhysAddrWidth);
2061	pVM->pgm.s.fLessThan52PhysicalAddressBits = cMaxPhysAddrWidth < 52;
2062	for (uint32_t iBit = cMaxPhysAddrWidth; iBit < 52; iBit++)
2063	pVM->pgm.s.HCPhysInvMmioPg \|= RT_BIT_64(iBit);
2064	}
2065	else
2066	{
2067	LogRel(("PGM: ASSUMING CPU physical address width of 48 bits (uMaxExtLeaf=%#x)\n", uMaxExtLeaf));
2068	cMaxPhysAddrWidth = 48;
2069	pVM->pgm.s.fLessThan52PhysicalAddressBits = true;
2070	pVM->pgm.s.HCPhysInvMmioPg \|= UINT64_C(0x000f0000000000);
2071	}
2072
2073	pVM->pgm.s.GCPhysInvAddrMask = 0;
2074	for (uint32_t iBit = cMaxPhysAddrWidth; iBit < 64; iBit++)
2075	pVM->pgm.s.GCPhysInvAddrMask \|= RT_BIT_64(iBit);
2076
2077	/*
2078	* Initialize the invalid paging entry masks, assuming NX is disabled.
2079	*/
2080	uint64_t fMbzPageFrameMask = pVM->pgm.s.GCPhysInvAddrMask & UINT64_C(0x000ffffffffff000);
2081	for (VMCPUID iCpu = 0; iCpu < pVM->cCpus; iCpu++)
2082	{
2083	PVMCPU pVCpu = &pVM->aCpus[iCpu];
2084
2085	/** @todo The manuals are not entirely clear whether the physical
2086	* address width is relevant. See table 5-9 in the intel
2087	* manual vs the PDE4M descriptions. Write testcase (NP). */
2088	pVCpu->pgm.s.fGst32BitMbzBigPdeMask = ((uint32_t)(fMbzPageFrameMask >> (32 - 13)) & X86_PDE4M_PG_HIGH_MASK)
2089	\| X86_PDE4M_MBZ_MASK;
2090
2091	pVCpu->pgm.s.fGstPaeMbzPteMask = fMbzPageFrameMask \| X86_PTE_PAE_MBZ_MASK_NO_NX;
2092	pVCpu->pgm.s.fGstPaeMbzPdeMask = fMbzPageFrameMask \| X86_PDE_PAE_MBZ_MASK_NO_NX;
2093	pVCpu->pgm.s.fGstPaeMbzBigPdeMask = fMbzPageFrameMask \| X86_PDE2M_PAE_MBZ_MASK_NO_NX;
2094	pVCpu->pgm.s.fGstPaeMbzPdpeMask = fMbzPageFrameMask \| X86_PDPE_PAE_MBZ_MASK;
2095
2096	pVCpu->pgm.s.fGstAmd64MbzPteMask = fMbzPageFrameMask \| X86_PTE_LM_MBZ_MASK_NO_NX;
2097	pVCpu->pgm.s.fGstAmd64MbzPdeMask = fMbzPageFrameMask \| X86_PDE_LM_MBZ_MASK_NX;
2098	pVCpu->pgm.s.fGstAmd64MbzBigPdeMask = fMbzPageFrameMask \| X86_PDE2M_LM_MBZ_MASK_NX;
2099	pVCpu->pgm.s.fGstAmd64MbzPdpeMask = fMbzPageFrameMask \| X86_PDPE_LM_MBZ_MASK_NO_NX;
2100	pVCpu->pgm.s.fGstAmd64MbzBigPdpeMask = fMbzPageFrameMask \| X86_PDPE1G_LM_MBZ_MASK_NO_NX;
2101	pVCpu->pgm.s.fGstAmd64MbzPml4eMask = fMbzPageFrameMask \| X86_PML4E_MBZ_MASK_NO_NX;
2102
2103	pVCpu->pgm.s.fGst64ShadowedPteMask = X86_PTE_P \| X86_PTE_RW \| X86_PTE_US \| X86_PTE_G \| X86_PTE_A \| X86_PTE_D;
2104	pVCpu->pgm.s.fGst64ShadowedPdeMask = X86_PDE_P \| X86_PDE_RW \| X86_PDE_US \| X86_PDE_A;
2105	pVCpu->pgm.s.fGst64ShadowedBigPdeMask = X86_PDE4M_P \| X86_PDE4M_RW \| X86_PDE4M_US \| X86_PDE4M_A;
2106	pVCpu->pgm.s.fGst64ShadowedBigPde4PteMask =
2107	X86_PDE4M_P \| X86_PDE4M_RW \| X86_PDE4M_US \| X86_PDE4M_G \| X86_PDE4M_A \| X86_PDE4M_D;
2108	pVCpu->pgm.s.fGstAmd64ShadowedPdpeMask = X86_PDPE_P \| X86_PDPE_RW \| X86_PDPE_US \| X86_PDPE_A;
2109	pVCpu->pgm.s.fGstAmd64ShadowedPml4eMask = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_US \| X86_PML4E_A;
2110	}
2111
2112	/*
2113	* Note that AMD uses all the 8 reserved bits for the address (so 40 bits in total);
2114	* Intel only goes up to 36 bits, so we stick to 36 as well.
2115	* Update: More recent intel manuals specifies 40 bits just like AMD.
2116	*/
2117	uint32_t u32Dummy, u32Features;
2118	CPUMGetGuestCpuId(VMMGetCpu(pVM), 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
2119	if (u32Features & X86_CPUID_FEATURE_EDX_PSE36)
2120	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(RT_MAX(36, cMaxPhysAddrWidth)) - 1;
2121	else
2122	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(32) - 1;
2123
2124	/*
2125	* Allocate memory if we're supposed to do that.
2126	*/
2127	if (pVM->pgm.s.fRamPreAlloc)
2128	rc = pgmR3PhysRamPreAllocate(pVM);
2129
2130	LogRel(("PGMR3InitFinalize: 4 MB PSE mask %RGp\n", pVM->pgm.s.GCPhys4MBPSEMask));
2131	return rc;
2132	}
2133
2134
2135	/**
2136	* Applies relocations to data and code managed by this component.
2137	*
2138	* This function will be called at init and whenever the VMM need to relocate it
2139	* self inside the GC.
2140	*
2141	* @param pVM The VM.
2142	* @param offDelta Relocation delta relative to old location.
2143	*/
2144	VMMR3DECL(void) PGMR3Relocate(PVM pVM, RTGCINTPTR offDelta)
2145	{
2146	LogFlow(("PGMR3Relocate %RGv to %RGv\n", pVM->pgm.s.GCPtrCR3Mapping, pVM->pgm.s.GCPtrCR3Mapping + offDelta));
2147
2148	/*
2149	* Paging stuff.
2150	*/
2151	pVM->pgm.s.GCPtrCR3Mapping += offDelta;
2152
2153	pgmR3ModeDataInit(pVM, true /* resolve GC/R0 symbols */);
2154
2155	/* Shadow, guest and both mode switch & relocation for each VCPU. */
2156	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2157	{
2158	PVMCPU pVCpu = &pVM->aCpus[i];
2159
2160	pgmR3ModeDataSwitch(pVM, pVCpu, pVCpu->pgm.s.enmShadowMode, pVCpu->pgm.s.enmGuestMode);
2161
2162	PGM_SHW_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2163	PGM_GST_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2164	PGM_BTH_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2165	}
2166
2167	/*
2168	* Trees.
2169	*/
2170	pVM->pgm.s.pTreesRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pTreesR3);
2171
2172	/*
2173	* Ram ranges.
2174	*/
2175	if (pVM->pgm.s.pRamRangesR3)
2176	{
2177	/* Update the pSelfRC pointers and relink them. */
2178	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesR3; pCur; pCur = pCur->pNextR3)
2179	if (!(pCur->fFlags & PGM_RAM_RANGE_FLAGS_FLOATING))
2180	pCur->pSelfRC = MMHyperCCToRC(pVM, pCur);
2181	pgmR3PhysRelinkRamRanges(pVM);
2182	}
2183
2184	/*
2185	* Update the pSelfRC pointer of the MMIO2 ram ranges since they might not
2186	* be mapped and thus not included in the above exercise.
2187	*/
2188	for (PPGMMMIO2RANGE pCur = pVM->pgm.s.pMmio2RangesR3; pCur; pCur = pCur->pNextR3)
2189	if (!(pCur->RamRange.fFlags & PGM_RAM_RANGE_FLAGS_FLOATING))
2190	pCur->RamRange.pSelfRC = MMHyperCCToRC(pVM, &pCur->RamRange);
2191
2192	/*
2193	* Update the two page directories with all page table mappings.
2194	* (One or more of them have changed, that's why we're here.)
2195	*/
2196	pVM->pgm.s.pMappingsRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pMappingsR3);
2197	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur->pNextR3; pCur = pCur->pNextR3)
2198	pCur->pNextRC = MMHyperR3ToRC(pVM, pCur->pNextR3);
2199
2200	/* Relocate GC addresses of Page Tables. */
2201	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur; pCur = pCur->pNextR3)
2202	{
2203	for (RTHCUINT i = 0; i < pCur->cPTs; i++)
2204	{
2205	pCur->aPTs[i].pPTRC = MMHyperR3ToRC(pVM, pCur->aPTs[i].pPTR3);
2206	pCur->aPTs[i].paPaePTsRC = MMHyperR3ToRC(pVM, pCur->aPTs[i].paPaePTsR3);
2207	}
2208	}
2209
2210	/*
2211	* Dynamic page mapping area.
2212	*/
2213	pVM->pgm.s.paDynPageMap32BitPTEsGC += offDelta;
2214	pVM->pgm.s.paDynPageMapPaePTEsGC += offDelta;
2215	pVM->pgm.s.pbDynPageMapBaseGC += offDelta;
2216
2217	if (pVM->pgm.s.pRCDynMap)
2218	{
2219	pVM->pgm.s.pRCDynMap += offDelta;
2220	PPGMRCDYNMAP pDynMap = (PPGMRCDYNMAP)MMHyperRCToCC(pVM, pVM->pgm.s.pRCDynMap);
2221
2222	pDynMap->paPages += offDelta;
2223	PPGMRCDYNMAPENTRY paPages = (PPGMRCDYNMAPENTRY)MMHyperRCToCC(pVM, pDynMap->paPages);
2224
2225	for (uint32_t iPage = 0; iPage < pDynMap->cPages; iPage++)
2226	{
2227	paPages[iPage].pvPage += offDelta;
2228	paPages[iPage].uPte.pv += offDelta;
2229	}
2230	}
2231
2232	/*
2233	* The Zero page.
2234	*/
2235	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
2236	#ifdef VBOX_WITH_2X_4GB_ADDR_SPACE
2237	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTR0PTR \|\| !VMMIsHwVirtExtForced(pVM));
2238	#else
2239	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTR0PTR);
2240	#endif
2241
2242	/*
2243	* Physical and virtual handlers.
2244	*/
2245	RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, true, pgmR3RelocatePhysHandler, &offDelta);
2246	pVM->pgm.s.pLastPhysHandlerRC = NIL_RTRCPTR;
2247	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesR3->VirtHandlers, true, pgmR3RelocateVirtHandler, &offDelta);
2248	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesR3->HyperVirtHandlers, true, pgmR3RelocateHyperVirtHandler, &offDelta);
2249
2250	/*
2251	* The page pool.
2252	*/
2253	pgmR3PoolRelocate(pVM);
2254
2255	#ifdef VBOX_WITH_STATISTICS
2256	/*
2257	* Statistics.
2258	*/
2259	pVM->pgm.s.pStatsRC = MMHyperCCToRC(pVM, pVM->pgm.s.pStatsR3);
2260	for (VMCPUID iCpu = 0; iCpu < pVM->cCpus; iCpu++)
2261	pVM->aCpus[iCpu].pgm.s.pStatsRC = MMHyperCCToRC(pVM, pVM->aCpus[iCpu].pgm.s.pStatsR3);
2262	#endif
2263	}
2264
2265
2266	/**
2267	* Callback function for relocating a physical access handler.
2268	*
2269	* @returns 0 (continue enum)
2270	* @param pNode Pointer to a PGMPHYSHANDLER node.
2271	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2272	* not certain the delta will fit in a void pointer for all possible configs.
2273	*/
2274	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser)
2275	{
2276	PPGMPHYSHANDLER pHandler = (PPGMPHYSHANDLER)pNode;
2277	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2278	if (pHandler->pfnHandlerRC)
2279	pHandler->pfnHandlerRC += offDelta;
2280	if (pHandler->pvUserRC >= 0x10000)
2281	pHandler->pvUserRC += offDelta;
2282	return 0;
2283	}
2284
2285
2286	/**
2287	* Callback function for relocating a virtual access handler.
2288	*
2289	* @returns 0 (continue enum)
2290	* @param pNode Pointer to a PGMVIRTHANDLER node.
2291	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2292	* not certain the delta will fit in a void pointer for all possible configs.
2293	*/
2294	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
2295	{
2296	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
2297	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2298	Assert( pHandler->enmType == PGMVIRTHANDLERTYPE_ALL
2299	\|\| pHandler->enmType == PGMVIRTHANDLERTYPE_WRITE);
2300	Assert(pHandler->pfnHandlerRC);
2301	pHandler->pfnHandlerRC += offDelta;
2302	return 0;
2303	}
2304
2305
2306	/**
2307	* Callback function for relocating a virtual access handler for the hypervisor mapping.
2308	*
2309	* @returns 0 (continue enum)
2310	* @param pNode Pointer to a PGMVIRTHANDLER node.
2311	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2312	* not certain the delta will fit in a void pointer for all possible configs.
2313	*/
2314	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
2315	{
2316	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
2317	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2318	Assert(pHandler->enmType == PGMVIRTHANDLERTYPE_HYPERVISOR);
2319	Assert(pHandler->pfnHandlerRC);
2320	pHandler->pfnHandlerRC += offDelta;
2321	return 0;
2322	}
2323
2324
2325	/**
2326	* Resets a virtual CPU when unplugged.
2327	*
2328	* @param pVM The VM handle.
2329	* @param pVCpu The virtual CPU handle.
2330	*/
2331	VMMR3DECL(void) PGMR3ResetUnpluggedCpu(PVM pVM, PVMCPU pVCpu)
2332	{
2333	int rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
2334	AssertRC(rc);
2335
2336	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
2337	AssertRC(rc);
2338
2339	STAM_REL_COUNTER_RESET(&pVCpu->pgm.s.cGuestModeChanges);
2340
2341	pgmR3PoolResetUnpluggedCpu(pVM, pVCpu);
2342
2343	/*
2344	* Re-init other members.
2345	*/
2346	pVCpu->pgm.s.fA20Enabled = true;
2347
2348	/*
2349	* Clear the FFs PGM owns.
2350	*/
2351	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
2352	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3_NON_GLOBAL);
2353	}
2354
2355
2356	/**
2357	* The VM is being reset.
2358	*
2359	* For the PGM component this means that any PD write monitors
2360	* needs to be removed.
2361	*
2362	* @param pVM VM handle.
2363	*/
2364	VMMR3DECL(void) PGMR3Reset(PVM pVM)
2365	{
2366	int rc;
2367
2368	LogFlow(("PGMR3Reset:\n"));
2369	VM_ASSERT_EMT(pVM);
2370
2371	pgmLock(pVM);
2372
2373	/*
2374	* Unfix any fixed mappings and disable CR3 monitoring.
2375	*/
2376	pVM->pgm.s.fMappingsFixed = false;
2377	pVM->pgm.s.fMappingsFixedRestored = false;
2378	pVM->pgm.s.GCPtrMappingFixed = NIL_RTGCPTR;
2379	pVM->pgm.s.cbMappingFixed = 0;
2380
2381	/*
2382	* Exit the guest paging mode before the pgm pool gets reset.
2383	* Important to clean up the amd64 case.
2384	*/
2385	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2386	{
2387	PVMCPU pVCpu = &pVM->aCpus[i];
2388	rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
2389	AssertRC(rc);
2390	}
2391
2392	#ifdef DEBUG
2393	DBGFR3InfoLog(pVM, "mappings", NULL);
2394	DBGFR3InfoLog(pVM, "handlers", "all nostat");
2395	#endif
2396
2397	/*
2398	* Switch mode back to real mode. (before resetting the pgm pool!)
2399	*/
2400	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2401	{
2402	PVMCPU pVCpu = &pVM->aCpus[i];
2403
2404	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
2405	AssertRC(rc);
2406
2407	STAM_REL_COUNTER_RESET(&pVCpu->pgm.s.cGuestModeChanges);
2408	}
2409
2410	/*
2411	* Reset the shadow page pool.
2412	*/
2413	pgmR3PoolReset(pVM);
2414
2415	/*
2416	* Re-init various other members and clear the FFs that PGM owns.
2417	*/
2418	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2419	{
2420	PVMCPU pVCpu = &pVM->aCpus[i];
2421
2422	pVCpu->pgm.s.fA20Enabled = true;
2423	pVCpu->pgm.s.fGst32BitPageSizeExtension = false;
2424	PGMNotifyNxeChanged(pVCpu, false);
2425
2426	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
2427	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3_NON_GLOBAL);
2428	}
2429
2430	/*
2431	* Reset (zero) RAM and shadow ROM pages.
2432	*/
2433	rc = pgmR3PhysRamReset(pVM);
2434	if (RT_SUCCESS(rc))
2435	rc = pgmR3PhysRomReset(pVM);
2436
2437
2438	pgmUnlock(pVM);
2439	AssertReleaseRC(rc);
2440	}
2441
2442
2443	#ifdef VBOX_STRICT
2444	/**
2445	* VM state change callback for clearing fNoMorePhysWrites after
2446	* a snapshot has been created.
2447	*/
2448	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser)
2449	{
2450	if ( enmState == VMSTATE_RUNNING
2451	\|\| enmState == VMSTATE_RESUMING)
2452	pVM->pgm.s.fNoMorePhysWrites = false;
2453	}
2454	#endif
2455
2456	/**
2457	* Private API to reset fNoMorePhysWrites.
2458	*/
2459	VMMR3DECL(void) PGMR3ResetNoMorePhysWritesFlag(PVM pVM)
2460	{
2461	pVM->pgm.s.fNoMorePhysWrites = false;
2462	}
2463
2464	/**
2465	* Terminates the PGM.
2466	*
2467	* @returns VBox status code.
2468	* @param pVM Pointer to VM structure.
2469	*/
2470	VMMR3DECL(int) PGMR3Term(PVM pVM)
2471	{
2472	/* Must free shared pages here. */
2473	pgmLock(pVM);
2474	pgmR3PhysRamTerm(pVM);
2475	pgmR3PhysRomTerm(pVM);
2476	pgmUnlock(pVM);
2477
2478	PGMDeregisterStringFormatTypes();
2479	return PDMR3CritSectDelete(&pVM->pgm.s.CritSect);
2480	}
2481
2482
2483	/**
2484	* Show paging mode.
2485	*
2486	* @param pVM VM Handle.
2487	* @param pHlp The info helpers.
2488	* @param pszArgs "all" (default), "guest", "shadow" or "host".
2489	*/
2490	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2491	{
2492	/* digest argument. */
2493	bool fGuest, fShadow, fHost;
2494	if (pszArgs)
2495	pszArgs = RTStrStripL(pszArgs);
2496	if (!pszArgs \|\| !*pszArgs \|\| strstr(pszArgs, "all"))
2497	fShadow = fHost = fGuest = true;
2498	else
2499	{
2500	fShadow = fHost = fGuest = false;
2501	if (strstr(pszArgs, "guest"))
2502	fGuest = true;
2503	if (strstr(pszArgs, "shadow"))
2504	fShadow = true;
2505	if (strstr(pszArgs, "host"))
2506	fHost = true;
2507	}
2508
2509	/** @todo SMP support! */
2510	/* print info. */
2511	if (fGuest)
2512	pHlp->pfnPrintf(pHlp, "Guest paging mode: %s, changed %RU64 times, A20 %s\n",
2513	PGMGetModeName(pVM->aCpus[0].pgm.s.enmGuestMode), pVM->aCpus[0].pgm.s.cGuestModeChanges.c,
2514	pVM->aCpus[0].pgm.s.fA20Enabled ? "enabled" : "disabled");
2515	if (fShadow)
2516	pHlp->pfnPrintf(pHlp, "Shadow paging mode: %s\n", PGMGetModeName(pVM->aCpus[0].pgm.s.enmShadowMode));
2517	if (fHost)
2518	{
2519	const char *psz;
2520	switch (pVM->pgm.s.enmHostMode)
2521	{
2522	case SUPPAGINGMODE_INVALID: psz = "invalid"; break;
2523	case SUPPAGINGMODE_32_BIT: psz = "32-bit"; break;
2524	case SUPPAGINGMODE_32_BIT_GLOBAL: psz = "32-bit+G"; break;
2525	case SUPPAGINGMODE_PAE: psz = "PAE"; break;
2526	case SUPPAGINGMODE_PAE_GLOBAL: psz = "PAE+G"; break;
2527	case SUPPAGINGMODE_PAE_NX: psz = "PAE+NX"; break;
2528	case SUPPAGINGMODE_PAE_GLOBAL_NX: psz = "PAE+G+NX"; break;
2529	case SUPPAGINGMODE_AMD64: psz = "AMD64"; break;
2530	case SUPPAGINGMODE_AMD64_GLOBAL: psz = "AMD64+G"; break;
2531	case SUPPAGINGMODE_AMD64_NX: psz = "AMD64+NX"; break;
2532	case SUPPAGINGMODE_AMD64_GLOBAL_NX: psz = "AMD64+G+NX"; break;
2533	default: psz = "unknown"; break;
2534	}
2535	pHlp->pfnPrintf(pHlp, "Host paging mode: %s\n", psz);
2536	}
2537	}
2538
2539
2540	/**
2541	* Dump registered MMIO ranges to the log.
2542	*
2543	* @param pVM VM Handle.
2544	* @param pHlp The info helpers.
2545	* @param pszArgs Arguments, ignored.
2546	*/
2547	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2548	{
2549	NOREF(pszArgs);
2550	pHlp->pfnPrintf(pHlp,
2551	"RAM ranges (pVM=%p)\n"
2552	"%.s %.s\n",
2553	pVM,
2554	sizeof(RTGCPHYS) * 4 + 1, "GC Phys Range ",
2555	sizeof(RTHCPTR) * 2, "pvHC ");
2556
2557	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesR3; pCur; pCur = pCur->pNextR3)
2558	pHlp->pfnPrintf(pHlp,
2559	"%RGp-%RGp %RHv %s\n",
2560	pCur->GCPhys,
2561	pCur->GCPhysLast,
2562	pCur->pvR3,
2563	pCur->pszDesc);
2564	}
2565
2566	/**
2567	* Dump the page directory to the log.
2568	*
2569	* @param pVM VM Handle.
2570	* @param pHlp The info helpers.
2571	* @param pszArgs Arguments, ignored.
2572	*/
2573	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2574	{
2575	/** @todo SMP support!! */
2576	PVMCPU pVCpu = &pVM->aCpus[0];
2577
2578	/** @todo fix this! Convert the PGMR3DumpHierarchyHC functions to do guest stuff. */
2579	/* Big pages supported? */
2580	const bool fPSE = !!(CPUMGetGuestCR4(pVCpu) & X86_CR4_PSE);
2581
2582	/* Global pages supported? */
2583	const bool fPGE = !!(CPUMGetGuestCR4(pVCpu) & X86_CR4_PGE);
2584
2585	NOREF(pszArgs);
2586
2587	/*
2588	* Get page directory addresses.
2589	*/
2590	PX86PD pPDSrc = pgmGstGet32bitPDPtr(pVCpu);
2591	Assert(pPDSrc);
2592	Assert(PGMPhysGCPhys2R3PtrAssert(pVM, (RTGCPHYS)(CPUMGetGuestCR3(pVCpu) & X86_CR3_PAGE_MASK), sizeof(*pPDSrc)) == pPDSrc);
2593
2594	/*
2595	* Iterate the page directory.
2596	*/
2597	for (unsigned iPD = 0; iPD < RT_ELEMENTS(pPDSrc->a); iPD++)
2598	{
2599	X86PDE PdeSrc = pPDSrc->a[iPD];
2600	if (PdeSrc.n.u1Present)
2601	{
2602	if (PdeSrc.b.u1Size && fPSE)
2603	pHlp->pfnPrintf(pHlp,
2604	"%04X - %RGp P=%d U=%d RW=%d G=%d - BIG\n",
2605	iPD,
2606	pgmGstGet4MBPhysPage(&pVM->pgm.s, PdeSrc),
2607	PdeSrc.b.u1Present, PdeSrc.b.u1User, PdeSrc.b.u1Write, PdeSrc.b.u1Global && fPGE);
2608	else
2609	pHlp->pfnPrintf(pHlp,
2610	"%04X - %RGp P=%d U=%d RW=%d [G=%d]\n",
2611	iPD,
2612	(RTGCPHYS)(PdeSrc.u & X86_PDE_PG_MASK),
2613	PdeSrc.n.u1Present, PdeSrc.n.u1User, PdeSrc.n.u1Write, PdeSrc.b.u1Global && fPGE);
2614	}
2615	}
2616	}
2617
2618
2619	/**
2620	* Service a VMMCALLRING3_PGM_LOCK call.
2621	*
2622	* @returns VBox status code.
2623	* @param pVM The VM handle.
2624	*/
2625	VMMR3DECL(int) PGMR3LockCall(PVM pVM)
2626	{
2627	int rc = PDMR3CritSectEnterEx(&pVM->pgm.s.CritSect, true /* fHostCall */);
2628	AssertRC(rc);
2629	return rc;
2630	}
2631
2632
2633	/**
2634	* Converts a PGMMODE value to a PGM_TYPE_* \#define.
2635	*
2636	* @returns PGM_TYPE_*.
2637	* @param pgmMode The mode value to convert.
2638	*/
2639	DECLINLINE(unsigned) pgmModeToType(PGMMODE pgmMode)
2640	{
2641	switch (pgmMode)
2642	{
2643	case PGMMODE_REAL: return PGM_TYPE_REAL;
2644	case PGMMODE_PROTECTED: return PGM_TYPE_PROT;
2645	case PGMMODE_32_BIT: return PGM_TYPE_32BIT;
2646	case PGMMODE_PAE:
2647	case PGMMODE_PAE_NX: return PGM_TYPE_PAE;
2648	case PGMMODE_AMD64:
2649	case PGMMODE_AMD64_NX: return PGM_TYPE_AMD64;
2650	case PGMMODE_NESTED: return PGM_TYPE_NESTED;
2651	case PGMMODE_EPT: return PGM_TYPE_EPT;
2652	default:
2653	AssertFatalMsgFailed(("pgmMode=%d\n", pgmMode));
2654	}
2655	}
2656
2657
2658	/**
2659	* Gets the index into the paging mode data array of a SHW+GST mode.
2660	*
2661	* @returns PGM::paPagingData index.
2662	* @param uShwType The shadow paging mode type.
2663	* @param uGstType The guest paging mode type.
2664	*/
2665	DECLINLINE(unsigned) pgmModeDataIndex(unsigned uShwType, unsigned uGstType)
2666	{
2667	Assert(uShwType >= PGM_TYPE_32BIT && uShwType <= PGM_TYPE_MAX);
2668	Assert(uGstType >= PGM_TYPE_REAL && uGstType <= PGM_TYPE_AMD64);
2669	return (uShwType - PGM_TYPE_32BIT) * (PGM_TYPE_AMD64 - PGM_TYPE_REAL + 1)
2670	+ (uGstType - PGM_TYPE_REAL);
2671	}
2672
2673
2674	/**
2675	* Gets the index into the paging mode data array of a SHW+GST mode.
2676	*
2677	* @returns PGM::paPagingData index.
2678	* @param enmShw The shadow paging mode.
2679	* @param enmGst The guest paging mode.
2680	*/
2681	DECLINLINE(unsigned) pgmModeDataIndexByMode(PGMMODE enmShw, PGMMODE enmGst)
2682	{
2683	Assert(enmShw >= PGMMODE_32_BIT && enmShw <= PGMMODE_MAX);
2684	Assert(enmGst > PGMMODE_INVALID && enmGst < PGMMODE_MAX);
2685	return pgmModeDataIndex(pgmModeToType(enmShw), pgmModeToType(enmGst));
2686	}
2687
2688
2689	/**
2690	* Calculates the max data index.
2691	* @returns The number of entries in the paging data array.
2692	*/
2693	DECLINLINE(unsigned) pgmModeDataMaxIndex(void)
2694	{
2695	return pgmModeDataIndex(PGM_TYPE_MAX, PGM_TYPE_AMD64) + 1;
2696	}
2697
2698
2699	/**
2700	* Initializes the paging mode data kept in PGM::paModeData.
2701	*
2702	* @param pVM The VM handle.
2703	* @param fResolveGCAndR0 Indicate whether or not GC and Ring-0 symbols can be resolved now.
2704	* This is used early in the init process to avoid trouble with PDM
2705	* not being initialized yet.
2706	*/
2707	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0)
2708	{
2709	PPGMMODEDATA pModeData;
2710	int rc;
2711
2712	/*
2713	* Allocate the array on the first call.
2714	*/
2715	if (!pVM->pgm.s.paModeData)
2716	{
2717	pVM->pgm.s.paModeData = (PPGMMODEDATA)MMR3HeapAllocZ(pVM, MM_TAG_PGM, sizeof(PGMMODEDATA) * pgmModeDataMaxIndex());
2718	AssertReturn(pVM->pgm.s.paModeData, VERR_NO_MEMORY);
2719	}
2720
2721	/*
2722	* Initialize the array entries.
2723	*/
2724	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_REAL)];
2725	pModeData->uShwType = PGM_TYPE_32BIT;
2726	pModeData->uGstType = PGM_TYPE_REAL;
2727	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2728	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2729	rc = PGM_BTH_NAME_32BIT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2730
2731	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGMMODE_PROTECTED)];
2732	pModeData->uShwType = PGM_TYPE_32BIT;
2733	pModeData->uGstType = PGM_TYPE_PROT;
2734	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2735	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2736	rc = PGM_BTH_NAME_32BIT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2737
2738	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_32BIT)];
2739	pModeData->uShwType = PGM_TYPE_32BIT;
2740	pModeData->uGstType = PGM_TYPE_32BIT;
2741	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2742	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2743	rc = PGM_BTH_NAME_32BIT_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2744
2745	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_REAL)];
2746	pModeData->uShwType = PGM_TYPE_PAE;
2747	pModeData->uGstType = PGM_TYPE_REAL;
2748	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2749	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2750	rc = PGM_BTH_NAME_PAE_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2751
2752	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PROT)];
2753	pModeData->uShwType = PGM_TYPE_PAE;
2754	pModeData->uGstType = PGM_TYPE_PROT;
2755	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2756	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2757	rc = PGM_BTH_NAME_PAE_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2758
2759	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_32BIT)];
2760	pModeData->uShwType = PGM_TYPE_PAE;
2761	pModeData->uGstType = PGM_TYPE_32BIT;
2762	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2763	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2764	rc = PGM_BTH_NAME_PAE_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2765
2766	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PAE)];
2767	pModeData->uShwType = PGM_TYPE_PAE;
2768	pModeData->uGstType = PGM_TYPE_PAE;
2769	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2770	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2771	rc = PGM_BTH_NAME_PAE_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2772
2773	#ifdef VBOX_WITH_64_BITS_GUESTS
2774	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_AMD64, PGM_TYPE_AMD64)];
2775	pModeData->uShwType = PGM_TYPE_AMD64;
2776	pModeData->uGstType = PGM_TYPE_AMD64;
2777	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2778	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2779	rc = PGM_BTH_NAME_AMD64_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2780	#endif
2781
2782	/* The nested paging mode. */
2783	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_REAL)];
2784	pModeData->uShwType = PGM_TYPE_NESTED;
2785	pModeData->uGstType = PGM_TYPE_REAL;
2786	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2787	rc = PGM_BTH_NAME_NESTED_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2788
2789	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGMMODE_PROTECTED)];
2790	pModeData->uShwType = PGM_TYPE_NESTED;
2791	pModeData->uGstType = PGM_TYPE_PROT;
2792	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2793	rc = PGM_BTH_NAME_NESTED_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2794
2795	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_32BIT)];
2796	pModeData->uShwType = PGM_TYPE_NESTED;
2797	pModeData->uGstType = PGM_TYPE_32BIT;
2798	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2799	rc = PGM_BTH_NAME_NESTED_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2800
2801	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_PAE)];
2802	pModeData->uShwType = PGM_TYPE_NESTED;
2803	pModeData->uGstType = PGM_TYPE_PAE;
2804	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2805	rc = PGM_BTH_NAME_NESTED_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2806
2807	#ifdef VBOX_WITH_64_BITS_GUESTS
2808	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2809	pModeData->uShwType = PGM_TYPE_NESTED;
2810	pModeData->uGstType = PGM_TYPE_AMD64;
2811	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2812	rc = PGM_BTH_NAME_NESTED_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2813	#endif
2814
2815	/* The shadow part of the nested callback mode depends on the host paging mode (AMD-V only). */
2816	switch (pVM->pgm.s.enmHostMode)
2817	{
2818	#if HC_ARCH_BITS == 32
2819	case SUPPAGINGMODE_32_BIT:
2820	case SUPPAGINGMODE_32_BIT_GLOBAL:
2821	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2822	{
2823	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2824	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2825	}
2826	# ifdef VBOX_WITH_64_BITS_GUESTS
2827	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2828	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2829	# endif
2830	break;
2831
2832	case SUPPAGINGMODE_PAE:
2833	case SUPPAGINGMODE_PAE_NX:
2834	case SUPPAGINGMODE_PAE_GLOBAL:
2835	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2836	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2837	{
2838	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2839	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2840	}
2841	# ifdef VBOX_WITH_64_BITS_GUESTS
2842	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2843	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2844	# endif
2845	break;
2846	#endif /* HC_ARCH_BITS == 32 */
2847
2848	#if HC_ARCH_BITS == 64 \|\| defined(RT_OS_DARWIN)
2849	case SUPPAGINGMODE_AMD64:
2850	case SUPPAGINGMODE_AMD64_GLOBAL:
2851	case SUPPAGINGMODE_AMD64_NX:
2852	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2853	# ifdef VBOX_WITH_64_BITS_GUESTS
2854	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_AMD64; i++)
2855	# else
2856	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2857	# endif
2858	{
2859	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2860	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2861	}
2862	break;
2863	#endif /* HC_ARCH_BITS == 64 \|\| RT_OS_DARWIN */
2864
2865	default:
2866	AssertFailed();
2867	break;
2868	}
2869
2870	/* Extended paging (EPT) / Intel VT-x */
2871	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_REAL)];
2872	pModeData->uShwType = PGM_TYPE_EPT;
2873	pModeData->uGstType = PGM_TYPE_REAL;
2874	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2875	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2876	rc = PGM_BTH_NAME_EPT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2877
2878	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_PROT)];
2879	pModeData->uShwType = PGM_TYPE_EPT;
2880	pModeData->uGstType = PGM_TYPE_PROT;
2881	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2882	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2883	rc = PGM_BTH_NAME_EPT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2884
2885	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_32BIT)];
2886	pModeData->uShwType = PGM_TYPE_EPT;
2887	pModeData->uGstType = PGM_TYPE_32BIT;
2888	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2889	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2890	rc = PGM_BTH_NAME_EPT_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2891
2892	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_PAE)];
2893	pModeData->uShwType = PGM_TYPE_EPT;
2894	pModeData->uGstType = PGM_TYPE_PAE;
2895	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2896	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2897	rc = PGM_BTH_NAME_EPT_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2898
2899	#ifdef VBOX_WITH_64_BITS_GUESTS
2900	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_AMD64)];
2901	pModeData->uShwType = PGM_TYPE_EPT;
2902	pModeData->uGstType = PGM_TYPE_AMD64;
2903	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2904	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2905	rc = PGM_BTH_NAME_EPT_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2906	#endif
2907	return VINF_SUCCESS;
2908	}
2909
2910
2911	/**
2912	* Switch to different (or relocated in the relocate case) mode data.
2913	*
2914	* @param pVM The VM handle.
2915	* @param pVCpu The VMCPU to operate on.
2916	* @param enmShw The the shadow paging mode.
2917	* @param enmGst The the guest paging mode.
2918	*/
2919	static void pgmR3ModeDataSwitch(PVM pVM, PVMCPU pVCpu, PGMMODE enmShw, PGMMODE enmGst)
2920	{
2921	PPGMMODEDATA pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndexByMode(enmShw, enmGst)];
2922
2923	Assert(pModeData->uGstType == pgmModeToType(enmGst));
2924	Assert(pModeData->uShwType == pgmModeToType(enmShw));
2925
2926	/* shadow */
2927	pVCpu->pgm.s.pfnR3ShwRelocate = pModeData->pfnR3ShwRelocate;
2928	pVCpu->pgm.s.pfnR3ShwExit = pModeData->pfnR3ShwExit;
2929	pVCpu->pgm.s.pfnR3ShwGetPage = pModeData->pfnR3ShwGetPage;
2930	Assert(pVCpu->pgm.s.pfnR3ShwGetPage);
2931	pVCpu->pgm.s.pfnR3ShwModifyPage = pModeData->pfnR3ShwModifyPage;
2932
2933	pVCpu->pgm.s.pfnRCShwGetPage = pModeData->pfnRCShwGetPage;
2934	pVCpu->pgm.s.pfnRCShwModifyPage = pModeData->pfnRCShwModifyPage;
2935
2936	pVCpu->pgm.s.pfnR0ShwGetPage = pModeData->pfnR0ShwGetPage;
2937	pVCpu->pgm.s.pfnR0ShwModifyPage = pModeData->pfnR0ShwModifyPage;
2938
2939
2940	/* guest */
2941	pVCpu->pgm.s.pfnR3GstRelocate = pModeData->pfnR3GstRelocate;
2942	pVCpu->pgm.s.pfnR3GstExit = pModeData->pfnR3GstExit;
2943	pVCpu->pgm.s.pfnR3GstGetPage = pModeData->pfnR3GstGetPage;
2944	Assert(pVCpu->pgm.s.pfnR3GstGetPage);
2945	pVCpu->pgm.s.pfnR3GstModifyPage = pModeData->pfnR3GstModifyPage;
2946	pVCpu->pgm.s.pfnR3GstGetPDE = pModeData->pfnR3GstGetPDE;
2947	pVCpu->pgm.s.pfnRCGstGetPage = pModeData->pfnRCGstGetPage;
2948	pVCpu->pgm.s.pfnRCGstModifyPage = pModeData->pfnRCGstModifyPage;
2949	pVCpu->pgm.s.pfnRCGstGetPDE = pModeData->pfnRCGstGetPDE;
2950	pVCpu->pgm.s.pfnR0GstGetPage = pModeData->pfnR0GstGetPage;
2951	pVCpu->pgm.s.pfnR0GstModifyPage = pModeData->pfnR0GstModifyPage;
2952	pVCpu->pgm.s.pfnR0GstGetPDE = pModeData->pfnR0GstGetPDE;
2953
2954	/* both */
2955	pVCpu->pgm.s.pfnR3BthRelocate = pModeData->pfnR3BthRelocate;
2956	pVCpu->pgm.s.pfnR3BthInvalidatePage = pModeData->pfnR3BthInvalidatePage;
2957	pVCpu->pgm.s.pfnR3BthSyncCR3 = pModeData->pfnR3BthSyncCR3;
2958	Assert(pVCpu->pgm.s.pfnR3BthSyncCR3);
2959	pVCpu->pgm.s.pfnR3BthPrefetchPage = pModeData->pfnR3BthPrefetchPage;
2960	pVCpu->pgm.s.pfnR3BthVerifyAccessSyncPage = pModeData->pfnR3BthVerifyAccessSyncPage;
2961	#ifdef VBOX_STRICT
2962	pVCpu->pgm.s.pfnR3BthAssertCR3 = pModeData->pfnR3BthAssertCR3;
2963	#endif
2964	pVCpu->pgm.s.pfnR3BthMapCR3 = pModeData->pfnR3BthMapCR3;
2965	pVCpu->pgm.s.pfnR3BthUnmapCR3 = pModeData->pfnR3BthUnmapCR3;
2966
2967	pVCpu->pgm.s.pfnRCBthTrap0eHandler = pModeData->pfnRCBthTrap0eHandler;
2968	pVCpu->pgm.s.pfnRCBthInvalidatePage = pModeData->pfnRCBthInvalidatePage;
2969	pVCpu->pgm.s.pfnRCBthSyncCR3 = pModeData->pfnRCBthSyncCR3;
2970	pVCpu->pgm.s.pfnRCBthPrefetchPage = pModeData->pfnRCBthPrefetchPage;
2971	pVCpu->pgm.s.pfnRCBthVerifyAccessSyncPage = pModeData->pfnRCBthVerifyAccessSyncPage;
2972	#ifdef VBOX_STRICT
2973	pVCpu->pgm.s.pfnRCBthAssertCR3 = pModeData->pfnRCBthAssertCR3;
2974	#endif
2975	pVCpu->pgm.s.pfnRCBthMapCR3 = pModeData->pfnRCBthMapCR3;
2976	pVCpu->pgm.s.pfnRCBthUnmapCR3 = pModeData->pfnRCBthUnmapCR3;
2977
2978	pVCpu->pgm.s.pfnR0BthTrap0eHandler = pModeData->pfnR0BthTrap0eHandler;
2979	pVCpu->pgm.s.pfnR0BthInvalidatePage = pModeData->pfnR0BthInvalidatePage;
2980	pVCpu->pgm.s.pfnR0BthSyncCR3 = pModeData->pfnR0BthSyncCR3;
2981	pVCpu->pgm.s.pfnR0BthPrefetchPage = pModeData->pfnR0BthPrefetchPage;
2982	pVCpu->pgm.s.pfnR0BthVerifyAccessSyncPage = pModeData->pfnR0BthVerifyAccessSyncPage;
2983	#ifdef VBOX_STRICT
2984	pVCpu->pgm.s.pfnR0BthAssertCR3 = pModeData->pfnR0BthAssertCR3;
2985	#endif
2986	pVCpu->pgm.s.pfnR0BthMapCR3 = pModeData->pfnR0BthMapCR3;
2987	pVCpu->pgm.s.pfnR0BthUnmapCR3 = pModeData->pfnR0BthUnmapCR3;
2988	}
2989
2990
2991	/**
2992	* Calculates the shadow paging mode.
2993	*
2994	* @returns The shadow paging mode.
2995	* @param pVM VM handle.
2996	* @param enmGuestMode The guest mode.
2997	* @param enmHostMode The host mode.
2998	* @param enmShadowMode The current shadow mode.
2999	* @param penmSwitcher Where to store the switcher to use.
3000	* VMMSWITCHER_INVALID means no change.
3001	*/
3002	static PGMMODE pgmR3CalcShadowMode(PVM pVM, PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher)
3003	{
3004	VMMSWITCHER enmSwitcher = VMMSWITCHER_INVALID;
3005	switch (enmGuestMode)
3006	{
3007	/*
3008	* When switching to real or protected mode we don't change
3009	* anything since it's likely that we'll switch back pretty soon.
3010	*
3011	* During pgmR3InitPaging we'll end up here with PGMMODE_INVALID
3012	* and is supposed to determine which shadow paging and switcher to
3013	* use during init.
3014	*/
3015	case PGMMODE_REAL:
3016	case PGMMODE_PROTECTED:
3017	if ( enmShadowMode != PGMMODE_INVALID
3018	&& !HWACCMIsEnabled(pVM) /* always switch in hwaccm mode! */)
3019	break; /* (no change) */
3020
3021	switch (enmHostMode)
3022	{
3023	case SUPPAGINGMODE_32_BIT:
3024	case SUPPAGINGMODE_32_BIT_GLOBAL:
3025	enmShadowMode = PGMMODE_32_BIT;
3026	enmSwitcher = VMMSWITCHER_32_TO_32;
3027	break;
3028
3029	case SUPPAGINGMODE_PAE:
3030	case SUPPAGINGMODE_PAE_NX:
3031	case SUPPAGINGMODE_PAE_GLOBAL:
3032	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3033	enmShadowMode = PGMMODE_PAE;
3034	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3035	#ifdef DEBUG_bird
3036	if (RTEnvExist("VBOX_32BIT"))
3037	{
3038	enmShadowMode = PGMMODE_32_BIT;
3039	enmSwitcher = VMMSWITCHER_PAE_TO_32;
3040	}
3041	#endif
3042	break;
3043
3044	case SUPPAGINGMODE_AMD64:
3045	case SUPPAGINGMODE_AMD64_GLOBAL:
3046	case SUPPAGINGMODE_AMD64_NX:
3047	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3048	enmShadowMode = PGMMODE_PAE;
3049	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3050	#ifdef DEBUG_bird
3051	if (RTEnvExist("VBOX_32BIT"))
3052	{
3053	enmShadowMode = PGMMODE_32_BIT;
3054	enmSwitcher = VMMSWITCHER_AMD64_TO_32;
3055	}
3056	#endif
3057	break;
3058
3059	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3060	}
3061	break;
3062
3063	case PGMMODE_32_BIT:
3064	switch (enmHostMode)
3065	{
3066	case SUPPAGINGMODE_32_BIT:
3067	case SUPPAGINGMODE_32_BIT_GLOBAL:
3068	enmShadowMode = PGMMODE_32_BIT;
3069	enmSwitcher = VMMSWITCHER_32_TO_32;
3070	break;
3071
3072	case SUPPAGINGMODE_PAE:
3073	case SUPPAGINGMODE_PAE_NX:
3074	case SUPPAGINGMODE_PAE_GLOBAL:
3075	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3076	enmShadowMode = PGMMODE_PAE;
3077	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3078	#ifdef DEBUG_bird
3079	if (RTEnvExist("VBOX_32BIT"))
3080	{
3081	enmShadowMode = PGMMODE_32_BIT;
3082	enmSwitcher = VMMSWITCHER_PAE_TO_32;
3083	}
3084	#endif
3085	break;
3086
3087	case SUPPAGINGMODE_AMD64:
3088	case SUPPAGINGMODE_AMD64_GLOBAL:
3089	case SUPPAGINGMODE_AMD64_NX:
3090	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3091	enmShadowMode = PGMMODE_PAE;
3092	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3093	#ifdef DEBUG_bird
3094	if (RTEnvExist("VBOX_32BIT"))
3095	{
3096	enmShadowMode = PGMMODE_32_BIT;
3097	enmSwitcher = VMMSWITCHER_AMD64_TO_32;
3098	}
3099	#endif
3100	break;
3101
3102	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3103	}
3104	break;
3105
3106	case PGMMODE_PAE:
3107	case PGMMODE_PAE_NX: /** @todo This might require more switchers and guest+both modes. */
3108	switch (enmHostMode)
3109	{
3110	case SUPPAGINGMODE_32_BIT:
3111	case SUPPAGINGMODE_32_BIT_GLOBAL:
3112	enmShadowMode = PGMMODE_PAE;
3113	enmSwitcher = VMMSWITCHER_32_TO_PAE;
3114	break;
3115
3116	case SUPPAGINGMODE_PAE:
3117	case SUPPAGINGMODE_PAE_NX:
3118	case SUPPAGINGMODE_PAE_GLOBAL:
3119	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3120	enmShadowMode = PGMMODE_PAE;
3121	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3122	break;
3123
3124	case SUPPAGINGMODE_AMD64:
3125	case SUPPAGINGMODE_AMD64_GLOBAL:
3126	case SUPPAGINGMODE_AMD64_NX:
3127	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3128	enmShadowMode = PGMMODE_PAE;
3129	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3130	break;
3131
3132	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3133	}
3134	break;
3135
3136	case PGMMODE_AMD64:
3137	case PGMMODE_AMD64_NX:
3138	switch (enmHostMode)
3139	{
3140	case SUPPAGINGMODE_32_BIT:
3141	case SUPPAGINGMODE_32_BIT_GLOBAL:
3142	enmShadowMode = PGMMODE_AMD64;
3143	enmSwitcher = VMMSWITCHER_32_TO_AMD64;
3144	break;
3145
3146	case SUPPAGINGMODE_PAE:
3147	case SUPPAGINGMODE_PAE_NX:
3148	case SUPPAGINGMODE_PAE_GLOBAL:
3149	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3150	enmShadowMode = PGMMODE_AMD64;
3151	enmSwitcher = VMMSWITCHER_PAE_TO_AMD64;
3152	break;
3153
3154	case SUPPAGINGMODE_AMD64:
3155	case SUPPAGINGMODE_AMD64_GLOBAL:
3156	case SUPPAGINGMODE_AMD64_NX:
3157	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3158	enmShadowMode = PGMMODE_AMD64;
3159	enmSwitcher = VMMSWITCHER_AMD64_TO_AMD64;
3160	break;
3161
3162	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3163	}
3164	break;
3165
3166
3167	default:
3168	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
3169	*penmSwitcher = VMMSWITCHER_INVALID;
3170	return PGMMODE_INVALID;
3171	}
3172	/* Override the shadow mode is nested paging is active. */
3173	pVM->pgm.s.fNestedPaging = HWACCMIsNestedPagingActive(pVM);
3174	if (pVM->pgm.s.fNestedPaging)
3175	enmShadowMode = HWACCMGetShwPagingMode(pVM);
3176
3177	*penmSwitcher = enmSwitcher;
3178	return enmShadowMode;
3179	}
3180
3181
3182	/**
3183	* Performs the actual mode change.
3184	* This is called by PGMChangeMode and pgmR3InitPaging().
3185	*
3186	* @returns VBox status code. May suspend or power off the VM on error, but this
3187	* will trigger using FFs and not status codes.
3188	*
3189	* @param pVM VM handle.
3190	* @param pVCpu The VMCPU to operate on.
3191	* @param enmGuestMode The new guest mode. This is assumed to be different from
3192	* the current mode.
3193	*/
3194	VMMR3DECL(int) PGMR3ChangeMode(PVM pVM, PVMCPU pVCpu, PGMMODE enmGuestMode)
3195	{
3196	bool fIsOldGuestPagingMode64Bits = (pVCpu->pgm.s.enmGuestMode >= PGMMODE_AMD64);
3197	bool fIsNewGuestPagingMode64Bits = (enmGuestMode >= PGMMODE_AMD64);
3198
3199	Log(("PGMR3ChangeMode: Guest mode: %s -> %s\n", PGMGetModeName(pVCpu->pgm.s.enmGuestMode), PGMGetModeName(enmGuestMode)));
3200	STAM_REL_COUNTER_INC(&pVCpu->pgm.s.cGuestModeChanges);
3201
3202	/*
3203	* Calc the shadow mode and switcher.
3204	*/
3205	VMMSWITCHER enmSwitcher;
3206	PGMMODE enmShadowMode = pgmR3CalcShadowMode(pVM, enmGuestMode, pVM->pgm.s.enmHostMode, pVCpu->pgm.s.enmShadowMode, &enmSwitcher);
3207
3208	#ifdef VBOX_WITH_RAW_MODE
3209	if (enmSwitcher != VMMSWITCHER_INVALID)
3210	{
3211	/*
3212	* Select new switcher.
3213	*/
3214	int rc = VMMR3SelectSwitcher(pVM, enmSwitcher);
3215	if (RT_FAILURE(rc))
3216	{
3217	AssertReleaseMsgFailed(("VMMR3SelectSwitcher(%d) -> %Rrc\n", enmSwitcher, rc));
3218	return rc;
3219	}
3220	}
3221	#endif
3222
3223	/*
3224	* Exit old mode(s).
3225	*/
3226	#if HC_ARCH_BITS == 32
3227	/* The nested shadow paging mode for AMD-V does change when running 64 bits guests on 32 bits hosts; typically PAE <-> AMD64 */
3228	const bool fForceShwEnterExit = ( fIsOldGuestPagingMode64Bits != fIsNewGuestPagingMode64Bits
3229	&& enmShadowMode == PGMMODE_NESTED);
3230	#else
3231	const bool fForceShwEnterExit = false;
3232	#endif
3233	/* shadow */
3234	if ( enmShadowMode != pVCpu->pgm.s.enmShadowMode
3235	\|\| fForceShwEnterExit)
3236	{
3237	LogFlow(("PGMR3ChangeMode: Shadow mode: %s -> %s\n", PGMGetModeName(pVCpu->pgm.s.enmShadowMode), PGMGetModeName(enmShadowMode)));
3238	if (PGM_SHW_PFN(Exit, pVCpu))
3239	{
3240	int rc = PGM_SHW_PFN(Exit, pVCpu)(pVCpu);
3241	if (RT_FAILURE(rc))
3242	{
3243	AssertMsgFailed(("Exit failed for shadow mode %d: %Rrc\n", pVCpu->pgm.s.enmShadowMode, rc));
3244	return rc;
3245	}
3246	}
3247
3248	}
3249	else
3250	LogFlow(("PGMR3ChangeMode: Shadow mode remains: %s\n", PGMGetModeName(pVCpu->pgm.s.enmShadowMode)));
3251
3252	/* guest */
3253	if (PGM_GST_PFN(Exit, pVCpu))
3254	{
3255	int rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
3256	if (RT_FAILURE(rc))
3257	{
3258	AssertMsgFailed(("Exit failed for guest mode %d: %Rrc\n", pVCpu->pgm.s.enmGuestMode, rc));
3259	return rc;
3260	}
3261	}
3262
3263	/*
3264	* Load new paging mode data.
3265	*/
3266	pgmR3ModeDataSwitch(pVM, pVCpu, enmShadowMode, enmGuestMode);
3267
3268	/*
3269	* Enter new shadow mode (if changed).
3270	*/
3271	if ( enmShadowMode != pVCpu->pgm.s.enmShadowMode
3272	\|\| fForceShwEnterExit)
3273	{
3274	int rc;
3275	pVCpu->pgm.s.enmShadowMode = enmShadowMode;
3276	switch (enmShadowMode)
3277	{
3278	case PGMMODE_32_BIT:
3279	rc = PGM_SHW_NAME_32BIT(Enter)(pVCpu, false);
3280	break;
3281	case PGMMODE_PAE:
3282	case PGMMODE_PAE_NX:
3283	rc = PGM_SHW_NAME_PAE(Enter)(pVCpu, false);
3284	break;
3285	case PGMMODE_AMD64:
3286	case PGMMODE_AMD64_NX:
3287	rc = PGM_SHW_NAME_AMD64(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3288	break;
3289	case PGMMODE_NESTED:
3290	rc = PGM_SHW_NAME_NESTED(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3291	break;
3292	case PGMMODE_EPT:
3293	rc = PGM_SHW_NAME_EPT(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3294	break;
3295	case PGMMODE_REAL:
3296	case PGMMODE_PROTECTED:
3297	default:
3298	AssertReleaseMsgFailed(("enmShadowMode=%d\n", enmShadowMode));
3299	return VERR_INTERNAL_ERROR;
3300	}
3301	if (RT_FAILURE(rc))
3302	{
3303	AssertReleaseMsgFailed(("Entering enmShadowMode=%d failed: %Rrc\n", enmShadowMode, rc));
3304	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
3305	return rc;
3306	}
3307	}
3308
3309	/*
3310	* Always flag the necessary updates
3311	*/
3312	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3313
3314	/*
3315	* Enter the new guest and shadow+guest modes.
3316	*/
3317	int rc = -1;
3318	int rc2 = -1;
3319	RTGCPHYS GCPhysCR3 = NIL_RTGCPHYS;
3320	pVCpu->pgm.s.enmGuestMode = enmGuestMode;
3321	switch (enmGuestMode)
3322	{
3323	case PGMMODE_REAL:
3324	rc = PGM_GST_NAME_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3325	switch (pVCpu->pgm.s.enmShadowMode)
3326	{
3327	case PGMMODE_32_BIT:
3328	rc2 = PGM_BTH_NAME_32BIT_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3329	break;
3330	case PGMMODE_PAE:
3331	case PGMMODE_PAE_NX:
3332	rc2 = PGM_BTH_NAME_PAE_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3333	break;
3334	case PGMMODE_NESTED:
3335	rc2 = PGM_BTH_NAME_NESTED_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3336	break;
3337	case PGMMODE_EPT:
3338	rc2 = PGM_BTH_NAME_EPT_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3339	break;
3340	case PGMMODE_AMD64:
3341	case PGMMODE_AMD64_NX:
3342	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3343	default: AssertFailed(); break;
3344	}
3345	break;
3346
3347	case PGMMODE_PROTECTED:
3348	rc = PGM_GST_NAME_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3349	switch (pVCpu->pgm.s.enmShadowMode)
3350	{
3351	case PGMMODE_32_BIT:
3352	rc2 = PGM_BTH_NAME_32BIT_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3353	break;
3354	case PGMMODE_PAE:
3355	case PGMMODE_PAE_NX:
3356	rc2 = PGM_BTH_NAME_PAE_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3357	break;
3358	case PGMMODE_NESTED:
3359	rc2 = PGM_BTH_NAME_NESTED_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3360	break;
3361	case PGMMODE_EPT:
3362	rc2 = PGM_BTH_NAME_EPT_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3363	break;
3364	case PGMMODE_AMD64:
3365	case PGMMODE_AMD64_NX:
3366	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3367	default: AssertFailed(); break;
3368	}
3369	break;
3370
3371	case PGMMODE_32_BIT:
3372	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & X86_CR3_PAGE_MASK;
3373	rc = PGM_GST_NAME_32BIT(Enter)(pVCpu, GCPhysCR3);
3374	switch (pVCpu->pgm.s.enmShadowMode)
3375	{
3376	case PGMMODE_32_BIT:
3377	rc2 = PGM_BTH_NAME_32BIT_32BIT(Enter)(pVCpu, GCPhysCR3);
3378	break;
3379	case PGMMODE_PAE:
3380	case PGMMODE_PAE_NX:
3381	rc2 = PGM_BTH_NAME_PAE_32BIT(Enter)(pVCpu, GCPhysCR3);
3382	break;
3383	case PGMMODE_NESTED:
3384	rc2 = PGM_BTH_NAME_NESTED_32BIT(Enter)(pVCpu, GCPhysCR3);
3385	break;
3386	case PGMMODE_EPT:
3387	rc2 = PGM_BTH_NAME_EPT_32BIT(Enter)(pVCpu, GCPhysCR3);
3388	break;
3389	case PGMMODE_AMD64:
3390	case PGMMODE_AMD64_NX:
3391	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3392	default: AssertFailed(); break;
3393	}
3394	break;
3395
3396	case PGMMODE_PAE_NX:
3397	case PGMMODE_PAE:
3398	{
3399	uint32_t u32Dummy, u32Features;
3400
3401	CPUMGetGuestCpuId(pVCpu, 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
3402	if (!(u32Features & X86_CPUID_FEATURE_EDX_PAE))
3403	return VMSetRuntimeError(pVM, VMSETRTERR_FLAGS_FATAL, "PAEmode",
3404	N_("The guest is trying to switch to the PAE mode which is currently disabled by default in VirtualBox. PAE support can be enabled using the VM settings (General/Advanced)"));
3405
3406	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & X86_CR3_PAE_PAGE_MASK;
3407	rc = PGM_GST_NAME_PAE(Enter)(pVCpu, GCPhysCR3);
3408	switch (pVCpu->pgm.s.enmShadowMode)
3409	{
3410	case PGMMODE_PAE:
3411	case PGMMODE_PAE_NX:
3412	rc2 = PGM_BTH_NAME_PAE_PAE(Enter)(pVCpu, GCPhysCR3);
3413	break;
3414	case PGMMODE_NESTED:
3415	rc2 = PGM_BTH_NAME_NESTED_PAE(Enter)(pVCpu, GCPhysCR3);
3416	break;
3417	case PGMMODE_EPT:
3418	rc2 = PGM_BTH_NAME_EPT_PAE(Enter)(pVCpu, GCPhysCR3);
3419	break;
3420	case PGMMODE_32_BIT:
3421	case PGMMODE_AMD64:
3422	case PGMMODE_AMD64_NX:
3423	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3424	default: AssertFailed(); break;
3425	}
3426	break;
3427	}
3428
3429	#ifdef VBOX_WITH_64_BITS_GUESTS
3430	case PGMMODE_AMD64_NX:
3431	case PGMMODE_AMD64:
3432	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & UINT64_C(0xfffffffffffff000); /** @todo define this mask! */
3433	rc = PGM_GST_NAME_AMD64(Enter)(pVCpu, GCPhysCR3);
3434	switch (pVCpu->pgm.s.enmShadowMode)
3435	{
3436	case PGMMODE_AMD64:
3437	case PGMMODE_AMD64_NX:
3438	rc2 = PGM_BTH_NAME_AMD64_AMD64(Enter)(pVCpu, GCPhysCR3);
3439	break;
3440	case PGMMODE_NESTED:
3441	rc2 = PGM_BTH_NAME_NESTED_AMD64(Enter)(pVCpu, GCPhysCR3);
3442	break;
3443	case PGMMODE_EPT:
3444	rc2 = PGM_BTH_NAME_EPT_AMD64(Enter)(pVCpu, GCPhysCR3);
3445	break;
3446	case PGMMODE_32_BIT:
3447	case PGMMODE_PAE:
3448	case PGMMODE_PAE_NX:
3449	AssertMsgFailed(("Should use AMD64 shadow mode!\n"));
3450	default: AssertFailed(); break;
3451	}
3452	break;
3453	#endif
3454
3455	default:
3456	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
3457	rc = VERR_NOT_IMPLEMENTED;
3458	break;
3459	}
3460
3461	/* status codes. */
3462	AssertRC(rc);
3463	AssertRC(rc2);
3464	if (RT_SUCCESS(rc))
3465	{
3466	rc = rc2;
3467	if (RT_SUCCESS(rc)) /* no informational status codes. */
3468	rc = VINF_SUCCESS;
3469	}
3470
3471	/* Notify HWACCM as well. */
3472	HWACCMR3PagingModeChanged(pVM, pVCpu, pVCpu->pgm.s.enmShadowMode, pVCpu->pgm.s.enmGuestMode);
3473	return rc;
3474	}
3475
3476
3477	/**
3478	* Called by pgmPoolFlushAllInt prior to flushing the pool.
3479	*
3480	* @returns VBox status code, fully asserted.
3481	* @param pVM The VM handle.
3482	* @param pVCpu The VMCPU to operate on.
3483	*/
3484	int pgmR3ExitShadowModeBeforePoolFlush(PVM pVM, PVMCPU pVCpu)
3485	{
3486	/* Unmap the old CR3 value before flushing everything. */
3487	int rc = PGM_BTH_PFN(UnmapCR3, pVCpu)(pVCpu);
3488	AssertRC(rc);
3489
3490	/* Exit the current shadow paging mode as well; nested paging and EPT use a root CR3 which will get flushed here. */
3491	rc = PGM_SHW_PFN(Exit, pVCpu)(pVCpu);
3492	AssertRC(rc);
3493	Assert(pVCpu->pgm.s.pShwPageCR3R3 == NULL);
3494	return rc;
3495	}
3496
3497
3498	/**
3499	* Called by pgmPoolFlushAllInt after flushing the pool.
3500	*
3501	* @returns VBox status code, fully asserted.
3502	* @param pVM The VM handle.
3503	* @param pVCpu The VMCPU to operate on.
3504	*/
3505	int pgmR3ReEnterShadowModeAfterPoolFlush(PVM pVM, PVMCPU pVCpu)
3506	{
3507	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
3508	int rc = PGMR3ChangeMode(pVM, pVCpu, PGMGetGuestMode(pVCpu));
3509	Assert(VMCPU_FF_ISSET(pVCpu, VMCPU_FF_PGM_SYNC_CR3));
3510	AssertRCReturn(rc, rc);
3511	AssertRCSuccessReturn(rc, VERR_IPE_UNEXPECTED_INFO_STATUS);
3512
3513	Assert(pVCpu->pgm.s.pShwPageCR3R3 != NULL);
3514	AssertMsg( pVCpu->pgm.s.enmShadowMode >= PGMMODE_NESTED
3515	\|\| CPUMGetHyperCR3(pVCpu) == PGMGetHyperCR3(pVCpu),
3516	("%RHp != %RHp %s\n", (RTHCPHYS)CPUMGetHyperCR3(pVCpu), PGMGetHyperCR3(pVCpu), PGMGetModeName(pVCpu->pgm.s.enmShadowMode)));
3517	return rc;
3518	}
3519
3520	#ifdef VBOX_WITH_DEBUGGER
3521
3522	/**
3523	* The '.pgmram' command.
3524	*
3525	* @returns VBox status.
3526	* @param pCmd Pointer to the command descriptor (as registered).
3527	* @param pCmdHlp Pointer to command helper functions.
3528	* @param pVM Pointer to the current VM (if any).
3529	* @param paArgs Pointer to (readonly) array of arguments.
3530	* @param cArgs Number of arguments in the array.
3531	*/
3532	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
3533	{
3534	/*
3535	* Validate input.
3536	*/
3537	if (!pVM)
3538	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3539	if (!pVM->pgm.s.pRamRangesRC)
3540	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Sorry, no Ram is registered.\n");
3541
3542	/*
3543	* Dump the ranges.
3544	*/
3545	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "From - To (incl) pvHC\n");
3546	PPGMRAMRANGE pRam;
3547	for (pRam = pVM->pgm.s.pRamRangesR3; pRam; pRam = pRam->pNextR3)
3548	{
3549	rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL,
3550	"%RGp - %RGp %p\n",
3551	pRam->GCPhys, pRam->GCPhysLast, pRam->pvR3);
3552	if (RT_FAILURE(rc))
3553	return rc;
3554	}
3555
3556	return VINF_SUCCESS;
3557	}
3558
3559
3560	/**
3561	* The '.pgmerror' and '.pgmerroroff' commands.
3562	*
3563	* @returns VBox status.
3564	* @param pCmd Pointer to the command descriptor (as registered).
3565	* @param pCmdHlp Pointer to command helper functions.
3566	* @param pVM Pointer to the current VM (if any).
3567	* @param paArgs Pointer to (readonly) array of arguments.
3568	* @param cArgs Number of arguments in the array.
3569	*/
3570	static DECLCALLBACK(int) pgmR3CmdError(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
3571	{
3572	/*
3573	* Validate input.
3574	*/
3575	if (!pVM)
3576	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3577	AssertReturn(cArgs == 0 \|\| (cArgs == 1 && paArgs[0].enmType == DBGCVAR_TYPE_STRING),
3578	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Hit bug in the parser.\n"));
3579
3580	if (!cArgs)
3581	{
3582	/*
3583	* Print the list of error injection locations with status.
3584	*/
3585	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "PGM error inject locations:\n");
3586	pCmdHlp->pfnPrintf(pCmdHlp, NULL, " handy - %RTbool\n", pVM->pgm.s.fErrInjHandyPages);
3587	}
3588	else
3589	{
3590
3591	/*
3592	* String switch on where to inject the error.
3593	*/
3594	bool const fNewState = !strcmp(pCmd->pszCmd, "pgmerror");
3595	const char *pszWhere = paArgs[0].u.pszString;
3596	if (!strcmp(pszWhere, "handy"))
3597	ASMAtomicWriteBool(&pVM->pgm.s.fErrInjHandyPages, fNewState);
3598	else
3599	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Invalid 'where' value: %s.\n", pszWhere);
3600	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "done\n");
3601	}
3602	return VINF_SUCCESS;
3603	}
3604
3605
3606	/**
3607	* The '.pgmsync' command.
3608	*
3609	* @returns VBox status.
3610	* @param pCmd Pointer to the command descriptor (as registered).
3611	* @param pCmdHlp Pointer to command helper functions.
3612	* @param pVM Pointer to the current VM (if any).
3613	* @param paArgs Pointer to (readonly) array of arguments.
3614	* @param cArgs Number of arguments in the array.
3615	*/
3616	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
3617	{
3618	/** @todo SMP support */
3619
3620	/*
3621	* Validate input.
3622	*/
3623	if (!pVM)
3624	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3625
3626	PVMCPU pVCpu = &pVM->aCpus[0];
3627
3628	/*
3629	* Force page directory sync.
3630	*/
3631	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3632
3633	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Forcing page directory sync.\n");
3634	if (RT_FAILURE(rc))
3635	return rc;
3636
3637	return VINF_SUCCESS;
3638	}
3639
3640
3641	#ifdef VBOX_STRICT
3642	/**
3643	* The '.pgmassertcr3' command.
3644	*
3645	* @returns VBox status.
3646	* @param pCmd Pointer to the command descriptor (as registered).
3647	* @param pCmdHlp Pointer to command helper functions.
3648	* @param pVM Pointer to the current VM (if any).
3649	* @param paArgs Pointer to (readonly) array of arguments.
3650	* @param cArgs Number of arguments in the array.
3651	*/
3652	static DECLCALLBACK(int) pgmR3CmdAssertCR3(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
3653	{
3654	/** @todo SMP support!! */
3655
3656	/*
3657	* Validate input.
3658	*/
3659	if (!pVM)
3660	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3661
3662	PVMCPU pVCpu = &pVM->aCpus[0];
3663
3664	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Checking shadow CR3 page tables for consistency.\n");
3665	if (RT_FAILURE(rc))
3666	return rc;
3667
3668	PGMAssertCR3(pVM, pVCpu, CPUMGetGuestCR3(pVCpu), CPUMGetGuestCR4(pVCpu));
3669
3670	return VINF_SUCCESS;
3671	}
3672	#endif /* VBOX_STRICT */
3673
3674
3675	/**
3676	* The '.pgmsyncalways' command.
3677	*
3678	* @returns VBox status.
3679	* @param pCmd Pointer to the command descriptor (as registered).
3680	* @param pCmdHlp Pointer to command helper functions.
3681	* @param pVM Pointer to the current VM (if any).
3682	* @param paArgs Pointer to (readonly) array of arguments.
3683	* @param cArgs Number of arguments in the array.
3684	*/
3685	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
3686	{
3687	/** @todo SMP support!! */
3688	PVMCPU pVCpu = &pVM->aCpus[0];
3689
3690	/*
3691	* Validate input.
3692	*/
3693	if (!pVM)
3694	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3695
3696	/*
3697	* Force page directory sync.
3698	*/
3699	if (pVCpu->pgm.s.fSyncFlags & PGM_SYNC_ALWAYS)
3700	{
3701	ASMAtomicAndU32(&pVCpu->pgm.s.fSyncFlags, ~PGM_SYNC_ALWAYS);
3702	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Disabled permanent forced page directory syncing.\n");
3703	}
3704	else
3705	{
3706	ASMAtomicOrU32(&pVCpu->pgm.s.fSyncFlags, PGM_SYNC_ALWAYS);
3707	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3708	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Enabled permanent forced page directory syncing.\n");
3709	}
3710	}
3711
3712
3713	/**
3714	* The '.pgmphystofile' command.
3715	*
3716	* @returns VBox status.
3717	* @param pCmd Pointer to the command descriptor (as registered).
3718	* @param pCmdHlp Pointer to command helper functions.
3719	* @param pVM Pointer to the current VM (if any).
3720	* @param paArgs Pointer to (readonly) array of arguments.
3721	* @param cArgs Number of arguments in the array.
3722	*/
3723	static DECLCALLBACK(int) pgmR3CmdPhysToFile(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs, PDBGCVAR pResult)
3724	{
3725	/*
3726	* Validate input.
3727	*/
3728	if (!pVM)
3729	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3730	if ( cArgs < 1
3731	\|\| cArgs > 2
3732	\|\| paArgs[0].enmType != DBGCVAR_TYPE_STRING
3733	\|\| ( cArgs > 1
3734	&& paArgs[1].enmType != DBGCVAR_TYPE_STRING))
3735	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: parser error, invalid arguments.\n");
3736	if ( cArgs >= 2
3737	&& strcmp(paArgs[1].u.pszString, "nozero"))
3738	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Invalid 2nd argument '%s', must be 'nozero'.\n", paArgs[1].u.pszString);
3739	bool fIncZeroPgs = cArgs < 2;
3740
3741	/*
3742	* Open the output file and get the ram parameters.
3743	*/
3744	RTFILE hFile;
3745	int rc = RTFileOpen(&hFile, paArgs[0].u.pszString, RTFILE_O_WRITE \| RTFILE_O_CREATE_REPLACE \| RTFILE_O_DENY_WRITE);
3746	if (RT_FAILURE(rc))
3747	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileOpen(,'%s',) -> %Rrc.\n", paArgs[0].u.pszString, rc);
3748
3749	uint32_t cbRamHole = 0;
3750	CFGMR3QueryU32Def(CFGMR3GetRoot(pVM), "RamHoleSize", &cbRamHole, MM_RAM_HOLE_SIZE_DEFAULT);
3751	uint64_t cbRam = 0;
3752	CFGMR3QueryU64Def(CFGMR3GetRoot(pVM), "RamSize", &cbRam, 0);
3753	RTGCPHYS GCPhysEnd = cbRam + cbRamHole;
3754
3755	/*
3756	* Dump the physical memory, page by page.
3757	*/
3758	RTGCPHYS GCPhys = 0;
3759	char abZeroPg[PAGE_SIZE];
3760	RT_ZERO(abZeroPg);
3761
3762	pgmLock(pVM);
3763	for (PPGMRAMRANGE pRam = pVM->pgm.s.pRamRangesR3;
3764	pRam && pRam->GCPhys < GCPhysEnd && RT_SUCCESS(rc);
3765	pRam = pRam->pNextR3)
3766	{
3767	/* fill the gap */
3768	if (pRam->GCPhys > GCPhys && fIncZeroPgs)
3769	{
3770	while (pRam->GCPhys > GCPhys && RT_SUCCESS(rc))
3771	{
3772	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
3773	GCPhys += PAGE_SIZE;
3774	}
3775	}
3776
3777	PCPGMPAGE pPage = &pRam->aPages[0];
3778	while (GCPhys < pRam->GCPhysLast && RT_SUCCESS(rc))
3779	{
3780	if ( PGM_PAGE_IS_ZERO(pPage)
3781	\|\| PGM_PAGE_IS_BALLOONED(pPage))
3782	{
3783	if (fIncZeroPgs)
3784	{
3785	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
3786	if (RT_FAILURE(rc))
3787	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3788	}
3789	}
3790	else
3791	{
3792	switch (PGM_PAGE_GET_TYPE(pPage))
3793	{
3794	case PGMPAGETYPE_RAM:
3795	case PGMPAGETYPE_ROM_SHADOW: /* trouble?? */
3796	case PGMPAGETYPE_ROM:
3797	case PGMPAGETYPE_MMIO2:
3798	{
3799	void const *pvPage;
3800	PGMPAGEMAPLOCK Lock;
3801	rc = PGMPhysGCPhys2CCPtrReadOnly(pVM, GCPhys, &pvPage, &Lock);
3802	if (RT_SUCCESS(rc))
3803	{
3804	rc = RTFileWrite(hFile, pvPage, PAGE_SIZE, NULL);
3805	PGMPhysReleasePageMappingLock(pVM, &Lock);
3806	if (RT_FAILURE(rc))
3807	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3808	}
3809	else
3810	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: PGMPhysGCPhys2CCPtrReadOnly -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3811	break;
3812	}
3813
3814	default:
3815	AssertFailed();
3816	case PGMPAGETYPE_MMIO2_ALIAS_MMIO:
3817	case PGMPAGETYPE_MMIO:
3818	if (fIncZeroPgs)
3819	{
3820	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
3821	if (RT_FAILURE(rc))
3822	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3823	}
3824	break;
3825	}
3826	}
3827
3828
3829	/* advance */
3830	GCPhys += PAGE_SIZE;
3831	pPage++;
3832	}
3833	}
3834	pgmUnlock(pVM);
3835
3836	RTFileClose(hFile);
3837	if (RT_SUCCESS(rc))
3838	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Successfully saved physical memory to '%s'.\n", paArgs[0].u.pszString);
3839	return VINF_SUCCESS;
3840	}
3841
3842	#endif /* VBOX_WITH_DEBUGGER */
3843
3844	/**
3845	* pvUser argument of the pgmR3CheckIntegrity*Node callbacks.
3846	*/
3847	typedef struct PGMCHECKINTARGS
3848	{
3849	bool fLeftToRight; /*< true: left-to-right; false: right-to-left. /
3850	PPGMPHYSHANDLER pPrevPhys;
3851	PPGMVIRTHANDLER pPrevVirt;
3852	PPGMPHYS2VIRTHANDLER pPrevPhys2Virt;
3853	PVM pVM;
3854	} PGMCHECKINTARGS, *PPGMCHECKINTARGS;
3855
3856	/**
3857	* Validate a node in the physical handler tree.
3858	*
3859	* @returns 0 on if ok, other wise 1.
3860	* @param pNode The handler node.
3861	* @param pvUser pVM.
3862	*/
3863	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
3864	{
3865	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3866	PPGMPHYSHANDLER pCur = (PPGMPHYSHANDLER)pNode;
3867	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
3868	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGp-%RGp %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3869	AssertReleaseMsg( !pArgs->pPrevPhys
3870	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys->Core.KeyLast > pCur->Core.Key),
3871	("pPrevPhys=%p %RGp-%RGp %s\n"
3872	" pCur=%p %RGp-%RGp %s\n",
3873	pArgs->pPrevPhys, pArgs->pPrevPhys->Core.Key, pArgs->pPrevPhys->Core.KeyLast, pArgs->pPrevPhys->pszDesc,
3874	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3875	pArgs->pPrevPhys = pCur;
3876	return 0;
3877	}
3878
3879
3880	/**
3881	* Validate a node in the virtual handler tree.
3882	*
3883	* @returns 0 on if ok, other wise 1.
3884	* @param pNode The handler node.
3885	* @param pvUser pVM.
3886	*/
3887	static DECLCALLBACK(int) pgmR3CheckIntegrityVirtHandlerNode(PAVLROGCPTRNODECORE pNode, void *pvUser)
3888	{
3889	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3890	PPGMVIRTHANDLER pCur = (PPGMVIRTHANDLER)pNode;
3891	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
3892	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGv-%RGv %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3893	AssertReleaseMsg( !pArgs->pPrevVirt
3894	\|\| (pArgs->fLeftToRight ? pArgs->pPrevVirt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevVirt->Core.KeyLast > pCur->Core.Key),
3895	("pPrevVirt=%p %RGv-%RGv %s\n"
3896	" pCur=%p %RGv-%RGv %s\n",
3897	pArgs->pPrevVirt, pArgs->pPrevVirt->Core.Key, pArgs->pPrevVirt->Core.KeyLast, pArgs->pPrevVirt->pszDesc,
3898	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3899	for (unsigned iPage = 0; iPage < pCur->cPages; iPage++)
3900	{
3901	AssertReleaseMsg(pCur->aPhysToVirt[iPage].offVirtHandler == -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage]),
3902	("pCur=%p %RGv-%RGv %s\n"
3903	"iPage=%d offVirtHandle=%#x expected %#x\n",
3904	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc,
3905	iPage, pCur->aPhysToVirt[iPage].offVirtHandler, -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage])));
3906	}
3907	pArgs->pPrevVirt = pCur;
3908	return 0;
3909	}
3910
3911
3912	/**
3913	* Validate a node in the virtual handler tree.
3914	*
3915	* @returns 0 on if ok, other wise 1.
3916	* @param pNode The handler node.
3917	* @param pvUser pVM.
3918	*/
3919	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysToVirtHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
3920	{
3921	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3922	PPGMPHYS2VIRTHANDLER pCur = (PPGMPHYS2VIRTHANDLER)pNode;
3923	AssertReleaseMsgReturn(!((uintptr_t)pCur & 3), ("\n"), 1);
3924	AssertReleaseMsgReturn(!(pCur->offVirtHandler & 3), ("\n"), 1);
3925	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGp-%RGp\n", pCur, pCur->Core.Key, pCur->Core.KeyLast));
3926	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
3927	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
3928	("pPrevPhys2Virt=%p %RGp-%RGp\n"
3929	" pCur=%p %RGp-%RGp\n",
3930	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
3931	pCur, pCur->Core.Key, pCur->Core.KeyLast));
3932	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
3933	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
3934	("pPrevPhys2Virt=%p %RGp-%RGp\n"
3935	" pCur=%p %RGp-%RGp\n",
3936	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
3937	pCur, pCur->Core.Key, pCur->Core.KeyLast));
3938	AssertReleaseMsg((pCur->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD),
3939	("pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
3940	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
3941	if (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK)
3942	{
3943	PPGMPHYS2VIRTHANDLER pCur2 = pCur;
3944	for (;;)
3945	{
3946	pCur2 = (PPGMPHYS2VIRTHANDLER)((intptr_t)pCur + (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK));
3947	AssertReleaseMsg(pCur2 != pCur,
3948	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
3949	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
3950	AssertReleaseMsg((pCur2->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == PGMPHYS2VIRTHANDLER_IN_TREE,
3951	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
3952	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
3953	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
3954	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
3955	AssertReleaseMsg((pCur2->Core.Key ^ pCur->Core.Key) < PAGE_SIZE,
3956	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
3957	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
3958	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
3959	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
3960	AssertReleaseMsg((pCur2->Core.KeyLast ^ pCur->Core.KeyLast) < PAGE_SIZE,
3961	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
3962	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
3963	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
3964	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
3965	if (!(pCur2->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK))
3966	break;
3967	}
3968	}
3969
3970	pArgs->pPrevPhys2Virt = pCur;
3971	return 0;
3972	}
3973
3974
3975	/**
3976	* Perform an integrity check on the PGM component.
3977	*
3978	* @returns VINF_SUCCESS if everything is fine.
3979	* @returns VBox error status after asserting on integrity breach.
3980	* @param pVM The VM handle.
3981	*/
3982	VMMR3DECL(int) PGMR3CheckIntegrity(PVM pVM)
3983	{
3984	AssertReleaseReturn(pVM->pgm.s.offVM, VERR_INTERNAL_ERROR);
3985
3986	/*
3987	* Check the trees.
3988	*/
3989	int cErrors = 0;
3990	const static PGMCHECKINTARGS s_LeftToRight = { true, NULL, NULL, NULL, pVM };
3991	const static PGMCHECKINTARGS s_RightToLeft = { false, NULL, NULL, NULL, pVM };
3992	PGMCHECKINTARGS Args = s_LeftToRight;
3993	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, true, pgmR3CheckIntegrityPhysHandlerNode, &Args);
3994	Args = s_RightToLeft;
3995	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, false, pgmR3CheckIntegrityPhysHandlerNode, &Args);
3996	Args = s_LeftToRight;
3997	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->VirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
3998	Args = s_RightToLeft;
3999	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->VirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4000	Args = s_LeftToRight;
4001	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->HyperVirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4002	Args = s_RightToLeft;
4003	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->HyperVirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4004	Args = s_LeftToRight;
4005	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysToVirtHandlers, true, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
4006	Args = s_RightToLeft;
4007	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysToVirtHandlers, false, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
4008
4009	return !cErrors ? VINF_SUCCESS : VERR_INTERNAL_ERROR;
4010	}
4011

Note: See TracBrowser for help on using the repository browser.

source: vbox/trunk/src/VBox/VMM/PGM.cpp@ 34942

Download in other formats: