VirtualBox

source: vbox/trunk/src/VBox/VMM/VMMR0/GMMR0.cpp@ 41801

Last change on this file since 41801 was 41776, checked in by vboxsync, 13 years ago

doxygen comments.

  • Property svn:eol-style set to native
  • Property svn:keywords set to Id
File size: 186.3 KB
Line 
1/* $Id: GMMR0.cpp 41776 2012-06-16 18:36:56Z vboxsync $ */
2/** @file
3 * GMM - Global Memory Manager.
4 */
5
6/*
7 * Copyright (C) 2007-2012 Oracle Corporation
8 *
9 * This file is part of VirtualBox Open Source Edition (OSE), as
10 * available from http://www.virtualbox.org. This file is free software;
11 * you can redistribute it and/or modify it under the terms of the GNU
12 * General Public License (GPL) as published by the Free Software
13 * Foundation, in version 2 as it comes in the "COPYING" file of the
14 * VirtualBox OSE distribution. VirtualBox OSE is distributed in the
15 * hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
16 */
17
18
19/** @page pg_gmm GMM - The Global Memory Manager
20 *
21 * As the name indicates, this component is responsible for global memory
22 * management. Currently only guest RAM is allocated from the GMM, but this
23 * may change to include shadow page tables and other bits later.
24 *
25 * Guest RAM is managed as individual pages, but allocated from the host OS
26 * in chunks for reasons of portability / efficiency. To minimize the memory
27 * footprint all tracking structure must be as small as possible without
28 * unnecessary performance penalties.
29 *
30 * The allocation chunks has fixed sized, the size defined at compile time
31 * by the #GMM_CHUNK_SIZE \#define.
32 *
33 * Each chunk is given an unique ID. Each page also has a unique ID. The
34 * relation ship between the two IDs is:
35 * @code
36 * GMM_CHUNK_SHIFT = log2(GMM_CHUNK_SIZE / PAGE_SIZE);
37 * idPage = (idChunk << GMM_CHUNK_SHIFT) | iPage;
38 * @endcode
39 * Where iPage is the index of the page within the chunk. This ID scheme
40 * permits for efficient chunk and page lookup, but it relies on the chunk size
41 * to be set at compile time. The chunks are organized in an AVL tree with their
42 * IDs being the keys.
43 *
44 * The physical address of each page in an allocation chunk is maintained by
45 * the #RTR0MEMOBJ and obtained using #RTR0MemObjGetPagePhysAddr. There is no
46 * need to duplicate this information (it'll cost 8-bytes per page if we did).
47 *
48 * So what do we need to track per page? Most importantly we need to know
49 * which state the page is in:
50 * - Private - Allocated for (eventually) backing one particular VM page.
51 * - Shared - Readonly page that is used by one or more VMs and treated
52 * as COW by PGM.
53 * - Free - Not used by anyone.
54 *
55 * For the page replacement operations (sharing, defragmenting and freeing)
56 * to be somewhat efficient, private pages needs to be associated with a
57 * particular page in a particular VM.
58 *
59 * Tracking the usage of shared pages is impractical and expensive, so we'll
60 * settle for a reference counting system instead.
61 *
62 * Free pages will be chained on LIFOs
63 *
64 * On 64-bit systems we will use a 64-bit bitfield per page, while on 32-bit
65 * systems a 32-bit bitfield will have to suffice because of address space
66 * limitations. The #GMMPAGE structure shows the details.
67 *
68 *
69 * @section sec_gmm_alloc_strat Page Allocation Strategy
70 *
71 * The strategy for allocating pages has to take fragmentation and shared
72 * pages into account, or we may end up with with 2000 chunks with only
73 * a few pages in each. Shared pages cannot easily be reallocated because
74 * of the inaccurate usage accounting (see above). Private pages can be
75 * reallocated by a defragmentation thread in the same manner that sharing
76 * is done.
77 *
78 * The first approach is to manage the free pages in two sets depending on
79 * whether they are mainly for the allocation of shared or private pages.
80 * In the initial implementation there will be almost no possibility for
81 * mixing shared and private pages in the same chunk (only if we're really
82 * stressed on memory), but when we implement forking of VMs and have to
83 * deal with lots of COW pages it'll start getting kind of interesting.
84 *
85 * The sets are lists of chunks with approximately the same number of
86 * free pages. Say the chunk size is 1MB, meaning 256 pages, and a set
87 * consists of 16 lists. So, the first list will contain the chunks with
88 * 1-7 free pages, the second covers 8-15, and so on. The chunks will be
89 * moved between the lists as pages are freed up or allocated.
90 *
91 *
92 * @section sec_gmm_costs Costs
93 *
94 * The per page cost in kernel space is 32-bit plus whatever RTR0MEMOBJ
95 * entails. In addition there is the chunk cost of approximately
96 * (sizeof(RT0MEMOBJ) + sizeof(CHUNK)) / 2^CHUNK_SHIFT bytes per page.
97 *
98 * On Windows the per page #RTR0MEMOBJ cost is 32-bit on 32-bit windows
99 * and 64-bit on 64-bit windows (a PFN_NUMBER in the MDL). So, 64-bit per page.
100 * The cost on Linux is identical, but here it's because of sizeof(struct page *).
101 *
102 *
103 * @section sec_gmm_legacy Legacy Mode for Non-Tier-1 Platforms
104 *
105 * In legacy mode the page source is locked user pages and not
106 * #RTR0MemObjAllocPhysNC, this means that a page can only be allocated
107 * by the VM that locked it. We will make no attempt at implementing
108 * page sharing on these systems, just do enough to make it all work.
109 *
110 *
111 * @subsection sub_gmm_locking Serializing
112 *
113 * One simple fast mutex will be employed in the initial implementation, not
114 * two as mentioned in @ref subsec_pgmPhys_Serializing.
115 *
116 * @see @ref subsec_pgmPhys_Serializing
117 *
118 *
119 * @section sec_gmm_overcommit Memory Over-Commitment Management
120 *
121 * The GVM will have to do the system wide memory over-commitment
122 * management. My current ideas are:
123 * - Per VM oc policy that indicates how much to initially commit
124 * to it and what to do in a out-of-memory situation.
125 * - Prevent overtaxing the host.
126 *
127 * There are some challenges here, the main ones are configurability and
128 * security. Should we for instance permit anyone to request 100% memory
129 * commitment? Who should be allowed to do runtime adjustments of the
130 * config. And how to prevent these settings from being lost when the last
131 * VM process exits? The solution is probably to have an optional root
132 * daemon the will keep VMMR0.r0 in memory and enable the security measures.
133 *
134 *
135 *
136 * @section sec_gmm_numa NUMA
137 *
138 * NUMA considerations will be designed and implemented a bit later.
139 *
140 * The preliminary guesses is that we will have to try allocate memory as
141 * close as possible to the CPUs the VM is executed on (EMT and additional CPU
142 * threads). Which means it's mostly about allocation and sharing policies.
143 * Both the scheduler and allocator interface will to supply some NUMA info
144 * and we'll need to have a way to calc access costs.
145 *
146 */
147
148
149/*******************************************************************************
150* Header Files *
151*******************************************************************************/
152#define LOG_GROUP LOG_GROUP_GMM
153#include <VBox/rawpci.h>
154#include <VBox/vmm/vm.h>
155#include <VBox/vmm/gmm.h>
156#include "GMMR0Internal.h"
157#include <VBox/vmm/gvm.h>
158#include <VBox/vmm/pgm.h>
159#include <VBox/log.h>
160#include <VBox/param.h>
161#include <VBox/err.h>
162#include <iprt/asm.h>
163#include <iprt/avl.h>
164#ifdef VBOX_STRICT
165# include <iprt/crc.h>
166#endif
167#include <iprt/list.h>
168#include <iprt/mem.h>
169#include <iprt/memobj.h>
170#include <iprt/mp.h>
171#include <iprt/semaphore.h>
172#include <iprt/string.h>
173#include <iprt/time.h>
174
175
176/*******************************************************************************
177* Structures and Typedefs *
178*******************************************************************************/
179/** Pointer to set of free chunks. */
180typedef struct GMMCHUNKFREESET *PGMMCHUNKFREESET;
181
182/**
183 * The per-page tracking structure employed by the GMM.
184 *
185 * On 32-bit hosts we'll some trickery is necessary to compress all
186 * the information into 32-bits. When the fSharedFree member is set,
187 * the 30th bit decides whether it's a free page or not.
188 *
189 * Because of the different layout on 32-bit and 64-bit hosts, macros
190 * are used to get and set some of the data.
191 */
192typedef union GMMPAGE
193{
194#if HC_ARCH_BITS == 64
195 /** Unsigned integer view. */
196 uint64_t u;
197
198 /** The common view. */
199 struct GMMPAGECOMMON
200 {
201 uint32_t uStuff1 : 32;
202 uint32_t uStuff2 : 30;
203 /** The page state. */
204 uint32_t u2State : 2;
205 } Common;
206
207 /** The view of a private page. */
208 struct GMMPAGEPRIVATE
209 {
210 /** The guest page frame number. (Max addressable: 2 ^ 44 - 16) */
211 uint32_t pfn;
212 /** The GVM handle. (64K VMs) */
213 uint32_t hGVM : 16;
214 /** Reserved. */
215 uint32_t u16Reserved : 14;
216 /** The page state. */
217 uint32_t u2State : 2;
218 } Private;
219
220 /** The view of a shared page. */
221 struct GMMPAGESHARED
222 {
223 /** The host page frame number. (Max addressable: 2 ^ 44 - 16) */
224 uint32_t pfn;
225 /** The reference count (64K VMs). */
226 uint32_t cRefs : 16;
227 /** Used for debug checksumming. */
228 uint32_t u14Checksum : 14;
229 /** The page state. */
230 uint32_t u2State : 2;
231 } Shared;
232
233 /** The view of a free page. */
234 struct GMMPAGEFREE
235 {
236 /** The index of the next page in the free list. UINT16_MAX is NIL. */
237 uint16_t iNext;
238 /** Reserved. Checksum or something? */
239 uint16_t u16Reserved0;
240 /** Reserved. Checksum or something? */
241 uint32_t u30Reserved1 : 30;
242 /** The page state. */
243 uint32_t u2State : 2;
244 } Free;
245
246#else /* 32-bit */
247 /** Unsigned integer view. */
248 uint32_t u;
249
250 /** The common view. */
251 struct GMMPAGECOMMON
252 {
253 uint32_t uStuff : 30;
254 /** The page state. */
255 uint32_t u2State : 2;
256 } Common;
257
258 /** The view of a private page. */
259 struct GMMPAGEPRIVATE
260 {
261 /** The guest page frame number. (Max addressable: 2 ^ 36) */
262 uint32_t pfn : 24;
263 /** The GVM handle. (127 VMs) */
264 uint32_t hGVM : 7;
265 /** The top page state bit, MBZ. */
266 uint32_t fZero : 1;
267 } Private;
268
269 /** The view of a shared page. */
270 struct GMMPAGESHARED
271 {
272 /** The reference count. */
273 uint32_t cRefs : 30;
274 /** The page state. */
275 uint32_t u2State : 2;
276 } Shared;
277
278 /** The view of a free page. */
279 struct GMMPAGEFREE
280 {
281 /** The index of the next page in the free list. UINT16_MAX is NIL. */
282 uint32_t iNext : 16;
283 /** Reserved. Checksum or something? */
284 uint32_t u14Reserved : 14;
285 /** The page state. */
286 uint32_t u2State : 2;
287 } Free;
288#endif
289} GMMPAGE;
290AssertCompileSize(GMMPAGE, sizeof(RTHCUINTPTR));
291/** Pointer to a GMMPAGE. */
292typedef GMMPAGE *PGMMPAGE;
293
294
295/** @name The Page States.
296 * @{ */
297/** A private page. */
298#define GMM_PAGE_STATE_PRIVATE 0
299/** A private page - alternative value used on the 32-bit implementation.
300 * This will never be used on 64-bit hosts. */
301#define GMM_PAGE_STATE_PRIVATE_32 1
302/** A shared page. */
303#define GMM_PAGE_STATE_SHARED 2
304/** A free page. */
305#define GMM_PAGE_STATE_FREE 3
306/** @} */
307
308
309/** @def GMM_PAGE_IS_PRIVATE
310 *
311 * @returns true if private, false if not.
312 * @param pPage The GMM page.
313 */
314#if HC_ARCH_BITS == 64
315# define GMM_PAGE_IS_PRIVATE(pPage) ( (pPage)->Common.u2State == GMM_PAGE_STATE_PRIVATE )
316#else
317# define GMM_PAGE_IS_PRIVATE(pPage) ( (pPage)->Private.fZero == 0 )
318#endif
319
320/** @def GMM_PAGE_IS_SHARED
321 *
322 * @returns true if shared, false if not.
323 * @param pPage The GMM page.
324 */
325#define GMM_PAGE_IS_SHARED(pPage) ( (pPage)->Common.u2State == GMM_PAGE_STATE_SHARED )
326
327/** @def GMM_PAGE_IS_FREE
328 *
329 * @returns true if free, false if not.
330 * @param pPage The GMM page.
331 */
332#define GMM_PAGE_IS_FREE(pPage) ( (pPage)->Common.u2State == GMM_PAGE_STATE_FREE )
333
334/** @def GMM_PAGE_PFN_LAST
335 * The last valid guest pfn range.
336 * @remark Some of the values outside the range has special meaning,
337 * see GMM_PAGE_PFN_UNSHAREABLE.
338 */
339#if HC_ARCH_BITS == 64
340# define GMM_PAGE_PFN_LAST UINT32_C(0xfffffff0)
341#else
342# define GMM_PAGE_PFN_LAST UINT32_C(0x00fffff0)
343#endif
344AssertCompile(GMM_PAGE_PFN_LAST == (GMM_GCPHYS_LAST >> PAGE_SHIFT));
345
346/** @def GMM_PAGE_PFN_UNSHAREABLE
347 * Indicates that this page isn't used for normal guest memory and thus isn't shareable.
348 */
349#if HC_ARCH_BITS == 64
350# define GMM_PAGE_PFN_UNSHAREABLE UINT32_C(0xfffffff1)
351#else
352# define GMM_PAGE_PFN_UNSHAREABLE UINT32_C(0x00fffff1)
353#endif
354AssertCompile(GMM_PAGE_PFN_UNSHAREABLE == (GMM_GCPHYS_UNSHAREABLE >> PAGE_SHIFT));
355
356
357/**
358 * A GMM allocation chunk ring-3 mapping record.
359 *
360 * This should really be associated with a session and not a VM, but
361 * it's simpler to associated with a VM and cleanup with the VM object
362 * is destroyed.
363 */
364typedef struct GMMCHUNKMAP
365{
366 /** The mapping object. */
367 RTR0MEMOBJ hMapObj;
368 /** The VM owning the mapping. */
369 PGVM pGVM;
370} GMMCHUNKMAP;
371/** Pointer to a GMM allocation chunk mapping. */
372typedef struct GMMCHUNKMAP *PGMMCHUNKMAP;
373
374
375/**
376 * A GMM allocation chunk.
377 */
378typedef struct GMMCHUNK
379{
380 /** The AVL node core.
381 * The Key is the chunk ID. (Giant mtx.) */
382 AVLU32NODECORE Core;
383 /** The memory object.
384 * Either from RTR0MemObjAllocPhysNC or RTR0MemObjLockUser depending on
385 * what the host can dish up with. (Chunk mtx protects mapping accesses
386 * and related frees.) */
387 RTR0MEMOBJ hMemObj;
388 /** Pointer to the next chunk in the free list. (Giant mtx.) */
389 PGMMCHUNK pFreeNext;
390 /** Pointer to the previous chunk in the free list. (Giant mtx.) */
391 PGMMCHUNK pFreePrev;
392 /** Pointer to the free set this chunk belongs to. NULL for
393 * chunks with no free pages. (Giant mtx.) */
394 PGMMCHUNKFREESET pSet;
395 /** List node in the chunk list (GMM::ChunkList). (Giant mtx.) */
396 RTLISTNODE ListNode;
397 /** Pointer to an array of mappings. (Chunk mtx.) */
398 PGMMCHUNKMAP paMappingsX;
399 /** The number of mappings. (Chunk mtx.) */
400 uint16_t cMappingsX;
401 /** The mapping lock this chunk is using using. UINT16_MAX if nobody is
402 * mapping or freeing anything. (Giant mtx.) */
403 uint8_t volatile iChunkMtx;
404 /** Flags field reserved for future use (like eliminating enmType).
405 * (Giant mtx.) */
406 uint8_t fFlags;
407 /** The head of the list of free pages. UINT16_MAX is the NIL value.
408 * (Giant mtx.) */
409 uint16_t iFreeHead;
410 /** The number of free pages. (Giant mtx.) */
411 uint16_t cFree;
412 /** The GVM handle of the VM that first allocated pages from this chunk, this
413 * is used as a preference when there are several chunks to choose from.
414 * When in bound memory mode this isn't a preference any longer. (Giant
415 * mtx.) */
416 uint16_t hGVM;
417 /** The ID of the NUMA node the memory mostly resides on. (Reserved for
418 * future use.) (Giant mtx.) */
419 uint16_t idNumaNode;
420 /** The number of private pages. (Giant mtx.) */
421 uint16_t cPrivate;
422 /** The number of shared pages. (Giant mtx.) */
423 uint16_t cShared;
424 /** The pages. (Giant mtx.) */
425 GMMPAGE aPages[GMM_CHUNK_SIZE >> PAGE_SHIFT];
426} GMMCHUNK;
427
428/** Indicates that the NUMA properies of the memory is unknown. */
429#define GMM_CHUNK_NUMA_ID_UNKNOWN UINT16_C(0xfffe)
430
431/** @name GMM_CHUNK_FLAGS_XXX - chunk flags.
432 * @{ */
433/** Indicates that the chunk is a large page (2MB). */
434#define GMM_CHUNK_FLAGS_LARGE_PAGE UINT16_C(0x0001)
435/** @} */
436
437
438/**
439 * An allocation chunk TLB entry.
440 */
441typedef struct GMMCHUNKTLBE
442{
443 /** The chunk id. */
444 uint32_t idChunk;
445 /** Pointer to the chunk. */
446 PGMMCHUNK pChunk;
447} GMMCHUNKTLBE;
448/** Pointer to an allocation chunk TLB entry. */
449typedef GMMCHUNKTLBE *PGMMCHUNKTLBE;
450
451
452/** The number of entries tin the allocation chunk TLB. */
453#define GMM_CHUNKTLB_ENTRIES 32
454/** Gets the TLB entry index for the given Chunk ID. */
455#define GMM_CHUNKTLB_IDX(idChunk) ( (idChunk) & (GMM_CHUNKTLB_ENTRIES - 1) )
456
457/**
458 * An allocation chunk TLB.
459 */
460typedef struct GMMCHUNKTLB
461{
462 /** The TLB entries. */
463 GMMCHUNKTLBE aEntries[GMM_CHUNKTLB_ENTRIES];
464} GMMCHUNKTLB;
465/** Pointer to an allocation chunk TLB. */
466typedef GMMCHUNKTLB *PGMMCHUNKTLB;
467
468
469/**
470 * The GMM instance data.
471 */
472typedef struct GMM
473{
474 /** Magic / eye catcher. GMM_MAGIC */
475 uint32_t u32Magic;
476 /** The number of threads waiting on the mutex. */
477 uint32_t cMtxContenders;
478 /** The fast mutex protecting the GMM.
479 * More fine grained locking can be implemented later if necessary. */
480 RTSEMFASTMUTEX hMtx;
481#ifdef VBOX_STRICT
482 /** The current mutex owner. */
483 RTNATIVETHREAD hMtxOwner;
484#endif
485 /** The chunk tree. */
486 PAVLU32NODECORE pChunks;
487 /** The chunk TLB. */
488 GMMCHUNKTLB ChunkTLB;
489 /** The private free set. */
490 GMMCHUNKFREESET PrivateX;
491 /** The shared free set. */
492 GMMCHUNKFREESET Shared;
493
494 /** Shared module tree (global).
495 * @todo separate trees for distinctly different guest OSes. */
496 PAVLLU32NODECORE pGlobalSharedModuleTree;
497 /** Sharable modules (count of nodes in pGlobalSharedModuleTree). */
498 uint32_t cShareableModules;
499
500 /** The chunk list. For simplifying the cleanup process. */
501 RTLISTANCHOR ChunkList;
502
503 /** The maximum number of pages we're allowed to allocate.
504 * @gcfgm 64-bit GMM/MaxPages Direct.
505 * @gcfgm 32-bit GMM/PctPages Relative to the number of host pages. */
506 uint64_t cMaxPages;
507 /** The number of pages that has been reserved.
508 * The deal is that cReservedPages - cOverCommittedPages <= cMaxPages. */
509 uint64_t cReservedPages;
510 /** The number of pages that we have over-committed in reservations. */
511 uint64_t cOverCommittedPages;
512 /** The number of actually allocated (committed if you like) pages. */
513 uint64_t cAllocatedPages;
514 /** The number of pages that are shared. A subset of cAllocatedPages. */
515 uint64_t cSharedPages;
516 /** The number of pages that are actually shared between VMs. */
517 uint64_t cDuplicatePages;
518 /** The number of pages that are shared that has been left behind by
519 * VMs not doing proper cleanups. */
520 uint64_t cLeftBehindSharedPages;
521 /** The number of allocation chunks.
522 * (The number of pages we've allocated from the host can be derived from this.) */
523 uint32_t cChunks;
524 /** The number of current ballooned pages. */
525 uint64_t cBalloonedPages;
526
527 /** The legacy allocation mode indicator.
528 * This is determined at initialization time. */
529 bool fLegacyAllocationMode;
530 /** The bound memory mode indicator.
531 * When set, the memory will be bound to a specific VM and never
532 * shared. This is always set if fLegacyAllocationMode is set.
533 * (Also determined at initialization time.) */
534 bool fBoundMemoryMode;
535 /** The number of registered VMs. */
536 uint16_t cRegisteredVMs;
537
538 /** The number of freed chunks ever. This is used a list generation to
539 * avoid restarting the cleanup scanning when the list wasn't modified. */
540 uint32_t cFreedChunks;
541 /** The previous allocated Chunk ID.
542 * Used as a hint to avoid scanning the whole bitmap. */
543 uint32_t idChunkPrev;
544 /** Chunk ID allocation bitmap.
545 * Bits of allocated IDs are set, free ones are clear.
546 * The NIL id (0) is marked allocated. */
547 uint32_t bmChunkId[(GMM_CHUNKID_LAST + 1 + 31) / 32];
548
549 /** The index of the next mutex to use. */
550 uint32_t iNextChunkMtx;
551 /** Chunk locks for reducing lock contention without having to allocate
552 * one lock per chunk. */
553 struct
554 {
555 /** The mutex */
556 RTSEMFASTMUTEX hMtx;
557 /** The number of threads currently using this mutex. */
558 uint32_t volatile cUsers;
559 } aChunkMtx[64];
560} GMM;
561/** Pointer to the GMM instance. */
562typedef GMM *PGMM;
563
564/** The value of GMM::u32Magic (Katsuhiro Otomo). */
565#define GMM_MAGIC UINT32_C(0x19540414)
566
567
568/**
569 * GMM chunk mutex state.
570 *
571 * This is returned by gmmR0ChunkMutexAcquire and is used by the other
572 * gmmR0ChunkMutex* methods.
573 */
574typedef struct GMMR0CHUNKMTXSTATE
575{
576 PGMM pGMM;
577 /** The index of the chunk mutex. */
578 uint8_t iChunkMtx;
579 /** The relevant flags (GMMR0CHUNK_MTX_XXX). */
580 uint8_t fFlags;
581} GMMR0CHUNKMTXSTATE;
582/** Pointer to a chunk mutex state. */
583typedef GMMR0CHUNKMTXSTATE *PGMMR0CHUNKMTXSTATE;
584
585/** @name GMMR0CHUNK_MTX_XXX
586 * @{ */
587#define GMMR0CHUNK_MTX_INVALID UINT32_C(0)
588#define GMMR0CHUNK_MTX_KEEP_GIANT UINT32_C(1)
589#define GMMR0CHUNK_MTX_RETAKE_GIANT UINT32_C(2)
590#define GMMR0CHUNK_MTX_DROP_GIANT UINT32_C(3)
591#define GMMR0CHUNK_MTX_END UINT32_C(4)
592/** @} */
593
594
595/** The maximum number of shared modules per-vm. */
596#define GMM_MAX_SHARED_PER_VM_MODULES 2048
597/** The maximum number of shared modules GMM is allowed to track. */
598#define GMM_MAX_SHARED_GLOBAL_MODULES 16834
599
600
601/**
602 * Argument packet for gmmR0SharedModuleCleanup.
603 */
604typedef struct GMMR0SHMODPERVMDTORARGS
605{
606 PGVM pGVM;
607 PGMM pGMM;
608} GMMR0SHMODPERVMDTORARGS;
609
610/**
611 * Argument packet for gmmR0CheckSharedModule.
612 */
613typedef struct GMMCHECKSHAREDMODULEINFO
614{
615 PGVM pGVM;
616 VMCPUID idCpu;
617} GMMCHECKSHAREDMODULEINFO;
618
619/**
620 * Argument packet for gmmR0FindDupPageInChunk by GMMR0FindDuplicatePage.
621 */
622typedef struct GMMFINDDUPPAGEINFO
623{
624 PGVM pGVM;
625 PGMM pGMM;
626 uint8_t *pSourcePage;
627 bool fFoundDuplicate;
628} GMMFINDDUPPAGEINFO;
629
630
631/*******************************************************************************
632* Global Variables *
633*******************************************************************************/
634/** Pointer to the GMM instance data. */
635static PGMM g_pGMM = NULL;
636
637/** Macro for obtaining and validating the g_pGMM pointer.
638 *
639 * On failure it will return from the invoking function with the specified
640 * return value.
641 *
642 * @param pGMM The name of the pGMM variable.
643 * @param rc The return value on failure. Use VERR_GMM_INSTANCE for VBox
644 * status codes.
645 */
646#define GMM_GET_VALID_INSTANCE(pGMM, rc) \
647 do { \
648 (pGMM) = g_pGMM; \
649 AssertPtrReturn((pGMM), (rc)); \
650 AssertMsgReturn((pGMM)->u32Magic == GMM_MAGIC, ("%p - %#x\n", (pGMM), (pGMM)->u32Magic), (rc)); \
651 } while (0)
652
653/** Macro for obtaining and validating the g_pGMM pointer, void function
654 * variant.
655 *
656 * On failure it will return from the invoking function.
657 *
658 * @param pGMM The name of the pGMM variable.
659 */
660#define GMM_GET_VALID_INSTANCE_VOID(pGMM) \
661 do { \
662 (pGMM) = g_pGMM; \
663 AssertPtrReturnVoid((pGMM)); \
664 AssertMsgReturnVoid((pGMM)->u32Magic == GMM_MAGIC, ("%p - %#x\n", (pGMM), (pGMM)->u32Magic)); \
665 } while (0)
666
667
668/** @def GMM_CHECK_SANITY_UPON_ENTERING
669 * Checks the sanity of the GMM instance data before making changes.
670 *
671 * This is macro is a stub by default and must be enabled manually in the code.
672 *
673 * @returns true if sane, false if not.
674 * @param pGMM The name of the pGMM variable.
675 */
676#if defined(VBOX_STRICT) && defined(GMMR0_WITH_SANITY_CHECK) && 0
677# define GMM_CHECK_SANITY_UPON_ENTERING(pGMM) (gmmR0SanityCheck((pGMM), __PRETTY_FUNCTION__, __LINE__) == 0)
678#else
679# define GMM_CHECK_SANITY_UPON_ENTERING(pGMM) (true)
680#endif
681
682/** @def GMM_CHECK_SANITY_UPON_LEAVING
683 * Checks the sanity of the GMM instance data after making changes.
684 *
685 * This is macro is a stub by default and must be enabled manually in the code.
686 *
687 * @returns true if sane, false if not.
688 * @param pGMM The name of the pGMM variable.
689 */
690#if defined(VBOX_STRICT) && defined(GMMR0_WITH_SANITY_CHECK) && 0
691# define GMM_CHECK_SANITY_UPON_LEAVING(pGMM) (gmmR0SanityCheck((pGMM), __PRETTY_FUNCTION__, __LINE__) == 0)
692#else
693# define GMM_CHECK_SANITY_UPON_LEAVING(pGMM) (true)
694#endif
695
696/** @def GMM_CHECK_SANITY_IN_LOOPS
697 * Checks the sanity of the GMM instance in the allocation loops.
698 *
699 * This is macro is a stub by default and must be enabled manually in the code.
700 *
701 * @returns true if sane, false if not.
702 * @param pGMM The name of the pGMM variable.
703 */
704#if defined(VBOX_STRICT) && defined(GMMR0_WITH_SANITY_CHECK) && 0
705# define GMM_CHECK_SANITY_IN_LOOPS(pGMM) (gmmR0SanityCheck((pGMM), __PRETTY_FUNCTION__, __LINE__) == 0)
706#else
707# define GMM_CHECK_SANITY_IN_LOOPS(pGMM) (true)
708#endif
709
710
711/*******************************************************************************
712* Internal Functions *
713*******************************************************************************/
714static DECLCALLBACK(int) gmmR0TermDestroyChunk(PAVLU32NODECORE pNode, void *pvGMM);
715static bool gmmR0CleanupVMScanChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk);
716DECLINLINE(void) gmmR0UnlinkChunk(PGMMCHUNK pChunk);
717DECLINLINE(void) gmmR0LinkChunk(PGMMCHUNK pChunk, PGMMCHUNKFREESET pSet);
718DECLINLINE(void) gmmR0SelectSetAndLinkChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk);
719#ifdef GMMR0_WITH_SANITY_CHECK
720static uint32_t gmmR0SanityCheck(PGMM pGMM, const char *pszFunction, unsigned uLineNo);
721#endif
722static bool gmmR0FreeChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem);
723DECLINLINE(void) gmmR0FreePrivatePage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage);
724DECLINLINE(void) gmmR0FreeSharedPage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage);
725static int gmmR0UnmapChunkLocked(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk);
726#ifdef VBOX_WITH_PAGE_SHARING
727static void gmmR0SharedModuleCleanup(PGMM pGMM, PGVM pGVM);
728# ifdef VBOX_STRICT
729static uint32_t gmmR0StrictPageChecksum(PGMM pGMM, PGVM pGVM, uint32_t idPage);
730# endif
731#endif
732
733
734
735/**
736 * Initializes the GMM component.
737 *
738 * This is called when the VMMR0.r0 module is loaded and protected by the
739 * loader semaphore.
740 *
741 * @returns VBox status code.
742 */
743GMMR0DECL(int) GMMR0Init(void)
744{
745 LogFlow(("GMMInit:\n"));
746
747 /*
748 * Allocate the instance data and the locks.
749 */
750 PGMM pGMM = (PGMM)RTMemAllocZ(sizeof(*pGMM));
751 if (!pGMM)
752 return VERR_NO_MEMORY;
753
754 pGMM->u32Magic = GMM_MAGIC;
755 for (unsigned i = 0; i < RT_ELEMENTS(pGMM->ChunkTLB.aEntries); i++)
756 pGMM->ChunkTLB.aEntries[i].idChunk = NIL_GMM_CHUNKID;
757 RTListInit(&pGMM->ChunkList);
758 ASMBitSet(&pGMM->bmChunkId[0], NIL_GMM_CHUNKID);
759
760 int rc = RTSemFastMutexCreate(&pGMM->hMtx);
761 if (RT_SUCCESS(rc))
762 {
763 unsigned iMtx;
764 for (iMtx = 0; iMtx < RT_ELEMENTS(pGMM->aChunkMtx); iMtx++)
765 {
766 rc = RTSemFastMutexCreate(&pGMM->aChunkMtx[iMtx].hMtx);
767 if (RT_FAILURE(rc))
768 break;
769 }
770 if (RT_SUCCESS(rc))
771 {
772 /*
773 * Check and see if RTR0MemObjAllocPhysNC works.
774 */
775#if 0 /* later, see @bufref{3170}. */
776 RTR0MEMOBJ MemObj;
777 rc = RTR0MemObjAllocPhysNC(&MemObj, _64K, NIL_RTHCPHYS);
778 if (RT_SUCCESS(rc))
779 {
780 rc = RTR0MemObjFree(MemObj, true);
781 AssertRC(rc);
782 }
783 else if (rc == VERR_NOT_SUPPORTED)
784 pGMM->fLegacyAllocationMode = pGMM->fBoundMemoryMode = true;
785 else
786 SUPR0Printf("GMMR0Init: RTR0MemObjAllocPhysNC(,64K,Any) -> %d!\n", rc);
787#else
788# if defined(RT_OS_WINDOWS) || (defined(RT_OS_SOLARIS) && ARCH_BITS == 64) || defined(RT_OS_LINUX) || defined(RT_OS_FREEBSD)
789 pGMM->fLegacyAllocationMode = false;
790# if ARCH_BITS == 32
791 /* Don't reuse possibly partial chunks because of the virtual
792 address space limitation. */
793 pGMM->fBoundMemoryMode = true;
794# else
795 pGMM->fBoundMemoryMode = false;
796# endif
797# else
798 pGMM->fLegacyAllocationMode = true;
799 pGMM->fBoundMemoryMode = true;
800# endif
801#endif
802
803 /*
804 * Query system page count and guess a reasonable cMaxPages value.
805 */
806 pGMM->cMaxPages = UINT32_MAX; /** @todo IPRT function for query ram size and such. */
807
808 g_pGMM = pGMM;
809 LogFlow(("GMMInit: pGMM=%p fLegacyAllocationMode=%RTbool fBoundMemoryMode=%RTbool\n", pGMM, pGMM->fLegacyAllocationMode, pGMM->fBoundMemoryMode));
810 return VINF_SUCCESS;
811 }
812
813 /*
814 * Bail out.
815 */
816 while (iMtx-- > 0)
817 RTSemFastMutexDestroy(pGMM->aChunkMtx[iMtx].hMtx);
818 RTSemFastMutexDestroy(pGMM->hMtx);
819 }
820
821 pGMM->u32Magic = 0;
822 RTMemFree(pGMM);
823 SUPR0Printf("GMMR0Init: failed! rc=%d\n", rc);
824 return rc;
825}
826
827
828/**
829 * Terminates the GMM component.
830 */
831GMMR0DECL(void) GMMR0Term(void)
832{
833 LogFlow(("GMMTerm:\n"));
834
835 /*
836 * Take care / be paranoid...
837 */
838 PGMM pGMM = g_pGMM;
839 if (!VALID_PTR(pGMM))
840 return;
841 if (pGMM->u32Magic != GMM_MAGIC)
842 {
843 SUPR0Printf("GMMR0Term: u32Magic=%#x\n", pGMM->u32Magic);
844 return;
845 }
846
847 /*
848 * Undo what init did and free all the resources we've acquired.
849 */
850 /* Destroy the fundamentals. */
851 g_pGMM = NULL;
852 pGMM->u32Magic = ~GMM_MAGIC;
853 RTSemFastMutexDestroy(pGMM->hMtx);
854 pGMM->hMtx = NIL_RTSEMFASTMUTEX;
855
856 /* Free any chunks still hanging around. */
857 RTAvlU32Destroy(&pGMM->pChunks, gmmR0TermDestroyChunk, pGMM);
858
859 /* Destroy the chunk locks. */
860 for (unsigned iMtx = 0; iMtx < RT_ELEMENTS(pGMM->aChunkMtx); iMtx++)
861 {
862 Assert(pGMM->aChunkMtx[iMtx].cUsers == 0);
863 RTSemFastMutexDestroy(pGMM->aChunkMtx[iMtx].hMtx);
864 pGMM->aChunkMtx[iMtx].hMtx = NIL_RTSEMFASTMUTEX;
865 }
866
867 /* Finally the instance data itself. */
868 RTMemFree(pGMM);
869 LogFlow(("GMMTerm: done\n"));
870}
871
872
873/**
874 * RTAvlU32Destroy callback.
875 *
876 * @returns 0
877 * @param pNode The node to destroy.
878 * @param pvGMM The GMM handle.
879 */
880static DECLCALLBACK(int) gmmR0TermDestroyChunk(PAVLU32NODECORE pNode, void *pvGMM)
881{
882 PGMMCHUNK pChunk = (PGMMCHUNK)pNode;
883
884 if (pChunk->cFree != (GMM_CHUNK_SIZE >> PAGE_SHIFT))
885 SUPR0Printf("GMMR0Term: %p/%#x: cFree=%d cPrivate=%d cShared=%d cMappings=%d\n", pChunk,
886 pChunk->Core.Key, pChunk->cFree, pChunk->cPrivate, pChunk->cShared, pChunk->cMappingsX);
887
888 int rc = RTR0MemObjFree(pChunk->hMemObj, true /* fFreeMappings */);
889 if (RT_FAILURE(rc))
890 {
891 SUPR0Printf("GMMR0Term: %p/%#x: RTRMemObjFree(%p,true) -> %d (cMappings=%d)\n", pChunk,
892 pChunk->Core.Key, pChunk->hMemObj, rc, pChunk->cMappingsX);
893 AssertRC(rc);
894 }
895 pChunk->hMemObj = NIL_RTR0MEMOBJ;
896
897 RTMemFree(pChunk->paMappingsX);
898 pChunk->paMappingsX = NULL;
899
900 RTMemFree(pChunk);
901 NOREF(pvGMM);
902 return 0;
903}
904
905
906/**
907 * Initializes the per-VM data for the GMM.
908 *
909 * This is called from within the GVMM lock (from GVMMR0CreateVM)
910 * and should only initialize the data members so GMMR0CleanupVM
911 * can deal with them. We reserve no memory or anything here,
912 * that's done later in GMMR0InitVM.
913 *
914 * @param pGVM Pointer to the Global VM structure.
915 */
916GMMR0DECL(void) GMMR0InitPerVMData(PGVM pGVM)
917{
918 AssertCompile(RT_SIZEOFMEMB(GVM,gmm.s) <= RT_SIZEOFMEMB(GVM,gmm.padding));
919
920 pGVM->gmm.s.Stats.enmPolicy = GMMOCPOLICY_INVALID;
921 pGVM->gmm.s.Stats.enmPriority = GMMPRIORITY_INVALID;
922 pGVM->gmm.s.Stats.fMayAllocate = false;
923}
924
925
926/**
927 * Acquires the GMM giant lock.
928 *
929 * @returns Assert status code from RTSemFastMutexRequest.
930 * @param pGMM Pointer to the GMM instance.
931 */
932static int gmmR0MutexAcquire(PGMM pGMM)
933{
934 ASMAtomicIncU32(&pGMM->cMtxContenders);
935 int rc = RTSemFastMutexRequest(pGMM->hMtx);
936 ASMAtomicDecU32(&pGMM->cMtxContenders);
937 AssertRC(rc);
938#ifdef VBOX_STRICT
939 pGMM->hMtxOwner = RTThreadNativeSelf();
940#endif
941 return rc;
942}
943
944
945/**
946 * Releases the GMM giant lock.
947 *
948 * @returns Assert status code from RTSemFastMutexRequest.
949 * @param pGMM Pointer to the GMM instance.
950 */
951static int gmmR0MutexRelease(PGMM pGMM)
952{
953#ifdef VBOX_STRICT
954 pGMM->hMtxOwner = NIL_RTNATIVETHREAD;
955#endif
956 int rc = RTSemFastMutexRelease(pGMM->hMtx);
957 AssertRC(rc);
958 return rc;
959}
960
961
962/**
963 * Yields the GMM giant lock if there is contention and a certain minimum time
964 * has elapsed since we took it.
965 *
966 * @returns @c true if the mutex was yielded, @c false if not.
967 * @param pGMM Pointer to the GMM instance.
968 * @param puLockNanoTS Where the lock acquisition time stamp is kept
969 * (in/out).
970 */
971static bool gmmR0MutexYield(PGMM pGMM, uint64_t *puLockNanoTS)
972{
973 /*
974 * If nobody is contending the mutex, don't bother checking the time.
975 */
976 if (ASMAtomicReadU32(&pGMM->cMtxContenders) == 0)
977 return false;
978
979 /*
980 * Don't yield if we haven't executed for at least 2 milliseconds.
981 */
982 uint64_t uNanoNow = RTTimeSystemNanoTS();
983 if (uNanoNow - *puLockNanoTS < UINT32_C(2000000))
984 return false;
985
986 /*
987 * Yield the mutex.
988 */
989#ifdef VBOX_STRICT
990 pGMM->hMtxOwner = NIL_RTNATIVETHREAD;
991#endif
992 ASMAtomicIncU32(&pGMM->cMtxContenders);
993 int rc1 = RTSemFastMutexRelease(pGMM->hMtx); AssertRC(rc1);
994
995 RTThreadYield();
996
997 int rc2 = RTSemFastMutexRequest(pGMM->hMtx); AssertRC(rc2);
998 *puLockNanoTS = RTTimeSystemNanoTS();
999 ASMAtomicDecU32(&pGMM->cMtxContenders);
1000#ifdef VBOX_STRICT
1001 pGMM->hMtxOwner = RTThreadNativeSelf();
1002#endif
1003
1004 return true;
1005}
1006
1007
1008/**
1009 * Acquires a chunk lock.
1010 *
1011 * The caller must own the giant lock.
1012 *
1013 * @returns Assert status code from RTSemFastMutexRequest.
1014 * @param pMtxState The chunk mutex state info. (Avoids
1015 * passing the same flags and stuff around
1016 * for subsequent release and drop-giant
1017 * calls.)
1018 * @param pGMM Pointer to the GMM instance.
1019 * @param pChunk Pointer to the chunk.
1020 * @param fFlags Flags regarding the giant lock, GMMR0CHUNK_MTX_XXX.
1021 */
1022static int gmmR0ChunkMutexAcquire(PGMMR0CHUNKMTXSTATE pMtxState, PGMM pGMM, PGMMCHUNK pChunk, uint32_t fFlags)
1023{
1024 Assert(fFlags > GMMR0CHUNK_MTX_INVALID && fFlags < GMMR0CHUNK_MTX_END);
1025 Assert(pGMM->hMtxOwner == RTThreadNativeSelf());
1026
1027 pMtxState->pGMM = pGMM;
1028 pMtxState->fFlags = (uint8_t)fFlags;
1029
1030 /*
1031 * Get the lock index and reference the lock.
1032 */
1033 Assert(pGMM->hMtxOwner == RTThreadNativeSelf());
1034 uint32_t iChunkMtx = pChunk->iChunkMtx;
1035 if (iChunkMtx == UINT8_MAX)
1036 {
1037 iChunkMtx = pGMM->iNextChunkMtx++;
1038 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1039
1040 /* Try get an unused one... */
1041 if (pGMM->aChunkMtx[iChunkMtx].cUsers)
1042 {
1043 iChunkMtx = pGMM->iNextChunkMtx++;
1044 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1045 if (pGMM->aChunkMtx[iChunkMtx].cUsers)
1046 {
1047 iChunkMtx = pGMM->iNextChunkMtx++;
1048 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1049 if (pGMM->aChunkMtx[iChunkMtx].cUsers)
1050 {
1051 iChunkMtx = pGMM->iNextChunkMtx++;
1052 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1053 }
1054 }
1055 }
1056
1057 pChunk->iChunkMtx = iChunkMtx;
1058 }
1059 AssertCompile(RT_ELEMENTS(pGMM->aChunkMtx) < UINT8_MAX);
1060 pMtxState->iChunkMtx = (uint8_t)iChunkMtx;
1061 ASMAtomicIncU32(&pGMM->aChunkMtx[iChunkMtx].cUsers);
1062
1063 /*
1064 * Drop the giant?
1065 */
1066 if (fFlags != GMMR0CHUNK_MTX_KEEP_GIANT)
1067 {
1068 /** @todo GMM life cycle cleanup (we may race someone
1069 * destroying and cleaning up GMM)? */
1070 gmmR0MutexRelease(pGMM);
1071 }
1072
1073 /*
1074 * Take the chunk mutex.
1075 */
1076 int rc = RTSemFastMutexRequest(pGMM->aChunkMtx[iChunkMtx].hMtx);
1077 AssertRC(rc);
1078 return rc;
1079}
1080
1081
1082/**
1083 * Releases the GMM giant lock.
1084 *
1085 * @returns Assert status code from RTSemFastMutexRequest.
1086 * @param pGMM Pointer to the GMM instance.
1087 * @param pChunk Pointer to the chunk if it's still
1088 * alive, NULL if it isn't. This is used to deassociate
1089 * the chunk from the mutex on the way out so a new one
1090 * can be selected next time, thus avoiding contented
1091 * mutexes.
1092 */
1093static int gmmR0ChunkMutexRelease(PGMMR0CHUNKMTXSTATE pMtxState, PGMMCHUNK pChunk)
1094{
1095 PGMM pGMM = pMtxState->pGMM;
1096
1097 /*
1098 * Release the chunk mutex and reacquire the giant if requested.
1099 */
1100 int rc = RTSemFastMutexRelease(pGMM->aChunkMtx[pMtxState->iChunkMtx].hMtx);
1101 AssertRC(rc);
1102 if (pMtxState->fFlags == GMMR0CHUNK_MTX_RETAKE_GIANT)
1103 rc = gmmR0MutexAcquire(pGMM);
1104 else
1105 Assert((pMtxState->fFlags != GMMR0CHUNK_MTX_DROP_GIANT) == (pGMM->hMtxOwner == RTThreadNativeSelf()));
1106
1107 /*
1108 * Drop the chunk mutex user reference and deassociate it from the chunk
1109 * when possible.
1110 */
1111 if ( ASMAtomicDecU32(&pGMM->aChunkMtx[pMtxState->iChunkMtx].cUsers) == 0
1112 && pChunk
1113 && RT_SUCCESS(rc) )
1114 {
1115 if (pMtxState->fFlags != GMMR0CHUNK_MTX_DROP_GIANT)
1116 pChunk->iChunkMtx = UINT8_MAX;
1117 else
1118 {
1119 rc = gmmR0MutexAcquire(pGMM);
1120 if (RT_SUCCESS(rc))
1121 {
1122 if (pGMM->aChunkMtx[pMtxState->iChunkMtx].cUsers == 0)
1123 pChunk->iChunkMtx = UINT8_MAX;
1124 rc = gmmR0MutexRelease(pGMM);
1125 }
1126 }
1127 }
1128
1129 pMtxState->pGMM = NULL;
1130 return rc;
1131}
1132
1133
1134/**
1135 * Drops the giant GMM lock we kept in gmmR0ChunkMutexAcquire while keeping the
1136 * chunk locked.
1137 *
1138 * This only works if gmmR0ChunkMutexAcquire was called with
1139 * GMMR0CHUNK_MTX_KEEP_GIANT. gmmR0ChunkMutexRelease will retake the giant
1140 * mutex, i.e. behave as if GMMR0CHUNK_MTX_RETAKE_GIANT was used.
1141 *
1142 * @returns VBox status code (assuming success is ok).
1143 * @param pMtxState Pointer to the chunk mutex state.
1144 */
1145static int gmmR0ChunkMutexDropGiant(PGMMR0CHUNKMTXSTATE pMtxState)
1146{
1147 AssertReturn(pMtxState->fFlags == GMMR0CHUNK_MTX_KEEP_GIANT, VERR_GMM_MTX_FLAGS);
1148 Assert(pMtxState->pGMM->hMtxOwner == RTThreadNativeSelf());
1149 pMtxState->fFlags = GMMR0CHUNK_MTX_RETAKE_GIANT;
1150 /** @todo GMM life cycle cleanup (we may race someone
1151 * destroying and cleaning up GMM)? */
1152 return gmmR0MutexRelease(pMtxState->pGMM);
1153}
1154
1155
1156/**
1157 * For experimenting with NUMA affinity and such.
1158 *
1159 * @returns The current NUMA Node ID.
1160 */
1161static uint16_t gmmR0GetCurrentNumaNodeId(void)
1162{
1163#if 1
1164 return GMM_CHUNK_NUMA_ID_UNKNOWN;
1165#else
1166 return RTMpCpuId() / 16;
1167#endif
1168}
1169
1170
1171
1172/**
1173 * Cleans up when a VM is terminating.
1174 *
1175 * @param pGVM Pointer to the Global VM structure.
1176 */
1177GMMR0DECL(void) GMMR0CleanupVM(PGVM pGVM)
1178{
1179 LogFlow(("GMMR0CleanupVM: pGVM=%p:{.pVM=%p, .hSelf=%#x}\n", pGVM, pGVM->pVM, pGVM->hSelf));
1180
1181 PGMM pGMM;
1182 GMM_GET_VALID_INSTANCE_VOID(pGMM);
1183
1184#ifdef VBOX_WITH_PAGE_SHARING
1185 /*
1186 * Clean up all registered shared modules first.
1187 */
1188 gmmR0SharedModuleCleanup(pGMM, pGVM);
1189#endif
1190
1191 gmmR0MutexAcquire(pGMM);
1192 uint64_t uLockNanoTS = RTTimeSystemNanoTS();
1193 GMM_CHECK_SANITY_UPON_ENTERING(pGMM);
1194
1195 /*
1196 * The policy is 'INVALID' until the initial reservation
1197 * request has been serviced.
1198 */
1199 if ( pGVM->gmm.s.Stats.enmPolicy > GMMOCPOLICY_INVALID
1200 && pGVM->gmm.s.Stats.enmPolicy < GMMOCPOLICY_END)
1201 {
1202 /*
1203 * If it's the last VM around, we can skip walking all the chunk looking
1204 * for the pages owned by this VM and instead flush the whole shebang.
1205 *
1206 * This takes care of the eventuality that a VM has left shared page
1207 * references behind (shouldn't happen of course, but you never know).
1208 */
1209 Assert(pGMM->cRegisteredVMs);
1210 pGMM->cRegisteredVMs--;
1211
1212 /*
1213 * Walk the entire pool looking for pages that belong to this VM
1214 * and leftover mappings. (This'll only catch private pages,
1215 * shared pages will be 'left behind'.)
1216 */
1217 uint64_t cPrivatePages = pGVM->gmm.s.Stats.cPrivatePages; /* save */
1218
1219 unsigned iCountDown = 64;
1220 bool fRedoFromStart;
1221 PGMMCHUNK pChunk;
1222 do
1223 {
1224 fRedoFromStart = false;
1225 RTListForEachReverse(&pGMM->ChunkList, pChunk, GMMCHUNK, ListNode)
1226 {
1227 uint32_t const cFreeChunksOld = pGMM->cFreedChunks;
1228 if (gmmR0CleanupVMScanChunk(pGMM, pGVM, pChunk))
1229 {
1230 /* We left the giant mutex, so reset the yield counters. */
1231 uLockNanoTS = RTTimeSystemNanoTS();
1232 iCountDown = 64;
1233 }
1234 else
1235 {
1236 /* Didn't leave it, so do normal yielding. */
1237 if (!iCountDown)
1238 gmmR0MutexYield(pGMM, &uLockNanoTS);
1239 else
1240 iCountDown--;
1241 }
1242 if (pGMM->cFreedChunks != cFreeChunksOld)
1243 break;
1244 }
1245 } while (fRedoFromStart);
1246
1247 if (pGVM->gmm.s.Stats.cPrivatePages)
1248 SUPR0Printf("GMMR0CleanupVM: hGVM=%#x has %#x private pages that cannot be found!\n", pGVM->hSelf, pGVM->gmm.s.Stats.cPrivatePages);
1249
1250 pGMM->cAllocatedPages -= cPrivatePages;
1251
1252 /*
1253 * Free empty chunks.
1254 */
1255 PGMMCHUNKFREESET pPrivateSet = pGMM->fBoundMemoryMode ? &pGVM->gmm.s.Private : &pGMM->PrivateX;
1256 do
1257 {
1258 fRedoFromStart = false;
1259 iCountDown = 10240;
1260 pChunk = pPrivateSet->apLists[GMM_CHUNK_FREE_SET_UNUSED_LIST];
1261 while (pChunk)
1262 {
1263 PGMMCHUNK pNext = pChunk->pFreeNext;
1264 Assert(pChunk->cFree == GMM_CHUNK_NUM_PAGES);
1265 if ( !pGMM->fBoundMemoryMode
1266 || pChunk->hGVM == pGVM->hSelf)
1267 {
1268 uint64_t const idGenerationOld = pPrivateSet->idGeneration;
1269 if (gmmR0FreeChunk(pGMM, pGVM, pChunk, true /*fRelaxedSem*/))
1270 {
1271 /* We've left the giant mutex, restart? (+1 for our unlink) */
1272 fRedoFromStart = pPrivateSet->idGeneration != idGenerationOld + 1;
1273 if (fRedoFromStart)
1274 break;
1275 uLockNanoTS = RTTimeSystemNanoTS();
1276 iCountDown = 10240;
1277 }
1278 }
1279
1280 /* Advance and maybe yield the lock. */
1281 pChunk = pNext;
1282 if (--iCountDown == 0)
1283 {
1284 uint64_t const idGenerationOld = pPrivateSet->idGeneration;
1285 fRedoFromStart = gmmR0MutexYield(pGMM, &uLockNanoTS)
1286 && pPrivateSet->idGeneration != idGenerationOld;
1287 if (fRedoFromStart)
1288 break;
1289 iCountDown = 10240;
1290 }
1291 }
1292 } while (fRedoFromStart);
1293
1294 /*
1295 * Account for shared pages that weren't freed.
1296 */
1297 if (pGVM->gmm.s.Stats.cSharedPages)
1298 {
1299 Assert(pGMM->cSharedPages >= pGVM->gmm.s.Stats.cSharedPages);
1300 SUPR0Printf("GMMR0CleanupVM: hGVM=%#x left %#x shared pages behind!\n", pGVM->hSelf, pGVM->gmm.s.Stats.cSharedPages);
1301 pGMM->cLeftBehindSharedPages += pGVM->gmm.s.Stats.cSharedPages;
1302 }
1303
1304 /*
1305 * Clean up balloon statistics in case the VM process crashed.
1306 */
1307 Assert(pGMM->cBalloonedPages >= pGVM->gmm.s.Stats.cBalloonedPages);
1308 pGMM->cBalloonedPages -= pGVM->gmm.s.Stats.cBalloonedPages;
1309
1310 /*
1311 * Update the over-commitment management statistics.
1312 */
1313 pGMM->cReservedPages -= pGVM->gmm.s.Stats.Reserved.cBasePages
1314 + pGVM->gmm.s.Stats.Reserved.cFixedPages
1315 + pGVM->gmm.s.Stats.Reserved.cShadowPages;
1316 switch (pGVM->gmm.s.Stats.enmPolicy)
1317 {
1318 case GMMOCPOLICY_NO_OC:
1319 break;
1320 default:
1321 /** @todo Update GMM->cOverCommittedPages */
1322 break;
1323 }
1324 }
1325
1326 /* zap the GVM data. */
1327 pGVM->gmm.s.Stats.enmPolicy = GMMOCPOLICY_INVALID;
1328 pGVM->gmm.s.Stats.enmPriority = GMMPRIORITY_INVALID;
1329 pGVM->gmm.s.Stats.fMayAllocate = false;
1330
1331 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
1332 gmmR0MutexRelease(pGMM);
1333
1334 LogFlow(("GMMR0CleanupVM: returns\n"));
1335}
1336
1337
1338/**
1339 * Scan one chunk for private pages belonging to the specified VM.
1340 *
1341 * @note This function may drop the giant mutex!
1342 *
1343 * @returns @c true if we've temporarily dropped the giant mutex, @c false if
1344 * we didn't.
1345 * @param pGMM Pointer to the GMM instance.
1346 * @param pGVM The global VM handle.
1347 * @param pChunk The chunk to scan.
1348 */
1349static bool gmmR0CleanupVMScanChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk)
1350{
1351 /*
1352 * Look for pages belonging to the VM.
1353 * (Perform some internal checks while we're scanning.)
1354 */
1355#ifndef VBOX_STRICT
1356 if (pChunk->cFree != (GMM_CHUNK_SIZE >> PAGE_SHIFT))
1357#endif
1358 {
1359 unsigned cPrivate = 0;
1360 unsigned cShared = 0;
1361 unsigned cFree = 0;
1362
1363 gmmR0UnlinkChunk(pChunk); /* avoiding cFreePages updates. */
1364
1365 uint16_t hGVM = pGVM->hSelf;
1366 unsigned iPage = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
1367 while (iPage-- > 0)
1368 if (GMM_PAGE_IS_PRIVATE(&pChunk->aPages[iPage]))
1369 {
1370 if (pChunk->aPages[iPage].Private.hGVM == hGVM)
1371 {
1372 /*
1373 * Free the page.
1374 *
1375 * The reason for not using gmmR0FreePrivatePage here is that we
1376 * must *not* cause the chunk to be freed from under us - we're in
1377 * an AVL tree walk here.
1378 */
1379 pChunk->aPages[iPage].u = 0;
1380 pChunk->aPages[iPage].Free.iNext = pChunk->iFreeHead;
1381 pChunk->aPages[iPage].Free.u2State = GMM_PAGE_STATE_FREE;
1382 pChunk->iFreeHead = iPage;
1383 pChunk->cPrivate--;
1384 pChunk->cFree++;
1385 pGVM->gmm.s.Stats.cPrivatePages--;
1386 cFree++;
1387 }
1388 else
1389 cPrivate++;
1390 }
1391 else if (GMM_PAGE_IS_FREE(&pChunk->aPages[iPage]))
1392 cFree++;
1393 else
1394 cShared++;
1395
1396 gmmR0SelectSetAndLinkChunk(pGMM, pGVM, pChunk);
1397
1398 /*
1399 * Did it add up?
1400 */
1401 if (RT_UNLIKELY( pChunk->cFree != cFree
1402 || pChunk->cPrivate != cPrivate
1403 || pChunk->cShared != cShared))
1404 {
1405 SUPR0Printf("gmmR0CleanupVMScanChunk: Chunk %p/%#x has bogus stats - free=%d/%d private=%d/%d shared=%d/%d\n",
1406 pChunk->cFree, cFree, pChunk->cPrivate, cPrivate, pChunk->cShared, cShared);
1407 pChunk->cFree = cFree;
1408 pChunk->cPrivate = cPrivate;
1409 pChunk->cShared = cShared;
1410 }
1411 }
1412
1413 /*
1414 * If not in bound memory mode, we should reset the hGVM field
1415 * if it has our handle in it.
1416 */
1417 if (pChunk->hGVM == pGVM->hSelf)
1418 {
1419 if (!g_pGMM->fBoundMemoryMode)
1420 pChunk->hGVM = NIL_GVM_HANDLE;
1421 else if (pChunk->cFree != GMM_CHUNK_NUM_PAGES)
1422 {
1423 SUPR0Printf("gmmR0CleanupVMScanChunk: %p/%#x: cFree=%#x - it should be 0 in bound mode!\n",
1424 pChunk, pChunk->Core.Key, pChunk->cFree);
1425 AssertMsgFailed(("%p/%#x: cFree=%#x - it should be 0 in bound mode!\n", pChunk, pChunk->Core.Key, pChunk->cFree));
1426
1427 gmmR0UnlinkChunk(pChunk);
1428 pChunk->cFree = GMM_CHUNK_NUM_PAGES;
1429 gmmR0SelectSetAndLinkChunk(pGMM, pGVM, pChunk);
1430 }
1431 }
1432
1433 /*
1434 * Look for a mapping belonging to the terminating VM.
1435 */
1436 GMMR0CHUNKMTXSTATE MtxState;
1437 gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk, GMMR0CHUNK_MTX_KEEP_GIANT);
1438 unsigned cMappings = pChunk->cMappingsX;
1439 for (unsigned i = 0; i < cMappings; i++)
1440 if (pChunk->paMappingsX[i].pGVM == pGVM)
1441 {
1442 gmmR0ChunkMutexDropGiant(&MtxState);
1443
1444 RTR0MEMOBJ hMemObj = pChunk->paMappingsX[i].hMapObj;
1445
1446 cMappings--;
1447 if (i < cMappings)
1448 pChunk->paMappingsX[i] = pChunk->paMappingsX[cMappings];
1449 pChunk->paMappingsX[cMappings].pGVM = NULL;
1450 pChunk->paMappingsX[cMappings].hMapObj = NIL_RTR0MEMOBJ;
1451 Assert(pChunk->cMappingsX - 1U == cMappings);
1452 pChunk->cMappingsX = cMappings;
1453
1454 int rc = RTR0MemObjFree(hMemObj, false /* fFreeMappings (NA) */);
1455 if (RT_FAILURE(rc))
1456 {
1457 SUPR0Printf("gmmR0CleanupVMScanChunk: %p/%#x: mapping #%x: RTRMemObjFree(%p,false) -> %d \n",
1458 pChunk, pChunk->Core.Key, i, hMemObj, rc);
1459 AssertRC(rc);
1460 }
1461
1462 gmmR0ChunkMutexRelease(&MtxState, pChunk);
1463 return true;
1464 }
1465
1466 gmmR0ChunkMutexRelease(&MtxState, pChunk);
1467 return false;
1468}
1469
1470
1471/**
1472 * The initial resource reservations.
1473 *
1474 * This will make memory reservations according to policy and priority. If there aren't
1475 * sufficient resources available to sustain the VM this function will fail and all
1476 * future allocations requests will fail as well.
1477 *
1478 * These are just the initial reservations made very very early during the VM creation
1479 * process and will be adjusted later in the GMMR0UpdateReservation call after the
1480 * ring-3 init has completed.
1481 *
1482 * @returns VBox status code.
1483 * @retval VERR_GMM_MEMORY_RESERVATION_DECLINED
1484 * @retval VERR_GMM_
1485 *
1486 * @param pVM Pointer to the VM.
1487 * @param idCpu The VCPU id.
1488 * @param cBasePages The number of pages that may be allocated for the base RAM and ROMs.
1489 * This does not include MMIO2 and similar.
1490 * @param cShadowPages The number of pages that may be allocated for shadow paging structures.
1491 * @param cFixedPages The number of pages that may be allocated for fixed objects like the
1492 * hyper heap, MMIO2 and similar.
1493 * @param enmPolicy The OC policy to use on this VM.
1494 * @param enmPriority The priority in an out-of-memory situation.
1495 *
1496 * @thread The creator thread / EMT.
1497 */
1498GMMR0DECL(int) GMMR0InitialReservation(PVM pVM, VMCPUID idCpu, uint64_t cBasePages, uint32_t cShadowPages, uint32_t cFixedPages,
1499 GMMOCPOLICY enmPolicy, GMMPRIORITY enmPriority)
1500{
1501 LogFlow(("GMMR0InitialReservation: pVM=%p cBasePages=%#llx cShadowPages=%#x cFixedPages=%#x enmPolicy=%d enmPriority=%d\n",
1502 pVM, cBasePages, cShadowPages, cFixedPages, enmPolicy, enmPriority));
1503
1504 /*
1505 * Validate, get basics and take the semaphore.
1506 */
1507 PGMM pGMM;
1508 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
1509 PGVM pGVM;
1510 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
1511 if (RT_FAILURE(rc))
1512 return rc;
1513
1514 AssertReturn(cBasePages, VERR_INVALID_PARAMETER);
1515 AssertReturn(cShadowPages, VERR_INVALID_PARAMETER);
1516 AssertReturn(cFixedPages, VERR_INVALID_PARAMETER);
1517 AssertReturn(enmPolicy > GMMOCPOLICY_INVALID && enmPolicy < GMMOCPOLICY_END, VERR_INVALID_PARAMETER);
1518 AssertReturn(enmPriority > GMMPRIORITY_INVALID && enmPriority < GMMPRIORITY_END, VERR_INVALID_PARAMETER);
1519
1520 gmmR0MutexAcquire(pGMM);
1521 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
1522 {
1523 if ( !pGVM->gmm.s.Stats.Reserved.cBasePages
1524 && !pGVM->gmm.s.Stats.Reserved.cFixedPages
1525 && !pGVM->gmm.s.Stats.Reserved.cShadowPages)
1526 {
1527 /*
1528 * Check if we can accommodate this.
1529 */
1530 /* ... later ... */
1531 if (RT_SUCCESS(rc))
1532 {
1533 /*
1534 * Update the records.
1535 */
1536 pGVM->gmm.s.Stats.Reserved.cBasePages = cBasePages;
1537 pGVM->gmm.s.Stats.Reserved.cFixedPages = cFixedPages;
1538 pGVM->gmm.s.Stats.Reserved.cShadowPages = cShadowPages;
1539 pGVM->gmm.s.Stats.enmPolicy = enmPolicy;
1540 pGVM->gmm.s.Stats.enmPriority = enmPriority;
1541 pGVM->gmm.s.Stats.fMayAllocate = true;
1542
1543 pGMM->cReservedPages += cBasePages + cFixedPages + cShadowPages;
1544 pGMM->cRegisteredVMs++;
1545 }
1546 }
1547 else
1548 rc = VERR_WRONG_ORDER;
1549 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
1550 }
1551 else
1552 rc = VERR_GMM_IS_NOT_SANE;
1553 gmmR0MutexRelease(pGMM);
1554 LogFlow(("GMMR0InitialReservation: returns %Rrc\n", rc));
1555 return rc;
1556}
1557
1558
1559/**
1560 * VMMR0 request wrapper for GMMR0InitialReservation.
1561 *
1562 * @returns see GMMR0InitialReservation.
1563 * @param pVM Pointer to the VM.
1564 * @param idCpu The VCPU id.
1565 * @param pReq Pointer to the request packet.
1566 */
1567GMMR0DECL(int) GMMR0InitialReservationReq(PVM pVM, VMCPUID idCpu, PGMMINITIALRESERVATIONREQ pReq)
1568{
1569 /*
1570 * Validate input and pass it on.
1571 */
1572 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
1573 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
1574 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
1575
1576 return GMMR0InitialReservation(pVM, idCpu, pReq->cBasePages, pReq->cShadowPages, pReq->cFixedPages, pReq->enmPolicy, pReq->enmPriority);
1577}
1578
1579
1580/**
1581 * This updates the memory reservation with the additional MMIO2 and ROM pages.
1582 *
1583 * @returns VBox status code.
1584 * @retval VERR_GMM_MEMORY_RESERVATION_DECLINED
1585 *
1586 * @param pVM Pointer to the VM.
1587 * @param idCpu The VCPU id.
1588 * @param cBasePages The number of pages that may be allocated for the base RAM and ROMs.
1589 * This does not include MMIO2 and similar.
1590 * @param cShadowPages The number of pages that may be allocated for shadow paging structures.
1591 * @param cFixedPages The number of pages that may be allocated for fixed objects like the
1592 * hyper heap, MMIO2 and similar.
1593 *
1594 * @thread EMT.
1595 */
1596GMMR0DECL(int) GMMR0UpdateReservation(PVM pVM, VMCPUID idCpu, uint64_t cBasePages, uint32_t cShadowPages, uint32_t cFixedPages)
1597{
1598 LogFlow(("GMMR0UpdateReservation: pVM=%p cBasePages=%#llx cShadowPages=%#x cFixedPages=%#x\n",
1599 pVM, cBasePages, cShadowPages, cFixedPages));
1600
1601 /*
1602 * Validate, get basics and take the semaphore.
1603 */
1604 PGMM pGMM;
1605 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
1606 PGVM pGVM;
1607 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
1608 if (RT_FAILURE(rc))
1609 return rc;
1610
1611 AssertReturn(cBasePages, VERR_INVALID_PARAMETER);
1612 AssertReturn(cShadowPages, VERR_INVALID_PARAMETER);
1613 AssertReturn(cFixedPages, VERR_INVALID_PARAMETER);
1614
1615 gmmR0MutexAcquire(pGMM);
1616 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
1617 {
1618 if ( pGVM->gmm.s.Stats.Reserved.cBasePages
1619 && pGVM->gmm.s.Stats.Reserved.cFixedPages
1620 && pGVM->gmm.s.Stats.Reserved.cShadowPages)
1621 {
1622 /*
1623 * Check if we can accommodate this.
1624 */
1625 /* ... later ... */
1626 if (RT_SUCCESS(rc))
1627 {
1628 /*
1629 * Update the records.
1630 */
1631 pGMM->cReservedPages -= pGVM->gmm.s.Stats.Reserved.cBasePages
1632 + pGVM->gmm.s.Stats.Reserved.cFixedPages
1633 + pGVM->gmm.s.Stats.Reserved.cShadowPages;
1634 pGMM->cReservedPages += cBasePages + cFixedPages + cShadowPages;
1635
1636 pGVM->gmm.s.Stats.Reserved.cBasePages = cBasePages;
1637 pGVM->gmm.s.Stats.Reserved.cFixedPages = cFixedPages;
1638 pGVM->gmm.s.Stats.Reserved.cShadowPages = cShadowPages;
1639 }
1640 }
1641 else
1642 rc = VERR_WRONG_ORDER;
1643 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
1644 }
1645 else
1646 rc = VERR_GMM_IS_NOT_SANE;
1647 gmmR0MutexRelease(pGMM);
1648 LogFlow(("GMMR0UpdateReservation: returns %Rrc\n", rc));
1649 return rc;
1650}
1651
1652
1653/**
1654 * VMMR0 request wrapper for GMMR0UpdateReservation.
1655 *
1656 * @returns see GMMR0UpdateReservation.
1657 * @param pVM Pointer to the VM.
1658 * @param idCpu The VCPU id.
1659 * @param pReq Pointer to the request packet.
1660 */
1661GMMR0DECL(int) GMMR0UpdateReservationReq(PVM pVM, VMCPUID idCpu, PGMMUPDATERESERVATIONREQ pReq)
1662{
1663 /*
1664 * Validate input and pass it on.
1665 */
1666 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
1667 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
1668 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
1669
1670 return GMMR0UpdateReservation(pVM, idCpu, pReq->cBasePages, pReq->cShadowPages, pReq->cFixedPages);
1671}
1672
1673#ifdef GMMR0_WITH_SANITY_CHECK
1674
1675/**
1676 * Performs sanity checks on a free set.
1677 *
1678 * @returns Error count.
1679 *
1680 * @param pGMM Pointer to the GMM instance.
1681 * @param pSet Pointer to the set.
1682 * @param pszSetName The set name.
1683 * @param pszFunction The function from which it was called.
1684 * @param uLine The line number.
1685 */
1686static uint32_t gmmR0SanityCheckSet(PGMM pGMM, PGMMCHUNKFREESET pSet, const char *pszSetName,
1687 const char *pszFunction, unsigned uLineNo)
1688{
1689 uint32_t cErrors = 0;
1690
1691 /*
1692 * Count the free pages in all the chunks and match it against pSet->cFreePages.
1693 */
1694 uint32_t cPages = 0;
1695 for (unsigned i = 0; i < RT_ELEMENTS(pSet->apLists); i++)
1696 {
1697 for (PGMMCHUNK pCur = pSet->apLists[i]; pCur; pCur = pCur->pFreeNext)
1698 {
1699 /** @todo check that the chunk is hash into the right set. */
1700 cPages += pCur->cFree;
1701 }
1702 }
1703 if (RT_UNLIKELY(cPages != pSet->cFreePages))
1704 {
1705 SUPR0Printf("GMM insanity: found %#x pages in the %s set, expected %#x. (%s, line %u)\n",
1706 cPages, pszSetName, pSet->cFreePages, pszFunction, uLineNo);
1707 cErrors++;
1708 }
1709
1710 return cErrors;
1711}
1712
1713
1714/**
1715 * Performs some sanity checks on the GMM while owning lock.
1716 *
1717 * @returns Error count.
1718 *
1719 * @param pGMM Pointer to the GMM instance.
1720 * @param pszFunction The function from which it is called.
1721 * @param uLineNo The line number.
1722 */
1723static uint32_t gmmR0SanityCheck(PGMM pGMM, const char *pszFunction, unsigned uLineNo)
1724{
1725 uint32_t cErrors = 0;
1726
1727 cErrors += gmmR0SanityCheckSet(pGMM, &pGMM->PrivateX, "private", pszFunction, uLineNo);
1728 cErrors += gmmR0SanityCheckSet(pGMM, &pGMM->Shared, "shared", pszFunction, uLineNo);
1729 /** @todo add more sanity checks. */
1730
1731 return cErrors;
1732}
1733
1734#endif /* GMMR0_WITH_SANITY_CHECK */
1735
1736/**
1737 * Looks up a chunk in the tree and fill in the TLB entry for it.
1738 *
1739 * This is not expected to fail and will bitch if it does.
1740 *
1741 * @returns Pointer to the allocation chunk, NULL if not found.
1742 * @param pGMM Pointer to the GMM instance.
1743 * @param idChunk The ID of the chunk to find.
1744 * @param pTlbe Pointer to the TLB entry.
1745 */
1746static PGMMCHUNK gmmR0GetChunkSlow(PGMM pGMM, uint32_t idChunk, PGMMCHUNKTLBE pTlbe)
1747{
1748 PGMMCHUNK pChunk = (PGMMCHUNK)RTAvlU32Get(&pGMM->pChunks, idChunk);
1749 AssertMsgReturn(pChunk, ("Chunk %#x not found!\n", idChunk), NULL);
1750 pTlbe->idChunk = idChunk;
1751 pTlbe->pChunk = pChunk;
1752 return pChunk;
1753}
1754
1755
1756/**
1757 * Finds a allocation chunk.
1758 *
1759 * This is not expected to fail and will bitch if it does.
1760 *
1761 * @returns Pointer to the allocation chunk, NULL if not found.
1762 * @param pGMM Pointer to the GMM instance.
1763 * @param idChunk The ID of the chunk to find.
1764 */
1765DECLINLINE(PGMMCHUNK) gmmR0GetChunk(PGMM pGMM, uint32_t idChunk)
1766{
1767 /*
1768 * Do a TLB lookup, branch if not in the TLB.
1769 */
1770 PGMMCHUNKTLBE pTlbe = &pGMM->ChunkTLB.aEntries[GMM_CHUNKTLB_IDX(idChunk)];
1771 if ( pTlbe->idChunk != idChunk
1772 || !pTlbe->pChunk)
1773 return gmmR0GetChunkSlow(pGMM, idChunk, pTlbe);
1774 return pTlbe->pChunk;
1775}
1776
1777
1778/**
1779 * Finds a page.
1780 *
1781 * This is not expected to fail and will bitch if it does.
1782 *
1783 * @returns Pointer to the page, NULL if not found.
1784 * @param pGMM Pointer to the GMM instance.
1785 * @param idPage The ID of the page to find.
1786 */
1787DECLINLINE(PGMMPAGE) gmmR0GetPage(PGMM pGMM, uint32_t idPage)
1788{
1789 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
1790 if (RT_LIKELY(pChunk))
1791 return &pChunk->aPages[idPage & GMM_PAGEID_IDX_MASK];
1792 return NULL;
1793}
1794
1795
1796/**
1797 * Gets the host physical address for a page given by it's ID.
1798 *
1799 * @returns The host physical address or NIL_RTHCPHYS.
1800 * @param pGMM Pointer to the GMM instance.
1801 * @param idPage The ID of the page to find.
1802 */
1803DECLINLINE(RTHCPHYS) gmmR0GetPageHCPhys(PGMM pGMM, uint32_t idPage)
1804{
1805 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
1806 if (RT_LIKELY(pChunk))
1807 return RTR0MemObjGetPagePhysAddr(pChunk->hMemObj, idPage & GMM_PAGEID_IDX_MASK);
1808 return NIL_RTHCPHYS;
1809}
1810
1811
1812/**
1813 * Selects the appropriate free list given the number of free pages.
1814 *
1815 * @returns Free list index.
1816 * @param cFree The number of free pages in the chunk.
1817 */
1818DECLINLINE(unsigned) gmmR0SelectFreeSetList(unsigned cFree)
1819{
1820 unsigned iList = cFree >> GMM_CHUNK_FREE_SET_SHIFT;
1821 AssertMsg(iList < RT_SIZEOFMEMB(GMMCHUNKFREESET, apLists) / RT_SIZEOFMEMB(GMMCHUNKFREESET, apLists[0]),
1822 ("%d (%u)\n", iList, cFree));
1823 return iList;
1824}
1825
1826
1827/**
1828 * Unlinks the chunk from the free list it's currently on (if any).
1829 *
1830 * @param pChunk The allocation chunk.
1831 */
1832DECLINLINE(void) gmmR0UnlinkChunk(PGMMCHUNK pChunk)
1833{
1834 PGMMCHUNKFREESET pSet = pChunk->pSet;
1835 if (RT_LIKELY(pSet))
1836 {
1837 pSet->cFreePages -= pChunk->cFree;
1838 pSet->idGeneration++;
1839
1840 PGMMCHUNK pPrev = pChunk->pFreePrev;
1841 PGMMCHUNK pNext = pChunk->pFreeNext;
1842 if (pPrev)
1843 pPrev->pFreeNext = pNext;
1844 else
1845 pSet->apLists[gmmR0SelectFreeSetList(pChunk->cFree)] = pNext;
1846 if (pNext)
1847 pNext->pFreePrev = pPrev;
1848
1849 pChunk->pSet = NULL;
1850 pChunk->pFreeNext = NULL;
1851 pChunk->pFreePrev = NULL;
1852 }
1853 else
1854 {
1855 Assert(!pChunk->pFreeNext);
1856 Assert(!pChunk->pFreePrev);
1857 Assert(!pChunk->cFree);
1858 }
1859}
1860
1861
1862/**
1863 * Links the chunk onto the appropriate free list in the specified free set.
1864 *
1865 * If no free entries, it's not linked into any list.
1866 *
1867 * @param pChunk The allocation chunk.
1868 * @param pSet The free set.
1869 */
1870DECLINLINE(void) gmmR0LinkChunk(PGMMCHUNK pChunk, PGMMCHUNKFREESET pSet)
1871{
1872 Assert(!pChunk->pSet);
1873 Assert(!pChunk->pFreeNext);
1874 Assert(!pChunk->pFreePrev);
1875
1876 if (pChunk->cFree > 0)
1877 {
1878 pChunk->pSet = pSet;
1879 pChunk->pFreePrev = NULL;
1880 unsigned const iList = gmmR0SelectFreeSetList(pChunk->cFree);
1881 pChunk->pFreeNext = pSet->apLists[iList];
1882 if (pChunk->pFreeNext)
1883 pChunk->pFreeNext->pFreePrev = pChunk;
1884 pSet->apLists[iList] = pChunk;
1885
1886 pSet->cFreePages += pChunk->cFree;
1887 pSet->idGeneration++;
1888 }
1889}
1890
1891
1892/**
1893 * Links the chunk onto the appropriate free list in the specified free set.
1894 *
1895 * If no free entries, it's not linked into any list.
1896 *
1897 * @param pChunk The allocation chunk.
1898 */
1899DECLINLINE(void) gmmR0SelectSetAndLinkChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk)
1900{
1901 PGMMCHUNKFREESET pSet;
1902 if (pGMM->fBoundMemoryMode)
1903 pSet = &pGVM->gmm.s.Private;
1904 else if (pChunk->cShared)
1905 pSet = &pGMM->Shared;
1906 else
1907 pSet = &pGMM->PrivateX;
1908 gmmR0LinkChunk(pChunk, pSet);
1909}
1910
1911
1912/**
1913 * Frees a Chunk ID.
1914 *
1915 * @param pGMM Pointer to the GMM instance.
1916 * @param idChunk The Chunk ID to free.
1917 */
1918static void gmmR0FreeChunkId(PGMM pGMM, uint32_t idChunk)
1919{
1920 AssertReturnVoid(idChunk != NIL_GMM_CHUNKID);
1921 AssertMsg(ASMBitTest(&pGMM->bmChunkId[0], idChunk), ("%#x\n", idChunk));
1922 ASMAtomicBitClear(&pGMM->bmChunkId[0], idChunk);
1923}
1924
1925
1926/**
1927 * Allocates a new Chunk ID.
1928 *
1929 * @returns The Chunk ID.
1930 * @param pGMM Pointer to the GMM instance.
1931 */
1932static uint32_t gmmR0AllocateChunkId(PGMM pGMM)
1933{
1934 AssertCompile(!((GMM_CHUNKID_LAST + 1) & 31)); /* must be a multiple of 32 */
1935 AssertCompile(NIL_GMM_CHUNKID == 0);
1936
1937 /*
1938 * Try the next sequential one.
1939 */
1940 int32_t idChunk = ++pGMM->idChunkPrev;
1941#if 0 /** @todo enable this code */
1942 if ( idChunk <= GMM_CHUNKID_LAST
1943 && idChunk > NIL_GMM_CHUNKID
1944 && !ASMAtomicBitTestAndSet(&pVMM->bmChunkId[0], idChunk))
1945 return idChunk;
1946#endif
1947
1948 /*
1949 * Scan sequentially from the last one.
1950 */
1951 if ( (uint32_t)idChunk < GMM_CHUNKID_LAST
1952 && idChunk > NIL_GMM_CHUNKID)
1953 {
1954 idChunk = ASMBitNextClear(&pGMM->bmChunkId[0], GMM_CHUNKID_LAST + 1, idChunk);
1955 if (idChunk > NIL_GMM_CHUNKID)
1956 {
1957 AssertMsgReturn(!ASMAtomicBitTestAndSet(&pGMM->bmChunkId[0], idChunk), ("%#x\n", idChunk), NIL_GMM_CHUNKID);
1958 return pGMM->idChunkPrev = idChunk;
1959 }
1960 }
1961
1962 /*
1963 * Ok, scan from the start.
1964 * We're not racing anyone, so there is no need to expect failures or have restart loops.
1965 */
1966 idChunk = ASMBitFirstClear(&pGMM->bmChunkId[0], GMM_CHUNKID_LAST + 1);
1967 AssertMsgReturn(idChunk > NIL_GMM_CHUNKID, ("%#x\n", idChunk), NIL_GVM_HANDLE);
1968 AssertMsgReturn(!ASMAtomicBitTestAndSet(&pGMM->bmChunkId[0], idChunk), ("%#x\n", idChunk), NIL_GMM_CHUNKID);
1969
1970 return pGMM->idChunkPrev = idChunk;
1971}
1972
1973
1974/**
1975 * Allocates one private page.
1976 *
1977 * Worker for gmmR0AllocatePages.
1978 *
1979 * @param pChunk The chunk to allocate it from.
1980 * @param hGVM The GVM handle of the VM requesting memory.
1981 * @param pPageDesc The page descriptor.
1982 */
1983static void gmmR0AllocatePage(PGMMCHUNK pChunk, uint32_t hGVM, PGMMPAGEDESC pPageDesc)
1984{
1985 /* update the chunk stats. */
1986 if (pChunk->hGVM == NIL_GVM_HANDLE)
1987 pChunk->hGVM = hGVM;
1988 Assert(pChunk->cFree);
1989 pChunk->cFree--;
1990 pChunk->cPrivate++;
1991
1992 /* unlink the first free page. */
1993 const uint32_t iPage = pChunk->iFreeHead;
1994 AssertReleaseMsg(iPage < RT_ELEMENTS(pChunk->aPages), ("%d\n", iPage));
1995 PGMMPAGE pPage = &pChunk->aPages[iPage];
1996 Assert(GMM_PAGE_IS_FREE(pPage));
1997 pChunk->iFreeHead = pPage->Free.iNext;
1998 Log3(("A pPage=%p iPage=%#x/%#x u2State=%d iFreeHead=%#x iNext=%#x\n",
1999 pPage, iPage, (pChunk->Core.Key << GMM_CHUNKID_SHIFT) | iPage,
2000 pPage->Common.u2State, pChunk->iFreeHead, pPage->Free.iNext));
2001
2002 /* make the page private. */
2003 pPage->u = 0;
2004 AssertCompile(GMM_PAGE_STATE_PRIVATE == 0);
2005 pPage->Private.hGVM = hGVM;
2006 AssertCompile(NIL_RTHCPHYS >= GMM_GCPHYS_LAST);
2007 AssertCompile(GMM_GCPHYS_UNSHAREABLE >= GMM_GCPHYS_LAST);
2008 if (pPageDesc->HCPhysGCPhys <= GMM_GCPHYS_LAST)
2009 pPage->Private.pfn = pPageDesc->HCPhysGCPhys >> PAGE_SHIFT;
2010 else
2011 pPage->Private.pfn = GMM_PAGE_PFN_UNSHAREABLE; /* unshareable / unassigned - same thing. */
2012
2013 /* update the page descriptor. */
2014 pPageDesc->HCPhysGCPhys = RTR0MemObjGetPagePhysAddr(pChunk->hMemObj, iPage);
2015 Assert(pPageDesc->HCPhysGCPhys != NIL_RTHCPHYS);
2016 pPageDesc->idPage = (pChunk->Core.Key << GMM_CHUNKID_SHIFT) | iPage;
2017 pPageDesc->idSharedPage = NIL_GMM_PAGEID;
2018}
2019
2020
2021/**
2022 * Picks the free pages from a chunk.
2023 *
2024 * @returns The new page descriptor table index.
2025 * @param pGMM Pointer to the GMM instance data.
2026 * @param hGVM The global VM handle.
2027 * @param pChunk The chunk.
2028 * @param iPage The current page descriptor table index.
2029 * @param cPages The total number of pages to allocate.
2030 * @param paPages The page descriptor table (input + ouput).
2031 */
2032static uint32_t gmmR0AllocatePagesFromChunk(PGMMCHUNK pChunk, uint16_t const hGVM, uint32_t iPage, uint32_t cPages,
2033 PGMMPAGEDESC paPages)
2034{
2035 PGMMCHUNKFREESET pSet = pChunk->pSet; Assert(pSet);
2036 gmmR0UnlinkChunk(pChunk);
2037
2038 for (; pChunk->cFree && iPage < cPages; iPage++)
2039 gmmR0AllocatePage(pChunk, hGVM, &paPages[iPage]);
2040
2041 gmmR0LinkChunk(pChunk, pSet);
2042 return iPage;
2043}
2044
2045
2046/**
2047 * Registers a new chunk of memory.
2048 *
2049 * This is called by both gmmR0AllocateOneChunk and GMMR0SeedChunk.
2050 *
2051 * @returns VBox status code. On success, the giant GMM lock will be held, the
2052 * caller must release it (ugly).
2053 * @param pGMM Pointer to the GMM instance.
2054 * @param pSet Pointer to the set.
2055 * @param MemObj The memory object for the chunk.
2056 * @param hGVM The affinity of the chunk. NIL_GVM_HANDLE for no
2057 * affinity.
2058 * @param fChunkFlags The chunk flags, GMM_CHUNK_FLAGS_XXX.
2059 * @param ppChunk Chunk address (out). Optional.
2060 *
2061 * @remarks The caller must not own the giant GMM mutex.
2062 * The giant GMM mutex will be acquired and returned acquired in
2063 * the success path. On failure, no locks will be held.
2064 */
2065static int gmmR0RegisterChunk(PGMM pGMM, PGMMCHUNKFREESET pSet, RTR0MEMOBJ MemObj, uint16_t hGVM, uint16_t fChunkFlags,
2066 PGMMCHUNK *ppChunk)
2067{
2068 Assert(pGMM->hMtxOwner != RTThreadNativeSelf());
2069 Assert(hGVM != NIL_GVM_HANDLE || pGMM->fBoundMemoryMode);
2070 Assert(fChunkFlags == 0 || fChunkFlags == GMM_CHUNK_FLAGS_LARGE_PAGE);
2071
2072 int rc;
2073 PGMMCHUNK pChunk = (PGMMCHUNK)RTMemAllocZ(sizeof(*pChunk));
2074 if (pChunk)
2075 {
2076 /*
2077 * Initialize it.
2078 */
2079 pChunk->hMemObj = MemObj;
2080 pChunk->cFree = GMM_CHUNK_NUM_PAGES;
2081 pChunk->hGVM = hGVM;
2082 /*pChunk->iFreeHead = 0;*/
2083 pChunk->idNumaNode = gmmR0GetCurrentNumaNodeId();
2084 pChunk->iChunkMtx = UINT8_MAX;
2085 pChunk->fFlags = fChunkFlags;
2086 for (unsigned iPage = 0; iPage < RT_ELEMENTS(pChunk->aPages) - 1; iPage++)
2087 {
2088 pChunk->aPages[iPage].Free.u2State = GMM_PAGE_STATE_FREE;
2089 pChunk->aPages[iPage].Free.iNext = iPage + 1;
2090 }
2091 pChunk->aPages[RT_ELEMENTS(pChunk->aPages) - 1].Free.u2State = GMM_PAGE_STATE_FREE;
2092 pChunk->aPages[RT_ELEMENTS(pChunk->aPages) - 1].Free.iNext = UINT16_MAX;
2093
2094 /*
2095 * Allocate a Chunk ID and insert it into the tree.
2096 * This has to be done behind the mutex of course.
2097 */
2098 rc = gmmR0MutexAcquire(pGMM);
2099 if (RT_SUCCESS(rc))
2100 {
2101 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
2102 {
2103 pChunk->Core.Key = gmmR0AllocateChunkId(pGMM);
2104 if ( pChunk->Core.Key != NIL_GMM_CHUNKID
2105 && pChunk->Core.Key <= GMM_CHUNKID_LAST
2106 && RTAvlU32Insert(&pGMM->pChunks, &pChunk->Core))
2107 {
2108 pGMM->cChunks++;
2109 RTListAppend(&pGMM->ChunkList, &pChunk->ListNode);
2110 gmmR0LinkChunk(pChunk, pSet);
2111 LogFlow(("gmmR0RegisterChunk: pChunk=%p id=%#x cChunks=%d\n", pChunk, pChunk->Core.Key, pGMM->cChunks));
2112
2113 if (ppChunk)
2114 *ppChunk = pChunk;
2115 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
2116 return VINF_SUCCESS;
2117 }
2118
2119 /* bail out */
2120 rc = VERR_GMM_CHUNK_INSERT;
2121 }
2122 else
2123 rc = VERR_GMM_IS_NOT_SANE;
2124 gmmR0MutexRelease(pGMM);
2125 }
2126
2127 RTMemFree(pChunk);
2128 }
2129 else
2130 rc = VERR_NO_MEMORY;
2131 return rc;
2132}
2133
2134
2135/**
2136 * Allocate a new chunk, immediately pick the requested pages from it, and adds
2137 * what's remaining to the specified free set.
2138 *
2139 * @note This will leave the giant mutex while allocating the new chunk!
2140 *
2141 * @returns VBox status code.
2142 * @param pGMM Pointer to the GMM instance data.
2143 * @param pGVM Pointer to the kernel-only VM instace data.
2144 * @param pSet Pointer to the free set.
2145 * @param cPages The number of pages requested.
2146 * @param paPages The page descriptor table (input + output).
2147 * @param piPage The pointer to the page descriptor table index
2148 * variable. This will be updated.
2149 */
2150static int gmmR0AllocateChunkNew(PGMM pGMM, PGVM pGVM, PGMMCHUNKFREESET pSet, uint32_t cPages,
2151 PGMMPAGEDESC paPages, uint32_t *piPage)
2152{
2153 gmmR0MutexRelease(pGMM);
2154
2155 RTR0MEMOBJ hMemObj;
2156 int rc = RTR0MemObjAllocPhysNC(&hMemObj, GMM_CHUNK_SIZE, NIL_RTHCPHYS);
2157 if (RT_SUCCESS(rc))
2158 {
2159/** @todo Duplicate gmmR0RegisterChunk here so we can avoid chaining up the
2160 * free pages first and then unchaining them right afterwards. Instead
2161 * do as much work as possible without holding the giant lock. */
2162 PGMMCHUNK pChunk;
2163 rc = gmmR0RegisterChunk(pGMM, pSet, hMemObj, pGVM->hSelf, 0 /*fChunkFlags*/, &pChunk);
2164 if (RT_SUCCESS(rc))
2165 {
2166 *piPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, *piPage, cPages, paPages);
2167 return VINF_SUCCESS;
2168 }
2169
2170 /* bail out */
2171 RTR0MemObjFree(hMemObj, false /* fFreeMappings */);
2172 }
2173
2174 int rc2 = gmmR0MutexAcquire(pGMM);
2175 AssertRCReturn(rc2, RT_FAILURE(rc) ? rc : rc2);
2176 return rc;
2177
2178}
2179
2180
2181/**
2182 * As a last restort we'll pick any page we can get.
2183 *
2184 * @returns The new page descriptor table index.
2185 * @param pSet The set to pick from.
2186 * @param pGVM Pointer to the global VM structure.
2187 * @param iPage The current page descriptor table index.
2188 * @param cPages The total number of pages to allocate.
2189 * @param paPages The page descriptor table (input + ouput).
2190 */
2191static uint32_t gmmR0AllocatePagesIndiscriminately(PGMMCHUNKFREESET pSet, PGVM pGVM,
2192 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2193{
2194 unsigned iList = RT_ELEMENTS(pSet->apLists);
2195 while (iList-- > 0)
2196 {
2197 PGMMCHUNK pChunk = pSet->apLists[iList];
2198 while (pChunk)
2199 {
2200 PGMMCHUNK pNext = pChunk->pFreeNext;
2201
2202 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2203 if (iPage >= cPages)
2204 return iPage;
2205
2206 pChunk = pNext;
2207 }
2208 }
2209 return iPage;
2210}
2211
2212
2213/**
2214 * Pick pages from empty chunks on the same NUMA node.
2215 *
2216 * @returns The new page descriptor table index.
2217 * @param pSet The set to pick from.
2218 * @param pGVM Pointer to the global VM structure.
2219 * @param iPage The current page descriptor table index.
2220 * @param cPages The total number of pages to allocate.
2221 * @param paPages The page descriptor table (input + ouput).
2222 */
2223static uint32_t gmmR0AllocatePagesFromEmptyChunksOnSameNode(PGMMCHUNKFREESET pSet, PGVM pGVM,
2224 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2225{
2226 PGMMCHUNK pChunk = pSet->apLists[GMM_CHUNK_FREE_SET_UNUSED_LIST];
2227 if (pChunk)
2228 {
2229 uint16_t const idNumaNode = gmmR0GetCurrentNumaNodeId();
2230 while (pChunk)
2231 {
2232 PGMMCHUNK pNext = pChunk->pFreeNext;
2233
2234 if (pChunk->idNumaNode == idNumaNode)
2235 {
2236 pChunk->hGVM = pGVM->hSelf;
2237 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2238 if (iPage >= cPages)
2239 {
2240 pGVM->gmm.s.idLastChunkHint = pChunk->cFree ? pChunk->Core.Key : NIL_GMM_CHUNKID;
2241 return iPage;
2242 }
2243 }
2244
2245 pChunk = pNext;
2246 }
2247 }
2248 return iPage;
2249}
2250
2251
2252/**
2253 * Pick pages from non-empty chunks on the same NUMA node.
2254 *
2255 * @returns The new page descriptor table index.
2256 * @param pSet The set to pick from.
2257 * @param pGVM Pointer to the global VM structure.
2258 * @param iPage The current page descriptor table index.
2259 * @param cPages The total number of pages to allocate.
2260 * @param paPages The page descriptor table (input + ouput).
2261 */
2262static uint32_t gmmR0AllocatePagesFromSameNode(PGMMCHUNKFREESET pSet, PGVM pGVM,
2263 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2264{
2265 /** @todo start by picking from chunks with about the right size first? */
2266 uint16_t const idNumaNode = gmmR0GetCurrentNumaNodeId();
2267 unsigned iList = GMM_CHUNK_FREE_SET_UNUSED_LIST;
2268 while (iList-- > 0)
2269 {
2270 PGMMCHUNK pChunk = pSet->apLists[iList];
2271 while (pChunk)
2272 {
2273 PGMMCHUNK pNext = pChunk->pFreeNext;
2274
2275 if (pChunk->idNumaNode == idNumaNode)
2276 {
2277 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2278 if (iPage >= cPages)
2279 {
2280 pGVM->gmm.s.idLastChunkHint = pChunk->cFree ? pChunk->Core.Key : NIL_GMM_CHUNKID;
2281 return iPage;
2282 }
2283 }
2284
2285 pChunk = pNext;
2286 }
2287 }
2288 return iPage;
2289}
2290
2291
2292/**
2293 * Pick pages that are in chunks already associated with the VM.
2294 *
2295 * @returns The new page descriptor table index.
2296 * @param pGMM Pointer to the GMM instance data.
2297 * @param pGVM Pointer to the global VM structure.
2298 * @param pSet The set to pick from.
2299 * @param iPage The current page descriptor table index.
2300 * @param cPages The total number of pages to allocate.
2301 * @param paPages The page descriptor table (input + ouput).
2302 */
2303static uint32_t gmmR0AllocatePagesAssociatedWithVM(PGMM pGMM, PGVM pGVM, PGMMCHUNKFREESET pSet,
2304 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2305{
2306 uint16_t const hGVM = pGVM->hSelf;
2307
2308 /* Hint. */
2309 if (pGVM->gmm.s.idLastChunkHint != NIL_GMM_CHUNKID)
2310 {
2311 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, pGVM->gmm.s.idLastChunkHint);
2312 if (pChunk && pChunk->cFree)
2313 {
2314 iPage = gmmR0AllocatePagesFromChunk(pChunk, hGVM, iPage, cPages, paPages);
2315 if (iPage >= cPages)
2316 return iPage;
2317 }
2318 }
2319
2320 /* Scan. */
2321 for (unsigned iList = 0; iList < RT_ELEMENTS(pSet->apLists); iList++)
2322 {
2323 PGMMCHUNK pChunk = pSet->apLists[iList];
2324 while (pChunk)
2325 {
2326 PGMMCHUNK pNext = pChunk->pFreeNext;
2327
2328 if (pChunk->hGVM == hGVM)
2329 {
2330 iPage = gmmR0AllocatePagesFromChunk(pChunk, hGVM, iPage, cPages, paPages);
2331 if (iPage >= cPages)
2332 {
2333 pGVM->gmm.s.idLastChunkHint = pChunk->cFree ? pChunk->Core.Key : NIL_GMM_CHUNKID;
2334 return iPage;
2335 }
2336 }
2337
2338 pChunk = pNext;
2339 }
2340 }
2341 return iPage;
2342}
2343
2344
2345
2346/**
2347 * Pick pages in bound memory mode.
2348 *
2349 * @returns The new page descriptor table index.
2350 * @param pGVM Pointer to the global VM structure.
2351 * @param iPage The current page descriptor table index.
2352 * @param cPages The total number of pages to allocate.
2353 * @param paPages The page descriptor table (input + ouput).
2354 */
2355static uint32_t gmmR0AllocatePagesInBoundMode(PGVM pGVM, uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2356{
2357 for (unsigned iList = 0; iList < RT_ELEMENTS(pGVM->gmm.s.Private.apLists); iList++)
2358 {
2359 PGMMCHUNK pChunk = pGVM->gmm.s.Private.apLists[iList];
2360 while (pChunk)
2361 {
2362 Assert(pChunk->hGVM == pGVM->hSelf);
2363 PGMMCHUNK pNext = pChunk->pFreeNext;
2364 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2365 if (iPage >= cPages)
2366 return iPage;
2367 pChunk = pNext;
2368 }
2369 }
2370 return iPage;
2371}
2372
2373
2374/**
2375 * Checks if we should start picking pages from chunks of other VMs.
2376 *
2377 * @returns @c true if we should, @c false if we should first try allocate more
2378 * chunks.
2379 */
2380static bool gmmR0ShouldAllocatePagesInOtherChunks(PGVM pGVM)
2381{
2382 /*
2383 * Don't allocate a new chunk if we're
2384 */
2385 uint64_t cPgReserved = pGVM->gmm.s.Stats.Reserved.cBasePages
2386 + pGVM->gmm.s.Stats.Reserved.cFixedPages
2387 - pGVM->gmm.s.Stats.cBalloonedPages
2388 /** @todo what about shared pages? */;
2389 uint64_t cPgAllocated = pGVM->gmm.s.Stats.Allocated.cBasePages
2390 + pGVM->gmm.s.Stats.Allocated.cFixedPages;
2391 uint64_t cPgDelta = cPgReserved - cPgAllocated;
2392 if (cPgDelta < GMM_CHUNK_NUM_PAGES * 4)
2393 return true;
2394 /** @todo make the threshold configurable, also test the code to see if
2395 * this ever kicks in (we might be reserving too much or smth). */
2396
2397 /*
2398 * Check how close we're to the max memory limit and how many fragments
2399 * there are?...
2400 */
2401 /** @todo. */
2402
2403 return false;
2404}
2405
2406
2407/**
2408 * Common worker for GMMR0AllocateHandyPages and GMMR0AllocatePages.
2409 *
2410 * @returns VBox status code:
2411 * @retval VINF_SUCCESS on success.
2412 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk or
2413 * gmmR0AllocateMoreChunks is necessary.
2414 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
2415 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
2416 * that is we're trying to allocate more than we've reserved.
2417 *
2418 * @param pGMM Pointer to the GMM instance data.
2419 * @param pGVM Pointer to the VM.
2420 * @param cPages The number of pages to allocate.
2421 * @param paPages Pointer to the page descriptors.
2422 * See GMMPAGEDESC for details on what is expected on input.
2423 * @param enmAccount The account to charge.
2424 *
2425 * @remarks Call takes the giant GMM lock.
2426 */
2427static int gmmR0AllocatePagesNew(PGMM pGMM, PGVM pGVM, uint32_t cPages, PGMMPAGEDESC paPages, GMMACCOUNT enmAccount)
2428{
2429 Assert(pGMM->hMtxOwner == RTThreadNativeSelf());
2430
2431 /*
2432 * Check allocation limits.
2433 */
2434 if (RT_UNLIKELY(pGMM->cAllocatedPages + cPages > pGMM->cMaxPages))
2435 return VERR_GMM_HIT_GLOBAL_LIMIT;
2436
2437 switch (enmAccount)
2438 {
2439 case GMMACCOUNT_BASE:
2440 if (RT_UNLIKELY( pGVM->gmm.s.Stats.Allocated.cBasePages + pGVM->gmm.s.Stats.cBalloonedPages + cPages
2441 > pGVM->gmm.s.Stats.Reserved.cBasePages))
2442 {
2443 Log(("gmmR0AllocatePages:Base: Reserved=%#llx Allocated+Ballooned+Requested=%#llx+%#llx+%#x!\n",
2444 pGVM->gmm.s.Stats.Reserved.cBasePages, pGVM->gmm.s.Stats.Allocated.cBasePages,
2445 pGVM->gmm.s.Stats.cBalloonedPages, cPages));
2446 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
2447 }
2448 break;
2449 case GMMACCOUNT_SHADOW:
2450 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cShadowPages + cPages > pGVM->gmm.s.Stats.Reserved.cShadowPages))
2451 {
2452 Log(("gmmR0AllocatePages:Shadow: Reserved=%#x Allocated+Requested=%#x+%#x!\n",
2453 pGVM->gmm.s.Stats.Reserved.cShadowPages, pGVM->gmm.s.Stats.Allocated.cShadowPages, cPages));
2454 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
2455 }
2456 break;
2457 case GMMACCOUNT_FIXED:
2458 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cFixedPages + cPages > pGVM->gmm.s.Stats.Reserved.cFixedPages))
2459 {
2460 Log(("gmmR0AllocatePages:Fixed: Reserved=%#x Allocated+Requested=%#x+%#x!\n",
2461 pGVM->gmm.s.Stats.Reserved.cFixedPages, pGVM->gmm.s.Stats.Allocated.cFixedPages, cPages));
2462 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
2463 }
2464 break;
2465 default:
2466 AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
2467 }
2468
2469 /*
2470 * If we're in legacy memory mode, it's easy to figure if we have
2471 * sufficient number of pages up-front.
2472 */
2473 if ( pGMM->fLegacyAllocationMode
2474 && pGVM->gmm.s.Private.cFreePages < cPages)
2475 {
2476 Assert(pGMM->fBoundMemoryMode);
2477 return VERR_GMM_SEED_ME;
2478 }
2479
2480 /*
2481 * Update the accounts before we proceed because we might be leaving the
2482 * protection of the global mutex and thus run the risk of permitting
2483 * too much memory to be allocated.
2484 */
2485 switch (enmAccount)
2486 {
2487 case GMMACCOUNT_BASE: pGVM->gmm.s.Stats.Allocated.cBasePages += cPages; break;
2488 case GMMACCOUNT_SHADOW: pGVM->gmm.s.Stats.Allocated.cShadowPages += cPages; break;
2489 case GMMACCOUNT_FIXED: pGVM->gmm.s.Stats.Allocated.cFixedPages += cPages; break;
2490 default: AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
2491 }
2492 pGVM->gmm.s.Stats.cPrivatePages += cPages;
2493 pGMM->cAllocatedPages += cPages;
2494
2495 /*
2496 * Part two of it's-easy-in-legacy-memory-mode.
2497 */
2498 uint32_t iPage = 0;
2499 if (pGMM->fLegacyAllocationMode)
2500 {
2501 iPage = gmmR0AllocatePagesInBoundMode(pGVM, iPage, cPages, paPages);
2502 AssertReleaseReturn(iPage == cPages, VERR_GMM_ALLOC_PAGES_IPE);
2503 return VINF_SUCCESS;
2504 }
2505
2506 /*
2507 * Bound mode is also relatively straightforward.
2508 */
2509 int rc = VINF_SUCCESS;
2510 if (pGMM->fBoundMemoryMode)
2511 {
2512 iPage = gmmR0AllocatePagesInBoundMode(pGVM, iPage, cPages, paPages);
2513 if (iPage < cPages)
2514 do
2515 rc = gmmR0AllocateChunkNew(pGMM, pGVM, &pGVM->gmm.s.Private, cPages, paPages, &iPage);
2516 while (iPage < cPages && RT_SUCCESS(rc));
2517 }
2518 /*
2519 * Shared mode is trickier as we should try archive the same locality as
2520 * in bound mode, but smartly make use of non-full chunks allocated by
2521 * other VMs if we're low on memory.
2522 */
2523 else
2524 {
2525 /* Pick the most optimal pages first. */
2526 iPage = gmmR0AllocatePagesAssociatedWithVM(pGMM, pGVM, &pGMM->PrivateX, iPage, cPages, paPages);
2527 if (iPage < cPages)
2528 {
2529 /* Maybe we should try getting pages from chunks "belonging" to
2530 other VMs before allocating more chunks? */
2531 if (gmmR0ShouldAllocatePagesInOtherChunks(pGVM))
2532 iPage = gmmR0AllocatePagesFromSameNode(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2533
2534 /* Allocate memory from empty chunks. */
2535 if (iPage < cPages)
2536 iPage = gmmR0AllocatePagesFromEmptyChunksOnSameNode(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2537
2538 /* Grab empty shared chunks. */
2539 if (iPage < cPages)
2540 iPage = gmmR0AllocatePagesFromEmptyChunksOnSameNode(&pGMM->Shared, pGVM, iPage, cPages, paPages);
2541
2542 /*
2543 * Ok, try allocate new chunks.
2544 */
2545 if (iPage < cPages)
2546 {
2547 do
2548 rc = gmmR0AllocateChunkNew(pGMM, pGVM, &pGMM->PrivateX, cPages, paPages, &iPage);
2549 while (iPage < cPages && RT_SUCCESS(rc));
2550
2551 /* If the host is out of memory, take whatever we can get. */
2552 if ( (rc == VERR_NO_MEMORY || rc == VERR_NO_PHYS_MEMORY)
2553 && pGMM->PrivateX.cFreePages + pGMM->Shared.cFreePages >= cPages - iPage)
2554 {
2555 iPage = gmmR0AllocatePagesIndiscriminately(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2556 if (iPage < cPages)
2557 iPage = gmmR0AllocatePagesIndiscriminately(&pGMM->Shared, pGVM, iPage, cPages, paPages);
2558 AssertRelease(iPage == cPages);
2559 rc = VINF_SUCCESS;
2560 }
2561 }
2562 }
2563 }
2564
2565 /*
2566 * Clean up on failure. Since this is bound to be a low-memory condition
2567 * we will give back any empty chunks that might be hanging around.
2568 */
2569 if (RT_FAILURE(rc))
2570 {
2571 /* Update the statistics. */
2572 pGVM->gmm.s.Stats.cPrivatePages -= cPages;
2573 pGMM->cAllocatedPages -= cPages - iPage;
2574 switch (enmAccount)
2575 {
2576 case GMMACCOUNT_BASE: pGVM->gmm.s.Stats.Allocated.cBasePages -= cPages; break;
2577 case GMMACCOUNT_SHADOW: pGVM->gmm.s.Stats.Allocated.cShadowPages -= cPages; break;
2578 case GMMACCOUNT_FIXED: pGVM->gmm.s.Stats.Allocated.cFixedPages -= cPages; break;
2579 default: AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
2580 }
2581
2582 /* Release the pages. */
2583 while (iPage-- > 0)
2584 {
2585 uint32_t idPage = paPages[iPage].idPage;
2586 PGMMPAGE pPage = gmmR0GetPage(pGMM, idPage);
2587 if (RT_LIKELY(pPage))
2588 {
2589 Assert(GMM_PAGE_IS_PRIVATE(pPage));
2590 Assert(pPage->Private.hGVM == pGVM->hSelf);
2591 gmmR0FreePrivatePage(pGMM, pGVM, idPage, pPage);
2592 }
2593 else
2594 AssertMsgFailed(("idPage=%#x\n", idPage));
2595
2596 paPages[iPage].idPage = NIL_GMM_PAGEID;
2597 paPages[iPage].idSharedPage = NIL_GMM_PAGEID;
2598 paPages[iPage].HCPhysGCPhys = NIL_RTHCPHYS;
2599 }
2600
2601 /* Free empty chunks. */
2602 /** @todo */
2603
2604 /* return the fail status on failure */
2605 return rc;
2606 }
2607 return VINF_SUCCESS;
2608}
2609
2610
2611/**
2612 * Updates the previous allocations and allocates more pages.
2613 *
2614 * The handy pages are always taken from the 'base' memory account.
2615 * The allocated pages are not cleared and will contains random garbage.
2616 *
2617 * @returns VBox status code:
2618 * @retval VINF_SUCCESS on success.
2619 * @retval VERR_NOT_OWNER if the caller is not an EMT.
2620 * @retval VERR_GMM_PAGE_NOT_FOUND if one of the pages to update wasn't found.
2621 * @retval VERR_GMM_PAGE_NOT_PRIVATE if one of the pages to update wasn't a
2622 * private page.
2623 * @retval VERR_GMM_PAGE_NOT_SHARED if one of the pages to update wasn't a
2624 * shared page.
2625 * @retval VERR_GMM_NOT_PAGE_OWNER if one of the pages to be updated wasn't
2626 * owned by the VM.
2627 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk is necessary.
2628 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
2629 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
2630 * that is we're trying to allocate more than we've reserved.
2631 *
2632 * @param pVM Pointer to the VM.
2633 * @param idCpu The VCPU id.
2634 * @param cPagesToUpdate The number of pages to update (starting from the head).
2635 * @param cPagesToAlloc The number of pages to allocate (starting from the head).
2636 * @param paPages The array of page descriptors.
2637 * See GMMPAGEDESC for details on what is expected on input.
2638 * @thread EMT.
2639 */
2640GMMR0DECL(int) GMMR0AllocateHandyPages(PVM pVM, VMCPUID idCpu, uint32_t cPagesToUpdate, uint32_t cPagesToAlloc, PGMMPAGEDESC paPages)
2641{
2642 LogFlow(("GMMR0AllocateHandyPages: pVM=%p cPagesToUpdate=%#x cPagesToAlloc=%#x paPages=%p\n",
2643 pVM, cPagesToUpdate, cPagesToAlloc, paPages));
2644
2645 /*
2646 * Validate, get basics and take the semaphore.
2647 * (This is a relatively busy path, so make predictions where possible.)
2648 */
2649 PGMM pGMM;
2650 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
2651 PGVM pGVM;
2652 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
2653 if (RT_FAILURE(rc))
2654 return rc;
2655
2656 AssertPtrReturn(paPages, VERR_INVALID_PARAMETER);
2657 AssertMsgReturn( (cPagesToUpdate && cPagesToUpdate < 1024)
2658 || (cPagesToAlloc && cPagesToAlloc < 1024),
2659 ("cPagesToUpdate=%#x cPagesToAlloc=%#x\n", cPagesToUpdate, cPagesToAlloc),
2660 VERR_INVALID_PARAMETER);
2661
2662 unsigned iPage = 0;
2663 for (; iPage < cPagesToUpdate; iPage++)
2664 {
2665 AssertMsgReturn( ( paPages[iPage].HCPhysGCPhys <= GMM_GCPHYS_LAST
2666 && !(paPages[iPage].HCPhysGCPhys & PAGE_OFFSET_MASK))
2667 || paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS
2668 || paPages[iPage].HCPhysGCPhys == GMM_GCPHYS_UNSHAREABLE,
2669 ("#%#x: %RHp\n", iPage, paPages[iPage].HCPhysGCPhys),
2670 VERR_INVALID_PARAMETER);
2671 AssertMsgReturn( paPages[iPage].idPage <= GMM_PAGEID_LAST
2672 /*|| paPages[iPage].idPage == NIL_GMM_PAGEID*/,
2673 ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
2674 AssertMsgReturn( paPages[iPage].idPage <= GMM_PAGEID_LAST
2675 /*|| paPages[iPage].idSharedPage == NIL_GMM_PAGEID*/,
2676 ("#%#x: %#x\n", iPage, paPages[iPage].idSharedPage), VERR_INVALID_PARAMETER);
2677 }
2678
2679 for (; iPage < cPagesToAlloc; iPage++)
2680 {
2681 AssertMsgReturn(paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS, ("#%#x: %RHp\n", iPage, paPages[iPage].HCPhysGCPhys), VERR_INVALID_PARAMETER);
2682 AssertMsgReturn(paPages[iPage].idPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
2683 AssertMsgReturn(paPages[iPage].idSharedPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idSharedPage), VERR_INVALID_PARAMETER);
2684 }
2685
2686 gmmR0MutexAcquire(pGMM);
2687 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
2688 {
2689 /* No allocations before the initial reservation has been made! */
2690 if (RT_LIKELY( pGVM->gmm.s.Stats.Reserved.cBasePages
2691 && pGVM->gmm.s.Stats.Reserved.cFixedPages
2692 && pGVM->gmm.s.Stats.Reserved.cShadowPages))
2693 {
2694 /*
2695 * Perform the updates.
2696 * Stop on the first error.
2697 */
2698 for (iPage = 0; iPage < cPagesToUpdate; iPage++)
2699 {
2700 if (paPages[iPage].idPage != NIL_GMM_PAGEID)
2701 {
2702 PGMMPAGE pPage = gmmR0GetPage(pGMM, paPages[iPage].idPage);
2703 if (RT_LIKELY(pPage))
2704 {
2705 if (RT_LIKELY(GMM_PAGE_IS_PRIVATE(pPage)))
2706 {
2707 if (RT_LIKELY(pPage->Private.hGVM == pGVM->hSelf))
2708 {
2709 AssertCompile(NIL_RTHCPHYS > GMM_GCPHYS_LAST && GMM_GCPHYS_UNSHAREABLE > GMM_GCPHYS_LAST);
2710 if (RT_LIKELY(paPages[iPage].HCPhysGCPhys <= GMM_GCPHYS_LAST))
2711 pPage->Private.pfn = paPages[iPage].HCPhysGCPhys >> PAGE_SHIFT;
2712 else if (paPages[iPage].HCPhysGCPhys == GMM_GCPHYS_UNSHAREABLE)
2713 pPage->Private.pfn = GMM_PAGE_PFN_UNSHAREABLE;
2714 /* else: NIL_RTHCPHYS nothing */
2715
2716 paPages[iPage].idPage = NIL_GMM_PAGEID;
2717 paPages[iPage].HCPhysGCPhys = NIL_RTHCPHYS;
2718 }
2719 else
2720 {
2721 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not owner! hGVM=%#x hSelf=%#x\n",
2722 iPage, paPages[iPage].idPage, pPage->Private.hGVM, pGVM->hSelf));
2723 rc = VERR_GMM_NOT_PAGE_OWNER;
2724 break;
2725 }
2726 }
2727 else
2728 {
2729 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not private! %.*Rhxs (type %d)\n", iPage, paPages[iPage].idPage, sizeof(*pPage), pPage, pPage->Common.u2State));
2730 rc = VERR_GMM_PAGE_NOT_PRIVATE;
2731 break;
2732 }
2733 }
2734 else
2735 {
2736 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not found! (private)\n", iPage, paPages[iPage].idPage));
2737 rc = VERR_GMM_PAGE_NOT_FOUND;
2738 break;
2739 }
2740 }
2741
2742 if (paPages[iPage].idSharedPage != NIL_GMM_PAGEID)
2743 {
2744 PGMMPAGE pPage = gmmR0GetPage(pGMM, paPages[iPage].idSharedPage);
2745 if (RT_LIKELY(pPage))
2746 {
2747 if (RT_LIKELY(GMM_PAGE_IS_SHARED(pPage)))
2748 {
2749 AssertCompile(NIL_RTHCPHYS > GMM_GCPHYS_LAST && GMM_GCPHYS_UNSHAREABLE > GMM_GCPHYS_LAST);
2750 Assert(pPage->Shared.cRefs);
2751 Assert(pGVM->gmm.s.Stats.cSharedPages);
2752 Assert(pGVM->gmm.s.Stats.Allocated.cBasePages);
2753
2754 Log(("GMMR0AllocateHandyPages: free shared page %x cRefs=%d\n", paPages[iPage].idSharedPage, pPage->Shared.cRefs));
2755 pGVM->gmm.s.Stats.cSharedPages--;
2756 pGVM->gmm.s.Stats.Allocated.cBasePages--;
2757 if (!--pPage->Shared.cRefs)
2758 gmmR0FreeSharedPage(pGMM, pGVM, paPages[iPage].idSharedPage, pPage);
2759 else
2760 {
2761 Assert(pGMM->cDuplicatePages);
2762 pGMM->cDuplicatePages--;
2763 }
2764
2765 paPages[iPage].idSharedPage = NIL_GMM_PAGEID;
2766 }
2767 else
2768 {
2769 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not shared!\n", iPage, paPages[iPage].idSharedPage));
2770 rc = VERR_GMM_PAGE_NOT_SHARED;
2771 break;
2772 }
2773 }
2774 else
2775 {
2776 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not found! (shared)\n", iPage, paPages[iPage].idSharedPage));
2777 rc = VERR_GMM_PAGE_NOT_FOUND;
2778 break;
2779 }
2780 }
2781 } /* for each page to update */
2782
2783 if (RT_SUCCESS(rc))
2784 {
2785#if defined(VBOX_STRICT) && 0 /** @todo re-test this later. Appeared to be a PGM init bug. */
2786 for (iPage = 0; iPage < cPagesToAlloc; iPage++)
2787 {
2788 Assert(paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS);
2789 Assert(paPages[iPage].idPage == NIL_GMM_PAGEID);
2790 Assert(paPages[iPage].idSharedPage == NIL_GMM_PAGEID);
2791 }
2792#endif
2793
2794 /*
2795 * Join paths with GMMR0AllocatePages for the allocation.
2796 * Note! gmmR0AllocateMoreChunks may leave the protection of the mutex!
2797 */
2798 rc = gmmR0AllocatePagesNew(pGMM, pGVM, cPagesToAlloc, paPages, GMMACCOUNT_BASE);
2799 }
2800 }
2801 else
2802 rc = VERR_WRONG_ORDER;
2803 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
2804 }
2805 else
2806 rc = VERR_GMM_IS_NOT_SANE;
2807 gmmR0MutexRelease(pGMM);
2808 LogFlow(("GMMR0AllocateHandyPages: returns %Rrc\n", rc));
2809 return rc;
2810}
2811
2812
2813/**
2814 * Allocate one or more pages.
2815 *
2816 * This is typically used for ROMs and MMIO2 (VRAM) during VM creation.
2817 * The allocated pages are not cleared and will contain random garbage.
2818 *
2819 * @returns VBox status code:
2820 * @retval VINF_SUCCESS on success.
2821 * @retval VERR_NOT_OWNER if the caller is not an EMT.
2822 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk is necessary.
2823 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
2824 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
2825 * that is we're trying to allocate more than we've reserved.
2826 *
2827 * @param pVM Pointer to the VM.
2828 * @param idCpu The VCPU id.
2829 * @param cPages The number of pages to allocate.
2830 * @param paPages Pointer to the page descriptors.
2831 * See GMMPAGEDESC for details on what is expected on input.
2832 * @param enmAccount The account to charge.
2833 *
2834 * @thread EMT.
2835 */
2836GMMR0DECL(int) GMMR0AllocatePages(PVM pVM, VMCPUID idCpu, uint32_t cPages, PGMMPAGEDESC paPages, GMMACCOUNT enmAccount)
2837{
2838 LogFlow(("GMMR0AllocatePages: pVM=%p cPages=%#x paPages=%p enmAccount=%d\n", pVM, cPages, paPages, enmAccount));
2839
2840 /*
2841 * Validate, get basics and take the semaphore.
2842 */
2843 PGMM pGMM;
2844 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
2845 PGVM pGVM;
2846 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
2847 if (RT_FAILURE(rc))
2848 return rc;
2849
2850 AssertPtrReturn(paPages, VERR_INVALID_PARAMETER);
2851 AssertMsgReturn(enmAccount > GMMACCOUNT_INVALID && enmAccount < GMMACCOUNT_END, ("%d\n", enmAccount), VERR_INVALID_PARAMETER);
2852 AssertMsgReturn(cPages > 0 && cPages < RT_BIT(32 - PAGE_SHIFT), ("%#x\n", cPages), VERR_INVALID_PARAMETER);
2853
2854 for (unsigned iPage = 0; iPage < cPages; iPage++)
2855 {
2856 AssertMsgReturn( paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS
2857 || paPages[iPage].HCPhysGCPhys == GMM_GCPHYS_UNSHAREABLE
2858 || ( enmAccount == GMMACCOUNT_BASE
2859 && paPages[iPage].HCPhysGCPhys <= GMM_GCPHYS_LAST
2860 && !(paPages[iPage].HCPhysGCPhys & PAGE_OFFSET_MASK)),
2861 ("#%#x: %RHp enmAccount=%d\n", iPage, paPages[iPage].HCPhysGCPhys, enmAccount),
2862 VERR_INVALID_PARAMETER);
2863 AssertMsgReturn(paPages[iPage].idPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
2864 AssertMsgReturn(paPages[iPage].idSharedPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idSharedPage), VERR_INVALID_PARAMETER);
2865 }
2866
2867 gmmR0MutexAcquire(pGMM);
2868 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
2869 {
2870
2871 /* No allocations before the initial reservation has been made! */
2872 if (RT_LIKELY( pGVM->gmm.s.Stats.Reserved.cBasePages
2873 && pGVM->gmm.s.Stats.Reserved.cFixedPages
2874 && pGVM->gmm.s.Stats.Reserved.cShadowPages))
2875 rc = gmmR0AllocatePagesNew(pGMM, pGVM, cPages, paPages, enmAccount);
2876 else
2877 rc = VERR_WRONG_ORDER;
2878 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
2879 }
2880 else
2881 rc = VERR_GMM_IS_NOT_SANE;
2882 gmmR0MutexRelease(pGMM);
2883 LogFlow(("GMMR0AllocatePages: returns %Rrc\n", rc));
2884 return rc;
2885}
2886
2887
2888/**
2889 * VMMR0 request wrapper for GMMR0AllocatePages.
2890 *
2891 * @returns see GMMR0AllocatePages.
2892 * @param pVM Pointer to the VM.
2893 * @param idCpu The VCPU id.
2894 * @param pReq Pointer to the request packet.
2895 */
2896GMMR0DECL(int) GMMR0AllocatePagesReq(PVM pVM, VMCPUID idCpu, PGMMALLOCATEPAGESREQ pReq)
2897{
2898 /*
2899 * Validate input and pass it on.
2900 */
2901 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
2902 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
2903 AssertMsgReturn(pReq->Hdr.cbReq >= RT_UOFFSETOF(GMMALLOCATEPAGESREQ, aPages[0]),
2904 ("%#x < %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF(GMMALLOCATEPAGESREQ, aPages[0])),
2905 VERR_INVALID_PARAMETER);
2906 AssertMsgReturn(pReq->Hdr.cbReq == RT_UOFFSETOF(GMMALLOCATEPAGESREQ, aPages[pReq->cPages]),
2907 ("%#x != %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF(GMMALLOCATEPAGESREQ, aPages[pReq->cPages])),
2908 VERR_INVALID_PARAMETER);
2909
2910 return GMMR0AllocatePages(pVM, idCpu, pReq->cPages, &pReq->aPages[0], pReq->enmAccount);
2911}
2912
2913
2914/**
2915 * Allocate a large page to represent guest RAM
2916 *
2917 * The allocated pages are not cleared and will contains random garbage.
2918 *
2919 * @returns VBox status code:
2920 * @retval VINF_SUCCESS on success.
2921 * @retval VERR_NOT_OWNER if the caller is not an EMT.
2922 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk is necessary.
2923 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
2924 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
2925 * that is we're trying to allocate more than we've reserved.
2926 * @returns see GMMR0AllocatePages.
2927 * @param pVM Pointer to the VM.
2928 * @param idCpu The VCPU id.
2929 * @param cbPage Large page size.
2930 */
2931GMMR0DECL(int) GMMR0AllocateLargePage(PVM pVM, VMCPUID idCpu, uint32_t cbPage, uint32_t *pIdPage, RTHCPHYS *pHCPhys)
2932{
2933 LogFlow(("GMMR0AllocateLargePage: pVM=%p cbPage=%x\n", pVM, cbPage));
2934
2935 AssertReturn(cbPage == GMM_CHUNK_SIZE, VERR_INVALID_PARAMETER);
2936 AssertPtrReturn(pIdPage, VERR_INVALID_PARAMETER);
2937 AssertPtrReturn(pHCPhys, VERR_INVALID_PARAMETER);
2938
2939 /*
2940 * Validate, get basics and take the semaphore.
2941 */
2942 PGMM pGMM;
2943 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
2944 PGVM pGVM;
2945 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
2946 if (RT_FAILURE(rc))
2947 return rc;
2948
2949 /* Not supported in legacy mode where we allocate the memory in ring 3 and lock it in ring 0. */
2950 if (pGMM->fLegacyAllocationMode)
2951 return VERR_NOT_SUPPORTED;
2952
2953 *pHCPhys = NIL_RTHCPHYS;
2954 *pIdPage = NIL_GMM_PAGEID;
2955
2956 gmmR0MutexAcquire(pGMM);
2957 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
2958 {
2959 const unsigned cPages = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
2960 if (RT_UNLIKELY( pGVM->gmm.s.Stats.Allocated.cBasePages + pGVM->gmm.s.Stats.cBalloonedPages + cPages
2961 > pGVM->gmm.s.Stats.Reserved.cBasePages))
2962 {
2963 Log(("GMMR0AllocateLargePage: Reserved=%#llx Allocated+Requested=%#llx+%#x!\n",
2964 pGVM->gmm.s.Stats.Reserved.cBasePages, pGVM->gmm.s.Stats.Allocated.cBasePages, cPages));
2965 gmmR0MutexRelease(pGMM);
2966 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
2967 }
2968
2969 /*
2970 * Allocate a new large page chunk.
2971 *
2972 * Note! We leave the giant GMM lock temporarily as the allocation might
2973 * take a long time. gmmR0RegisterChunk will retake it (ugly).
2974 */
2975 AssertCompile(GMM_CHUNK_SIZE == _2M);
2976 gmmR0MutexRelease(pGMM);
2977
2978 RTR0MEMOBJ hMemObj;
2979 rc = RTR0MemObjAllocPhysEx(&hMemObj, GMM_CHUNK_SIZE, NIL_RTHCPHYS, GMM_CHUNK_SIZE);
2980 if (RT_SUCCESS(rc))
2981 {
2982 PGMMCHUNKFREESET pSet = pGMM->fBoundMemoryMode ? &pGVM->gmm.s.Private : &pGMM->PrivateX;
2983 PGMMCHUNK pChunk;
2984 rc = gmmR0RegisterChunk(pGMM, pSet, hMemObj, pGVM->hSelf, GMM_CHUNK_FLAGS_LARGE_PAGE, &pChunk);
2985 if (RT_SUCCESS(rc))
2986 {
2987 /*
2988 * Allocate all the pages in the chunk.
2989 */
2990 /* Unlink the new chunk from the free list. */
2991 gmmR0UnlinkChunk(pChunk);
2992
2993 /** @todo rewrite this to skip the looping. */
2994 /* Allocate all pages. */
2995 GMMPAGEDESC PageDesc;
2996 gmmR0AllocatePage(pChunk, pGVM->hSelf, &PageDesc);
2997
2998 /* Return the first page as we'll use the whole chunk as one big page. */
2999 *pIdPage = PageDesc.idPage;
3000 *pHCPhys = PageDesc.HCPhysGCPhys;
3001
3002 for (unsigned i = 1; i < cPages; i++)
3003 gmmR0AllocatePage(pChunk, pGVM->hSelf, &PageDesc);
3004
3005 /* Update accounting. */
3006 pGVM->gmm.s.Stats.Allocated.cBasePages += cPages;
3007 pGVM->gmm.s.Stats.cPrivatePages += cPages;
3008 pGMM->cAllocatedPages += cPages;
3009
3010 gmmR0LinkChunk(pChunk, pSet);
3011 gmmR0MutexRelease(pGMM);
3012 }
3013 else
3014 RTR0MemObjFree(hMemObj, false /* fFreeMappings */);
3015 }
3016 }
3017 else
3018 {
3019 gmmR0MutexRelease(pGMM);
3020 rc = VERR_GMM_IS_NOT_SANE;
3021 }
3022
3023 LogFlow(("GMMR0AllocateLargePage: returns %Rrc\n", rc));
3024 return rc;
3025}
3026
3027
3028/**
3029 * Free a large page.
3030 *
3031 * @returns VBox status code:
3032 * @param pVM Pointer to the VM.
3033 * @param idCpu The VCPU id.
3034 * @param idPage The large page id.
3035 */
3036GMMR0DECL(int) GMMR0FreeLargePage(PVM pVM, VMCPUID idCpu, uint32_t idPage)
3037{
3038 LogFlow(("GMMR0FreeLargePage: pVM=%p idPage=%x\n", pVM, idPage));
3039
3040 /*
3041 * Validate, get basics and take the semaphore.
3042 */
3043 PGMM pGMM;
3044 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3045 PGVM pGVM;
3046 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
3047 if (RT_FAILURE(rc))
3048 return rc;
3049
3050 /* Not supported in legacy mode where we allocate the memory in ring 3 and lock it in ring 0. */
3051 if (pGMM->fLegacyAllocationMode)
3052 return VERR_NOT_SUPPORTED;
3053
3054 gmmR0MutexAcquire(pGMM);
3055 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3056 {
3057 const unsigned cPages = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
3058
3059 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cBasePages < cPages))
3060 {
3061 Log(("GMMR0FreeLargePage: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cBasePages, cPages));
3062 gmmR0MutexRelease(pGMM);
3063 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3064 }
3065
3066 PGMMPAGE pPage = gmmR0GetPage(pGMM, idPage);
3067 if (RT_LIKELY( pPage
3068 && GMM_PAGE_IS_PRIVATE(pPage)))
3069 {
3070 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
3071 Assert(pChunk);
3072 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
3073 Assert(pChunk->cPrivate > 0);
3074
3075 /* Release the memory immediately. */
3076 gmmR0FreeChunk(pGMM, NULL, pChunk, false /*fRelaxedSem*/); /** @todo this can be relaxed too! */
3077
3078 /* Update accounting. */
3079 pGVM->gmm.s.Stats.Allocated.cBasePages -= cPages;
3080 pGVM->gmm.s.Stats.cPrivatePages -= cPages;
3081 pGMM->cAllocatedPages -= cPages;
3082 }
3083 else
3084 rc = VERR_GMM_PAGE_NOT_FOUND;
3085 }
3086 else
3087 rc = VERR_GMM_IS_NOT_SANE;
3088
3089 gmmR0MutexRelease(pGMM);
3090 LogFlow(("GMMR0FreeLargePage: returns %Rrc\n", rc));
3091 return rc;
3092}
3093
3094
3095/**
3096 * VMMR0 request wrapper for GMMR0FreeLargePage.
3097 *
3098 * @returns see GMMR0FreeLargePage.
3099 * @param pVM Pointer to the VM.
3100 * @param idCpu The VCPU id.
3101 * @param pReq Pointer to the request packet.
3102 */
3103GMMR0DECL(int) GMMR0FreeLargePageReq(PVM pVM, VMCPUID idCpu, PGMMFREELARGEPAGEREQ pReq)
3104{
3105 /*
3106 * Validate input and pass it on.
3107 */
3108 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
3109 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3110 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMFREEPAGESREQ),
3111 ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(GMMFREEPAGESREQ)),
3112 VERR_INVALID_PARAMETER);
3113
3114 return GMMR0FreeLargePage(pVM, idCpu, pReq->idPage);
3115}
3116
3117
3118/**
3119 * Frees a chunk, giving it back to the host OS.
3120 *
3121 * @param pGMM Pointer to the GMM instance.
3122 * @param pGVM This is set when called from GMMR0CleanupVM so we can
3123 * unmap and free the chunk in one go.
3124 * @param pChunk The chunk to free.
3125 * @param fRelaxedSem Whether we can release the semaphore while doing the
3126 * freeing (@c true) or not.
3127 */
3128static bool gmmR0FreeChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem)
3129{
3130 Assert(pChunk->Core.Key != NIL_GMM_CHUNKID);
3131
3132 GMMR0CHUNKMTXSTATE MtxState;
3133 gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk, GMMR0CHUNK_MTX_KEEP_GIANT);
3134
3135 /*
3136 * Cleanup hack! Unmap the chunk from the callers address space.
3137 * This shouldn't happen, so screw lock contention...
3138 */
3139 if ( pChunk->cMappingsX
3140 && !pGMM->fLegacyAllocationMode
3141 && pGVM)
3142 gmmR0UnmapChunkLocked(pGMM, pGVM, pChunk);
3143
3144 /*
3145 * If there are current mappings of the chunk, then request the
3146 * VMs to unmap them. Reposition the chunk in the free list so
3147 * it won't be a likely candidate for allocations.
3148 */
3149 if (pChunk->cMappingsX)
3150 {
3151 /** @todo R0 -> VM request */
3152 /* The chunk can be mapped by more than one VM if fBoundMemoryMode is false! */
3153 Log(("gmmR0FreeChunk: chunk still has %d/%d mappings; don't free!\n", pChunk->cMappingsX));
3154 gmmR0ChunkMutexRelease(&MtxState, pChunk);
3155 return false;
3156 }
3157
3158
3159 /*
3160 * Save and trash the handle.
3161 */
3162 RTR0MEMOBJ const hMemObj = pChunk->hMemObj;
3163 pChunk->hMemObj = NIL_RTR0MEMOBJ;
3164
3165 /*
3166 * Unlink it from everywhere.
3167 */
3168 gmmR0UnlinkChunk(pChunk);
3169
3170 RTListNodeRemove(&pChunk->ListNode);
3171
3172 PAVLU32NODECORE pCore = RTAvlU32Remove(&pGMM->pChunks, pChunk->Core.Key);
3173 Assert(pCore == &pChunk->Core); NOREF(pCore);
3174
3175 PGMMCHUNKTLBE pTlbe = &pGMM->ChunkTLB.aEntries[GMM_CHUNKTLB_IDX(pChunk->Core.Key)];
3176 if (pTlbe->pChunk == pChunk)
3177 {
3178 pTlbe->idChunk = NIL_GMM_CHUNKID;
3179 pTlbe->pChunk = NULL;
3180 }
3181
3182 Assert(pGMM->cChunks > 0);
3183 pGMM->cChunks--;
3184
3185 /*
3186 * Free the Chunk ID before dropping the locks and freeing the rest.
3187 */
3188 gmmR0FreeChunkId(pGMM, pChunk->Core.Key);
3189 pChunk->Core.Key = NIL_GMM_CHUNKID;
3190
3191 pGMM->cFreedChunks++;
3192
3193 gmmR0ChunkMutexRelease(&MtxState, NULL);
3194 if (fRelaxedSem)
3195 gmmR0MutexRelease(pGMM);
3196
3197 RTMemFree(pChunk->paMappingsX);
3198 pChunk->paMappingsX = NULL;
3199
3200 RTMemFree(pChunk);
3201
3202 int rc = RTR0MemObjFree(hMemObj, false /* fFreeMappings */);
3203 AssertLogRelRC(rc);
3204
3205 if (fRelaxedSem)
3206 gmmR0MutexAcquire(pGMM);
3207 return fRelaxedSem;
3208}
3209
3210
3211/**
3212 * Free page worker.
3213 *
3214 * The caller does all the statistic decrementing, we do all the incrementing.
3215 *
3216 * @param pGMM Pointer to the GMM instance data.
3217 * @param pGVM Pointer to the GVM instance.
3218 * @param pChunk Pointer to the chunk this page belongs to.
3219 * @param idPage The Page ID.
3220 * @param pPage Pointer to the page.
3221 */
3222static void gmmR0FreePageWorker(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, uint32_t idPage, PGMMPAGE pPage)
3223{
3224 Log3(("F pPage=%p iPage=%#x/%#x u2State=%d iFreeHead=%#x\n",
3225 pPage, pPage - &pChunk->aPages[0], idPage, pPage->Common.u2State, pChunk->iFreeHead)); NOREF(idPage);
3226
3227 /*
3228 * Put the page on the free list.
3229 */
3230 pPage->u = 0;
3231 pPage->Free.u2State = GMM_PAGE_STATE_FREE;
3232 Assert(pChunk->iFreeHead < RT_ELEMENTS(pChunk->aPages) || pChunk->iFreeHead == UINT16_MAX);
3233 pPage->Free.iNext = pChunk->iFreeHead;
3234 pChunk->iFreeHead = pPage - &pChunk->aPages[0];
3235
3236 /*
3237 * Update statistics (the cShared/cPrivate stats are up to date already),
3238 * and relink the chunk if necessary.
3239 */
3240 unsigned const cFree = pChunk->cFree;
3241 if ( !cFree
3242 || gmmR0SelectFreeSetList(cFree) != gmmR0SelectFreeSetList(cFree + 1))
3243 {
3244 gmmR0UnlinkChunk(pChunk);
3245 pChunk->cFree++;
3246 gmmR0SelectSetAndLinkChunk(pGMM, pGVM, pChunk);
3247 }
3248 else
3249 {
3250 pChunk->cFree = cFree + 1;
3251 pChunk->pSet->cFreePages++;
3252 }
3253
3254 /*
3255 * If the chunk becomes empty, consider giving memory back to the host OS.
3256 *
3257 * The current strategy is to try give it back if there are other chunks
3258 * in this free list, meaning if there are at least 240 free pages in this
3259 * category. Note that since there are probably mappings of the chunk,
3260 * it won't be freed up instantly, which probably screws up this logic
3261 * a bit...
3262 */
3263 /** @todo Do this on the way out. */
3264 if (RT_UNLIKELY( pChunk->cFree == GMM_CHUNK_NUM_PAGES
3265 && pChunk->pFreeNext
3266 && pChunk->pFreePrev /** @todo this is probably misfiring, see reset... */
3267 && !pGMM->fLegacyAllocationMode))
3268 gmmR0FreeChunk(pGMM, NULL, pChunk, false);
3269
3270}
3271
3272
3273/**
3274 * Frees a shared page, the page is known to exist and be valid and such.
3275 *
3276 * @param pGMM Pointer to the GMM instance.
3277 * @param pGVM Pointer to the GVM instance.
3278 * @param idPage The page id.
3279 * @param pPage The page structure.
3280 */
3281DECLINLINE(void) gmmR0FreeSharedPage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage)
3282{
3283 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
3284 Assert(pChunk);
3285 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
3286 Assert(pChunk->cShared > 0);
3287 Assert(pGMM->cSharedPages > 0);
3288 Assert(pGMM->cAllocatedPages > 0);
3289 Assert(!pPage->Shared.cRefs);
3290#if defined(VBOX_WITH_PAGE_SHARING) && defined(VBOX_STRICT) && HC_ARCH_BITS == 64
3291 if (pPage->Shared.u14Checksum)
3292 {
3293 uint32_t uChecksum = gmmR0StrictPageChecksum(pGMM, pGVM, idPage);
3294 uChecksum &= UINT32_C(0x00003fff);
3295 AssertMsg(!uChecksum || uChecksum == pPage->Shared.u14Checksum,
3296 ("%#x vs %#x - idPage=%#\n", uChecksum, pPage->Shared.u14Checksum, idPage));
3297 }
3298#endif
3299
3300 pChunk->cShared--;
3301 pGMM->cAllocatedPages--;
3302 pGMM->cSharedPages--;
3303 gmmR0FreePageWorker(pGMM, pGVM, pChunk, idPage, pPage);
3304}
3305
3306
3307/**
3308 * Frees a private page, the page is known to exist and be valid and such.
3309 *
3310 * @param pGMM Pointer to the GMM instance.
3311 * @param pGVM Pointer to the GVM instance.
3312 * @param idPage The page id.
3313 * @param pPage The page structure.
3314 */
3315DECLINLINE(void) gmmR0FreePrivatePage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage)
3316{
3317 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
3318 Assert(pChunk);
3319 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
3320 Assert(pChunk->cPrivate > 0);
3321 Assert(pGMM->cAllocatedPages > 0);
3322
3323 pChunk->cPrivate--;
3324 pGMM->cAllocatedPages--;
3325 gmmR0FreePageWorker(pGMM, pGVM, pChunk, idPage, pPage);
3326}
3327
3328
3329/**
3330 * Common worker for GMMR0FreePages and GMMR0BalloonedPages.
3331 *
3332 * @returns VBox status code:
3333 * @retval xxx
3334 *
3335 * @param pGMM Pointer to the GMM instance data.
3336 * @param pGVM Pointer to the VM.
3337 * @param cPages The number of pages to free.
3338 * @param paPages Pointer to the page descriptors.
3339 * @param enmAccount The account this relates to.
3340 */
3341static int gmmR0FreePages(PGMM pGMM, PGVM pGVM, uint32_t cPages, PGMMFREEPAGEDESC paPages, GMMACCOUNT enmAccount)
3342{
3343 /*
3344 * Check that the request isn't impossible wrt to the account status.
3345 */
3346 switch (enmAccount)
3347 {
3348 case GMMACCOUNT_BASE:
3349 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cBasePages < cPages))
3350 {
3351 Log(("gmmR0FreePages: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cBasePages, cPages));
3352 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3353 }
3354 break;
3355 case GMMACCOUNT_SHADOW:
3356 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cShadowPages < cPages))
3357 {
3358 Log(("gmmR0FreePages: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cShadowPages, cPages));
3359 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3360 }
3361 break;
3362 case GMMACCOUNT_FIXED:
3363 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cFixedPages < cPages))
3364 {
3365 Log(("gmmR0FreePages: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cFixedPages, cPages));
3366 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3367 }
3368 break;
3369 default:
3370 AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
3371 }
3372
3373 /*
3374 * Walk the descriptors and free the pages.
3375 *
3376 * Statistics (except the account) are being updated as we go along,
3377 * unlike the alloc code. Also, stop on the first error.
3378 */
3379 int rc = VINF_SUCCESS;
3380 uint32_t iPage;
3381 for (iPage = 0; iPage < cPages; iPage++)
3382 {
3383 uint32_t idPage = paPages[iPage].idPage;
3384 PGMMPAGE pPage = gmmR0GetPage(pGMM, idPage);
3385 if (RT_LIKELY(pPage))
3386 {
3387 if (RT_LIKELY(GMM_PAGE_IS_PRIVATE(pPage)))
3388 {
3389 if (RT_LIKELY(pPage->Private.hGVM == pGVM->hSelf))
3390 {
3391 Assert(pGVM->gmm.s.Stats.cPrivatePages);
3392 pGVM->gmm.s.Stats.cPrivatePages--;
3393 gmmR0FreePrivatePage(pGMM, pGVM, idPage, pPage);
3394 }
3395 else
3396 {
3397 Log(("gmmR0AllocatePages: #%#x/%#x: not owner! hGVM=%#x hSelf=%#x\n", iPage, idPage,
3398 pPage->Private.hGVM, pGVM->hSelf));
3399 rc = VERR_GMM_NOT_PAGE_OWNER;
3400 break;
3401 }
3402 }
3403 else if (RT_LIKELY(GMM_PAGE_IS_SHARED(pPage)))
3404 {
3405 Assert(pGVM->gmm.s.Stats.cSharedPages);
3406 pGVM->gmm.s.Stats.cSharedPages--;
3407 Assert(pPage->Shared.cRefs);
3408 if (!--pPage->Shared.cRefs)
3409 gmmR0FreeSharedPage(pGMM, pGVM, idPage, pPage);
3410 else
3411 {
3412 Assert(pGMM->cDuplicatePages);
3413 pGMM->cDuplicatePages--;
3414 }
3415 }
3416 else
3417 {
3418 Log(("gmmR0AllocatePages: #%#x/%#x: already free!\n", iPage, idPage));
3419 rc = VERR_GMM_PAGE_ALREADY_FREE;
3420 break;
3421 }
3422 }
3423 else
3424 {
3425 Log(("gmmR0AllocatePages: #%#x/%#x: not found!\n", iPage, idPage));
3426 rc = VERR_GMM_PAGE_NOT_FOUND;
3427 break;
3428 }
3429 paPages[iPage].idPage = NIL_GMM_PAGEID;
3430 }
3431
3432 /*
3433 * Update the account.
3434 */
3435 switch (enmAccount)
3436 {
3437 case GMMACCOUNT_BASE: pGVM->gmm.s.Stats.Allocated.cBasePages -= iPage; break;
3438 case GMMACCOUNT_SHADOW: pGVM->gmm.s.Stats.Allocated.cShadowPages -= iPage; break;
3439 case GMMACCOUNT_FIXED: pGVM->gmm.s.Stats.Allocated.cFixedPages -= iPage; break;
3440 default:
3441 AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
3442 }
3443
3444 /*
3445 * Any threshold stuff to be done here?
3446 */
3447
3448 return rc;
3449}
3450
3451
3452/**
3453 * Free one or more pages.
3454 *
3455 * This is typically used at reset time or power off.
3456 *
3457 * @returns VBox status code:
3458 * @retval xxx
3459 *
3460 * @param pVM Pointer to the VM.
3461 * @param idCpu The VCPU id.
3462 * @param cPages The number of pages to allocate.
3463 * @param paPages Pointer to the page descriptors containing the Page IDs for each page.
3464 * @param enmAccount The account this relates to.
3465 * @thread EMT.
3466 */
3467GMMR0DECL(int) GMMR0FreePages(PVM pVM, VMCPUID idCpu, uint32_t cPages, PGMMFREEPAGEDESC paPages, GMMACCOUNT enmAccount)
3468{
3469 LogFlow(("GMMR0FreePages: pVM=%p cPages=%#x paPages=%p enmAccount=%d\n", pVM, cPages, paPages, enmAccount));
3470
3471 /*
3472 * Validate input and get the basics.
3473 */
3474 PGMM pGMM;
3475 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3476 PGVM pGVM;
3477 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
3478 if (RT_FAILURE(rc))
3479 return rc;
3480
3481 AssertPtrReturn(paPages, VERR_INVALID_PARAMETER);
3482 AssertMsgReturn(enmAccount > GMMACCOUNT_INVALID && enmAccount < GMMACCOUNT_END, ("%d\n", enmAccount), VERR_INVALID_PARAMETER);
3483 AssertMsgReturn(cPages > 0 && cPages < RT_BIT(32 - PAGE_SHIFT), ("%#x\n", cPages), VERR_INVALID_PARAMETER);
3484
3485 for (unsigned iPage = 0; iPage < cPages; iPage++)
3486 AssertMsgReturn( paPages[iPage].idPage <= GMM_PAGEID_LAST
3487 /*|| paPages[iPage].idPage == NIL_GMM_PAGEID*/,
3488 ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
3489
3490 /*
3491 * Take the semaphore and call the worker function.
3492 */
3493 gmmR0MutexAcquire(pGMM);
3494 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3495 {
3496 rc = gmmR0FreePages(pGMM, pGVM, cPages, paPages, enmAccount);
3497 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
3498 }
3499 else
3500 rc = VERR_GMM_IS_NOT_SANE;
3501 gmmR0MutexRelease(pGMM);
3502 LogFlow(("GMMR0FreePages: returns %Rrc\n", rc));
3503 return rc;
3504}
3505
3506
3507/**
3508 * VMMR0 request wrapper for GMMR0FreePages.
3509 *
3510 * @returns see GMMR0FreePages.
3511 * @param pVM Pointer to the VM.
3512 * @param idCpu The VCPU id.
3513 * @param pReq Pointer to the request packet.
3514 */
3515GMMR0DECL(int) GMMR0FreePagesReq(PVM pVM, VMCPUID idCpu, PGMMFREEPAGESREQ pReq)
3516{
3517 /*
3518 * Validate input and pass it on.
3519 */
3520 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
3521 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3522 AssertMsgReturn(pReq->Hdr.cbReq >= RT_UOFFSETOF(GMMFREEPAGESREQ, aPages[0]),
3523 ("%#x < %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF(GMMFREEPAGESREQ, aPages[0])),
3524 VERR_INVALID_PARAMETER);
3525 AssertMsgReturn(pReq->Hdr.cbReq == RT_UOFFSETOF(GMMFREEPAGESREQ, aPages[pReq->cPages]),
3526 ("%#x != %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF(GMMFREEPAGESREQ, aPages[pReq->cPages])),
3527 VERR_INVALID_PARAMETER);
3528
3529 return GMMR0FreePages(pVM, idCpu, pReq->cPages, &pReq->aPages[0], pReq->enmAccount);
3530}
3531
3532
3533/**
3534 * Report back on a memory ballooning request.
3535 *
3536 * The request may or may not have been initiated by the GMM. If it was initiated
3537 * by the GMM it is important that this function is called even if no pages were
3538 * ballooned.
3539 *
3540 * @returns VBox status code:
3541 * @retval VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH
3542 * @retval VERR_GMM_ATTEMPT_TO_DEFLATE_TOO_MUCH
3543 * @retval VERR_GMM_OVERCOMMITTED_TRY_AGAIN_IN_A_BIT - reset condition
3544 * indicating that we won't necessarily have sufficient RAM to boot
3545 * the VM again and that it should pause until this changes (we'll try
3546 * balloon some other VM). (For standard deflate we have little choice
3547 * but to hope the VM won't use the memory that was returned to it.)
3548 *
3549 * @param pVM Pointer to the VM.
3550 * @param idCpu The VCPU id.
3551 * @param enmAction Inflate/deflate/reset.
3552 * @param cBalloonedPages The number of pages that was ballooned.
3553 *
3554 * @thread EMT.
3555 */
3556GMMR0DECL(int) GMMR0BalloonedPages(PVM pVM, VMCPUID idCpu, GMMBALLOONACTION enmAction, uint32_t cBalloonedPages)
3557{
3558 LogFlow(("GMMR0BalloonedPages: pVM=%p enmAction=%d cBalloonedPages=%#x\n",
3559 pVM, enmAction, cBalloonedPages));
3560
3561 AssertMsgReturn(cBalloonedPages < RT_BIT(32 - PAGE_SHIFT), ("%#x\n", cBalloonedPages), VERR_INVALID_PARAMETER);
3562
3563 /*
3564 * Validate input and get the basics.
3565 */
3566 PGMM pGMM;
3567 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3568 PGVM pGVM;
3569 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
3570 if (RT_FAILURE(rc))
3571 return rc;
3572
3573 /*
3574 * Take the semaphore and do some more validations.
3575 */
3576 gmmR0MutexAcquire(pGMM);
3577 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3578 {
3579 switch (enmAction)
3580 {
3581 case GMMBALLOONACTION_INFLATE:
3582 {
3583 if (RT_LIKELY(pGVM->gmm.s.Stats.Allocated.cBasePages + pGVM->gmm.s.Stats.cBalloonedPages + cBalloonedPages
3584 <= pGVM->gmm.s.Stats.Reserved.cBasePages))
3585 {
3586 /*
3587 * Record the ballooned memory.
3588 */
3589 pGMM->cBalloonedPages += cBalloonedPages;
3590 if (pGVM->gmm.s.Stats.cReqBalloonedPages)
3591 {
3592 /* Codepath never taken. Might be interesting in the future to request ballooned memory from guests in low memory conditions.. */
3593 AssertFailed();
3594
3595 pGVM->gmm.s.Stats.cBalloonedPages += cBalloonedPages;
3596 pGVM->gmm.s.Stats.cReqActuallyBalloonedPages += cBalloonedPages;
3597 Log(("GMMR0BalloonedPages: +%#x - Global=%#llx / VM: Total=%#llx Req=%#llx Actual=%#llx (pending)\n",
3598 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages,
3599 pGVM->gmm.s.Stats.cReqBalloonedPages, pGVM->gmm.s.Stats.cReqActuallyBalloonedPages));
3600 }
3601 else
3602 {
3603 pGVM->gmm.s.Stats.cBalloonedPages += cBalloonedPages;
3604 Log(("GMMR0BalloonedPages: +%#x - Global=%#llx / VM: Total=%#llx (user)\n",
3605 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages));
3606 }
3607 }
3608 else
3609 {
3610 Log(("GMMR0BalloonedPages: cBasePages=%#llx Total=%#llx cBalloonedPages=%#llx Reserved=%#llx\n",
3611 pGVM->gmm.s.Stats.Allocated.cBasePages, pGVM->gmm.s.Stats.cBalloonedPages, cBalloonedPages,
3612 pGVM->gmm.s.Stats.Reserved.cBasePages));
3613 rc = VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3614 }
3615 break;
3616 }
3617
3618 case GMMBALLOONACTION_DEFLATE:
3619 {
3620 /* Deflate. */
3621 if (pGVM->gmm.s.Stats.cBalloonedPages >= cBalloonedPages)
3622 {
3623 /*
3624 * Record the ballooned memory.
3625 */
3626 Assert(pGMM->cBalloonedPages >= cBalloonedPages);
3627 pGMM->cBalloonedPages -= cBalloonedPages;
3628 pGVM->gmm.s.Stats.cBalloonedPages -= cBalloonedPages;
3629 if (pGVM->gmm.s.Stats.cReqDeflatePages)
3630 {
3631 AssertFailed(); /* This is path is for later. */
3632 Log(("GMMR0BalloonedPages: -%#x - Global=%#llx / VM: Total=%#llx Req=%#llx\n",
3633 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages, pGVM->gmm.s.Stats.cReqDeflatePages));
3634
3635 /*
3636 * Anything we need to do here now when the request has been completed?
3637 */
3638 pGVM->gmm.s.Stats.cReqDeflatePages = 0;
3639 }
3640 else
3641 Log(("GMMR0BalloonedPages: -%#x - Global=%#llx / VM: Total=%#llx (user)\n",
3642 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages));
3643 }
3644 else
3645 {
3646 Log(("GMMR0BalloonedPages: Total=%#llx cBalloonedPages=%#llx\n", pGVM->gmm.s.Stats.cBalloonedPages, cBalloonedPages));
3647 rc = VERR_GMM_ATTEMPT_TO_DEFLATE_TOO_MUCH;
3648 }
3649 break;
3650 }
3651
3652 case GMMBALLOONACTION_RESET:
3653 {
3654 /* Reset to an empty balloon. */
3655 Assert(pGMM->cBalloonedPages >= pGVM->gmm.s.Stats.cBalloonedPages);
3656
3657 pGMM->cBalloonedPages -= pGVM->gmm.s.Stats.cBalloonedPages;
3658 pGVM->gmm.s.Stats.cBalloonedPages = 0;
3659 break;
3660 }
3661
3662 default:
3663 rc = VERR_INVALID_PARAMETER;
3664 break;
3665 }
3666 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
3667 }
3668 else
3669 rc = VERR_GMM_IS_NOT_SANE;
3670
3671 gmmR0MutexRelease(pGMM);
3672 LogFlow(("GMMR0BalloonedPages: returns %Rrc\n", rc));
3673 return rc;
3674}
3675
3676
3677/**
3678 * VMMR0 request wrapper for GMMR0BalloonedPages.
3679 *
3680 * @returns see GMMR0BalloonedPages.
3681 * @param pVM Pointer to the VM.
3682 * @param idCpu The VCPU id.
3683 * @param pReq Pointer to the request packet.
3684 */
3685GMMR0DECL(int) GMMR0BalloonedPagesReq(PVM pVM, VMCPUID idCpu, PGMMBALLOONEDPAGESREQ pReq)
3686{
3687 /*
3688 * Validate input and pass it on.
3689 */
3690 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
3691 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3692 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMBALLOONEDPAGESREQ),
3693 ("%#x < %#x\n", pReq->Hdr.cbReq, sizeof(GMMBALLOONEDPAGESREQ)),
3694 VERR_INVALID_PARAMETER);
3695
3696 return GMMR0BalloonedPages(pVM, idCpu, pReq->enmAction, pReq->cBalloonedPages);
3697}
3698
3699/**
3700 * Return memory statistics for the hypervisor
3701 *
3702 * @returns VBox status code:
3703 * @param pVM Pointer to the VM.
3704 * @param pReq Pointer to the request packet.
3705 */
3706GMMR0DECL(int) GMMR0QueryHypervisorMemoryStatsReq(PVM pVM, PGMMMEMSTATSREQ pReq)
3707{
3708 /*
3709 * Validate input and pass it on.
3710 */
3711 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
3712 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3713 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMMEMSTATSREQ),
3714 ("%#x < %#x\n", pReq->Hdr.cbReq, sizeof(GMMMEMSTATSREQ)),
3715 VERR_INVALID_PARAMETER);
3716
3717 /*
3718 * Validate input and get the basics.
3719 */
3720 PGMM pGMM;
3721 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3722 pReq->cAllocPages = pGMM->cAllocatedPages;
3723 pReq->cFreePages = (pGMM->cChunks << (GMM_CHUNK_SHIFT- PAGE_SHIFT)) - pGMM->cAllocatedPages;
3724 pReq->cBalloonedPages = pGMM->cBalloonedPages;
3725 pReq->cMaxPages = pGMM->cMaxPages;
3726 pReq->cSharedPages = pGMM->cDuplicatePages;
3727 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
3728
3729 return VINF_SUCCESS;
3730}
3731
3732/**
3733 * Return memory statistics for the VM
3734 *
3735 * @returns VBox status code:
3736 * @param pVM Pointer to the VM.
3737 * @parma idCpu Cpu id.
3738 * @param pReq Pointer to the request packet.
3739 */
3740GMMR0DECL(int) GMMR0QueryMemoryStatsReq(PVM pVM, VMCPUID idCpu, PGMMMEMSTATSREQ pReq)
3741{
3742 /*
3743 * Validate input and pass it on.
3744 */
3745 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
3746 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3747 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMMEMSTATSREQ),
3748 ("%#x < %#x\n", pReq->Hdr.cbReq, sizeof(GMMMEMSTATSREQ)),
3749 VERR_INVALID_PARAMETER);
3750
3751 /*
3752 * Validate input and get the basics.
3753 */
3754 PGMM pGMM;
3755 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3756 PGVM pGVM;
3757 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
3758 if (RT_FAILURE(rc))
3759 return rc;
3760
3761 /*
3762 * Take the semaphore and do some more validations.
3763 */
3764 gmmR0MutexAcquire(pGMM);
3765 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3766 {
3767 pReq->cAllocPages = pGVM->gmm.s.Stats.Allocated.cBasePages;
3768 pReq->cBalloonedPages = pGVM->gmm.s.Stats.cBalloonedPages;
3769 pReq->cMaxPages = pGVM->gmm.s.Stats.Reserved.cBasePages;
3770 pReq->cFreePages = pReq->cMaxPages - pReq->cAllocPages;
3771 }
3772 else
3773 rc = VERR_GMM_IS_NOT_SANE;
3774
3775 gmmR0MutexRelease(pGMM);
3776 LogFlow(("GMMR3QueryVMMemoryStats: returns %Rrc\n", rc));
3777 return rc;
3778}
3779
3780
3781/**
3782 * Worker for gmmR0UnmapChunk and gmmr0FreeChunk.
3783 *
3784 * Don't call this in legacy allocation mode!
3785 *
3786 * @returns VBox status code.
3787 * @param pGMM Pointer to the GMM instance data.
3788 * @param pGVM Pointer to the Global VM structure.
3789 * @param pChunk Pointer to the chunk to be unmapped.
3790 */
3791static int gmmR0UnmapChunkLocked(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk)
3792{
3793 Assert(!pGMM->fLegacyAllocationMode);
3794
3795 /*
3796 * Find the mapping and try unmapping it.
3797 */
3798 uint32_t cMappings = pChunk->cMappingsX;
3799 for (uint32_t i = 0; i < cMappings; i++)
3800 {
3801 Assert(pChunk->paMappingsX[i].pGVM && pChunk->paMappingsX[i].hMapObj != NIL_RTR0MEMOBJ);
3802 if (pChunk->paMappingsX[i].pGVM == pGVM)
3803 {
3804 /* unmap */
3805 int rc = RTR0MemObjFree(pChunk->paMappingsX[i].hMapObj, false /* fFreeMappings (NA) */);
3806 if (RT_SUCCESS(rc))
3807 {
3808 /* update the record. */
3809 cMappings--;
3810 if (i < cMappings)
3811 pChunk->paMappingsX[i] = pChunk->paMappingsX[cMappings];
3812 pChunk->paMappingsX[cMappings].hMapObj = NIL_RTR0MEMOBJ;
3813 pChunk->paMappingsX[cMappings].pGVM = NULL;
3814 Assert(pChunk->cMappingsX - 1U == cMappings);
3815 pChunk->cMappingsX = cMappings;
3816 }
3817
3818 return rc;
3819 }
3820 }
3821
3822 Log(("gmmR0UnmapChunk: Chunk %#x is not mapped into pGVM=%p/%#x\n", pChunk->Core.Key, pGVM, pGVM->hSelf));
3823 return VERR_GMM_CHUNK_NOT_MAPPED;
3824}
3825
3826
3827/**
3828 * Unmaps a chunk previously mapped into the address space of the current process.
3829 *
3830 * @returns VBox status code.
3831 * @param pGMM Pointer to the GMM instance data.
3832 * @param pGVM Pointer to the Global VM structure.
3833 * @param pChunk Pointer to the chunk to be unmapped.
3834 */
3835static int gmmR0UnmapChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem)
3836{
3837 if (!pGMM->fLegacyAllocationMode)
3838 {
3839 /*
3840 * Lock the chunk and if possible leave the giant GMM lock.
3841 */
3842 GMMR0CHUNKMTXSTATE MtxState;
3843 int rc = gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk,
3844 fRelaxedSem ? GMMR0CHUNK_MTX_RETAKE_GIANT : GMMR0CHUNK_MTX_KEEP_GIANT);
3845 if (RT_SUCCESS(rc))
3846 {
3847 rc = gmmR0UnmapChunkLocked(pGMM, pGVM, pChunk);
3848 gmmR0ChunkMutexRelease(&MtxState, pChunk);
3849 }
3850 return rc;
3851 }
3852
3853 if (pChunk->hGVM == pGVM->hSelf)
3854 return VINF_SUCCESS;
3855
3856 Log(("gmmR0UnmapChunk: Chunk %#x is not mapped into pGVM=%p/%#x (legacy)\n", pChunk->Core.Key, pGVM, pGVM->hSelf));
3857 return VERR_GMM_CHUNK_NOT_MAPPED;
3858}
3859
3860
3861/**
3862 * Worker for gmmR0MapChunk.
3863 *
3864 * @returns VBox status code.
3865 * @param pGMM Pointer to the GMM instance data.
3866 * @param pGVM Pointer to the Global VM structure.
3867 * @param pChunk Pointer to the chunk to be mapped.
3868 * @param ppvR3 Where to store the ring-3 address of the mapping.
3869 * In the VERR_GMM_CHUNK_ALREADY_MAPPED case, this will be
3870 * contain the address of the existing mapping.
3871 */
3872static int gmmR0MapChunkLocked(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, PRTR3PTR ppvR3)
3873{
3874 /*
3875 * If we're in legacy mode this is simple.
3876 */
3877 if (pGMM->fLegacyAllocationMode)
3878 {
3879 if (pChunk->hGVM != pGVM->hSelf)
3880 {
3881 Log(("gmmR0MapChunk: chunk %#x is already mapped at %p!\n", pChunk->Core.Key, *ppvR3));
3882 return VERR_GMM_CHUNK_NOT_FOUND;
3883 }
3884
3885 *ppvR3 = RTR0MemObjAddressR3(pChunk->hMemObj);
3886 return VINF_SUCCESS;
3887 }
3888
3889 /*
3890 * Check to see if the chunk is already mapped.
3891 */
3892 for (uint32_t i = 0; i < pChunk->cMappingsX; i++)
3893 {
3894 Assert(pChunk->paMappingsX[i].pGVM && pChunk->paMappingsX[i].hMapObj != NIL_RTR0MEMOBJ);
3895 if (pChunk->paMappingsX[i].pGVM == pGVM)
3896 {
3897 *ppvR3 = RTR0MemObjAddressR3(pChunk->paMappingsX[i].hMapObj);
3898 Log(("gmmR0MapChunk: chunk %#x is already mapped at %p!\n", pChunk->Core.Key, *ppvR3));
3899#ifdef VBOX_WITH_PAGE_SHARING
3900 /* The ring-3 chunk cache can be out of sync; don't fail. */
3901 return VINF_SUCCESS;
3902#else
3903 return VERR_GMM_CHUNK_ALREADY_MAPPED;
3904#endif
3905 }
3906 }
3907
3908 /*
3909 * Do the mapping.
3910 */
3911 RTR0MEMOBJ hMapObj;
3912 int rc = RTR0MemObjMapUser(&hMapObj, pChunk->hMemObj, (RTR3PTR)-1, 0, RTMEM_PROT_READ | RTMEM_PROT_WRITE, NIL_RTR0PROCESS);
3913 if (RT_SUCCESS(rc))
3914 {
3915 /* reallocate the array? assumes few users per chunk (usually one). */
3916 unsigned iMapping = pChunk->cMappingsX;
3917 if ( iMapping <= 3
3918 || (iMapping & 3) == 0)
3919 {
3920 unsigned cNewSize = iMapping <= 3
3921 ? iMapping + 1
3922 : iMapping + 4;
3923 Assert(cNewSize < 4 || RT_ALIGN_32(cNewSize, 4) == cNewSize);
3924 if (RT_UNLIKELY(cNewSize > UINT16_MAX))
3925 {
3926 rc = RTR0MemObjFree(hMapObj, false /* fFreeMappings (NA) */); AssertRC(rc);
3927 return VERR_GMM_TOO_MANY_CHUNK_MAPPINGS;
3928 }
3929
3930 void *pvMappings = RTMemRealloc(pChunk->paMappingsX, cNewSize * sizeof(pChunk->paMappingsX[0]));
3931 if (RT_UNLIKELY(!pvMappings))
3932 {
3933 rc = RTR0MemObjFree(hMapObj, false /* fFreeMappings (NA) */); AssertRC(rc);
3934 return VERR_NO_MEMORY;
3935 }
3936 pChunk->paMappingsX = (PGMMCHUNKMAP)pvMappings;
3937 }
3938
3939 /* insert new entry */
3940 pChunk->paMappingsX[iMapping].hMapObj = hMapObj;
3941 pChunk->paMappingsX[iMapping].pGVM = pGVM;
3942 Assert(pChunk->cMappingsX == iMapping);
3943 pChunk->cMappingsX = iMapping + 1;
3944
3945 *ppvR3 = RTR0MemObjAddressR3(hMapObj);
3946 }
3947
3948 return rc;
3949}
3950
3951
3952/**
3953 * Maps a chunk into the user address space of the current process.
3954 *
3955 * @returns VBox status code.
3956 * @param pGMM Pointer to the GMM instance data.
3957 * @param pGVM Pointer to the Global VM structure.
3958 * @param pChunk Pointer to the chunk to be mapped.
3959 * @param fRelaxedSem Whether we can release the semaphore while doing the
3960 * mapping (@c true) or not.
3961 * @param ppvR3 Where to store the ring-3 address of the mapping.
3962 * In the VERR_GMM_CHUNK_ALREADY_MAPPED case, this will be
3963 * contain the address of the existing mapping.
3964 */
3965static int gmmR0MapChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem, PRTR3PTR ppvR3)
3966{
3967 /*
3968 * Take the chunk lock and leave the giant GMM lock when possible, then
3969 * call the worker function.
3970 */
3971 GMMR0CHUNKMTXSTATE MtxState;
3972 int rc = gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk,
3973 fRelaxedSem ? GMMR0CHUNK_MTX_RETAKE_GIANT : GMMR0CHUNK_MTX_KEEP_GIANT);
3974 if (RT_SUCCESS(rc))
3975 {
3976 rc = gmmR0MapChunkLocked(pGMM, pGVM, pChunk, ppvR3);
3977 gmmR0ChunkMutexRelease(&MtxState, pChunk);
3978 }
3979
3980 return rc;
3981}
3982
3983
3984
3985#if defined(VBOX_WITH_PAGE_SHARING) || (defined(VBOX_STRICT) && HC_ARCH_BITS == 64)
3986/**
3987 * Check if a chunk is mapped into the specified VM
3988 *
3989 * @returns mapped yes/no
3990 * @param pGMM Pointer to the GMM instance.
3991 * @param pGVM Pointer to the Global VM structure.
3992 * @param pChunk Pointer to the chunk to be mapped.
3993 * @param ppvR3 Where to store the ring-3 address of the mapping.
3994 */
3995static bool gmmR0IsChunkMapped(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, PRTR3PTR ppvR3)
3996{
3997 GMMR0CHUNKMTXSTATE MtxState;
3998 gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk, GMMR0CHUNK_MTX_KEEP_GIANT);
3999 for (uint32_t i = 0; i < pChunk->cMappingsX; i++)
4000 {
4001 Assert(pChunk->paMappingsX[i].pGVM && pChunk->paMappingsX[i].hMapObj != NIL_RTR0MEMOBJ);
4002 if (pChunk->paMappingsX[i].pGVM == pGVM)
4003 {
4004 *ppvR3 = RTR0MemObjAddressR3(pChunk->paMappingsX[i].hMapObj);
4005 gmmR0ChunkMutexRelease(&MtxState, pChunk);
4006 return true;
4007 }
4008 }
4009 *ppvR3 = NULL;
4010 gmmR0ChunkMutexRelease(&MtxState, pChunk);
4011 return false;
4012}
4013#endif /* VBOX_WITH_PAGE_SHARING || (VBOX_STRICT && 64-BIT) */
4014
4015
4016/**
4017 * Map a chunk and/or unmap another chunk.
4018 *
4019 * The mapping and unmapping applies to the current process.
4020 *
4021 * This API does two things because it saves a kernel call per mapping when
4022 * when the ring-3 mapping cache is full.
4023 *
4024 * @returns VBox status code.
4025 * @param pVM The VM.
4026 * @param idChunkMap The chunk to map. NIL_GMM_CHUNKID if nothing to map.
4027 * @param idChunkUnmap The chunk to unmap. NIL_GMM_CHUNKID if nothing to unmap.
4028 * @param ppvR3 Where to store the address of the mapped chunk. NULL is ok if nothing to map.
4029 * @thread EMT
4030 */
4031GMMR0DECL(int) GMMR0MapUnmapChunk(PVM pVM, uint32_t idChunkMap, uint32_t idChunkUnmap, PRTR3PTR ppvR3)
4032{
4033 LogFlow(("GMMR0MapUnmapChunk: pVM=%p idChunkMap=%#x idChunkUnmap=%#x ppvR3=%p\n",
4034 pVM, idChunkMap, idChunkUnmap, ppvR3));
4035
4036 /*
4037 * Validate input and get the basics.
4038 */
4039 PGMM pGMM;
4040 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4041 PGVM pGVM;
4042 int rc = GVMMR0ByVM(pVM, &pGVM);
4043 if (RT_FAILURE(rc))
4044 return rc;
4045
4046 AssertCompile(NIL_GMM_CHUNKID == 0);
4047 AssertMsgReturn(idChunkMap <= GMM_CHUNKID_LAST, ("%#x\n", idChunkMap), VERR_INVALID_PARAMETER);
4048 AssertMsgReturn(idChunkUnmap <= GMM_CHUNKID_LAST, ("%#x\n", idChunkUnmap), VERR_INVALID_PARAMETER);
4049
4050 if ( idChunkMap == NIL_GMM_CHUNKID
4051 && idChunkUnmap == NIL_GMM_CHUNKID)
4052 return VERR_INVALID_PARAMETER;
4053
4054 if (idChunkMap != NIL_GMM_CHUNKID)
4055 {
4056 AssertPtrReturn(ppvR3, VERR_INVALID_POINTER);
4057 *ppvR3 = NIL_RTR3PTR;
4058 }
4059
4060 /*
4061 * Take the semaphore and do the work.
4062 *
4063 * The unmapping is done last since it's easier to undo a mapping than
4064 * undoing an unmapping. The ring-3 mapping cache cannot not be so big
4065 * that it pushes the user virtual address space to within a chunk of
4066 * it it's limits, so, no problem here.
4067 */
4068 gmmR0MutexAcquire(pGMM);
4069 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
4070 {
4071 PGMMCHUNK pMap = NULL;
4072 if (idChunkMap != NIL_GVM_HANDLE)
4073 {
4074 pMap = gmmR0GetChunk(pGMM, idChunkMap);
4075 if (RT_LIKELY(pMap))
4076 rc = gmmR0MapChunk(pGMM, pGVM, pMap, true /*fRelaxedSem*/, ppvR3);
4077 else
4078 {
4079 Log(("GMMR0MapUnmapChunk: idChunkMap=%#x\n", idChunkMap));
4080 rc = VERR_GMM_CHUNK_NOT_FOUND;
4081 }
4082 }
4083/** @todo split this operation, the bail out might (theoretcially) not be
4084 * entirely safe. */
4085
4086 if ( idChunkUnmap != NIL_GMM_CHUNKID
4087 && RT_SUCCESS(rc))
4088 {
4089 PGMMCHUNK pUnmap = gmmR0GetChunk(pGMM, idChunkUnmap);
4090 if (RT_LIKELY(pUnmap))
4091 rc = gmmR0UnmapChunk(pGMM, pGVM, pUnmap, true /*fRelaxedSem*/);
4092 else
4093 {
4094 Log(("GMMR0MapUnmapChunk: idChunkUnmap=%#x\n", idChunkUnmap));
4095 rc = VERR_GMM_CHUNK_NOT_FOUND;
4096 }
4097
4098 if (RT_FAILURE(rc) && pMap)
4099 gmmR0UnmapChunk(pGMM, pGVM, pMap, false /*fRelaxedSem*/);
4100 }
4101
4102 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
4103 }
4104 else
4105 rc = VERR_GMM_IS_NOT_SANE;
4106 gmmR0MutexRelease(pGMM);
4107
4108 LogFlow(("GMMR0MapUnmapChunk: returns %Rrc\n", rc));
4109 return rc;
4110}
4111
4112
4113/**
4114 * VMMR0 request wrapper for GMMR0MapUnmapChunk.
4115 *
4116 * @returns see GMMR0MapUnmapChunk.
4117 * @param pVM Pointer to the VM.
4118 * @param pReq Pointer to the request packet.
4119 */
4120GMMR0DECL(int) GMMR0MapUnmapChunkReq(PVM pVM, PGMMMAPUNMAPCHUNKREQ pReq)
4121{
4122 /*
4123 * Validate input and pass it on.
4124 */
4125 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
4126 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
4127 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
4128
4129 return GMMR0MapUnmapChunk(pVM, pReq->idChunkMap, pReq->idChunkUnmap, &pReq->pvR3);
4130}
4131
4132
4133/**
4134 * Legacy mode API for supplying pages.
4135 *
4136 * The specified user address points to a allocation chunk sized block that
4137 * will be locked down and used by the GMM when the GM asks for pages.
4138 *
4139 * @returns VBox status code.
4140 * @param pVM Pointer to the VM.
4141 * @param idCpu The VCPU id.
4142 * @param pvR3 Pointer to the chunk size memory block to lock down.
4143 */
4144GMMR0DECL(int) GMMR0SeedChunk(PVM pVM, VMCPUID idCpu, RTR3PTR pvR3)
4145{
4146 /*
4147 * Validate input and get the basics.
4148 */
4149 PGMM pGMM;
4150 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4151 PGVM pGVM;
4152 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
4153 if (RT_FAILURE(rc))
4154 return rc;
4155
4156 AssertPtrReturn(pvR3, VERR_INVALID_POINTER);
4157 AssertReturn(!(PAGE_OFFSET_MASK & pvR3), VERR_INVALID_POINTER);
4158
4159 if (!pGMM->fLegacyAllocationMode)
4160 {
4161 Log(("GMMR0SeedChunk: not in legacy allocation mode!\n"));
4162 return VERR_NOT_SUPPORTED;
4163 }
4164
4165 /*
4166 * Lock the memory and add it as new chunk with our hGVM.
4167 * (The GMM locking is done inside gmmR0RegisterChunk.)
4168 */
4169 RTR0MEMOBJ MemObj;
4170 rc = RTR0MemObjLockUser(&MemObj, pvR3, GMM_CHUNK_SIZE, RTMEM_PROT_READ | RTMEM_PROT_WRITE, NIL_RTR0PROCESS);
4171 if (RT_SUCCESS(rc))
4172 {
4173 rc = gmmR0RegisterChunk(pGMM, &pGVM->gmm.s.Private, MemObj, pGVM->hSelf, 0 /*fChunkFlags*/, NULL);
4174 if (RT_SUCCESS(rc))
4175 gmmR0MutexRelease(pGMM);
4176 else
4177 RTR0MemObjFree(MemObj, false /* fFreeMappings */);
4178 }
4179
4180 LogFlow(("GMMR0SeedChunk: rc=%d (pvR3=%p)\n", rc, pvR3));
4181 return rc;
4182}
4183
4184#ifdef VBOX_WITH_PAGE_SHARING
4185
4186# ifdef VBOX_STRICT
4187/**
4188 * For checksumming shared pages in strict builds.
4189 *
4190 * The purpose is making sure that a page doesn't change.
4191 *
4192 * @returns Checksum, 0 on failure.
4193 * @param GMM The GMM instance data.
4194 * @param idPage The page ID.
4195 */
4196static uint32_t gmmR0StrictPageChecksum(PGMM pGMM, PGVM pGVM, uint32_t idPage)
4197{
4198 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
4199 AssertMsgReturn(pChunk, ("idPage=%#x\n", idPage), 0);
4200
4201 uint8_t *pbChunk;
4202 if (!gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
4203 return 0;
4204 uint8_t const *pbPage = pbChunk + ((idPage & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
4205
4206 return RTCrc32(pbPage, PAGE_SIZE);
4207}
4208# endif /* VBOX_STRICT */
4209
4210
4211/**
4212 * Calculates the module hash value.
4213 *
4214 * @returns Hash value.
4215 * @param pszModuleName The module name.
4216 * @param pszVersion The module version string.
4217 */
4218static uint32_t gmmR0ShModCalcHash(const char *pszModuleName, const char *pszVersion)
4219{
4220 return RTStrHash1ExN(3, pszModuleName, RTSTR_MAX, "::", (size_t)2, pszVersion, RTSTR_MAX);
4221}
4222
4223
4224/**
4225 * Finds a global module.
4226 *
4227 * @returns Pointer to the global module on success, NULL if not found.
4228 * @param pGMM The GMM instance data.
4229 * @param uHash The hash as calculated by gmmR0ShModCalcHash.
4230 * @param cbModule The module size.
4231 * @param enmGuestOS The guest OS type.
4232 * @param pszModuleName The module name.
4233 * @param pszVersion The module version.
4234 */
4235static PGMMSHAREDMODULE gmmR0ShModFindGlobal(PGMM pGMM, uint32_t uHash, uint32_t cbModule, VBOXOSFAMILY enmGuestOS,
4236 uint32_t cRegions, const char *pszModuleName, const char *pszVersion,
4237 struct VMMDEVSHAREDREGIONDESC const *paRegions)
4238{
4239 for (PGMMSHAREDMODULE pGblMod = (PGMMSHAREDMODULE)RTAvllU32Get(&pGMM->pGlobalSharedModuleTree, uHash);
4240 pGblMod;
4241 pGblMod = (PGMMSHAREDMODULE)pGblMod->Core.pList)
4242 {
4243 if (pGblMod->cbModule != cbModule)
4244 continue;
4245 if (pGblMod->enmGuestOS != enmGuestOS)
4246 continue;
4247 if (pGblMod->cRegions != cRegions)
4248 continue;
4249 if (strcmp(pGblMod->szName, pszModuleName))
4250 continue;
4251 if (strcmp(pGblMod->szVersion, pszVersion))
4252 continue;
4253
4254 uint32_t i;
4255 for (i = 0; i < cRegions; i++)
4256 {
4257 uint32_t off = paRegions[i].GCRegionAddr & PAGE_OFFSET_MASK;
4258 if (pGblMod->aRegions[i].off != off)
4259 break;
4260
4261 uint32_t cb = RT_ALIGN_32(paRegions[i].cbRegion + off, PAGE_SIZE);
4262 if (pGblMod->aRegions[i].cb != cb)
4263 break;
4264 }
4265
4266 if (i == cRegions)
4267 return pGblMod;
4268 }
4269
4270 return NULL;
4271}
4272
4273
4274/**
4275 * Creates a new global module.
4276 *
4277 * @returns VBox status code.
4278 * @param pGMM The GMM instance data.
4279 * @param uHash The hash as calculated by gmmR0ShModCalcHash.
4280 * @param cbModule The module size.
4281 * @param enmGuestOS The guest OS type.
4282 * @param cRegions The number of regions.
4283 * @param pszModuleName The module name.
4284 * @param pszVersion The module version.
4285 * @param paRegions The region descriptions.
4286 * @param ppGblMod Where to return the new module on success.
4287 */
4288static int gmmR0ShModNewGlobal(PGMM pGMM, uint32_t uHash, uint32_t cbModule, VBOXOSFAMILY enmGuestOS,
4289 uint32_t cRegions, const char *pszModuleName, const char *pszVersion,
4290 struct VMMDEVSHAREDREGIONDESC const *paRegions, PGMMSHAREDMODULE *ppGblMod)
4291{
4292 Log(("gmmR0ShModNewGlobal: %s %s size %#x os %u rgn %u\n", pszModuleName, pszVersion, cbModule, cRegions));
4293 if (pGMM->cShareableModules >= GMM_MAX_SHARED_GLOBAL_MODULES)
4294 {
4295 Log(("gmmR0ShModNewGlobal: Too many modules\n"));
4296 return VERR_GMM_TOO_MANY_GLOBAL_MODULES;
4297 }
4298
4299 PGMMSHAREDMODULE pGblMod = (PGMMSHAREDMODULE)RTMemAllocZ(RT_OFFSETOF(GMMSHAREDMODULE, aRegions[cRegions]));
4300 if (!pGblMod)
4301 {
4302 Log(("gmmR0ShModNewGlobal: No memory\n"));
4303 return VERR_NO_MEMORY;
4304 }
4305
4306 pGblMod->Core.Key = uHash;
4307 pGblMod->cbModule = cbModule;
4308 pGblMod->cRegions = cRegions;
4309 pGblMod->cUsers = 1;
4310 pGblMod->enmGuestOS = enmGuestOS;
4311 strcpy(pGblMod->szName, pszModuleName);
4312 strcpy(pGblMod->szVersion, pszVersion);
4313
4314 for (uint32_t i = 0; i < cRegions; i++)
4315 {
4316 Log(("gmmR0ShModNewGlobal: rgn[%u]=%RGvLB%#x\n", i, paRegions[i].GCRegionAddr, paRegions[i].cbRegion));
4317 pGblMod->aRegions[i].off = paRegions[i].GCRegionAddr & PAGE_OFFSET_MASK;
4318 pGblMod->aRegions[i].cb = paRegions[i].cbRegion + pGblMod->aRegions[i].off;
4319 pGblMod->aRegions[i].cb = RT_ALIGN_32(pGblMod->aRegions[i].cb, PAGE_SIZE);
4320 pGblMod->aRegions[i].paidPages = NULL; /* allocated when needed. */
4321 }
4322
4323 bool fInsert = RTAvllU32Insert(&pGMM->pGlobalSharedModuleTree, &pGblMod->Core);
4324 Assert(fInsert); NOREF(fInsert);
4325 pGMM->cShareableModules++;
4326
4327 *ppGblMod = pGblMod;
4328 return VINF_SUCCESS;
4329}
4330
4331
4332/**
4333 * Deletes a global module which is no longer referenced by anyone.
4334 *
4335 * @param pGMM The GMM instance data.
4336 * @param pGblMod The module to delete.
4337 */
4338static void gmmR0ShModDeleteGlobal(PGMM pGMM, PGMMSHAREDMODULE pGblMod)
4339{
4340 Assert(pGblMod->cUsers == 0);
4341 Assert(pGMM->cShareableModules > 0 && pGMM->cShareableModules <= GMM_MAX_SHARED_GLOBAL_MODULES);
4342
4343 void *pvTest = RTAvllU32RemoveNode(&pGMM->pGlobalSharedModuleTree, &pGblMod->Core);
4344 Assert(pvTest == pGblMod); NOREF(pvTest);
4345 pGMM->cShareableModules--;
4346
4347 uint32_t i = pGblMod->cRegions;
4348 while (i-- > 0)
4349 {
4350 if (pGblMod->aRegions[i].paidPages)
4351 {
4352 /* We don't doing anything to the pages as they are handled by the
4353 copy-on-write mechanism in PGM. */
4354 RTMemFree(pGblMod->aRegions[i].paidPages);
4355 pGblMod->aRegions[i].paidPages = NULL;
4356 }
4357 }
4358 RTMemFree(pGblMod);
4359}
4360
4361
4362static int gmmR0ShModNewPerVM(PGVM pGVM, RTGCPTR GCBaseAddr, uint32_t cRegions, const VMMDEVSHAREDREGIONDESC *paRegions,
4363 PGMMSHAREDMODULEPERVM *ppRecVM)
4364{
4365 if (pGVM->gmm.s.Stats.cShareableModules >= GMM_MAX_SHARED_PER_VM_MODULES)
4366 return VERR_GMM_TOO_MANY_PER_VM_MODULES;
4367
4368 PGMMSHAREDMODULEPERVM pRecVM;
4369 pRecVM = (PGMMSHAREDMODULEPERVM)RTMemAllocZ(RT_OFFSETOF(GMMSHAREDMODULEPERVM, aRegionsGCPtrs[cRegions]));
4370 if (!pRecVM)
4371 return VERR_NO_MEMORY;
4372
4373 pRecVM->Core.Key = GCBaseAddr;
4374 for (uint32_t i = 0; i < cRegions; i++)
4375 pRecVM->aRegionsGCPtrs[i] = paRegions[i].GCRegionAddr;
4376
4377 bool fInsert = RTAvlGCPtrInsert(&pGVM->gmm.s.pSharedModuleTree, &pRecVM->Core);
4378 Assert(fInsert); NOREF(fInsert);
4379 pGVM->gmm.s.Stats.cShareableModules++;
4380
4381 *ppRecVM = pRecVM;
4382 return VINF_SUCCESS;
4383}
4384
4385
4386static void gmmR0ShModDeletePerVM(PGMM pGMM, PGVM pGVM, PGMMSHAREDMODULEPERVM pRecVM, bool fRemove)
4387{
4388 /*
4389 * Free the per-VM module.
4390 */
4391 PGMMSHAREDMODULE pGblMod = pRecVM->pGlobalModule;
4392 pRecVM->pGlobalModule = NULL;
4393
4394 if (fRemove)
4395 {
4396 void *pvTest = RTAvlGCPtrRemove(&pGVM->gmm.s.pSharedModuleTree, pRecVM->Core.Key);
4397 Assert(pvTest == &pRecVM->Core);
4398 }
4399
4400 RTMemFree(pRecVM);
4401
4402 /*
4403 * Release the global module.
4404 * (In the registration bailout case, it might not be.)
4405 */
4406 if (pGblMod)
4407 {
4408 Assert(pGblMod->cUsers > 0);
4409 pGblMod->cUsers--;
4410 if (pGblMod->cUsers == 0)
4411 gmmR0ShModDeleteGlobal(pGMM, pGblMod);
4412 }
4413}
4414
4415#endif /* VBOX_WITH_PAGE_SHARING */
4416
4417/**
4418 * Registers a new shared module for the VM.
4419 *
4420 * @returns VBox status code.
4421 * @param pVM Pointer to the VM.
4422 * @param idCpu The VCPU id.
4423 * @param enmGuestOS The guest OS type.
4424 * @param pszModuleName The module name.
4425 * @param pszVersion The module version.
4426 * @param GCPtrModBase The module base address.
4427 * @param cbModule The module size.
4428 * @param cRegions The mumber of shared region descriptors.
4429 * @param paRegions Pointer to an array of shared region(s).
4430 */
4431GMMR0DECL(int) GMMR0RegisterSharedModule(PVM pVM, VMCPUID idCpu, VBOXOSFAMILY enmGuestOS, char *pszModuleName,
4432 char *pszVersion, RTGCPTR GCPtrModBase, uint32_t cbModule,
4433 uint32_t cRegions, struct VMMDEVSHAREDREGIONDESC const *paRegions)
4434{
4435#ifdef VBOX_WITH_PAGE_SHARING
4436 /*
4437 * Validate input and get the basics.
4438 *
4439 * Note! Turns out the module size does necessarily match the size of the
4440 * regions. (iTunes on XP)
4441 */
4442 PGMM pGMM;
4443 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4444 PGVM pGVM;
4445 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
4446 if (RT_FAILURE(rc))
4447 return rc;
4448
4449 if (RT_UNLIKELY(cRegions > VMMDEVSHAREDREGIONDESC_MAX))
4450 return VERR_GMM_TOO_MANY_REGIONS;
4451
4452 if (RT_UNLIKELY(cbModule == 0 || cbModule > _1G))
4453 return VERR_GMM_BAD_SHARED_MODULE_SIZE;
4454
4455 uint32_t cbTotal = 0;
4456 for (uint32_t i = 0; i < cRegions; i++)
4457 {
4458 if (RT_UNLIKELY(paRegions[i].cbRegion == 0 || paRegions[i].cbRegion > _1G))
4459 return VERR_GMM_SHARED_MODULE_BAD_REGIONS_SIZE;
4460
4461 cbTotal += paRegions[i].cbRegion;
4462 if (RT_UNLIKELY(cbTotal > _1G))
4463 return VERR_GMM_SHARED_MODULE_BAD_REGIONS_SIZE;
4464 }
4465
4466 AssertPtrReturn(pszModuleName, VERR_INVALID_POINTER);
4467 if (RT_UNLIKELY(!memchr(pszModuleName, '\0', GMM_SHARED_MODULE_MAX_NAME_STRING)))
4468 return VERR_GMM_MODULE_NAME_TOO_LONG;
4469
4470 AssertPtrReturn(pszVersion, VERR_INVALID_POINTER);
4471 if (RT_UNLIKELY(!memchr(pszVersion, '\0', GMM_SHARED_MODULE_MAX_VERSION_STRING)))
4472 return VERR_GMM_MODULE_NAME_TOO_LONG;
4473
4474 uint32_t const uHash = gmmR0ShModCalcHash(pszModuleName, pszVersion);
4475 Log(("GMMR0RegisterSharedModule %s %s base %RGv size %x hash %x\n", pszModuleName, pszVersion, GCPtrModBase, cbModule, uHash));
4476
4477 /*
4478 * Take the semaphore and do some more validations.
4479 */
4480 gmmR0MutexAcquire(pGMM);
4481 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
4482 {
4483 /*
4484 * Check if this module is already locally registered and register
4485 * it if it isn't. The base address is a unique module identifier
4486 * locally.
4487 */
4488 PGMMSHAREDMODULEPERVM pRecVM = (PGMMSHAREDMODULEPERVM)RTAvlGCPtrGet(&pGVM->gmm.s.pSharedModuleTree, GCPtrModBase);
4489 bool fNewModule = pRecVM == NULL;
4490 if (fNewModule)
4491 {
4492 rc = gmmR0ShModNewPerVM(pGVM, GCPtrModBase, cRegions, paRegions, &pRecVM);
4493 if (RT_SUCCESS(rc))
4494 {
4495 /*
4496 * Find a matching global module, register a new one if needed.
4497 */
4498 PGMMSHAREDMODULE pGblMod = gmmR0ShModFindGlobal(pGMM, uHash, cbModule, enmGuestOS, cRegions,
4499 pszModuleName, pszVersion, paRegions);
4500 if (!pGblMod)
4501 {
4502 Assert(fNewModule);
4503 rc = gmmR0ShModNewGlobal(pGMM, uHash, cbModule, enmGuestOS, cRegions,
4504 pszModuleName, pszVersion, paRegions, &pGblMod);
4505 if (RT_SUCCESS(rc))
4506 {
4507 pRecVM->pGlobalModule = pGblMod; /* (One referenced returned by gmmR0ShModNewGlobal.) */
4508 Log(("GMMR0RegisterSharedModule: new module %s %s\n", pszModuleName, pszVersion));
4509 }
4510 else
4511 gmmR0ShModDeletePerVM(pGMM, pGVM, pRecVM, true /*fRemove*/);
4512 }
4513 else
4514 {
4515 Assert(pGblMod->cUsers > 0 && pGblMod->cUsers < UINT32_MAX / 2);
4516 pGblMod->cUsers++;
4517 pRecVM->pGlobalModule = pGblMod;
4518
4519 Log(("GMMR0RegisterSharedModule: new per vm module %s %s, gbl users %d\n", pszModuleName, pszVersion, pGblMod->cUsers));
4520 }
4521 }
4522 }
4523 else
4524 {
4525 /*
4526 * Attempt to re-register an existing module.
4527 */
4528 PGMMSHAREDMODULE pGblMod = gmmR0ShModFindGlobal(pGMM, uHash, cbModule, enmGuestOS, cRegions,
4529 pszModuleName, pszVersion, paRegions);
4530 if (pRecVM->pGlobalModule == pGblMod)
4531 {
4532 Log(("GMMR0RegisterSharedModule: already registered %s %s, gbl users %d\n", pszModuleName, pszVersion, pGblMod->cUsers));
4533 rc = VINF_GMM_SHARED_MODULE_ALREADY_REGISTERED;
4534 }
4535 else
4536 {
4537 /** @todo may have to unregister+register when this happens in case it's caused
4538 * by VBoxService crashing and being restarted... */
4539 Log(("GMMR0RegisterSharedModule: Address clash!\n"
4540 " incoming at %RGvLB%#x %s %s rgns %u\n"
4541 " existing at %RGvLB%#x %s %s rgns %u\n",
4542 GCPtrModBase, cbModule, pszModuleName, pszVersion, cRegions,
4543 pRecVM->Core.Key, pRecVM->pGlobalModule->cbModule, pRecVM->pGlobalModule->szName,
4544 pRecVM->pGlobalModule->szVersion, pRecVM->pGlobalModule->cRegions));
4545 rc = VERR_GMM_SHARED_MODULE_ADDRESS_CLASH;
4546 }
4547 }
4548 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
4549 }
4550 else
4551 rc = VERR_GMM_IS_NOT_SANE;
4552
4553 gmmR0MutexRelease(pGMM);
4554 return rc;
4555#else
4556
4557 NOREF(pVM); NOREF(idCpu); NOREF(enmGuestOS); NOREF(pszModuleName); NOREF(pszVersion);
4558 NOREF(GCPtrModBase); NOREF(cbModule); NOREF(cRegions); NOREF(paRegions);
4559 return VERR_NOT_IMPLEMENTED;
4560#endif
4561}
4562
4563
4564/**
4565 * VMMR0 request wrapper for GMMR0RegisterSharedModule.
4566 *
4567 * @returns see GMMR0RegisterSharedModule.
4568 * @param pVM Pointer to the VM.
4569 * @param idCpu The VCPU id.
4570 * @param pReq Pointer to the request packet.
4571 */
4572GMMR0DECL(int) GMMR0RegisterSharedModuleReq(PVM pVM, VMCPUID idCpu, PGMMREGISTERSHAREDMODULEREQ pReq)
4573{
4574 /*
4575 * Validate input and pass it on.
4576 */
4577 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
4578 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
4579 AssertMsgReturn(pReq->Hdr.cbReq >= sizeof(*pReq) && pReq->Hdr.cbReq == RT_UOFFSETOF(GMMREGISTERSHAREDMODULEREQ, aRegions[pReq->cRegions]), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
4580
4581 /* Pass back return code in the request packet to preserve informational codes. (VMMR3CallR0 chokes on them) */
4582 pReq->rc = GMMR0RegisterSharedModule(pVM, idCpu, pReq->enmGuestOS, pReq->szName, pReq->szVersion,
4583 pReq->GCBaseAddr, pReq->cbModule, pReq->cRegions, pReq->aRegions);
4584 return VINF_SUCCESS;
4585}
4586
4587
4588/**
4589 * Unregisters a shared module for the VM
4590 *
4591 * @returns VBox status code.
4592 * @param pVM Pointer to the VM.
4593 * @param idCpu The VCPU id.
4594 * @param pszModuleName The module name.
4595 * @param pszVersion The module version.
4596 * @param GCPtrModBase The module base address.
4597 * @param cbModule The module size.
4598 */
4599GMMR0DECL(int) GMMR0UnregisterSharedModule(PVM pVM, VMCPUID idCpu, char *pszModuleName, char *pszVersion,
4600 RTGCPTR GCPtrModBase, uint32_t cbModule)
4601{
4602#ifdef VBOX_WITH_PAGE_SHARING
4603 /*
4604 * Validate input and get the basics.
4605 */
4606 PGMM pGMM;
4607 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4608 PGVM pGVM;
4609 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
4610 if (RT_FAILURE(rc))
4611 return rc;
4612
4613 AssertPtrReturn(pszModuleName, VERR_INVALID_POINTER);
4614 AssertPtrReturn(pszVersion, VERR_INVALID_POINTER);
4615 if (RT_UNLIKELY(!memchr(pszModuleName, '\0', GMM_SHARED_MODULE_MAX_NAME_STRING)))
4616 return VERR_GMM_MODULE_NAME_TOO_LONG;
4617 if (RT_UNLIKELY(!memchr(pszVersion, '\0', GMM_SHARED_MODULE_MAX_VERSION_STRING)))
4618 return VERR_GMM_MODULE_NAME_TOO_LONG;
4619
4620 Log(("GMMR0UnregisterSharedModule %s %s base=%RGv size %x\n", pszModuleName, pszVersion, GCPtrModBase, cbModule));
4621
4622 /*
4623 * Take the semaphore and do some more validations.
4624 */
4625 gmmR0MutexAcquire(pGMM);
4626 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
4627 {
4628 /*
4629 * Locate and remove the specified module.
4630 */
4631 PGMMSHAREDMODULEPERVM pRecVM = (PGMMSHAREDMODULEPERVM)RTAvlGCPtrGet(&pGVM->gmm.s.pSharedModuleTree, GCPtrModBase);
4632 if (pRecVM)
4633 {
4634 /** @todo Do we need to do more validations here, like that the
4635 * name + version + cbModule matches? */
4636 Assert(pRecVM->pGlobalModule);
4637 gmmR0ShModDeletePerVM(pGMM, pGVM, pRecVM, true /*fRemove*/);
4638 }
4639 else
4640 rc = VERR_GMM_SHARED_MODULE_NOT_FOUND;
4641
4642 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
4643 }
4644 else
4645 rc = VERR_GMM_IS_NOT_SANE;
4646
4647 gmmR0MutexRelease(pGMM);
4648 return rc;
4649#else
4650
4651 NOREF(pVM); NOREF(idCpu); NOREF(pszModuleName); NOREF(pszVersion); NOREF(GCPtrModBase); NOREF(cbModule);
4652 return VERR_NOT_IMPLEMENTED;
4653#endif
4654}
4655
4656
4657/**
4658 * VMMR0 request wrapper for GMMR0UnregisterSharedModule.
4659 *
4660 * @returns see GMMR0UnregisterSharedModule.
4661 * @param pVM Pointer to the VM.
4662 * @param idCpu The VCPU id.
4663 * @param pReq Pointer to the request packet.
4664 */
4665GMMR0DECL(int) GMMR0UnregisterSharedModuleReq(PVM pVM, VMCPUID idCpu, PGMMUNREGISTERSHAREDMODULEREQ pReq)
4666{
4667 /*
4668 * Validate input and pass it on.
4669 */
4670 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
4671 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
4672 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
4673
4674 return GMMR0UnregisterSharedModule(pVM, idCpu, pReq->szName, pReq->szVersion, pReq->GCBaseAddr, pReq->cbModule);
4675}
4676
4677#ifdef VBOX_WITH_PAGE_SHARING
4678
4679/**
4680 * Increase the use count of a shared page, the page is known to exist and be valid and such.
4681 *
4682 * @param pGMM Pointer to the GMM instance.
4683 * @param pGVM Pointer to the GVM instance.
4684 * @param pPage The page structure.
4685 */
4686DECLINLINE(void) gmmR0UseSharedPage(PGMM pGMM, PGVM pGVM, PGMMPAGE pPage)
4687{
4688 Assert(pGMM->cSharedPages > 0);
4689 Assert(pGMM->cAllocatedPages > 0);
4690
4691 pGMM->cDuplicatePages++;
4692
4693 pPage->Shared.cRefs++;
4694 pGVM->gmm.s.Stats.cSharedPages++;
4695 pGVM->gmm.s.Stats.Allocated.cBasePages++;
4696}
4697
4698
4699/**
4700 * Converts a private page to a shared page, the page is known to exist and be valid and such.
4701 *
4702 * @param pGMM Pointer to the GMM instance.
4703 * @param pGVM Pointer to the GVM instance.
4704 * @param HCPhys Host physical address
4705 * @param idPage The Page ID
4706 * @param pPage The page structure.
4707 */
4708DECLINLINE(void) gmmR0ConvertToSharedPage(PGMM pGMM, PGVM pGVM, RTHCPHYS HCPhys, uint32_t idPage, PGMMPAGE pPage)
4709{
4710 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
4711 Assert(pChunk);
4712 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
4713 Assert(GMM_PAGE_IS_PRIVATE(pPage));
4714
4715 pChunk->cPrivate--;
4716 pChunk->cShared++;
4717
4718 pGMM->cSharedPages++;
4719
4720 pGVM->gmm.s.Stats.cSharedPages++;
4721 pGVM->gmm.s.Stats.cPrivatePages--;
4722
4723 /* Modify the page structure. */
4724 pPage->Shared.pfn = (uint32_t)(uint64_t)(HCPhys >> PAGE_SHIFT);
4725 pPage->Shared.cRefs = 1;
4726#ifdef VBOX_STRICT
4727 pPage->Shared.u14Checksum = gmmR0StrictPageChecksum(pGMM, pGVM, idPage);
4728#else
4729 pPage->Shared.u14Checksum = 0;
4730#endif
4731 pPage->Shared.u2State = GMM_PAGE_STATE_SHARED;
4732}
4733
4734
4735static int gmmR0SharedModuleCheckPageFirstTime(PGMM pGMM, PGVM pGVM, PGMMSHAREDMODULE pModule,
4736 unsigned idxRegion, unsigned idxPage,
4737 PGMMSHAREDPAGEDESC pPageDesc, PGMMSHAREDREGIONDESC pGlobalRegion)
4738{
4739 /* Easy case: just change the internal page type. */
4740 PGMMPAGE pPage = gmmR0GetPage(pGMM, pPageDesc->idPage);
4741 AssertMsgReturn(pPage, ("idPage=%#x (GCPhys=%RGp HCPhys=%RHp idxRegion=%#x idxPage=%#x) #1\n",
4742 pPageDesc->idPage, pPageDesc->GCPhys, pPageDesc->HCPhys, idxRegion, idxPage),
4743 VERR_PGM_PHYS_INVALID_PAGE_ID);
4744
4745 AssertMsg(pPageDesc->GCPhys == (pPage->Private.pfn << 12), ("desc %RGp gmm %RGp\n", pPageDesc->HCPhys, (pPage->Private.pfn << 12)));
4746
4747 gmmR0ConvertToSharedPage(pGMM, pGVM, pPageDesc->HCPhys, pPageDesc->idPage, pPage);
4748
4749 /* Keep track of these references. */
4750 pGlobalRegion->paidPages[idxPage] = pPageDesc->idPage;
4751
4752 return VINF_SUCCESS;
4753}
4754
4755/**
4756 * Checks specified shared module range for changes
4757 *
4758 * Performs the following tasks:
4759 * - If a shared page is new, then it changes the GMM page type to shared and
4760 * returns it in the pPageDesc descriptor.
4761 * - If a shared page already exists, then it checks if the VM page is
4762 * identical and if so frees the VM page and returns the shared page in
4763 * pPageDesc descriptor.
4764 *
4765 * @remarks ASSUMES the caller has acquired the GMM semaphore!!
4766 *
4767 * @returns VBox status code.
4768 * @param pGMM Pointer to the GMM instance data.
4769 * @param pGVM Pointer to the GVM instance data.
4770 * @param pModule Module description
4771 * @param idxRegion Region index
4772 * @param idxPage Page index
4773 * @param paPageDesc Page descriptor
4774 */
4775GMMR0DECL(int) GMMR0SharedModuleCheckPage(PGVM pGVM, PGMMSHAREDMODULE pModule, uint32_t idxRegion, uint32_t idxPage,
4776 PGMMSHAREDPAGEDESC pPageDesc)
4777{
4778 int rc;
4779 PGMM pGMM;
4780 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4781
4782 AssertMsgReturn(idxRegion < pModule->cRegions,
4783 ("idxRegion=%#x cRegions=%#x %s %s\n", idxRegion, pModule->cRegions, pModule->szName, pModule->szVersion),
4784 VERR_INVALID_PARAMETER);
4785
4786 uint32_t const cPages = pModule->aRegions[idxRegion].cb >> PAGE_SHIFT;
4787 AssertMsgReturn(idxPage < cPages,
4788 ("idxRegion=%#x cRegions=%#x %s %s\n", idxRegion, pModule->cRegions, pModule->szName, pModule->szVersion),
4789 VERR_INVALID_PARAMETER);
4790
4791 LogFlow(("GMMR0SharedModuleCheckRange %s base %RGv region %d idxPage %d\n", pModule->szName, pModule->Core.Key, idxRegion, idxPage));
4792
4793 /*
4794 * First time; create a page descriptor array.
4795 */
4796 PGMMSHAREDREGIONDESC pGlobalRegion = &pModule->aRegions[idxRegion];
4797 if (!pGlobalRegion->paidPages)
4798 {
4799 Log(("Allocate page descriptor array for %d pages\n", cPages));
4800 pGlobalRegion->paidPages = (uint32_t *)RTMemAlloc(cPages * sizeof(pGlobalRegion->paidPages[0]));
4801 AssertReturn(pGlobalRegion->paidPages, VERR_NO_MEMORY);
4802
4803 /* Invalidate all descriptors. */
4804 uint32_t i = cPages;
4805 while (i-- > 0)
4806 pGlobalRegion->paidPages[i] = NIL_GMM_PAGEID;
4807 }
4808
4809 /*
4810 * We've seen this shared page for the first time?
4811 */
4812 if (pGlobalRegion->paidPages[idxPage] == NIL_GMM_PAGEID)
4813 {
4814 Log(("New shared page guest %RGp host %RHp\n", pPageDesc->GCPhys, pPageDesc->HCPhys));
4815 return gmmR0SharedModuleCheckPageFirstTime(pGMM, pGVM, pModule, idxRegion, idxPage, pPageDesc, pGlobalRegion);
4816 }
4817
4818 /*
4819 * We've seen it before...
4820 */
4821 Log(("Replace existing page guest %RGp host %RHp id %#x -> id %#x\n",
4822 pPageDesc->GCPhys, pPageDesc->HCPhys, pPageDesc->idPage, pGlobalRegion->paidPages[idxPage]));
4823 Assert(pPageDesc->idPage != pGlobalRegion->paidPages[idxPage]);
4824
4825 /*
4826 * Get the shared page source.
4827 */
4828 PGMMPAGE pPage = gmmR0GetPage(pGMM, pGlobalRegion->paidPages[idxPage]);
4829 AssertMsgReturn(pPage, ("idPage=%#x (idxRegion=%#x idxPage=%#x) #2\n", pPageDesc->idPage, idxRegion, idxPage),
4830 VERR_PGM_PHYS_INVALID_PAGE_ID);
4831
4832 if (pPage->Common.u2State != GMM_PAGE_STATE_SHARED)
4833 {
4834 /*
4835 * Page was freed at some point; invalidate this entry.
4836 */
4837 /** @todo this isn't really bullet proof. */
4838 Log(("Old shared page was freed -> create a new one\n"));
4839 pGlobalRegion->paidPages[idxPage] = NIL_GMM_PAGEID;
4840 return gmmR0SharedModuleCheckPageFirstTime(pGMM, pGVM, pModule, idxRegion, idxPage, pPageDesc, pGlobalRegion);
4841 }
4842
4843 Log(("Replace existing page guest host %RHp -> %RHp\n", pPageDesc->HCPhys, ((uint64_t)pPage->Shared.pfn) << PAGE_SHIFT));
4844
4845 /*
4846 * Calculate the virtual address of the local page.
4847 */
4848 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, pPageDesc->idPage >> GMM_CHUNKID_SHIFT);
4849 AssertMsgReturn(pChunk, ("idPage=%#x (idxRegion=%#x idxPage=%#x) #4\n", pPageDesc->idPage, idxRegion, idxPage),
4850 VERR_PGM_PHYS_INVALID_PAGE_ID);
4851
4852 uint8_t *pbChunk;
4853 AssertMsgReturn(gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk),
4854 ("idPage=%#x (idxRegion=%#x idxPage=%#x) #3\n", pPageDesc->idPage, idxRegion, idxPage),
4855 VERR_PGM_PHYS_INVALID_PAGE_ID);
4856 uint8_t *pbLocalPage = pbChunk + ((pPageDesc->idPage & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
4857
4858 /*
4859 * Calculate the virtual address of the shared page.
4860 */
4861 pChunk = gmmR0GetChunk(pGMM, pGlobalRegion->paidPages[idxPage] >> GMM_CHUNKID_SHIFT);
4862 Assert(pChunk); /* can't fail as gmmR0GetPage succeeded. */
4863
4864 /*
4865 * Get the virtual address of the physical page; map the chunk into the VM
4866 * process if not already done.
4867 */
4868 if (!gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
4869 {
4870 Log(("Map chunk into process!\n"));
4871 rc = gmmR0MapChunk(pGMM, pGVM, pChunk, false /*fRelaxedSem*/, (PRTR3PTR)&pbChunk);
4872 AssertRCReturn(rc, rc);
4873 }
4874 uint8_t *pbSharedPage = pbChunk + ((pGlobalRegion->paidPages[idxPage] & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
4875#ifdef VBOX_STRICT
4876 if (pPage->Shared.u14Checksum)
4877 {
4878 uint32_t uChecksum = RTCrc32(pbSharedPage, PAGE_SIZE) & UINT32_C(0x00003fff);
4879 AssertMsg(!uChecksum || uChecksum == pPage->Shared.u14Checksum,
4880 ("%#x vs %#x - idPage=%# - %s %s\n", uChecksum, pPage->Shared.u14Checksum,
4881 pGlobalRegion->paidPages[idxPage], pModule->szName, pModule->szVersion));
4882 }
4883#endif
4884
4885 /** @todo write ASMMemComparePage. */
4886 if (memcmp(pbSharedPage, pbLocalPage, PAGE_SIZE))
4887 {
4888 Log(("Unexpected differences found between local and shared page; skip\n"));
4889 /* Signal to the caller that this one hasn't changed. */
4890 pPageDesc->idPage = NIL_GMM_PAGEID;
4891 return VINF_SUCCESS;
4892 }
4893
4894 /*
4895 * Free the old local page.
4896 */
4897 GMMFREEPAGEDESC PageDesc;
4898 PageDesc.idPage = pPageDesc->idPage;
4899 rc = gmmR0FreePages(pGMM, pGVM, 1, &PageDesc, GMMACCOUNT_BASE);
4900 AssertRCReturn(rc, rc);
4901
4902 gmmR0UseSharedPage(pGMM, pGVM, pPage);
4903
4904 /*
4905 * Pass along the new physical address & page id.
4906 */
4907 pPageDesc->HCPhys = ((uint64_t)pPage->Shared.pfn) << PAGE_SHIFT;
4908 pPageDesc->idPage = pGlobalRegion->paidPages[idxPage];
4909
4910 return VINF_SUCCESS;
4911}
4912
4913
4914/**
4915 * RTAvlGCPtrDestroy callback.
4916 *
4917 * @returns 0 or VERR_GMM_INSTANCE.
4918 * @param pNode The node to destroy.
4919 * @param pvArgs Pointer to an argument packet.
4920 */
4921static DECLCALLBACK(int) gmmR0CleanupSharedModule(PAVLGCPTRNODECORE pNode, void *pvArgs)
4922{
4923 gmmR0ShModDeletePerVM(((GMMR0SHMODPERVMDTORARGS *)pvArgs)->pGMM,
4924 ((GMMR0SHMODPERVMDTORARGS *)pvArgs)->pGVM,
4925 (PGMMSHAREDMODULEPERVM)pNode,
4926 false /*fRemove*/);
4927 return VINF_SUCCESS;
4928}
4929
4930
4931/**
4932 * Used by GMMR0CleanupVM to clean up shared modules.
4933 *
4934 * This is called without taking the GMM lock so that it can be yielded as
4935 * needed here.
4936 *
4937 * @param pGMM The GMM handle.
4938 * @param pGVM The global VM handle.
4939 */
4940static void gmmR0SharedModuleCleanup(PGMM pGMM, PGVM pGVM)
4941{
4942 gmmR0MutexAcquire(pGMM);
4943 GMM_CHECK_SANITY_UPON_ENTERING(pGMM);
4944
4945 GMMR0SHMODPERVMDTORARGS Args;
4946 Args.pGVM = pGVM;
4947 Args.pGMM = pGMM;
4948 RTAvlGCPtrDestroy(&pGVM->gmm.s.pSharedModuleTree, gmmR0CleanupSharedModule, &Args);
4949
4950 Assert(pGVM->gmm.s.Stats.cShareableModules == 0);
4951 pGVM->gmm.s.Stats.cShareableModules = 0;
4952
4953 gmmR0MutexRelease(pGMM);
4954}
4955
4956#endif /* VBOX_WITH_PAGE_SHARING */
4957
4958/**
4959 * Removes all shared modules for the specified VM
4960 *
4961 * @returns VBox status code.
4962 * @param pVM Pointer to the VM.
4963 * @param idCpu The VCPU id.
4964 */
4965GMMR0DECL(int) GMMR0ResetSharedModules(PVM pVM, VMCPUID idCpu)
4966{
4967#ifdef VBOX_WITH_PAGE_SHARING
4968 /*
4969 * Validate input and get the basics.
4970 */
4971 PGMM pGMM;
4972 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4973 PGVM pGVM;
4974 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
4975 if (RT_FAILURE(rc))
4976 return rc;
4977
4978 /*
4979 * Take the semaphore and do some more validations.
4980 */
4981 gmmR0MutexAcquire(pGMM);
4982 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
4983 {
4984 Log(("GMMR0ResetSharedModules\n"));
4985 GMMR0SHMODPERVMDTORARGS Args;
4986 Args.pGVM = pGVM;
4987 Args.pGMM = pGMM;
4988 RTAvlGCPtrDestroy(&pGVM->gmm.s.pSharedModuleTree, gmmR0CleanupSharedModule, &Args);
4989 pGVM->gmm.s.Stats.cShareableModules = 0;
4990
4991 rc = VINF_SUCCESS;
4992 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
4993 }
4994 else
4995 rc = VERR_GMM_IS_NOT_SANE;
4996
4997 gmmR0MutexRelease(pGMM);
4998 return rc;
4999#else
5000 NOREF(pVM); NOREF(idCpu);
5001 return VERR_NOT_IMPLEMENTED;
5002#endif
5003}
5004
5005#ifdef VBOX_WITH_PAGE_SHARING
5006
5007/**
5008 * Tree enumeration callback for checking a shared module.
5009 */
5010static DECLCALLBACK(int) gmmR0CheckSharedModule(PAVLGCPTRNODECORE pNode, void *pvUser)
5011{
5012 GMMCHECKSHAREDMODULEINFO *pArgs = (GMMCHECKSHAREDMODULEINFO*)pvUser;
5013 PGMMSHAREDMODULEPERVM pRecVM = (PGMMSHAREDMODULEPERVM)pNode;
5014 PGMMSHAREDMODULE pGblMod = pRecVM->pGlobalModule;
5015
5016 Log(("gmmR0CheckSharedModule: check %s %s base=%RGv size=%x\n",
5017 pGblMod->szName, pGblMod->szVersion, pGblMod->Core.Key, pGblMod->cbModule));
5018
5019 int rc = PGMR0SharedModuleCheck(pArgs->pGVM->pVM, pArgs->pGVM, pArgs->idCpu, pGblMod, pRecVM->aRegionsGCPtrs);
5020 if (RT_FAILURE(rc))
5021 return rc;
5022 return VINF_SUCCESS;
5023}
5024
5025#endif /* VBOX_WITH_PAGE_SHARING */
5026#ifdef DEBUG_sandervl
5027
5028/**
5029 * Setup for a GMMR0CheckSharedModules call (to allow log flush jumps back to ring 3)
5030 *
5031 * @returns VBox status code.
5032 * @param pVM Pointer to the VM.
5033 */
5034GMMR0DECL(int) GMMR0CheckSharedModulesStart(PVM pVM)
5035{
5036 /*
5037 * Validate input and get the basics.
5038 */
5039 PGMM pGMM;
5040 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5041
5042 /*
5043 * Take the semaphore and do some more validations.
5044 */
5045 gmmR0MutexAcquire(pGMM);
5046 if (!GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
5047 rc = VERR_GMM_IS_NOT_SANE;
5048 else
5049 rc = VINF_SUCCESS;
5050
5051 return rc;
5052}
5053
5054/**
5055 * Clean up after a GMMR0CheckSharedModules call (to allow log flush jumps back to ring 3)
5056 *
5057 * @returns VBox status code.
5058 * @param pVM Pointer to the VM.
5059 */
5060GMMR0DECL(int) GMMR0CheckSharedModulesEnd(PVM pVM)
5061{
5062 /*
5063 * Validate input and get the basics.
5064 */
5065 PGMM pGMM;
5066 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5067
5068 gmmR0MutexRelease(pGMM);
5069 return VINF_SUCCESS;
5070}
5071
5072#endif /* DEBUG_sandervl */
5073
5074/**
5075 * Check all shared modules for the specified VM.
5076 *
5077 * @returns VBox status code.
5078 * @param pVM Pointer to the VM.
5079 * @param pVCpu Pointer to the VMCPU.
5080 */
5081GMMR0DECL(int) GMMR0CheckSharedModules(PVM pVM, PVMCPU pVCpu)
5082{
5083#ifdef VBOX_WITH_PAGE_SHARING
5084 /*
5085 * Validate input and get the basics.
5086 */
5087 PGMM pGMM;
5088 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5089 PGVM pGVM;
5090 int rc = GVMMR0ByVMAndEMT(pVM, pVCpu->idCpu, &pGVM);
5091 if (RT_FAILURE(rc))
5092 return rc;
5093
5094# ifndef DEBUG_sandervl
5095 /*
5096 * Take the semaphore and do some more validations.
5097 */
5098 gmmR0MutexAcquire(pGMM);
5099# endif
5100 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
5101 {
5102 /*
5103 * Walk the tree, checking each module.
5104 */
5105 Log(("GMMR0CheckSharedModules\n"));
5106
5107 GMMCHECKSHAREDMODULEINFO Args;
5108 Args.pGVM = pGVM;
5109 Args.idCpu = pVCpu->idCpu;
5110 rc = RTAvlGCPtrDoWithAll(&pGVM->gmm.s.pSharedModuleTree, true /* fFromLeft */, gmmR0CheckSharedModule, &Args);
5111
5112 Log(("GMMR0CheckSharedModules done!\n"));
5113 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
5114 }
5115 else
5116 rc = VERR_GMM_IS_NOT_SANE;
5117
5118# ifndef DEBUG_sandervl
5119 gmmR0MutexRelease(pGMM);
5120# endif
5121 return rc;
5122#else
5123 NOREF(pVM); NOREF(pVCpu);
5124 return VERR_NOT_IMPLEMENTED;
5125#endif
5126}
5127
5128#if defined(VBOX_STRICT) && HC_ARCH_BITS == 64
5129
5130/**
5131 * RTAvlU32DoWithAll callback.
5132 *
5133 * @returns 0
5134 * @param pNode The node to search.
5135 * @param pvUser Pointer to the input argument packet.
5136 */
5137static DECLCALLBACK(int) gmmR0FindDupPageInChunk(PAVLU32NODECORE pNode, void *pvUser)
5138{
5139 PGMMCHUNK pChunk = (PGMMCHUNK)pNode;
5140 GMMFINDDUPPAGEINFO *pArgs = (GMMFINDDUPPAGEINFO *)pvUser;
5141 PGVM pGVM = pArgs->pGVM;
5142 PGMM pGMM = pArgs->pGMM;
5143 uint8_t *pbChunk;
5144
5145 /* Only take chunks not mapped into this VM process; not entirely correct. */
5146 if (!gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
5147 {
5148 int rc = gmmR0MapChunk(pGMM, pGVM, pChunk, false /*fRelaxedSem*/, (PRTR3PTR)&pbChunk);
5149 if (RT_SUCCESS(rc))
5150 {
5151 /*
5152 * Look for duplicate pages
5153 */
5154 unsigned iPage = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
5155 while (iPage-- > 0)
5156 {
5157 if (GMM_PAGE_IS_PRIVATE(&pChunk->aPages[iPage]))
5158 {
5159 uint8_t *pbDestPage = pbChunk + (iPage << PAGE_SHIFT);
5160
5161 if (!memcmp(pArgs->pSourcePage, pbDestPage, PAGE_SIZE))
5162 {
5163 pArgs->fFoundDuplicate = true;
5164 break;
5165 }
5166 }
5167 }
5168 gmmR0UnmapChunk(pGMM, pGVM, pChunk, false /*fRelaxedSem*/);
5169 }
5170 }
5171 return pArgs->fFoundDuplicate; /* (stops search if true) */
5172}
5173
5174
5175/**
5176 * Find a duplicate of the specified page in other active VMs
5177 *
5178 * @returns VBox status code.
5179 * @param pVM Pointer to the VM.
5180 * @param pReq Pointer to the request packet.
5181 */
5182GMMR0DECL(int) GMMR0FindDuplicatePageReq(PVM pVM, PGMMFINDDUPLICATEPAGEREQ pReq)
5183{
5184 /*
5185 * Validate input and pass it on.
5186 */
5187 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
5188 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
5189 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
5190
5191 PGMM pGMM;
5192 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5193
5194 PGVM pGVM;
5195 int rc = GVMMR0ByVM(pVM, &pGVM);
5196 if (RT_FAILURE(rc))
5197 return rc;
5198
5199 /*
5200 * Take the semaphore and do some more validations.
5201 */
5202 rc = gmmR0MutexAcquire(pGMM);
5203 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
5204 {
5205 uint8_t *pbChunk;
5206 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, pReq->idPage >> GMM_CHUNKID_SHIFT);
5207 if (pChunk)
5208 {
5209 if (gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
5210 {
5211 uint8_t *pbSourcePage = pbChunk + ((pReq->idPage & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
5212 PGMMPAGE pPage = gmmR0GetPage(pGMM, pReq->idPage);
5213 if (pPage)
5214 {
5215 GMMFINDDUPPAGEINFO Args;
5216 Args.pGVM = pGVM;
5217 Args.pGMM = pGMM;
5218 Args.pSourcePage = pbSourcePage;
5219 Args.fFoundDuplicate = false;
5220 RTAvlU32DoWithAll(&pGMM->pChunks, true /* fFromLeft */, gmmR0FindDupPageInChunk, &Args);
5221
5222 pReq->fDuplicate = Args.fFoundDuplicate;
5223 }
5224 else
5225 {
5226 AssertFailed();
5227 rc = VERR_PGM_PHYS_INVALID_PAGE_ID;
5228 }
5229 }
5230 else
5231 AssertFailed();
5232 }
5233 else
5234 AssertFailed();
5235 }
5236 else
5237 rc = VERR_GMM_IS_NOT_SANE;
5238
5239 gmmR0MutexRelease(pGMM);
5240 return rc;
5241}
5242
5243#endif /* VBOX_STRICT && HC_ARCH_BITS == 64 */
5244
5245
5246/**
5247 * Retrieves the GMM statistics visible to the caller.
5248 *
5249 * @returns VBox status code.
5250 *
5251 * @param pStats Where to put the statistics.
5252 * @param pSession The current session.
5253 * @param pVM Pointer to the VM to obtain statistics for. Optional.
5254 */
5255GMMR0DECL(int) GMMR0QueryStatistics(PGMMSTATS pStats, PSUPDRVSESSION pSession, PVM pVM)
5256{
5257 LogFlow(("GVMMR0QueryStatistics: pStats=%p pSession=%p pVM=%p\n", pStats, pSession, pVM));
5258
5259 /*
5260 * Validate input.
5261 */
5262 AssertPtrReturn(pSession, VERR_INVALID_POINTER);
5263 AssertPtrReturn(pStats, VERR_INVALID_POINTER);
5264 pStats->cMaxPages = 0; /* (crash before taking the mutex...) */
5265
5266 PGMM pGMM;
5267 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5268
5269 /*
5270 * Resolve the VM handle, if not NULL, and lock the GMM.
5271 */
5272 int rc;
5273 PGVM pGVM;
5274 if (pVM)
5275 {
5276 rc = GVMMR0ByVM(pVM, &pGVM);
5277 if (RT_FAILURE(rc))
5278 return rc;
5279 }
5280 else
5281 pGVM = NULL;
5282
5283 rc = gmmR0MutexAcquire(pGMM);
5284 if (RT_FAILURE(rc))
5285 return rc;
5286
5287 /*
5288 * Copy out the GMM statistics.
5289 */
5290 pStats->cMaxPages = pGMM->cMaxPages;
5291 pStats->cReservedPages = pGMM->cReservedPages;
5292 pStats->cOverCommittedPages = pGMM->cOverCommittedPages;
5293 pStats->cAllocatedPages = pGMM->cAllocatedPages;
5294 pStats->cSharedPages = pGMM->cSharedPages;
5295 pStats->cDuplicatePages = pGMM->cDuplicatePages;
5296 pStats->cLeftBehindSharedPages = pGMM->cLeftBehindSharedPages;
5297 pStats->cBalloonedPages = pGMM->cBalloonedPages;
5298 pStats->cChunks = pGMM->cChunks;
5299 pStats->cFreedChunks = pGMM->cFreedChunks;
5300 pStats->cShareableModules = pGMM->cShareableModules;
5301 RT_ZERO(pStats->au64Reserved);
5302
5303 /*
5304 * Copy out the VM statistics.
5305 */
5306 if (pGVM)
5307 pStats->VMStats = pGVM->gmm.s.Stats;
5308 else
5309 RT_ZERO(pStats->VMStats);
5310
5311 gmmR0MutexRelease(pGMM);
5312 return rc;
5313}
5314
5315
5316/**
5317 * VMMR0 request wrapper for GMMR0QueryStatistics.
5318 *
5319 * @returns see GMMR0QueryStatistics.
5320 * @param pVM Pointer to the VM. Optional.
5321 * @param pReq Pointer to the request packet.
5322 */
5323GMMR0DECL(int) GMMR0QueryStatisticsReq(PVM pVM, PGMMQUERYSTATISTICSSREQ pReq)
5324{
5325 /*
5326 * Validate input and pass it on.
5327 */
5328 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
5329 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
5330
5331 return GMMR0QueryStatistics(&pReq->Stats, pReq->pSession, pVM);
5332}
5333
5334
5335/**
5336 * Resets the specified GMM statistics.
5337 *
5338 * @returns VBox status code.
5339 *
5340 * @param pStats Which statistics to reset, that is, non-zero fields
5341 * indicates which to reset.
5342 * @param pSession The current session.
5343 * @param pVM The VM to reset statistics for. Optional.
5344 */
5345GMMR0DECL(int) GMMR0ResetStatistics(PCGMMSTATS pStats, PSUPDRVSESSION pSession, PVM pVM)
5346{
5347 /* Currently nothing we can reset at the moment. */
5348 return VINF_SUCCESS;
5349}
5350
5351
5352/**
5353 * VMMR0 request wrapper for GMMR0ResetStatistics.
5354 *
5355 * @returns see GMMR0ResetStatistics.
5356 * @param pVM Pointer to the VM. Optional.
5357 * @param pReq Pointer to the request packet.
5358 */
5359GMMR0DECL(int) GMMR0ResetStatisticsReq(PVM pVM, PGMMRESETSTATISTICSSREQ pReq)
5360{
5361 /*
5362 * Validate input and pass it on.
5363 */
5364 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
5365 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
5366
5367 return GMMR0ResetStatistics(&pReq->Stats, pReq->pSession, pVM);
5368}
5369
Note: See TracBrowser for help on using the repository browser.

© 2024 Oracle Support Privacy / Do Not Sell My Info Terms of Use Trademark Policy Automated Access Etiquette