VirtualBox

source: vbox/trunk/src/VBox/VMM/VMMR0/GMMR0.cpp@ 66058

Last change on this file since 66058 was 63465, checked in by vboxsync, 8 years ago

VMM: warnings (clang)
  • Property svn:eol-style set to native
  • Property svn:keywords set to Id Revision
File size: 190.9 KB
Line 
1/* $Id: GMMR0.cpp 63465 2016-08-15 10:00:20Z vboxsync $ */
2/** @file
3 * GMM - Global Memory Manager.
4 */
5
6/*
7 * Copyright (C) 2007-2016 Oracle Corporation
8 *
9 * This file is part of VirtualBox Open Source Edition (OSE), as
10 * available from http://www.virtualbox.org. This file is free software;
11 * you can redistribute it and/or modify it under the terms of the GNU
12 * General Public License (GPL) as published by the Free Software
13 * Foundation, in version 2 as it comes in the "COPYING" file of the
14 * VirtualBox OSE distribution. VirtualBox OSE is distributed in the
15 * hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
16 */
17
18
19/** @page pg_gmm GMM - The Global Memory Manager
20 *
21 * As the name indicates, this component is responsible for global memory
22 * management. Currently only guest RAM is allocated from the GMM, but this
23 * may change to include shadow page tables and other bits later.
24 *
25 * Guest RAM is managed as individual pages, but allocated from the host OS
26 * in chunks for reasons of portability / efficiency. To minimize the memory
27 * footprint all tracking structure must be as small as possible without
28 * unnecessary performance penalties.
29 *
30 * The allocation chunks has fixed sized, the size defined at compile time
31 * by the #GMM_CHUNK_SIZE \#define.
32 *
33 * Each chunk is given an unique ID. Each page also has a unique ID. The
34 * relation ship between the two IDs is:
35 * @code
36 * GMM_CHUNK_SHIFT = log2(GMM_CHUNK_SIZE / PAGE_SIZE);
37 * idPage = (idChunk << GMM_CHUNK_SHIFT) | iPage;
38 * @endcode
39 * Where iPage is the index of the page within the chunk. This ID scheme
40 * permits for efficient chunk and page lookup, but it relies on the chunk size
41 * to be set at compile time. The chunks are organized in an AVL tree with their
42 * IDs being the keys.
43 *
44 * The physical address of each page in an allocation chunk is maintained by
45 * the #RTR0MEMOBJ and obtained using #RTR0MemObjGetPagePhysAddr. There is no
46 * need to duplicate this information (it'll cost 8-bytes per page if we did).
47 *
48 * So what do we need to track per page? Most importantly we need to know
49 * which state the page is in:
50 * - Private - Allocated for (eventually) backing one particular VM page.
51 * - Shared - Readonly page that is used by one or more VMs and treated
52 * as COW by PGM.
53 * - Free - Not used by anyone.
54 *
55 * For the page replacement operations (sharing, defragmenting and freeing)
56 * to be somewhat efficient, private pages needs to be associated with a
57 * particular page in a particular VM.
58 *
59 * Tracking the usage of shared pages is impractical and expensive, so we'll
60 * settle for a reference counting system instead.
61 *
62 * Free pages will be chained on LIFOs
63 *
64 * On 64-bit systems we will use a 64-bit bitfield per page, while on 32-bit
65 * systems a 32-bit bitfield will have to suffice because of address space
66 * limitations. The #GMMPAGE structure shows the details.
67 *
68 *
69 * @section sec_gmm_alloc_strat Page Allocation Strategy
70 *
71 * The strategy for allocating pages has to take fragmentation and shared
72 * pages into account, or we may end up with with 2000 chunks with only
73 * a few pages in each. Shared pages cannot easily be reallocated because
74 * of the inaccurate usage accounting (see above). Private pages can be
75 * reallocated by a defragmentation thread in the same manner that sharing
76 * is done.
77 *
78 * The first approach is to manage the free pages in two sets depending on
79 * whether they are mainly for the allocation of shared or private pages.
80 * In the initial implementation there will be almost no possibility for
81 * mixing shared and private pages in the same chunk (only if we're really
82 * stressed on memory), but when we implement forking of VMs and have to
83 * deal with lots of COW pages it'll start getting kind of interesting.
84 *
85 * The sets are lists of chunks with approximately the same number of
86 * free pages. Say the chunk size is 1MB, meaning 256 pages, and a set
87 * consists of 16 lists. So, the first list will contain the chunks with
88 * 1-7 free pages, the second covers 8-15, and so on. The chunks will be
89 * moved between the lists as pages are freed up or allocated.
90 *
91 *
92 * @section sec_gmm_costs Costs
93 *
94 * The per page cost in kernel space is 32-bit plus whatever RTR0MEMOBJ
95 * entails. In addition there is the chunk cost of approximately
96 * (sizeof(RT0MEMOBJ) + sizeof(CHUNK)) / 2^CHUNK_SHIFT bytes per page.
97 *
98 * On Windows the per page #RTR0MEMOBJ cost is 32-bit on 32-bit windows
99 * and 64-bit on 64-bit windows (a PFN_NUMBER in the MDL). So, 64-bit per page.
100 * The cost on Linux is identical, but here it's because of sizeof(struct page *).
101 *
102 *
103 * @section sec_gmm_legacy Legacy Mode for Non-Tier-1 Platforms
104 *
105 * In legacy mode the page source is locked user pages and not
106 * #RTR0MemObjAllocPhysNC, this means that a page can only be allocated
107 * by the VM that locked it. We will make no attempt at implementing
108 * page sharing on these systems, just do enough to make it all work.
109 *
110 *
111 * @subsection sub_gmm_locking Serializing
112 *
113 * One simple fast mutex will be employed in the initial implementation, not
114 * two as mentioned in @ref sec_pgmPhys_Serializing.
115 *
116 * @see @ref sec_pgmPhys_Serializing
117 *
118 *
119 * @section sec_gmm_overcommit Memory Over-Commitment Management
120 *
121 * The GVM will have to do the system wide memory over-commitment
122 * management. My current ideas are:
123 * - Per VM oc policy that indicates how much to initially commit
124 * to it and what to do in a out-of-memory situation.
125 * - Prevent overtaxing the host.
126 *
127 * There are some challenges here, the main ones are configurability and
128 * security. Should we for instance permit anyone to request 100% memory
129 * commitment? Who should be allowed to do runtime adjustments of the
130 * config. And how to prevent these settings from being lost when the last
131 * VM process exits? The solution is probably to have an optional root
132 * daemon the will keep VMMR0.r0 in memory and enable the security measures.
133 *
134 *
135 *
136 * @section sec_gmm_numa NUMA
137 *
138 * NUMA considerations will be designed and implemented a bit later.
139 *
140 * The preliminary guesses is that we will have to try allocate memory as
141 * close as possible to the CPUs the VM is executed on (EMT and additional CPU
142 * threads). Which means it's mostly about allocation and sharing policies.
143 * Both the scheduler and allocator interface will to supply some NUMA info
144 * and we'll need to have a way to calc access costs.
145 *
146 */
147
148
149/*********************************************************************************************************************************
150* Header Files *
151*********************************************************************************************************************************/
152#define LOG_GROUP LOG_GROUP_GMM
153#include <VBox/rawpci.h>
154#include <VBox/vmm/vm.h>
155#include <VBox/vmm/gmm.h>
156#include "GMMR0Internal.h"
157#include <VBox/vmm/gvm.h>
158#include <VBox/vmm/pgm.h>
159#include <VBox/log.h>
160#include <VBox/param.h>
161#include <VBox/err.h>
162#include <iprt/asm.h>
163#include <iprt/avl.h>
164#ifdef VBOX_STRICT
165# include <iprt/crc.h>
166#endif
167#include <iprt/critsect.h>
168#include <iprt/list.h>
169#include <iprt/mem.h>
170#include <iprt/memobj.h>
171#include <iprt/mp.h>
172#include <iprt/semaphore.h>
173#include <iprt/string.h>
174#include <iprt/time.h>
175
176
177/*********************************************************************************************************************************
178* Defined Constants And Macros *
179*********************************************************************************************************************************/
180/** @def VBOX_USE_CRIT_SECT_FOR_GIANT
181 * Use a critical section instead of a fast mutex for the giant GMM lock.
182 *
183 * @remarks This is primarily a way of avoiding the deadlock checks in the
184 * windows driver verifier. */
185#if defined(RT_OS_WINDOWS) || defined(DOXYGEN_RUNNING)
186# define VBOX_USE_CRIT_SECT_FOR_GIANT
187#endif
188
189
190/*********************************************************************************************************************************
191* Structures and Typedefs *
192*********************************************************************************************************************************/
193/** Pointer to set of free chunks. */
194typedef struct GMMCHUNKFREESET *PGMMCHUNKFREESET;
195
196/**
197 * The per-page tracking structure employed by the GMM.
198 *
199 * On 32-bit hosts we'll some trickery is necessary to compress all
200 * the information into 32-bits. When the fSharedFree member is set,
201 * the 30th bit decides whether it's a free page or not.
202 *
203 * Because of the different layout on 32-bit and 64-bit hosts, macros
204 * are used to get and set some of the data.
205 */
206typedef union GMMPAGE
207{
208#if HC_ARCH_BITS == 64
209 /** Unsigned integer view. */
210 uint64_t u;
211
212 /** The common view. */
213 struct GMMPAGECOMMON
214 {
215 uint32_t uStuff1 : 32;
216 uint32_t uStuff2 : 30;
217 /** The page state. */
218 uint32_t u2State : 2;
219 } Common;
220
221 /** The view of a private page. */
222 struct GMMPAGEPRIVATE
223 {
224 /** The guest page frame number. (Max addressable: 2 ^ 44 - 16) */
225 uint32_t pfn;
226 /** The GVM handle. (64K VMs) */
227 uint32_t hGVM : 16;
228 /** Reserved. */
229 uint32_t u16Reserved : 14;
230 /** The page state. */
231 uint32_t u2State : 2;
232 } Private;
233
234 /** The view of a shared page. */
235 struct GMMPAGESHARED
236 {
237 /** The host page frame number. (Max addressable: 2 ^ 44 - 16) */
238 uint32_t pfn;
239 /** The reference count (64K VMs). */
240 uint32_t cRefs : 16;
241 /** Used for debug checksumming. */
242 uint32_t u14Checksum : 14;
243 /** The page state. */
244 uint32_t u2State : 2;
245 } Shared;
246
247 /** The view of a free page. */
248 struct GMMPAGEFREE
249 {
250 /** The index of the next page in the free list. UINT16_MAX is NIL. */
251 uint16_t iNext;
252 /** Reserved. Checksum or something? */
253 uint16_t u16Reserved0;
254 /** Reserved. Checksum or something? */
255 uint32_t u30Reserved1 : 30;
256 /** The page state. */
257 uint32_t u2State : 2;
258 } Free;
259
260#else /* 32-bit */
261 /** Unsigned integer view. */
262 uint32_t u;
263
264 /** The common view. */
265 struct GMMPAGECOMMON
266 {
267 uint32_t uStuff : 30;
268 /** The page state. */
269 uint32_t u2State : 2;
270 } Common;
271
272 /** The view of a private page. */
273 struct GMMPAGEPRIVATE
274 {
275 /** The guest page frame number. (Max addressable: 2 ^ 36) */
276 uint32_t pfn : 24;
277 /** The GVM handle. (127 VMs) */
278 uint32_t hGVM : 7;
279 /** The top page state bit, MBZ. */
280 uint32_t fZero : 1;
281 } Private;
282
283 /** The view of a shared page. */
284 struct GMMPAGESHARED
285 {
286 /** The reference count. */
287 uint32_t cRefs : 30;
288 /** The page state. */
289 uint32_t u2State : 2;
290 } Shared;
291
292 /** The view of a free page. */
293 struct GMMPAGEFREE
294 {
295 /** The index of the next page in the free list. UINT16_MAX is NIL. */
296 uint32_t iNext : 16;
297 /** Reserved. Checksum or something? */
298 uint32_t u14Reserved : 14;
299 /** The page state. */
300 uint32_t u2State : 2;
301 } Free;
302#endif
303} GMMPAGE;
304AssertCompileSize(GMMPAGE, sizeof(RTHCUINTPTR));
305/** Pointer to a GMMPAGE. */
306typedef GMMPAGE *PGMMPAGE;
307
308
309/** @name The Page States.
310 * @{ */
311/** A private page. */
312#define GMM_PAGE_STATE_PRIVATE 0
313/** A private page - alternative value used on the 32-bit implementation.
314 * This will never be used on 64-bit hosts. */
315#define GMM_PAGE_STATE_PRIVATE_32 1
316/** A shared page. */
317#define GMM_PAGE_STATE_SHARED 2
318/** A free page. */
319#define GMM_PAGE_STATE_FREE 3
320/** @} */
321
322
323/** @def GMM_PAGE_IS_PRIVATE
324 *
325 * @returns true if private, false if not.
326 * @param pPage The GMM page.
327 */
328#if HC_ARCH_BITS == 64
329# define GMM_PAGE_IS_PRIVATE(pPage) ( (pPage)->Common.u2State == GMM_PAGE_STATE_PRIVATE )
330#else
331# define GMM_PAGE_IS_PRIVATE(pPage) ( (pPage)->Private.fZero == 0 )
332#endif
333
334/** @def GMM_PAGE_IS_SHARED
335 *
336 * @returns true if shared, false if not.
337 * @param pPage The GMM page.
338 */
339#define GMM_PAGE_IS_SHARED(pPage) ( (pPage)->Common.u2State == GMM_PAGE_STATE_SHARED )
340
341/** @def GMM_PAGE_IS_FREE
342 *
343 * @returns true if free, false if not.
344 * @param pPage The GMM page.
345 */
346#define GMM_PAGE_IS_FREE(pPage) ( (pPage)->Common.u2State == GMM_PAGE_STATE_FREE )
347
348/** @def GMM_PAGE_PFN_LAST
349 * The last valid guest pfn range.
350 * @remark Some of the values outside the range has special meaning,
351 * see GMM_PAGE_PFN_UNSHAREABLE.
352 */
353#if HC_ARCH_BITS == 64
354# define GMM_PAGE_PFN_LAST UINT32_C(0xfffffff0)
355#else
356# define GMM_PAGE_PFN_LAST UINT32_C(0x00fffff0)
357#endif
358AssertCompile(GMM_PAGE_PFN_LAST == (GMM_GCPHYS_LAST >> PAGE_SHIFT));
359
360/** @def GMM_PAGE_PFN_UNSHAREABLE
361 * Indicates that this page isn't used for normal guest memory and thus isn't shareable.
362 */
363#if HC_ARCH_BITS == 64
364# define GMM_PAGE_PFN_UNSHAREABLE UINT32_C(0xfffffff1)
365#else
366# define GMM_PAGE_PFN_UNSHAREABLE UINT32_C(0x00fffff1)
367#endif
368AssertCompile(GMM_PAGE_PFN_UNSHAREABLE == (GMM_GCPHYS_UNSHAREABLE >> PAGE_SHIFT));
369
370
371/**
372 * A GMM allocation chunk ring-3 mapping record.
373 *
374 * This should really be associated with a session and not a VM, but
375 * it's simpler to associated with a VM and cleanup with the VM object
376 * is destroyed.
377 */
378typedef struct GMMCHUNKMAP
379{
380 /** The mapping object. */
381 RTR0MEMOBJ hMapObj;
382 /** The VM owning the mapping. */
383 PGVM pGVM;
384} GMMCHUNKMAP;
385/** Pointer to a GMM allocation chunk mapping. */
386typedef struct GMMCHUNKMAP *PGMMCHUNKMAP;
387
388
389/**
390 * A GMM allocation chunk.
391 */
392typedef struct GMMCHUNK
393{
394 /** The AVL node core.
395 * The Key is the chunk ID. (Giant mtx.) */
396 AVLU32NODECORE Core;
397 /** The memory object.
398 * Either from RTR0MemObjAllocPhysNC or RTR0MemObjLockUser depending on
399 * what the host can dish up with. (Chunk mtx protects mapping accesses
400 * and related frees.) */
401 RTR0MEMOBJ hMemObj;
402 /** Pointer to the next chunk in the free list. (Giant mtx.) */
403 PGMMCHUNK pFreeNext;
404 /** Pointer to the previous chunk in the free list. (Giant mtx.) */
405 PGMMCHUNK pFreePrev;
406 /** Pointer to the free set this chunk belongs to. NULL for
407 * chunks with no free pages. (Giant mtx.) */
408 PGMMCHUNKFREESET pSet;
409 /** List node in the chunk list (GMM::ChunkList). (Giant mtx.) */
410 RTLISTNODE ListNode;
411 /** Pointer to an array of mappings. (Chunk mtx.) */
412 PGMMCHUNKMAP paMappingsX;
413 /** The number of mappings. (Chunk mtx.) */
414 uint16_t cMappingsX;
415 /** The mapping lock this chunk is using using. UINT16_MAX if nobody is
416 * mapping or freeing anything. (Giant mtx.) */
417 uint8_t volatile iChunkMtx;
418 /** Flags field reserved for future use (like eliminating enmType).
419 * (Giant mtx.) */
420 uint8_t fFlags;
421 /** The head of the list of free pages. UINT16_MAX is the NIL value.
422 * (Giant mtx.) */
423 uint16_t iFreeHead;
424 /** The number of free pages. (Giant mtx.) */
425 uint16_t cFree;
426 /** The GVM handle of the VM that first allocated pages from this chunk, this
427 * is used as a preference when there are several chunks to choose from.
428 * When in bound memory mode this isn't a preference any longer. (Giant
429 * mtx.) */
430 uint16_t hGVM;
431 /** The ID of the NUMA node the memory mostly resides on. (Reserved for
432 * future use.) (Giant mtx.) */
433 uint16_t idNumaNode;
434 /** The number of private pages. (Giant mtx.) */
435 uint16_t cPrivate;
436 /** The number of shared pages. (Giant mtx.) */
437 uint16_t cShared;
438 /** The pages. (Giant mtx.) */
439 GMMPAGE aPages[GMM_CHUNK_SIZE >> PAGE_SHIFT];
440} GMMCHUNK;
441
442/** Indicates that the NUMA properies of the memory is unknown. */
443#define GMM_CHUNK_NUMA_ID_UNKNOWN UINT16_C(0xfffe)
444
445/** @name GMM_CHUNK_FLAGS_XXX - chunk flags.
446 * @{ */
447/** Indicates that the chunk is a large page (2MB). */
448#define GMM_CHUNK_FLAGS_LARGE_PAGE UINT16_C(0x0001)
449/** @} */
450
451
452/**
453 * An allocation chunk TLB entry.
454 */
455typedef struct GMMCHUNKTLBE
456{
457 /** The chunk id. */
458 uint32_t idChunk;
459 /** Pointer to the chunk. */
460 PGMMCHUNK pChunk;
461} GMMCHUNKTLBE;
462/** Pointer to an allocation chunk TLB entry. */
463typedef GMMCHUNKTLBE *PGMMCHUNKTLBE;
464
465
466/** The number of entries tin the allocation chunk TLB. */
467#define GMM_CHUNKTLB_ENTRIES 32
468/** Gets the TLB entry index for the given Chunk ID. */
469#define GMM_CHUNKTLB_IDX(idChunk) ( (idChunk) & (GMM_CHUNKTLB_ENTRIES - 1) )
470
471/**
472 * An allocation chunk TLB.
473 */
474typedef struct GMMCHUNKTLB
475{
476 /** The TLB entries. */
477 GMMCHUNKTLBE aEntries[GMM_CHUNKTLB_ENTRIES];
478} GMMCHUNKTLB;
479/** Pointer to an allocation chunk TLB. */
480typedef GMMCHUNKTLB *PGMMCHUNKTLB;
481
482
483/**
484 * The GMM instance data.
485 */
486typedef struct GMM
487{
488 /** Magic / eye catcher. GMM_MAGIC */
489 uint32_t u32Magic;
490 /** The number of threads waiting on the mutex. */
491 uint32_t cMtxContenders;
492#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
493 /** The critical section protecting the GMM.
494 * More fine grained locking can be implemented later if necessary. */
495 RTCRITSECT GiantCritSect;
496#else
497 /** The fast mutex protecting the GMM.
498 * More fine grained locking can be implemented later if necessary. */
499 RTSEMFASTMUTEX hMtx;
500#endif
501#ifdef VBOX_STRICT
502 /** The current mutex owner. */
503 RTNATIVETHREAD hMtxOwner;
504#endif
505 /** The chunk tree. */
506 PAVLU32NODECORE pChunks;
507 /** The chunk TLB. */
508 GMMCHUNKTLB ChunkTLB;
509 /** The private free set. */
510 GMMCHUNKFREESET PrivateX;
511 /** The shared free set. */
512 GMMCHUNKFREESET Shared;
513
514 /** Shared module tree (global).
515 * @todo separate trees for distinctly different guest OSes. */
516 PAVLLU32NODECORE pGlobalSharedModuleTree;
517 /** Sharable modules (count of nodes in pGlobalSharedModuleTree). */
518 uint32_t cShareableModules;
519
520 /** The chunk list. For simplifying the cleanup process. */
521 RTLISTANCHOR ChunkList;
522
523 /** The maximum number of pages we're allowed to allocate.
524 * @gcfgm{GMM/MaxPages,64-bit, Direct.}
525 * @gcfgm{GMM/PctPages,32-bit, Relative to the number of host pages.} */
526 uint64_t cMaxPages;
527 /** The number of pages that has been reserved.
528 * The deal is that cReservedPages - cOverCommittedPages <= cMaxPages. */
529 uint64_t cReservedPages;
530 /** The number of pages that we have over-committed in reservations. */
531 uint64_t cOverCommittedPages;
532 /** The number of actually allocated (committed if you like) pages. */
533 uint64_t cAllocatedPages;
534 /** The number of pages that are shared. A subset of cAllocatedPages. */
535 uint64_t cSharedPages;
536 /** The number of pages that are actually shared between VMs. */
537 uint64_t cDuplicatePages;
538 /** The number of pages that are shared that has been left behind by
539 * VMs not doing proper cleanups. */
540 uint64_t cLeftBehindSharedPages;
541 /** The number of allocation chunks.
542 * (The number of pages we've allocated from the host can be derived from this.) */
543 uint32_t cChunks;
544 /** The number of current ballooned pages. */
545 uint64_t cBalloonedPages;
546
547 /** The legacy allocation mode indicator.
548 * This is determined at initialization time. */
549 bool fLegacyAllocationMode;
550 /** The bound memory mode indicator.
551 * When set, the memory will be bound to a specific VM and never
552 * shared. This is always set if fLegacyAllocationMode is set.
553 * (Also determined at initialization time.) */
554 bool fBoundMemoryMode;
555 /** The number of registered VMs. */
556 uint16_t cRegisteredVMs;
557
558 /** The number of freed chunks ever. This is used a list generation to
559 * avoid restarting the cleanup scanning when the list wasn't modified. */
560 uint32_t cFreedChunks;
561 /** The previous allocated Chunk ID.
562 * Used as a hint to avoid scanning the whole bitmap. */
563 uint32_t idChunkPrev;
564 /** Chunk ID allocation bitmap.
565 * Bits of allocated IDs are set, free ones are clear.
566 * The NIL id (0) is marked allocated. */
567 uint32_t bmChunkId[(GMM_CHUNKID_LAST + 1 + 31) / 32];
568
569 /** The index of the next mutex to use. */
570 uint32_t iNextChunkMtx;
571 /** Chunk locks for reducing lock contention without having to allocate
572 * one lock per chunk. */
573 struct
574 {
575 /** The mutex */
576 RTSEMFASTMUTEX hMtx;
577 /** The number of threads currently using this mutex. */
578 uint32_t volatile cUsers;
579 } aChunkMtx[64];
580} GMM;
581/** Pointer to the GMM instance. */
582typedef GMM *PGMM;
583
584/** The value of GMM::u32Magic (Katsuhiro Otomo). */
585#define GMM_MAGIC UINT32_C(0x19540414)
586
587
588/**
589 * GMM chunk mutex state.
590 *
591 * This is returned by gmmR0ChunkMutexAcquire and is used by the other
592 * gmmR0ChunkMutex* methods.
593 */
594typedef struct GMMR0CHUNKMTXSTATE
595{
596 PGMM pGMM;
597 /** The index of the chunk mutex. */
598 uint8_t iChunkMtx;
599 /** The relevant flags (GMMR0CHUNK_MTX_XXX). */
600 uint8_t fFlags;
601} GMMR0CHUNKMTXSTATE;
602/** Pointer to a chunk mutex state. */
603typedef GMMR0CHUNKMTXSTATE *PGMMR0CHUNKMTXSTATE;
604
605/** @name GMMR0CHUNK_MTX_XXX
606 * @{ */
607#define GMMR0CHUNK_MTX_INVALID UINT32_C(0)
608#define GMMR0CHUNK_MTX_KEEP_GIANT UINT32_C(1)
609#define GMMR0CHUNK_MTX_RETAKE_GIANT UINT32_C(2)
610#define GMMR0CHUNK_MTX_DROP_GIANT UINT32_C(3)
611#define GMMR0CHUNK_MTX_END UINT32_C(4)
612/** @} */
613
614
615/** The maximum number of shared modules per-vm. */
616#define GMM_MAX_SHARED_PER_VM_MODULES 2048
617/** The maximum number of shared modules GMM is allowed to track. */
618#define GMM_MAX_SHARED_GLOBAL_MODULES 16834
619
620
621/**
622 * Argument packet for gmmR0SharedModuleCleanup.
623 */
624typedef struct GMMR0SHMODPERVMDTORARGS
625{
626 PGVM pGVM;
627 PGMM pGMM;
628} GMMR0SHMODPERVMDTORARGS;
629
630/**
631 * Argument packet for gmmR0CheckSharedModule.
632 */
633typedef struct GMMCHECKSHAREDMODULEINFO
634{
635 PGVM pGVM;
636 VMCPUID idCpu;
637} GMMCHECKSHAREDMODULEINFO;
638
639/**
640 * Argument packet for gmmR0FindDupPageInChunk by GMMR0FindDuplicatePage.
641 */
642typedef struct GMMFINDDUPPAGEINFO
643{
644 PGVM pGVM;
645 PGMM pGMM;
646 uint8_t *pSourcePage;
647 bool fFoundDuplicate;
648} GMMFINDDUPPAGEINFO;
649
650
651/*********************************************************************************************************************************
652* Global Variables *
653*********************************************************************************************************************************/
654/** Pointer to the GMM instance data. */
655static PGMM g_pGMM = NULL;
656
657/** Macro for obtaining and validating the g_pGMM pointer.
658 *
659 * On failure it will return from the invoking function with the specified
660 * return value.
661 *
662 * @param pGMM The name of the pGMM variable.
663 * @param rc The return value on failure. Use VERR_GMM_INSTANCE for VBox
664 * status codes.
665 */
666#define GMM_GET_VALID_INSTANCE(pGMM, rc) \
667 do { \
668 (pGMM) = g_pGMM; \
669 AssertPtrReturn((pGMM), (rc)); \
670 AssertMsgReturn((pGMM)->u32Magic == GMM_MAGIC, ("%p - %#x\n", (pGMM), (pGMM)->u32Magic), (rc)); \
671 } while (0)
672
673/** Macro for obtaining and validating the g_pGMM pointer, void function
674 * variant.
675 *
676 * On failure it will return from the invoking function.
677 *
678 * @param pGMM The name of the pGMM variable.
679 */
680#define GMM_GET_VALID_INSTANCE_VOID(pGMM) \
681 do { \
682 (pGMM) = g_pGMM; \
683 AssertPtrReturnVoid((pGMM)); \
684 AssertMsgReturnVoid((pGMM)->u32Magic == GMM_MAGIC, ("%p - %#x\n", (pGMM), (pGMM)->u32Magic)); \
685 } while (0)
686
687
688/** @def GMM_CHECK_SANITY_UPON_ENTERING
689 * Checks the sanity of the GMM instance data before making changes.
690 *
691 * This is macro is a stub by default and must be enabled manually in the code.
692 *
693 * @returns true if sane, false if not.
694 * @param pGMM The name of the pGMM variable.
695 */
696#if defined(VBOX_STRICT) && defined(GMMR0_WITH_SANITY_CHECK) && 0
697# define GMM_CHECK_SANITY_UPON_ENTERING(pGMM) (gmmR0SanityCheck((pGMM), __PRETTY_FUNCTION__, __LINE__) == 0)
698#else
699# define GMM_CHECK_SANITY_UPON_ENTERING(pGMM) (true)
700#endif
701
702/** @def GMM_CHECK_SANITY_UPON_LEAVING
703 * Checks the sanity of the GMM instance data after making changes.
704 *
705 * This is macro is a stub by default and must be enabled manually in the code.
706 *
707 * @returns true if sane, false if not.
708 * @param pGMM The name of the pGMM variable.
709 */
710#if defined(VBOX_STRICT) && defined(GMMR0_WITH_SANITY_CHECK) && 0
711# define GMM_CHECK_SANITY_UPON_LEAVING(pGMM) (gmmR0SanityCheck((pGMM), __PRETTY_FUNCTION__, __LINE__) == 0)
712#else
713# define GMM_CHECK_SANITY_UPON_LEAVING(pGMM) (true)
714#endif
715
716/** @def GMM_CHECK_SANITY_IN_LOOPS
717 * Checks the sanity of the GMM instance in the allocation loops.
718 *
719 * This is macro is a stub by default and must be enabled manually in the code.
720 *
721 * @returns true if sane, false if not.
722 * @param pGMM The name of the pGMM variable.
723 */
724#if defined(VBOX_STRICT) && defined(GMMR0_WITH_SANITY_CHECK) && 0
725# define GMM_CHECK_SANITY_IN_LOOPS(pGMM) (gmmR0SanityCheck((pGMM), __PRETTY_FUNCTION__, __LINE__) == 0)
726#else
727# define GMM_CHECK_SANITY_IN_LOOPS(pGMM) (true)
728#endif
729
730
731/*********************************************************************************************************************************
732* Internal Functions *
733*********************************************************************************************************************************/
734static DECLCALLBACK(int) gmmR0TermDestroyChunk(PAVLU32NODECORE pNode, void *pvGMM);
735static bool gmmR0CleanupVMScanChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk);
736DECLINLINE(void) gmmR0UnlinkChunk(PGMMCHUNK pChunk);
737DECLINLINE(void) gmmR0LinkChunk(PGMMCHUNK pChunk, PGMMCHUNKFREESET pSet);
738DECLINLINE(void) gmmR0SelectSetAndLinkChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk);
739#ifdef GMMR0_WITH_SANITY_CHECK
740static uint32_t gmmR0SanityCheck(PGMM pGMM, const char *pszFunction, unsigned uLineNo);
741#endif
742static bool gmmR0FreeChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem);
743DECLINLINE(void) gmmR0FreePrivatePage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage);
744DECLINLINE(void) gmmR0FreeSharedPage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage);
745static int gmmR0UnmapChunkLocked(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk);
746#ifdef VBOX_WITH_PAGE_SHARING
747static void gmmR0SharedModuleCleanup(PGMM pGMM, PGVM pGVM);
748# ifdef VBOX_STRICT
749static uint32_t gmmR0StrictPageChecksum(PGMM pGMM, PGVM pGVM, uint32_t idPage);
750# endif
751#endif
752
753
754
755/**
756 * Initializes the GMM component.
757 *
758 * This is called when the VMMR0.r0 module is loaded and protected by the
759 * loader semaphore.
760 *
761 * @returns VBox status code.
762 */
763GMMR0DECL(int) GMMR0Init(void)
764{
765 LogFlow(("GMMInit:\n"));
766
767 /*
768 * Allocate the instance data and the locks.
769 */
770 PGMM pGMM = (PGMM)RTMemAllocZ(sizeof(*pGMM));
771 if (!pGMM)
772 return VERR_NO_MEMORY;
773
774 pGMM->u32Magic = GMM_MAGIC;
775 for (unsigned i = 0; i < RT_ELEMENTS(pGMM->ChunkTLB.aEntries); i++)
776 pGMM->ChunkTLB.aEntries[i].idChunk = NIL_GMM_CHUNKID;
777 RTListInit(&pGMM->ChunkList);
778 ASMBitSet(&pGMM->bmChunkId[0], NIL_GMM_CHUNKID);
779
780#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
781 int rc = RTCritSectInit(&pGMM->GiantCritSect);
782#else
783 int rc = RTSemFastMutexCreate(&pGMM->hMtx);
784#endif
785 if (RT_SUCCESS(rc))
786 {
787 unsigned iMtx;
788 for (iMtx = 0; iMtx < RT_ELEMENTS(pGMM->aChunkMtx); iMtx++)
789 {
790 rc = RTSemFastMutexCreate(&pGMM->aChunkMtx[iMtx].hMtx);
791 if (RT_FAILURE(rc))
792 break;
793 }
794 if (RT_SUCCESS(rc))
795 {
796 /*
797 * Check and see if RTR0MemObjAllocPhysNC works.
798 */
799#if 0 /* later, see @bufref{3170}. */
800 RTR0MEMOBJ MemObj;
801 rc = RTR0MemObjAllocPhysNC(&MemObj, _64K, NIL_RTHCPHYS);
802 if (RT_SUCCESS(rc))
803 {
804 rc = RTR0MemObjFree(MemObj, true);
805 AssertRC(rc);
806 }
807 else if (rc == VERR_NOT_SUPPORTED)
808 pGMM->fLegacyAllocationMode = pGMM->fBoundMemoryMode = true;
809 else
810 SUPR0Printf("GMMR0Init: RTR0MemObjAllocPhysNC(,64K,Any) -> %d!\n", rc);
811#else
812# if defined(RT_OS_WINDOWS) || (defined(RT_OS_SOLARIS) && ARCH_BITS == 64) || defined(RT_OS_LINUX) || defined(RT_OS_FREEBSD)
813 pGMM->fLegacyAllocationMode = false;
814# if ARCH_BITS == 32
815 /* Don't reuse possibly partial chunks because of the virtual
816 address space limitation. */
817 pGMM->fBoundMemoryMode = true;
818# else
819 pGMM->fBoundMemoryMode = false;
820# endif
821# else
822 pGMM->fLegacyAllocationMode = true;
823 pGMM->fBoundMemoryMode = true;
824# endif
825#endif
826
827 /*
828 * Query system page count and guess a reasonable cMaxPages value.
829 */
830 pGMM->cMaxPages = UINT32_MAX; /** @todo IPRT function for query ram size and such. */
831
832 g_pGMM = pGMM;
833 LogFlow(("GMMInit: pGMM=%p fLegacyAllocationMode=%RTbool fBoundMemoryMode=%RTbool\n", pGMM, pGMM->fLegacyAllocationMode, pGMM->fBoundMemoryMode));
834 return VINF_SUCCESS;
835 }
836
837 /*
838 * Bail out.
839 */
840 while (iMtx-- > 0)
841 RTSemFastMutexDestroy(pGMM->aChunkMtx[iMtx].hMtx);
842#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
843 RTCritSectDelete(&pGMM->GiantCritSect);
844#else
845 RTSemFastMutexDestroy(pGMM->hMtx);
846#endif
847 }
848
849 pGMM->u32Magic = 0;
850 RTMemFree(pGMM);
851 SUPR0Printf("GMMR0Init: failed! rc=%d\n", rc);
852 return rc;
853}
854
855
856/**
857 * Terminates the GMM component.
858 */
859GMMR0DECL(void) GMMR0Term(void)
860{
861 LogFlow(("GMMTerm:\n"));
862
863 /*
864 * Take care / be paranoid...
865 */
866 PGMM pGMM = g_pGMM;
867 if (!VALID_PTR(pGMM))
868 return;
869 if (pGMM->u32Magic != GMM_MAGIC)
870 {
871 SUPR0Printf("GMMR0Term: u32Magic=%#x\n", pGMM->u32Magic);
872 return;
873 }
874
875 /*
876 * Undo what init did and free all the resources we've acquired.
877 */
878 /* Destroy the fundamentals. */
879 g_pGMM = NULL;
880 pGMM->u32Magic = ~GMM_MAGIC;
881#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
882 RTCritSectDelete(&pGMM->GiantCritSect);
883#else
884 RTSemFastMutexDestroy(pGMM->hMtx);
885 pGMM->hMtx = NIL_RTSEMFASTMUTEX;
886#endif
887
888 /* Free any chunks still hanging around. */
889 RTAvlU32Destroy(&pGMM->pChunks, gmmR0TermDestroyChunk, pGMM);
890
891 /* Destroy the chunk locks. */
892 for (unsigned iMtx = 0; iMtx < RT_ELEMENTS(pGMM->aChunkMtx); iMtx++)
893 {
894 Assert(pGMM->aChunkMtx[iMtx].cUsers == 0);
895 RTSemFastMutexDestroy(pGMM->aChunkMtx[iMtx].hMtx);
896 pGMM->aChunkMtx[iMtx].hMtx = NIL_RTSEMFASTMUTEX;
897 }
898
899 /* Finally the instance data itself. */
900 RTMemFree(pGMM);
901 LogFlow(("GMMTerm: done\n"));
902}
903
904
905/**
906 * RTAvlU32Destroy callback.
907 *
908 * @returns 0
909 * @param pNode The node to destroy.
910 * @param pvGMM The GMM handle.
911 */
912static DECLCALLBACK(int) gmmR0TermDestroyChunk(PAVLU32NODECORE pNode, void *pvGMM)
913{
914 PGMMCHUNK pChunk = (PGMMCHUNK)pNode;
915
916 if (pChunk->cFree != (GMM_CHUNK_SIZE >> PAGE_SHIFT))
917 SUPR0Printf("GMMR0Term: %p/%#x: cFree=%d cPrivate=%d cShared=%d cMappings=%d\n", pChunk,
918 pChunk->Core.Key, pChunk->cFree, pChunk->cPrivate, pChunk->cShared, pChunk->cMappingsX);
919
920 int rc = RTR0MemObjFree(pChunk->hMemObj, true /* fFreeMappings */);
921 if (RT_FAILURE(rc))
922 {
923 SUPR0Printf("GMMR0Term: %p/%#x: RTRMemObjFree(%p,true) -> %d (cMappings=%d)\n", pChunk,
924 pChunk->Core.Key, pChunk->hMemObj, rc, pChunk->cMappingsX);
925 AssertRC(rc);
926 }
927 pChunk->hMemObj = NIL_RTR0MEMOBJ;
928
929 RTMemFree(pChunk->paMappingsX);
930 pChunk->paMappingsX = NULL;
931
932 RTMemFree(pChunk);
933 NOREF(pvGMM);
934 return 0;
935}
936
937
938/**
939 * Initializes the per-VM data for the GMM.
940 *
941 * This is called from within the GVMM lock (from GVMMR0CreateVM)
942 * and should only initialize the data members so GMMR0CleanupVM
943 * can deal with them. We reserve no memory or anything here,
944 * that's done later in GMMR0InitVM.
945 *
946 * @param pGVM Pointer to the Global VM structure.
947 */
948GMMR0DECL(void) GMMR0InitPerVMData(PGVM pGVM)
949{
950 AssertCompile(RT_SIZEOFMEMB(GVM,gmm.s) <= RT_SIZEOFMEMB(GVM,gmm.padding));
951
952 pGVM->gmm.s.Stats.enmPolicy = GMMOCPOLICY_INVALID;
953 pGVM->gmm.s.Stats.enmPriority = GMMPRIORITY_INVALID;
954 pGVM->gmm.s.Stats.fMayAllocate = false;
955}
956
957
958/**
959 * Acquires the GMM giant lock.
960 *
961 * @returns Assert status code from RTSemFastMutexRequest.
962 * @param pGMM Pointer to the GMM instance.
963 */
964static int gmmR0MutexAcquire(PGMM pGMM)
965{
966 ASMAtomicIncU32(&pGMM->cMtxContenders);
967#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
968 int rc = RTCritSectEnter(&pGMM->GiantCritSect);
969#else
970 int rc = RTSemFastMutexRequest(pGMM->hMtx);
971#endif
972 ASMAtomicDecU32(&pGMM->cMtxContenders);
973 AssertRC(rc);
974#ifdef VBOX_STRICT
975 pGMM->hMtxOwner = RTThreadNativeSelf();
976#endif
977 return rc;
978}
979
980
981/**
982 * Releases the GMM giant lock.
983 *
984 * @returns Assert status code from RTSemFastMutexRequest.
985 * @param pGMM Pointer to the GMM instance.
986 */
987static int gmmR0MutexRelease(PGMM pGMM)
988{
989#ifdef VBOX_STRICT
990 pGMM->hMtxOwner = NIL_RTNATIVETHREAD;
991#endif
992#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
993 int rc = RTCritSectLeave(&pGMM->GiantCritSect);
994#else
995 int rc = RTSemFastMutexRelease(pGMM->hMtx);
996 AssertRC(rc);
997#endif
998 return rc;
999}
1000
1001
1002/**
1003 * Yields the GMM giant lock if there is contention and a certain minimum time
1004 * has elapsed since we took it.
1005 *
1006 * @returns @c true if the mutex was yielded, @c false if not.
1007 * @param pGMM Pointer to the GMM instance.
1008 * @param puLockNanoTS Where the lock acquisition time stamp is kept
1009 * (in/out).
1010 */
1011static bool gmmR0MutexYield(PGMM pGMM, uint64_t *puLockNanoTS)
1012{
1013 /*
1014 * If nobody is contending the mutex, don't bother checking the time.
1015 */
1016 if (ASMAtomicReadU32(&pGMM->cMtxContenders) == 0)
1017 return false;
1018
1019 /*
1020 * Don't yield if we haven't executed for at least 2 milliseconds.
1021 */
1022 uint64_t uNanoNow = RTTimeSystemNanoTS();
1023 if (uNanoNow - *puLockNanoTS < UINT32_C(2000000))
1024 return false;
1025
1026 /*
1027 * Yield the mutex.
1028 */
1029#ifdef VBOX_STRICT
1030 pGMM->hMtxOwner = NIL_RTNATIVETHREAD;
1031#endif
1032 ASMAtomicIncU32(&pGMM->cMtxContenders);
1033#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
1034 int rc1 = RTCritSectLeave(&pGMM->GiantCritSect); AssertRC(rc1);
1035#else
1036 int rc1 = RTSemFastMutexRelease(pGMM->hMtx); AssertRC(rc1);
1037#endif
1038
1039 RTThreadYield();
1040
1041#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
1042 int rc2 = RTCritSectEnter(&pGMM->GiantCritSect); AssertRC(rc2);
1043#else
1044 int rc2 = RTSemFastMutexRequest(pGMM->hMtx); AssertRC(rc2);
1045#endif
1046 *puLockNanoTS = RTTimeSystemNanoTS();
1047 ASMAtomicDecU32(&pGMM->cMtxContenders);
1048#ifdef VBOX_STRICT
1049 pGMM->hMtxOwner = RTThreadNativeSelf();
1050#endif
1051
1052 return true;
1053}
1054
1055
1056/**
1057 * Acquires a chunk lock.
1058 *
1059 * The caller must own the giant lock.
1060 *
1061 * @returns Assert status code from RTSemFastMutexRequest.
1062 * @param pMtxState The chunk mutex state info. (Avoids
1063 * passing the same flags and stuff around
1064 * for subsequent release and drop-giant
1065 * calls.)
1066 * @param pGMM Pointer to the GMM instance.
1067 * @param pChunk Pointer to the chunk.
1068 * @param fFlags Flags regarding the giant lock, GMMR0CHUNK_MTX_XXX.
1069 */
1070static int gmmR0ChunkMutexAcquire(PGMMR0CHUNKMTXSTATE pMtxState, PGMM pGMM, PGMMCHUNK pChunk, uint32_t fFlags)
1071{
1072 Assert(fFlags > GMMR0CHUNK_MTX_INVALID && fFlags < GMMR0CHUNK_MTX_END);
1073 Assert(pGMM->hMtxOwner == RTThreadNativeSelf());
1074
1075 pMtxState->pGMM = pGMM;
1076 pMtxState->fFlags = (uint8_t)fFlags;
1077
1078 /*
1079 * Get the lock index and reference the lock.
1080 */
1081 Assert(pGMM->hMtxOwner == RTThreadNativeSelf());
1082 uint32_t iChunkMtx = pChunk->iChunkMtx;
1083 if (iChunkMtx == UINT8_MAX)
1084 {
1085 iChunkMtx = pGMM->iNextChunkMtx++;
1086 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1087
1088 /* Try get an unused one... */
1089 if (pGMM->aChunkMtx[iChunkMtx].cUsers)
1090 {
1091 iChunkMtx = pGMM->iNextChunkMtx++;
1092 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1093 if (pGMM->aChunkMtx[iChunkMtx].cUsers)
1094 {
1095 iChunkMtx = pGMM->iNextChunkMtx++;
1096 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1097 if (pGMM->aChunkMtx[iChunkMtx].cUsers)
1098 {
1099 iChunkMtx = pGMM->iNextChunkMtx++;
1100 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1101 }
1102 }
1103 }
1104
1105 pChunk->iChunkMtx = iChunkMtx;
1106 }
1107 AssertCompile(RT_ELEMENTS(pGMM->aChunkMtx) < UINT8_MAX);
1108 pMtxState->iChunkMtx = (uint8_t)iChunkMtx;
1109 ASMAtomicIncU32(&pGMM->aChunkMtx[iChunkMtx].cUsers);
1110
1111 /*
1112 * Drop the giant?
1113 */
1114 if (fFlags != GMMR0CHUNK_MTX_KEEP_GIANT)
1115 {
1116 /** @todo GMM life cycle cleanup (we may race someone
1117 * destroying and cleaning up GMM)? */
1118 gmmR0MutexRelease(pGMM);
1119 }
1120
1121 /*
1122 * Take the chunk mutex.
1123 */
1124 int rc = RTSemFastMutexRequest(pGMM->aChunkMtx[iChunkMtx].hMtx);
1125 AssertRC(rc);
1126 return rc;
1127}
1128
1129
1130/**
1131 * Releases the GMM giant lock.
1132 *
1133 * @returns Assert status code from RTSemFastMutexRequest.
1134 * @param pMtxState Pointer to the chunk mutex state.
1135 * @param pChunk Pointer to the chunk if it's still
1136 * alive, NULL if it isn't. This is used to deassociate
1137 * the chunk from the mutex on the way out so a new one
1138 * can be selected next time, thus avoiding contented
1139 * mutexes.
1140 */
1141static int gmmR0ChunkMutexRelease(PGMMR0CHUNKMTXSTATE pMtxState, PGMMCHUNK pChunk)
1142{
1143 PGMM pGMM = pMtxState->pGMM;
1144
1145 /*
1146 * Release the chunk mutex and reacquire the giant if requested.
1147 */
1148 int rc = RTSemFastMutexRelease(pGMM->aChunkMtx[pMtxState->iChunkMtx].hMtx);
1149 AssertRC(rc);
1150 if (pMtxState->fFlags == GMMR0CHUNK_MTX_RETAKE_GIANT)
1151 rc = gmmR0MutexAcquire(pGMM);
1152 else
1153 Assert((pMtxState->fFlags != GMMR0CHUNK_MTX_DROP_GIANT) == (pGMM->hMtxOwner == RTThreadNativeSelf()));
1154
1155 /*
1156 * Drop the chunk mutex user reference and deassociate it from the chunk
1157 * when possible.
1158 */
1159 if ( ASMAtomicDecU32(&pGMM->aChunkMtx[pMtxState->iChunkMtx].cUsers) == 0
1160 && pChunk
1161 && RT_SUCCESS(rc) )
1162 {
1163 if (pMtxState->fFlags != GMMR0CHUNK_MTX_DROP_GIANT)
1164 pChunk->iChunkMtx = UINT8_MAX;
1165 else
1166 {
1167 rc = gmmR0MutexAcquire(pGMM);
1168 if (RT_SUCCESS(rc))
1169 {
1170 if (pGMM->aChunkMtx[pMtxState->iChunkMtx].cUsers == 0)
1171 pChunk->iChunkMtx = UINT8_MAX;
1172 rc = gmmR0MutexRelease(pGMM);
1173 }
1174 }
1175 }
1176
1177 pMtxState->pGMM = NULL;
1178 return rc;
1179}
1180
1181
1182/**
1183 * Drops the giant GMM lock we kept in gmmR0ChunkMutexAcquire while keeping the
1184 * chunk locked.
1185 *
1186 * This only works if gmmR0ChunkMutexAcquire was called with
1187 * GMMR0CHUNK_MTX_KEEP_GIANT. gmmR0ChunkMutexRelease will retake the giant
1188 * mutex, i.e. behave as if GMMR0CHUNK_MTX_RETAKE_GIANT was used.
1189 *
1190 * @returns VBox status code (assuming success is ok).
1191 * @param pMtxState Pointer to the chunk mutex state.
1192 */
1193static int gmmR0ChunkMutexDropGiant(PGMMR0CHUNKMTXSTATE pMtxState)
1194{
1195 AssertReturn(pMtxState->fFlags == GMMR0CHUNK_MTX_KEEP_GIANT, VERR_GMM_MTX_FLAGS);
1196 Assert(pMtxState->pGMM->hMtxOwner == RTThreadNativeSelf());
1197 pMtxState->fFlags = GMMR0CHUNK_MTX_RETAKE_GIANT;
1198 /** @todo GMM life cycle cleanup (we may race someone
1199 * destroying and cleaning up GMM)? */
1200 return gmmR0MutexRelease(pMtxState->pGMM);
1201}
1202
1203
1204/**
1205 * For experimenting with NUMA affinity and such.
1206 *
1207 * @returns The current NUMA Node ID.
1208 */
1209static uint16_t gmmR0GetCurrentNumaNodeId(void)
1210{
1211#if 1
1212 return GMM_CHUNK_NUMA_ID_UNKNOWN;
1213#else
1214 return RTMpCpuId() / 16;
1215#endif
1216}
1217
1218
1219
1220/**
1221 * Cleans up when a VM is terminating.
1222 *
1223 * @param pGVM Pointer to the Global VM structure.
1224 */
1225GMMR0DECL(void) GMMR0CleanupVM(PGVM pGVM)
1226{
1227 LogFlow(("GMMR0CleanupVM: pGVM=%p:{.pVM=%p, .hSelf=%#x}\n", pGVM, pGVM->pVM, pGVM->hSelf));
1228
1229 PGMM pGMM;
1230 GMM_GET_VALID_INSTANCE_VOID(pGMM);
1231
1232#ifdef VBOX_WITH_PAGE_SHARING
1233 /*
1234 * Clean up all registered shared modules first.
1235 */
1236 gmmR0SharedModuleCleanup(pGMM, pGVM);
1237#endif
1238
1239 gmmR0MutexAcquire(pGMM);
1240 uint64_t uLockNanoTS = RTTimeSystemNanoTS();
1241 GMM_CHECK_SANITY_UPON_ENTERING(pGMM);
1242
1243 /*
1244 * The policy is 'INVALID' until the initial reservation
1245 * request has been serviced.
1246 */
1247 if ( pGVM->gmm.s.Stats.enmPolicy > GMMOCPOLICY_INVALID
1248 && pGVM->gmm.s.Stats.enmPolicy < GMMOCPOLICY_END)
1249 {
1250 /*
1251 * If it's the last VM around, we can skip walking all the chunk looking
1252 * for the pages owned by this VM and instead flush the whole shebang.
1253 *
1254 * This takes care of the eventuality that a VM has left shared page
1255 * references behind (shouldn't happen of course, but you never know).
1256 */
1257 Assert(pGMM->cRegisteredVMs);
1258 pGMM->cRegisteredVMs--;
1259
1260 /*
1261 * Walk the entire pool looking for pages that belong to this VM
1262 * and leftover mappings. (This'll only catch private pages,
1263 * shared pages will be 'left behind'.)
1264 */
1265 /** @todo r=bird: This scanning+freeing could be optimized in bound mode! */
1266 uint64_t cPrivatePages = pGVM->gmm.s.Stats.cPrivatePages; /* save */
1267
1268 unsigned iCountDown = 64;
1269 bool fRedoFromStart;
1270 PGMMCHUNK pChunk;
1271 do
1272 {
1273 fRedoFromStart = false;
1274 RTListForEachReverse(&pGMM->ChunkList, pChunk, GMMCHUNK, ListNode)
1275 {
1276 uint32_t const cFreeChunksOld = pGMM->cFreedChunks;
1277 if ( ( !pGMM->fBoundMemoryMode
1278 || pChunk->hGVM == pGVM->hSelf)
1279 && gmmR0CleanupVMScanChunk(pGMM, pGVM, pChunk))
1280 {
1281 /* We left the giant mutex, so reset the yield counters. */
1282 uLockNanoTS = RTTimeSystemNanoTS();
1283 iCountDown = 64;
1284 }
1285 else
1286 {
1287 /* Didn't leave it, so do normal yielding. */
1288 if (!iCountDown)
1289 gmmR0MutexYield(pGMM, &uLockNanoTS);
1290 else
1291 iCountDown--;
1292 }
1293 if (pGMM->cFreedChunks != cFreeChunksOld)
1294 {
1295 fRedoFromStart = true;
1296 break;
1297 }
1298 }
1299 } while (fRedoFromStart);
1300
1301 if (pGVM->gmm.s.Stats.cPrivatePages)
1302 SUPR0Printf("GMMR0CleanupVM: hGVM=%#x has %#x private pages that cannot be found!\n", pGVM->hSelf, pGVM->gmm.s.Stats.cPrivatePages);
1303
1304 pGMM->cAllocatedPages -= cPrivatePages;
1305
1306 /*
1307 * Free empty chunks.
1308 */
1309 PGMMCHUNKFREESET pPrivateSet = pGMM->fBoundMemoryMode ? &pGVM->gmm.s.Private : &pGMM->PrivateX;
1310 do
1311 {
1312 fRedoFromStart = false;
1313 iCountDown = 10240;
1314 pChunk = pPrivateSet->apLists[GMM_CHUNK_FREE_SET_UNUSED_LIST];
1315 while (pChunk)
1316 {
1317 PGMMCHUNK pNext = pChunk->pFreeNext;
1318 Assert(pChunk->cFree == GMM_CHUNK_NUM_PAGES);
1319 if ( !pGMM->fBoundMemoryMode
1320 || pChunk->hGVM == pGVM->hSelf)
1321 {
1322 uint64_t const idGenerationOld = pPrivateSet->idGeneration;
1323 if (gmmR0FreeChunk(pGMM, pGVM, pChunk, true /*fRelaxedSem*/))
1324 {
1325 /* We've left the giant mutex, restart? (+1 for our unlink) */
1326 fRedoFromStart = pPrivateSet->idGeneration != idGenerationOld + 1;
1327 if (fRedoFromStart)
1328 break;
1329 uLockNanoTS = RTTimeSystemNanoTS();
1330 iCountDown = 10240;
1331 }
1332 }
1333
1334 /* Advance and maybe yield the lock. */
1335 pChunk = pNext;
1336 if (--iCountDown == 0)
1337 {
1338 uint64_t const idGenerationOld = pPrivateSet->idGeneration;
1339 fRedoFromStart = gmmR0MutexYield(pGMM, &uLockNanoTS)
1340 && pPrivateSet->idGeneration != idGenerationOld;
1341 if (fRedoFromStart)
1342 break;
1343 iCountDown = 10240;
1344 }
1345 }
1346 } while (fRedoFromStart);
1347
1348 /*
1349 * Account for shared pages that weren't freed.
1350 */
1351 if (pGVM->gmm.s.Stats.cSharedPages)
1352 {
1353 Assert(pGMM->cSharedPages >= pGVM->gmm.s.Stats.cSharedPages);
1354 SUPR0Printf("GMMR0CleanupVM: hGVM=%#x left %#x shared pages behind!\n", pGVM->hSelf, pGVM->gmm.s.Stats.cSharedPages);
1355 pGMM->cLeftBehindSharedPages += pGVM->gmm.s.Stats.cSharedPages;
1356 }
1357
1358 /*
1359 * Clean up balloon statistics in case the VM process crashed.
1360 */
1361 Assert(pGMM->cBalloonedPages >= pGVM->gmm.s.Stats.cBalloonedPages);
1362 pGMM->cBalloonedPages -= pGVM->gmm.s.Stats.cBalloonedPages;
1363
1364 /*
1365 * Update the over-commitment management statistics.
1366 */
1367 pGMM->cReservedPages -= pGVM->gmm.s.Stats.Reserved.cBasePages
1368 + pGVM->gmm.s.Stats.Reserved.cFixedPages
1369 + pGVM->gmm.s.Stats.Reserved.cShadowPages;
1370 switch (pGVM->gmm.s.Stats.enmPolicy)
1371 {
1372 case GMMOCPOLICY_NO_OC:
1373 break;
1374 default:
1375 /** @todo Update GMM->cOverCommittedPages */
1376 break;
1377 }
1378 }
1379
1380 /* zap the GVM data. */
1381 pGVM->gmm.s.Stats.enmPolicy = GMMOCPOLICY_INVALID;
1382 pGVM->gmm.s.Stats.enmPriority = GMMPRIORITY_INVALID;
1383 pGVM->gmm.s.Stats.fMayAllocate = false;
1384
1385 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
1386 gmmR0MutexRelease(pGMM);
1387
1388 LogFlow(("GMMR0CleanupVM: returns\n"));
1389}
1390
1391
1392/**
1393 * Scan one chunk for private pages belonging to the specified VM.
1394 *
1395 * @note This function may drop the giant mutex!
1396 *
1397 * @returns @c true if we've temporarily dropped the giant mutex, @c false if
1398 * we didn't.
1399 * @param pGMM Pointer to the GMM instance.
1400 * @param pGVM The global VM handle.
1401 * @param pChunk The chunk to scan.
1402 */
1403static bool gmmR0CleanupVMScanChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk)
1404{
1405 Assert(!pGMM->fBoundMemoryMode || pChunk->hGVM == pGVM->hSelf);
1406
1407 /*
1408 * Look for pages belonging to the VM.
1409 * (Perform some internal checks while we're scanning.)
1410 */
1411#ifndef VBOX_STRICT
1412 if (pChunk->cFree != (GMM_CHUNK_SIZE >> PAGE_SHIFT))
1413#endif
1414 {
1415 unsigned cPrivate = 0;
1416 unsigned cShared = 0;
1417 unsigned cFree = 0;
1418
1419 gmmR0UnlinkChunk(pChunk); /* avoiding cFreePages updates. */
1420
1421 uint16_t hGVM = pGVM->hSelf;
1422 unsigned iPage = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
1423 while (iPage-- > 0)
1424 if (GMM_PAGE_IS_PRIVATE(&pChunk->aPages[iPage]))
1425 {
1426 if (pChunk->aPages[iPage].Private.hGVM == hGVM)
1427 {
1428 /*
1429 * Free the page.
1430 *
1431 * The reason for not using gmmR0FreePrivatePage here is that we
1432 * must *not* cause the chunk to be freed from under us - we're in
1433 * an AVL tree walk here.
1434 */
1435 pChunk->aPages[iPage].u = 0;
1436 pChunk->aPages[iPage].Free.iNext = pChunk->iFreeHead;
1437 pChunk->aPages[iPage].Free.u2State = GMM_PAGE_STATE_FREE;
1438 pChunk->iFreeHead = iPage;
1439 pChunk->cPrivate--;
1440 pChunk->cFree++;
1441 pGVM->gmm.s.Stats.cPrivatePages--;
1442 cFree++;
1443 }
1444 else
1445 cPrivate++;
1446 }
1447 else if (GMM_PAGE_IS_FREE(&pChunk->aPages[iPage]))
1448 cFree++;
1449 else
1450 cShared++;
1451
1452 gmmR0SelectSetAndLinkChunk(pGMM, pGVM, pChunk);
1453
1454 /*
1455 * Did it add up?
1456 */
1457 if (RT_UNLIKELY( pChunk->cFree != cFree
1458 || pChunk->cPrivate != cPrivate
1459 || pChunk->cShared != cShared))
1460 {
1461 SUPR0Printf("gmmR0CleanupVMScanChunk: Chunk %p/%#x has bogus stats - free=%d/%d private=%d/%d shared=%d/%d\n",
1462 pChunk, pChunk->Core.Key, pChunk->cFree, cFree, pChunk->cPrivate, cPrivate, pChunk->cShared, cShared);
1463 pChunk->cFree = cFree;
1464 pChunk->cPrivate = cPrivate;
1465 pChunk->cShared = cShared;
1466 }
1467 }
1468
1469 /*
1470 * If not in bound memory mode, we should reset the hGVM field
1471 * if it has our handle in it.
1472 */
1473 if (pChunk->hGVM == pGVM->hSelf)
1474 {
1475 if (!g_pGMM->fBoundMemoryMode)
1476 pChunk->hGVM = NIL_GVM_HANDLE;
1477 else if (pChunk->cFree != GMM_CHUNK_NUM_PAGES)
1478 {
1479 SUPR0Printf("gmmR0CleanupVMScanChunk: %p/%#x: cFree=%#x - it should be 0 in bound mode!\n",
1480 pChunk, pChunk->Core.Key, pChunk->cFree);
1481 AssertMsgFailed(("%p/%#x: cFree=%#x - it should be 0 in bound mode!\n", pChunk, pChunk->Core.Key, pChunk->cFree));
1482
1483 gmmR0UnlinkChunk(pChunk);
1484 pChunk->cFree = GMM_CHUNK_NUM_PAGES;
1485 gmmR0SelectSetAndLinkChunk(pGMM, pGVM, pChunk);
1486 }
1487 }
1488
1489 /*
1490 * Look for a mapping belonging to the terminating VM.
1491 */
1492 GMMR0CHUNKMTXSTATE MtxState;
1493 gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk, GMMR0CHUNK_MTX_KEEP_GIANT);
1494 unsigned cMappings = pChunk->cMappingsX;
1495 for (unsigned i = 0; i < cMappings; i++)
1496 if (pChunk->paMappingsX[i].pGVM == pGVM)
1497 {
1498 gmmR0ChunkMutexDropGiant(&MtxState);
1499
1500 RTR0MEMOBJ hMemObj = pChunk->paMappingsX[i].hMapObj;
1501
1502 cMappings--;
1503 if (i < cMappings)
1504 pChunk->paMappingsX[i] = pChunk->paMappingsX[cMappings];
1505 pChunk->paMappingsX[cMappings].pGVM = NULL;
1506 pChunk->paMappingsX[cMappings].hMapObj = NIL_RTR0MEMOBJ;
1507 Assert(pChunk->cMappingsX - 1U == cMappings);
1508 pChunk->cMappingsX = cMappings;
1509
1510 int rc = RTR0MemObjFree(hMemObj, false /* fFreeMappings (NA) */);
1511 if (RT_FAILURE(rc))
1512 {
1513 SUPR0Printf("gmmR0CleanupVMScanChunk: %p/%#x: mapping #%x: RTRMemObjFree(%p,false) -> %d \n",
1514 pChunk, pChunk->Core.Key, i, hMemObj, rc);
1515 AssertRC(rc);
1516 }
1517
1518 gmmR0ChunkMutexRelease(&MtxState, pChunk);
1519 return true;
1520 }
1521
1522 gmmR0ChunkMutexRelease(&MtxState, pChunk);
1523 return false;
1524}
1525
1526
1527/**
1528 * The initial resource reservations.
1529 *
1530 * This will make memory reservations according to policy and priority. If there aren't
1531 * sufficient resources available to sustain the VM this function will fail and all
1532 * future allocations requests will fail as well.
1533 *
1534 * These are just the initial reservations made very very early during the VM creation
1535 * process and will be adjusted later in the GMMR0UpdateReservation call after the
1536 * ring-3 init has completed.
1537 *
1538 * @returns VBox status code.
1539 * @retval VERR_GMM_MEMORY_RESERVATION_DECLINED
1540 * @retval VERR_GMM_
1541 *
1542 * @param pVM The cross context VM structure.
1543 * @param idCpu The VCPU id.
1544 * @param cBasePages The number of pages that may be allocated for the base RAM and ROMs.
1545 * This does not include MMIO2 and similar.
1546 * @param cShadowPages The number of pages that may be allocated for shadow paging structures.
1547 * @param cFixedPages The number of pages that may be allocated for fixed objects like the
1548 * hyper heap, MMIO2 and similar.
1549 * @param enmPolicy The OC policy to use on this VM.
1550 * @param enmPriority The priority in an out-of-memory situation.
1551 *
1552 * @thread The creator thread / EMT.
1553 */
1554GMMR0DECL(int) GMMR0InitialReservation(PVM pVM, VMCPUID idCpu, uint64_t cBasePages, uint32_t cShadowPages, uint32_t cFixedPages,
1555 GMMOCPOLICY enmPolicy, GMMPRIORITY enmPriority)
1556{
1557 LogFlow(("GMMR0InitialReservation: pVM=%p cBasePages=%#llx cShadowPages=%#x cFixedPages=%#x enmPolicy=%d enmPriority=%d\n",
1558 pVM, cBasePages, cShadowPages, cFixedPages, enmPolicy, enmPriority));
1559
1560 /*
1561 * Validate, get basics and take the semaphore.
1562 */
1563 PGMM pGMM;
1564 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
1565 PGVM pGVM;
1566 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
1567 if (RT_FAILURE(rc))
1568 return rc;
1569
1570 AssertReturn(cBasePages, VERR_INVALID_PARAMETER);
1571 AssertReturn(cShadowPages, VERR_INVALID_PARAMETER);
1572 AssertReturn(cFixedPages, VERR_INVALID_PARAMETER);
1573 AssertReturn(enmPolicy > GMMOCPOLICY_INVALID && enmPolicy < GMMOCPOLICY_END, VERR_INVALID_PARAMETER);
1574 AssertReturn(enmPriority > GMMPRIORITY_INVALID && enmPriority < GMMPRIORITY_END, VERR_INVALID_PARAMETER);
1575
1576 gmmR0MutexAcquire(pGMM);
1577 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
1578 {
1579 if ( !pGVM->gmm.s.Stats.Reserved.cBasePages
1580 && !pGVM->gmm.s.Stats.Reserved.cFixedPages
1581 && !pGVM->gmm.s.Stats.Reserved.cShadowPages)
1582 {
1583 /*
1584 * Check if we can accommodate this.
1585 */
1586 /* ... later ... */
1587 if (RT_SUCCESS(rc))
1588 {
1589 /*
1590 * Update the records.
1591 */
1592 pGVM->gmm.s.Stats.Reserved.cBasePages = cBasePages;
1593 pGVM->gmm.s.Stats.Reserved.cFixedPages = cFixedPages;
1594 pGVM->gmm.s.Stats.Reserved.cShadowPages = cShadowPages;
1595 pGVM->gmm.s.Stats.enmPolicy = enmPolicy;
1596 pGVM->gmm.s.Stats.enmPriority = enmPriority;
1597 pGVM->gmm.s.Stats.fMayAllocate = true;
1598
1599 pGMM->cReservedPages += cBasePages + cFixedPages + cShadowPages;
1600 pGMM->cRegisteredVMs++;
1601 }
1602 }
1603 else
1604 rc = VERR_WRONG_ORDER;
1605 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
1606 }
1607 else
1608 rc = VERR_GMM_IS_NOT_SANE;
1609 gmmR0MutexRelease(pGMM);
1610 LogFlow(("GMMR0InitialReservation: returns %Rrc\n", rc));
1611 return rc;
1612}
1613
1614
1615/**
1616 * VMMR0 request wrapper for GMMR0InitialReservation.
1617 *
1618 * @returns see GMMR0InitialReservation.
1619 * @param pVM The cross context VM structure.
1620 * @param idCpu The VCPU id.
1621 * @param pReq Pointer to the request packet.
1622 */
1623GMMR0DECL(int) GMMR0InitialReservationReq(PVM pVM, VMCPUID idCpu, PGMMINITIALRESERVATIONREQ pReq)
1624{
1625 /*
1626 * Validate input and pass it on.
1627 */
1628 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
1629 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
1630 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
1631
1632 return GMMR0InitialReservation(pVM, idCpu, pReq->cBasePages, pReq->cShadowPages, pReq->cFixedPages, pReq->enmPolicy, pReq->enmPriority);
1633}
1634
1635
1636/**
1637 * This updates the memory reservation with the additional MMIO2 and ROM pages.
1638 *
1639 * @returns VBox status code.
1640 * @retval VERR_GMM_MEMORY_RESERVATION_DECLINED
1641 *
1642 * @param pVM The cross context VM structure.
1643 * @param idCpu The VCPU id.
1644 * @param cBasePages The number of pages that may be allocated for the base RAM and ROMs.
1645 * This does not include MMIO2 and similar.
1646 * @param cShadowPages The number of pages that may be allocated for shadow paging structures.
1647 * @param cFixedPages The number of pages that may be allocated for fixed objects like the
1648 * hyper heap, MMIO2 and similar.
1649 *
1650 * @thread EMT.
1651 */
1652GMMR0DECL(int) GMMR0UpdateReservation(PVM pVM, VMCPUID idCpu, uint64_t cBasePages, uint32_t cShadowPages, uint32_t cFixedPages)
1653{
1654 LogFlow(("GMMR0UpdateReservation: pVM=%p cBasePages=%#llx cShadowPages=%#x cFixedPages=%#x\n",
1655 pVM, cBasePages, cShadowPages, cFixedPages));
1656
1657 /*
1658 * Validate, get basics and take the semaphore.
1659 */
1660 PGMM pGMM;
1661 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
1662 PGVM pGVM;
1663 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
1664 if (RT_FAILURE(rc))
1665 return rc;
1666
1667 AssertReturn(cBasePages, VERR_INVALID_PARAMETER);
1668 AssertReturn(cShadowPages, VERR_INVALID_PARAMETER);
1669 AssertReturn(cFixedPages, VERR_INVALID_PARAMETER);
1670
1671 gmmR0MutexAcquire(pGMM);
1672 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
1673 {
1674 if ( pGVM->gmm.s.Stats.Reserved.cBasePages
1675 && pGVM->gmm.s.Stats.Reserved.cFixedPages
1676 && pGVM->gmm.s.Stats.Reserved.cShadowPages)
1677 {
1678 /*
1679 * Check if we can accommodate this.
1680 */
1681 /* ... later ... */
1682 if (RT_SUCCESS(rc))
1683 {
1684 /*
1685 * Update the records.
1686 */
1687 pGMM->cReservedPages -= pGVM->gmm.s.Stats.Reserved.cBasePages
1688 + pGVM->gmm.s.Stats.Reserved.cFixedPages
1689 + pGVM->gmm.s.Stats.Reserved.cShadowPages;
1690 pGMM->cReservedPages += cBasePages + cFixedPages + cShadowPages;
1691
1692 pGVM->gmm.s.Stats.Reserved.cBasePages = cBasePages;
1693 pGVM->gmm.s.Stats.Reserved.cFixedPages = cFixedPages;
1694 pGVM->gmm.s.Stats.Reserved.cShadowPages = cShadowPages;
1695 }
1696 }
1697 else
1698 rc = VERR_WRONG_ORDER;
1699 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
1700 }
1701 else
1702 rc = VERR_GMM_IS_NOT_SANE;
1703 gmmR0MutexRelease(pGMM);
1704 LogFlow(("GMMR0UpdateReservation: returns %Rrc\n", rc));
1705 return rc;
1706}
1707
1708
1709/**
1710 * VMMR0 request wrapper for GMMR0UpdateReservation.
1711 *
1712 * @returns see GMMR0UpdateReservation.
1713 * @param pVM The cross context VM structure.
1714 * @param idCpu The VCPU id.
1715 * @param pReq Pointer to the request packet.
1716 */
1717GMMR0DECL(int) GMMR0UpdateReservationReq(PVM pVM, VMCPUID idCpu, PGMMUPDATERESERVATIONREQ pReq)
1718{
1719 /*
1720 * Validate input and pass it on.
1721 */
1722 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
1723 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
1724 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
1725
1726 return GMMR0UpdateReservation(pVM, idCpu, pReq->cBasePages, pReq->cShadowPages, pReq->cFixedPages);
1727}
1728
1729#ifdef GMMR0_WITH_SANITY_CHECK
1730
1731/**
1732 * Performs sanity checks on a free set.
1733 *
1734 * @returns Error count.
1735 *
1736 * @param pGMM Pointer to the GMM instance.
1737 * @param pSet Pointer to the set.
1738 * @param pszSetName The set name.
1739 * @param pszFunction The function from which it was called.
1740 * @param uLine The line number.
1741 */
1742static uint32_t gmmR0SanityCheckSet(PGMM pGMM, PGMMCHUNKFREESET pSet, const char *pszSetName,
1743 const char *pszFunction, unsigned uLineNo)
1744{
1745 uint32_t cErrors = 0;
1746
1747 /*
1748 * Count the free pages in all the chunks and match it against pSet->cFreePages.
1749 */
1750 uint32_t cPages = 0;
1751 for (unsigned i = 0; i < RT_ELEMENTS(pSet->apLists); i++)
1752 {
1753 for (PGMMCHUNK pCur = pSet->apLists[i]; pCur; pCur = pCur->pFreeNext)
1754 {
1755 /** @todo check that the chunk is hash into the right set. */
1756 cPages += pCur->cFree;
1757 }
1758 }
1759 if (RT_UNLIKELY(cPages != pSet->cFreePages))
1760 {
1761 SUPR0Printf("GMM insanity: found %#x pages in the %s set, expected %#x. (%s, line %u)\n",
1762 cPages, pszSetName, pSet->cFreePages, pszFunction, uLineNo);
1763 cErrors++;
1764 }
1765
1766 return cErrors;
1767}
1768
1769
1770/**
1771 * Performs some sanity checks on the GMM while owning lock.
1772 *
1773 * @returns Error count.
1774 *
1775 * @param pGMM Pointer to the GMM instance.
1776 * @param pszFunction The function from which it is called.
1777 * @param uLineNo The line number.
1778 */
1779static uint32_t gmmR0SanityCheck(PGMM pGMM, const char *pszFunction, unsigned uLineNo)
1780{
1781 uint32_t cErrors = 0;
1782
1783 cErrors += gmmR0SanityCheckSet(pGMM, &pGMM->PrivateX, "private", pszFunction, uLineNo);
1784 cErrors += gmmR0SanityCheckSet(pGMM, &pGMM->Shared, "shared", pszFunction, uLineNo);
1785 /** @todo add more sanity checks. */
1786
1787 return cErrors;
1788}
1789
1790#endif /* GMMR0_WITH_SANITY_CHECK */
1791
1792/**
1793 * Looks up a chunk in the tree and fill in the TLB entry for it.
1794 *
1795 * This is not expected to fail and will bitch if it does.
1796 *
1797 * @returns Pointer to the allocation chunk, NULL if not found.
1798 * @param pGMM Pointer to the GMM instance.
1799 * @param idChunk The ID of the chunk to find.
1800 * @param pTlbe Pointer to the TLB entry.
1801 */
1802static PGMMCHUNK gmmR0GetChunkSlow(PGMM pGMM, uint32_t idChunk, PGMMCHUNKTLBE pTlbe)
1803{
1804 PGMMCHUNK pChunk = (PGMMCHUNK)RTAvlU32Get(&pGMM->pChunks, idChunk);
1805 AssertMsgReturn(pChunk, ("Chunk %#x not found!\n", idChunk), NULL);
1806 pTlbe->idChunk = idChunk;
1807 pTlbe->pChunk = pChunk;
1808 return pChunk;
1809}
1810
1811
1812/**
1813 * Finds a allocation chunk.
1814 *
1815 * This is not expected to fail and will bitch if it does.
1816 *
1817 * @returns Pointer to the allocation chunk, NULL if not found.
1818 * @param pGMM Pointer to the GMM instance.
1819 * @param idChunk The ID of the chunk to find.
1820 */
1821DECLINLINE(PGMMCHUNK) gmmR0GetChunk(PGMM pGMM, uint32_t idChunk)
1822{
1823 /*
1824 * Do a TLB lookup, branch if not in the TLB.
1825 */
1826 PGMMCHUNKTLBE pTlbe = &pGMM->ChunkTLB.aEntries[GMM_CHUNKTLB_IDX(idChunk)];
1827 if ( pTlbe->idChunk != idChunk
1828 || !pTlbe->pChunk)
1829 return gmmR0GetChunkSlow(pGMM, idChunk, pTlbe);
1830 return pTlbe->pChunk;
1831}
1832
1833
1834/**
1835 * Finds a page.
1836 *
1837 * This is not expected to fail and will bitch if it does.
1838 *
1839 * @returns Pointer to the page, NULL if not found.
1840 * @param pGMM Pointer to the GMM instance.
1841 * @param idPage The ID of the page to find.
1842 */
1843DECLINLINE(PGMMPAGE) gmmR0GetPage(PGMM pGMM, uint32_t idPage)
1844{
1845 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
1846 if (RT_LIKELY(pChunk))
1847 return &pChunk->aPages[idPage & GMM_PAGEID_IDX_MASK];
1848 return NULL;
1849}
1850
1851
1852#if 0 /* unused */
1853/**
1854 * Gets the host physical address for a page given by it's ID.
1855 *
1856 * @returns The host physical address or NIL_RTHCPHYS.
1857 * @param pGMM Pointer to the GMM instance.
1858 * @param idPage The ID of the page to find.
1859 */
1860DECLINLINE(RTHCPHYS) gmmR0GetPageHCPhys(PGMM pGMM, uint32_t idPage)
1861{
1862 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
1863 if (RT_LIKELY(pChunk))
1864 return RTR0MemObjGetPagePhysAddr(pChunk->hMemObj, idPage & GMM_PAGEID_IDX_MASK);
1865 return NIL_RTHCPHYS;
1866}
1867#endif /* unused */
1868
1869
1870/**
1871 * Selects the appropriate free list given the number of free pages.
1872 *
1873 * @returns Free list index.
1874 * @param cFree The number of free pages in the chunk.
1875 */
1876DECLINLINE(unsigned) gmmR0SelectFreeSetList(unsigned cFree)
1877{
1878 unsigned iList = cFree >> GMM_CHUNK_FREE_SET_SHIFT;
1879 AssertMsg(iList < RT_SIZEOFMEMB(GMMCHUNKFREESET, apLists) / RT_SIZEOFMEMB(GMMCHUNKFREESET, apLists[0]),
1880 ("%d (%u)\n", iList, cFree));
1881 return iList;
1882}
1883
1884
1885/**
1886 * Unlinks the chunk from the free list it's currently on (if any).
1887 *
1888 * @param pChunk The allocation chunk.
1889 */
1890DECLINLINE(void) gmmR0UnlinkChunk(PGMMCHUNK pChunk)
1891{
1892 PGMMCHUNKFREESET pSet = pChunk->pSet;
1893 if (RT_LIKELY(pSet))
1894 {
1895 pSet->cFreePages -= pChunk->cFree;
1896 pSet->idGeneration++;
1897
1898 PGMMCHUNK pPrev = pChunk->pFreePrev;
1899 PGMMCHUNK pNext = pChunk->pFreeNext;
1900 if (pPrev)
1901 pPrev->pFreeNext = pNext;
1902 else
1903 pSet->apLists[gmmR0SelectFreeSetList(pChunk->cFree)] = pNext;
1904 if (pNext)
1905 pNext->pFreePrev = pPrev;
1906
1907 pChunk->pSet = NULL;
1908 pChunk->pFreeNext = NULL;
1909 pChunk->pFreePrev = NULL;
1910 }
1911 else
1912 {
1913 Assert(!pChunk->pFreeNext);
1914 Assert(!pChunk->pFreePrev);
1915 Assert(!pChunk->cFree);
1916 }
1917}
1918
1919
1920/**
1921 * Links the chunk onto the appropriate free list in the specified free set.
1922 *
1923 * If no free entries, it's not linked into any list.
1924 *
1925 * @param pChunk The allocation chunk.
1926 * @param pSet The free set.
1927 */
1928DECLINLINE(void) gmmR0LinkChunk(PGMMCHUNK pChunk, PGMMCHUNKFREESET pSet)
1929{
1930 Assert(!pChunk->pSet);
1931 Assert(!pChunk->pFreeNext);
1932 Assert(!pChunk->pFreePrev);
1933
1934 if (pChunk->cFree > 0)
1935 {
1936 pChunk->pSet = pSet;
1937 pChunk->pFreePrev = NULL;
1938 unsigned const iList = gmmR0SelectFreeSetList(pChunk->cFree);
1939 pChunk->pFreeNext = pSet->apLists[iList];
1940 if (pChunk->pFreeNext)
1941 pChunk->pFreeNext->pFreePrev = pChunk;
1942 pSet->apLists[iList] = pChunk;
1943
1944 pSet->cFreePages += pChunk->cFree;
1945 pSet->idGeneration++;
1946 }
1947}
1948
1949
1950/**
1951 * Links the chunk onto the appropriate free list in the specified free set.
1952 *
1953 * If no free entries, it's not linked into any list.
1954 *
1955 * @param pGMM Pointer to the GMM instance.
1956 * @param pGVM Pointer to the kernel-only VM instace data.
1957 * @param pChunk The allocation chunk.
1958 */
1959DECLINLINE(void) gmmR0SelectSetAndLinkChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk)
1960{
1961 PGMMCHUNKFREESET pSet;
1962 if (pGMM->fBoundMemoryMode)
1963 pSet = &pGVM->gmm.s.Private;
1964 else if (pChunk->cShared)
1965 pSet = &pGMM->Shared;
1966 else
1967 pSet = &pGMM->PrivateX;
1968 gmmR0LinkChunk(pChunk, pSet);
1969}
1970
1971
1972/**
1973 * Frees a Chunk ID.
1974 *
1975 * @param pGMM Pointer to the GMM instance.
1976 * @param idChunk The Chunk ID to free.
1977 */
1978static void gmmR0FreeChunkId(PGMM pGMM, uint32_t idChunk)
1979{
1980 AssertReturnVoid(idChunk != NIL_GMM_CHUNKID);
1981 AssertMsg(ASMBitTest(&pGMM->bmChunkId[0], idChunk), ("%#x\n", idChunk));
1982 ASMAtomicBitClear(&pGMM->bmChunkId[0], idChunk);
1983}
1984
1985
1986/**
1987 * Allocates a new Chunk ID.
1988 *
1989 * @returns The Chunk ID.
1990 * @param pGMM Pointer to the GMM instance.
1991 */
1992static uint32_t gmmR0AllocateChunkId(PGMM pGMM)
1993{
1994 AssertCompile(!((GMM_CHUNKID_LAST + 1) & 31)); /* must be a multiple of 32 */
1995 AssertCompile(NIL_GMM_CHUNKID == 0);
1996
1997 /*
1998 * Try the next sequential one.
1999 */
2000 int32_t idChunk = ++pGMM->idChunkPrev;
2001#if 0 /** @todo enable this code */
2002 if ( idChunk <= GMM_CHUNKID_LAST
2003 && idChunk > NIL_GMM_CHUNKID
2004 && !ASMAtomicBitTestAndSet(&pVMM->bmChunkId[0], idChunk))
2005 return idChunk;
2006#endif
2007
2008 /*
2009 * Scan sequentially from the last one.
2010 */
2011 if ( (uint32_t)idChunk < GMM_CHUNKID_LAST
2012 && idChunk > NIL_GMM_CHUNKID)
2013 {
2014 idChunk = ASMBitNextClear(&pGMM->bmChunkId[0], GMM_CHUNKID_LAST + 1, idChunk - 1);
2015 if (idChunk > NIL_GMM_CHUNKID)
2016 {
2017 AssertMsgReturn(!ASMAtomicBitTestAndSet(&pGMM->bmChunkId[0], idChunk), ("%#x\n", idChunk), NIL_GMM_CHUNKID);
2018 return pGMM->idChunkPrev = idChunk;
2019 }
2020 }
2021
2022 /*
2023 * Ok, scan from the start.
2024 * We're not racing anyone, so there is no need to expect failures or have restart loops.
2025 */
2026 idChunk = ASMBitFirstClear(&pGMM->bmChunkId[0], GMM_CHUNKID_LAST + 1);
2027 AssertMsgReturn(idChunk > NIL_GMM_CHUNKID, ("%#x\n", idChunk), NIL_GVM_HANDLE);
2028 AssertMsgReturn(!ASMAtomicBitTestAndSet(&pGMM->bmChunkId[0], idChunk), ("%#x\n", idChunk), NIL_GMM_CHUNKID);
2029
2030 return pGMM->idChunkPrev = idChunk;
2031}
2032
2033
2034/**
2035 * Allocates one private page.
2036 *
2037 * Worker for gmmR0AllocatePages.
2038 *
2039 * @param pChunk The chunk to allocate it from.
2040 * @param hGVM The GVM handle of the VM requesting memory.
2041 * @param pPageDesc The page descriptor.
2042 */
2043static void gmmR0AllocatePage(PGMMCHUNK pChunk, uint32_t hGVM, PGMMPAGEDESC pPageDesc)
2044{
2045 /* update the chunk stats. */
2046 if (pChunk->hGVM == NIL_GVM_HANDLE)
2047 pChunk->hGVM = hGVM;
2048 Assert(pChunk->cFree);
2049 pChunk->cFree--;
2050 pChunk->cPrivate++;
2051
2052 /* unlink the first free page. */
2053 const uint32_t iPage = pChunk->iFreeHead;
2054 AssertReleaseMsg(iPage < RT_ELEMENTS(pChunk->aPages), ("%d\n", iPage));
2055 PGMMPAGE pPage = &pChunk->aPages[iPage];
2056 Assert(GMM_PAGE_IS_FREE(pPage));
2057 pChunk->iFreeHead = pPage->Free.iNext;
2058 Log3(("A pPage=%p iPage=%#x/%#x u2State=%d iFreeHead=%#x iNext=%#x\n",
2059 pPage, iPage, (pChunk->Core.Key << GMM_CHUNKID_SHIFT) | iPage,
2060 pPage->Common.u2State, pChunk->iFreeHead, pPage->Free.iNext));
2061
2062 /* make the page private. */
2063 pPage->u = 0;
2064 AssertCompile(GMM_PAGE_STATE_PRIVATE == 0);
2065 pPage->Private.hGVM = hGVM;
2066 AssertCompile(NIL_RTHCPHYS >= GMM_GCPHYS_LAST);
2067 AssertCompile(GMM_GCPHYS_UNSHAREABLE >= GMM_GCPHYS_LAST);
2068 if (pPageDesc->HCPhysGCPhys <= GMM_GCPHYS_LAST)
2069 pPage->Private.pfn = pPageDesc->HCPhysGCPhys >> PAGE_SHIFT;
2070 else
2071 pPage->Private.pfn = GMM_PAGE_PFN_UNSHAREABLE; /* unshareable / unassigned - same thing. */
2072
2073 /* update the page descriptor. */
2074 pPageDesc->HCPhysGCPhys = RTR0MemObjGetPagePhysAddr(pChunk->hMemObj, iPage);
2075 Assert(pPageDesc->HCPhysGCPhys != NIL_RTHCPHYS);
2076 pPageDesc->idPage = (pChunk->Core.Key << GMM_CHUNKID_SHIFT) | iPage;
2077 pPageDesc->idSharedPage = NIL_GMM_PAGEID;
2078}
2079
2080
2081/**
2082 * Picks the free pages from a chunk.
2083 *
2084 * @returns The new page descriptor table index.
2085 * @param pChunk The chunk.
2086 * @param hGVM The affinity of the chunk. NIL_GVM_HANDLE for no
2087 * affinity.
2088 * @param iPage The current page descriptor table index.
2089 * @param cPages The total number of pages to allocate.
2090 * @param paPages The page descriptor table (input + ouput).
2091 */
2092static uint32_t gmmR0AllocatePagesFromChunk(PGMMCHUNK pChunk, uint16_t const hGVM, uint32_t iPage, uint32_t cPages,
2093 PGMMPAGEDESC paPages)
2094{
2095 PGMMCHUNKFREESET pSet = pChunk->pSet; Assert(pSet);
2096 gmmR0UnlinkChunk(pChunk);
2097
2098 for (; pChunk->cFree && iPage < cPages; iPage++)
2099 gmmR0AllocatePage(pChunk, hGVM, &paPages[iPage]);
2100
2101 gmmR0LinkChunk(pChunk, pSet);
2102 return iPage;
2103}
2104
2105
2106/**
2107 * Registers a new chunk of memory.
2108 *
2109 * This is called by both gmmR0AllocateOneChunk and GMMR0SeedChunk.
2110 *
2111 * @returns VBox status code. On success, the giant GMM lock will be held, the
2112 * caller must release it (ugly).
2113 * @param pGMM Pointer to the GMM instance.
2114 * @param pSet Pointer to the set.
2115 * @param MemObj The memory object for the chunk.
2116 * @param hGVM The affinity of the chunk. NIL_GVM_HANDLE for no
2117 * affinity.
2118 * @param fChunkFlags The chunk flags, GMM_CHUNK_FLAGS_XXX.
2119 * @param ppChunk Chunk address (out). Optional.
2120 *
2121 * @remarks The caller must not own the giant GMM mutex.
2122 * The giant GMM mutex will be acquired and returned acquired in
2123 * the success path. On failure, no locks will be held.
2124 */
2125static int gmmR0RegisterChunk(PGMM pGMM, PGMMCHUNKFREESET pSet, RTR0MEMOBJ MemObj, uint16_t hGVM, uint16_t fChunkFlags,
2126 PGMMCHUNK *ppChunk)
2127{
2128 Assert(pGMM->hMtxOwner != RTThreadNativeSelf());
2129 Assert(hGVM != NIL_GVM_HANDLE || pGMM->fBoundMemoryMode);
2130 Assert(fChunkFlags == 0 || fChunkFlags == GMM_CHUNK_FLAGS_LARGE_PAGE);
2131
2132 int rc;
2133 PGMMCHUNK pChunk = (PGMMCHUNK)RTMemAllocZ(sizeof(*pChunk));
2134 if (pChunk)
2135 {
2136 /*
2137 * Initialize it.
2138 */
2139 pChunk->hMemObj = MemObj;
2140 pChunk->cFree = GMM_CHUNK_NUM_PAGES;
2141 pChunk->hGVM = hGVM;
2142 /*pChunk->iFreeHead = 0;*/
2143 pChunk->idNumaNode = gmmR0GetCurrentNumaNodeId();
2144 pChunk->iChunkMtx = UINT8_MAX;
2145 pChunk->fFlags = fChunkFlags;
2146 for (unsigned iPage = 0; iPage < RT_ELEMENTS(pChunk->aPages) - 1; iPage++)
2147 {
2148 pChunk->aPages[iPage].Free.u2State = GMM_PAGE_STATE_FREE;
2149 pChunk->aPages[iPage].Free.iNext = iPage + 1;
2150 }
2151 pChunk->aPages[RT_ELEMENTS(pChunk->aPages) - 1].Free.u2State = GMM_PAGE_STATE_FREE;
2152 pChunk->aPages[RT_ELEMENTS(pChunk->aPages) - 1].Free.iNext = UINT16_MAX;
2153
2154 /*
2155 * Allocate a Chunk ID and insert it into the tree.
2156 * This has to be done behind the mutex of course.
2157 */
2158 rc = gmmR0MutexAcquire(pGMM);
2159 if (RT_SUCCESS(rc))
2160 {
2161 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
2162 {
2163 pChunk->Core.Key = gmmR0AllocateChunkId(pGMM);
2164 if ( pChunk->Core.Key != NIL_GMM_CHUNKID
2165 && pChunk->Core.Key <= GMM_CHUNKID_LAST
2166 && RTAvlU32Insert(&pGMM->pChunks, &pChunk->Core))
2167 {
2168 pGMM->cChunks++;
2169 RTListAppend(&pGMM->ChunkList, &pChunk->ListNode);
2170 gmmR0LinkChunk(pChunk, pSet);
2171 LogFlow(("gmmR0RegisterChunk: pChunk=%p id=%#x cChunks=%d\n", pChunk, pChunk->Core.Key, pGMM->cChunks));
2172
2173 if (ppChunk)
2174 *ppChunk = pChunk;
2175 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
2176 return VINF_SUCCESS;
2177 }
2178
2179 /* bail out */
2180 rc = VERR_GMM_CHUNK_INSERT;
2181 }
2182 else
2183 rc = VERR_GMM_IS_NOT_SANE;
2184 gmmR0MutexRelease(pGMM);
2185 }
2186
2187 RTMemFree(pChunk);
2188 }
2189 else
2190 rc = VERR_NO_MEMORY;
2191 return rc;
2192}
2193
2194
2195/**
2196 * Allocate a new chunk, immediately pick the requested pages from it, and adds
2197 * what's remaining to the specified free set.
2198 *
2199 * @note This will leave the giant mutex while allocating the new chunk!
2200 *
2201 * @returns VBox status code.
2202 * @param pGMM Pointer to the GMM instance data.
2203 * @param pGVM Pointer to the kernel-only VM instace data.
2204 * @param pSet Pointer to the free set.
2205 * @param cPages The number of pages requested.
2206 * @param paPages The page descriptor table (input + output).
2207 * @param piPage The pointer to the page descriptor table index variable.
2208 * This will be updated.
2209 */
2210static int gmmR0AllocateChunkNew(PGMM pGMM, PGVM pGVM, PGMMCHUNKFREESET pSet, uint32_t cPages,
2211 PGMMPAGEDESC paPages, uint32_t *piPage)
2212{
2213 gmmR0MutexRelease(pGMM);
2214
2215 RTR0MEMOBJ hMemObj;
2216 int rc = RTR0MemObjAllocPhysNC(&hMemObj, GMM_CHUNK_SIZE, NIL_RTHCPHYS);
2217 if (RT_SUCCESS(rc))
2218 {
2219/** @todo Duplicate gmmR0RegisterChunk here so we can avoid chaining up the
2220 * free pages first and then unchaining them right afterwards. Instead
2221 * do as much work as possible without holding the giant lock. */
2222 PGMMCHUNK pChunk;
2223 rc = gmmR0RegisterChunk(pGMM, pSet, hMemObj, pGVM->hSelf, 0 /*fChunkFlags*/, &pChunk);
2224 if (RT_SUCCESS(rc))
2225 {
2226 *piPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, *piPage, cPages, paPages);
2227 return VINF_SUCCESS;
2228 }
2229
2230 /* bail out */
2231 RTR0MemObjFree(hMemObj, false /* fFreeMappings */);
2232 }
2233
2234 int rc2 = gmmR0MutexAcquire(pGMM);
2235 AssertRCReturn(rc2, RT_FAILURE(rc) ? rc : rc2);
2236 return rc;
2237
2238}
2239
2240
2241/**
2242 * As a last restort we'll pick any page we can get.
2243 *
2244 * @returns The new page descriptor table index.
2245 * @param pSet The set to pick from.
2246 * @param pGVM Pointer to the global VM structure.
2247 * @param iPage The current page descriptor table index.
2248 * @param cPages The total number of pages to allocate.
2249 * @param paPages The page descriptor table (input + ouput).
2250 */
2251static uint32_t gmmR0AllocatePagesIndiscriminately(PGMMCHUNKFREESET pSet, PGVM pGVM,
2252 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2253{
2254 unsigned iList = RT_ELEMENTS(pSet->apLists);
2255 while (iList-- > 0)
2256 {
2257 PGMMCHUNK pChunk = pSet->apLists[iList];
2258 while (pChunk)
2259 {
2260 PGMMCHUNK pNext = pChunk->pFreeNext;
2261
2262 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2263 if (iPage >= cPages)
2264 return iPage;
2265
2266 pChunk = pNext;
2267 }
2268 }
2269 return iPage;
2270}
2271
2272
2273/**
2274 * Pick pages from empty chunks on the same NUMA node.
2275 *
2276 * @returns The new page descriptor table index.
2277 * @param pSet The set to pick from.
2278 * @param pGVM Pointer to the global VM structure.
2279 * @param iPage The current page descriptor table index.
2280 * @param cPages The total number of pages to allocate.
2281 * @param paPages The page descriptor table (input + ouput).
2282 */
2283static uint32_t gmmR0AllocatePagesFromEmptyChunksOnSameNode(PGMMCHUNKFREESET pSet, PGVM pGVM,
2284 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2285{
2286 PGMMCHUNK pChunk = pSet->apLists[GMM_CHUNK_FREE_SET_UNUSED_LIST];
2287 if (pChunk)
2288 {
2289 uint16_t const idNumaNode = gmmR0GetCurrentNumaNodeId();
2290 while (pChunk)
2291 {
2292 PGMMCHUNK pNext = pChunk->pFreeNext;
2293
2294 if (pChunk->idNumaNode == idNumaNode)
2295 {
2296 pChunk->hGVM = pGVM->hSelf;
2297 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2298 if (iPage >= cPages)
2299 {
2300 pGVM->gmm.s.idLastChunkHint = pChunk->cFree ? pChunk->Core.Key : NIL_GMM_CHUNKID;
2301 return iPage;
2302 }
2303 }
2304
2305 pChunk = pNext;
2306 }
2307 }
2308 return iPage;
2309}
2310
2311
2312/**
2313 * Pick pages from non-empty chunks on the same NUMA node.
2314 *
2315 * @returns The new page descriptor table index.
2316 * @param pSet The set to pick from.
2317 * @param pGVM Pointer to the global VM structure.
2318 * @param iPage The current page descriptor table index.
2319 * @param cPages The total number of pages to allocate.
2320 * @param paPages The page descriptor table (input + ouput).
2321 */
2322static uint32_t gmmR0AllocatePagesFromSameNode(PGMMCHUNKFREESET pSet, PGVM pGVM,
2323 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2324{
2325 /** @todo start by picking from chunks with about the right size first? */
2326 uint16_t const idNumaNode = gmmR0GetCurrentNumaNodeId();
2327 unsigned iList = GMM_CHUNK_FREE_SET_UNUSED_LIST;
2328 while (iList-- > 0)
2329 {
2330 PGMMCHUNK pChunk = pSet->apLists[iList];
2331 while (pChunk)
2332 {
2333 PGMMCHUNK pNext = pChunk->pFreeNext;
2334
2335 if (pChunk->idNumaNode == idNumaNode)
2336 {
2337 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2338 if (iPage >= cPages)
2339 {
2340 pGVM->gmm.s.idLastChunkHint = pChunk->cFree ? pChunk->Core.Key : NIL_GMM_CHUNKID;
2341 return iPage;
2342 }
2343 }
2344
2345 pChunk = pNext;
2346 }
2347 }
2348 return iPage;
2349}
2350
2351
2352/**
2353 * Pick pages that are in chunks already associated with the VM.
2354 *
2355 * @returns The new page descriptor table index.
2356 * @param pGMM Pointer to the GMM instance data.
2357 * @param pGVM Pointer to the global VM structure.
2358 * @param pSet The set to pick from.
2359 * @param iPage The current page descriptor table index.
2360 * @param cPages The total number of pages to allocate.
2361 * @param paPages The page descriptor table (input + ouput).
2362 */
2363static uint32_t gmmR0AllocatePagesAssociatedWithVM(PGMM pGMM, PGVM pGVM, PGMMCHUNKFREESET pSet,
2364 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2365{
2366 uint16_t const hGVM = pGVM->hSelf;
2367
2368 /* Hint. */
2369 if (pGVM->gmm.s.idLastChunkHint != NIL_GMM_CHUNKID)
2370 {
2371 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, pGVM->gmm.s.idLastChunkHint);
2372 if (pChunk && pChunk->cFree)
2373 {
2374 iPage = gmmR0AllocatePagesFromChunk(pChunk, hGVM, iPage, cPages, paPages);
2375 if (iPage >= cPages)
2376 return iPage;
2377 }
2378 }
2379
2380 /* Scan. */
2381 for (unsigned iList = 0; iList < RT_ELEMENTS(pSet->apLists); iList++)
2382 {
2383 PGMMCHUNK pChunk = pSet->apLists[iList];
2384 while (pChunk)
2385 {
2386 PGMMCHUNK pNext = pChunk->pFreeNext;
2387
2388 if (pChunk->hGVM == hGVM)
2389 {
2390 iPage = gmmR0AllocatePagesFromChunk(pChunk, hGVM, iPage, cPages, paPages);
2391 if (iPage >= cPages)
2392 {
2393 pGVM->gmm.s.idLastChunkHint = pChunk->cFree ? pChunk->Core.Key : NIL_GMM_CHUNKID;
2394 return iPage;
2395 }
2396 }
2397
2398 pChunk = pNext;
2399 }
2400 }
2401 return iPage;
2402}
2403
2404
2405
2406/**
2407 * Pick pages in bound memory mode.
2408 *
2409 * @returns The new page descriptor table index.
2410 * @param pGVM Pointer to the global VM structure.
2411 * @param iPage The current page descriptor table index.
2412 * @param cPages The total number of pages to allocate.
2413 * @param paPages The page descriptor table (input + ouput).
2414 */
2415static uint32_t gmmR0AllocatePagesInBoundMode(PGVM pGVM, uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2416{
2417 for (unsigned iList = 0; iList < RT_ELEMENTS(pGVM->gmm.s.Private.apLists); iList++)
2418 {
2419 PGMMCHUNK pChunk = pGVM->gmm.s.Private.apLists[iList];
2420 while (pChunk)
2421 {
2422 Assert(pChunk->hGVM == pGVM->hSelf);
2423 PGMMCHUNK pNext = pChunk->pFreeNext;
2424 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2425 if (iPage >= cPages)
2426 return iPage;
2427 pChunk = pNext;
2428 }
2429 }
2430 return iPage;
2431}
2432
2433
2434/**
2435 * Checks if we should start picking pages from chunks of other VMs because
2436 * we're getting close to the system memory or reserved limit.
2437 *
2438 * @returns @c true if we should, @c false if we should first try allocate more
2439 * chunks.
2440 */
2441static bool gmmR0ShouldAllocatePagesInOtherChunksBecauseOfLimits(PGVM pGVM)
2442{
2443 /*
2444 * Don't allocate a new chunk if we're
2445 */
2446 uint64_t cPgReserved = pGVM->gmm.s.Stats.Reserved.cBasePages
2447 + pGVM->gmm.s.Stats.Reserved.cFixedPages
2448 - pGVM->gmm.s.Stats.cBalloonedPages
2449 /** @todo what about shared pages? */;
2450 uint64_t cPgAllocated = pGVM->gmm.s.Stats.Allocated.cBasePages
2451 + pGVM->gmm.s.Stats.Allocated.cFixedPages;
2452 uint64_t cPgDelta = cPgReserved - cPgAllocated;
2453 if (cPgDelta < GMM_CHUNK_NUM_PAGES * 4)
2454 return true;
2455 /** @todo make the threshold configurable, also test the code to see if
2456 * this ever kicks in (we might be reserving too much or smth). */
2457
2458 /*
2459 * Check how close we're to the max memory limit and how many fragments
2460 * there are?...
2461 */
2462 /** @todo. */
2463
2464 return false;
2465}
2466
2467
2468/**
2469 * Checks if we should start picking pages from chunks of other VMs because
2470 * there is a lot of free pages around.
2471 *
2472 * @returns @c true if we should, @c false if we should first try allocate more
2473 * chunks.
2474 */
2475static bool gmmR0ShouldAllocatePagesInOtherChunksBecauseOfLotsFree(PGMM pGMM)
2476{
2477 /*
2478 * Setting the limit at 16 chunks (32 MB) at the moment.
2479 */
2480 if (pGMM->PrivateX.cFreePages >= GMM_CHUNK_NUM_PAGES * 16)
2481 return true;
2482 return false;
2483}
2484
2485
2486/**
2487 * Common worker for GMMR0AllocateHandyPages and GMMR0AllocatePages.
2488 *
2489 * @returns VBox status code:
2490 * @retval VINF_SUCCESS on success.
2491 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk or
2492 * gmmR0AllocateMoreChunks is necessary.
2493 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
2494 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
2495 * that is we're trying to allocate more than we've reserved.
2496 *
2497 * @param pGMM Pointer to the GMM instance data.
2498 * @param pGVM Pointer to the VM.
2499 * @param cPages The number of pages to allocate.
2500 * @param paPages Pointer to the page descriptors. See GMMPAGEDESC for
2501 * details on what is expected on input.
2502 * @param enmAccount The account to charge.
2503 *
2504 * @remarks Call takes the giant GMM lock.
2505 */
2506static int gmmR0AllocatePagesNew(PGMM pGMM, PGVM pGVM, uint32_t cPages, PGMMPAGEDESC paPages, GMMACCOUNT enmAccount)
2507{
2508 Assert(pGMM->hMtxOwner == RTThreadNativeSelf());
2509
2510 /*
2511 * Check allocation limits.
2512 */
2513 if (RT_UNLIKELY(pGMM->cAllocatedPages + cPages > pGMM->cMaxPages))
2514 return VERR_GMM_HIT_GLOBAL_LIMIT;
2515
2516 switch (enmAccount)
2517 {
2518 case GMMACCOUNT_BASE:
2519 if (RT_UNLIKELY( pGVM->gmm.s.Stats.Allocated.cBasePages + pGVM->gmm.s.Stats.cBalloonedPages + cPages
2520 > pGVM->gmm.s.Stats.Reserved.cBasePages))
2521 {
2522 Log(("gmmR0AllocatePages:Base: Reserved=%#llx Allocated+Ballooned+Requested=%#llx+%#llx+%#x!\n",
2523 pGVM->gmm.s.Stats.Reserved.cBasePages, pGVM->gmm.s.Stats.Allocated.cBasePages,
2524 pGVM->gmm.s.Stats.cBalloonedPages, cPages));
2525 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
2526 }
2527 break;
2528 case GMMACCOUNT_SHADOW:
2529 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cShadowPages + cPages > pGVM->gmm.s.Stats.Reserved.cShadowPages))
2530 {
2531 Log(("gmmR0AllocatePages:Shadow: Reserved=%#x Allocated+Requested=%#x+%#x!\n",
2532 pGVM->gmm.s.Stats.Reserved.cShadowPages, pGVM->gmm.s.Stats.Allocated.cShadowPages, cPages));
2533 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
2534 }
2535 break;
2536 case GMMACCOUNT_FIXED:
2537 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cFixedPages + cPages > pGVM->gmm.s.Stats.Reserved.cFixedPages))
2538 {
2539 Log(("gmmR0AllocatePages:Fixed: Reserved=%#x Allocated+Requested=%#x+%#x!\n",
2540 pGVM->gmm.s.Stats.Reserved.cFixedPages, pGVM->gmm.s.Stats.Allocated.cFixedPages, cPages));
2541 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
2542 }
2543 break;
2544 default:
2545 AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
2546 }
2547
2548 /*
2549 * If we're in legacy memory mode, it's easy to figure if we have
2550 * sufficient number of pages up-front.
2551 */
2552 if ( pGMM->fLegacyAllocationMode
2553 && pGVM->gmm.s.Private.cFreePages < cPages)
2554 {
2555 Assert(pGMM->fBoundMemoryMode);
2556 return VERR_GMM_SEED_ME;
2557 }
2558
2559 /*
2560 * Update the accounts before we proceed because we might be leaving the
2561 * protection of the global mutex and thus run the risk of permitting
2562 * too much memory to be allocated.
2563 */
2564 switch (enmAccount)
2565 {
2566 case GMMACCOUNT_BASE: pGVM->gmm.s.Stats.Allocated.cBasePages += cPages; break;
2567 case GMMACCOUNT_SHADOW: pGVM->gmm.s.Stats.Allocated.cShadowPages += cPages; break;
2568 case GMMACCOUNT_FIXED: pGVM->gmm.s.Stats.Allocated.cFixedPages += cPages; break;
2569 default: AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
2570 }
2571 pGVM->gmm.s.Stats.cPrivatePages += cPages;
2572 pGMM->cAllocatedPages += cPages;
2573
2574 /*
2575 * Part two of it's-easy-in-legacy-memory-mode.
2576 */
2577 uint32_t iPage = 0;
2578 if (pGMM->fLegacyAllocationMode)
2579 {
2580 iPage = gmmR0AllocatePagesInBoundMode(pGVM, iPage, cPages, paPages);
2581 AssertReleaseReturn(iPage == cPages, VERR_GMM_ALLOC_PAGES_IPE);
2582 return VINF_SUCCESS;
2583 }
2584
2585 /*
2586 * Bound mode is also relatively straightforward.
2587 */
2588 int rc = VINF_SUCCESS;
2589 if (pGMM->fBoundMemoryMode)
2590 {
2591 iPage = gmmR0AllocatePagesInBoundMode(pGVM, iPage, cPages, paPages);
2592 if (iPage < cPages)
2593 do
2594 rc = gmmR0AllocateChunkNew(pGMM, pGVM, &pGVM->gmm.s.Private, cPages, paPages, &iPage);
2595 while (iPage < cPages && RT_SUCCESS(rc));
2596 }
2597 /*
2598 * Shared mode is trickier as we should try archive the same locality as
2599 * in bound mode, but smartly make use of non-full chunks allocated by
2600 * other VMs if we're low on memory.
2601 */
2602 else
2603 {
2604 /* Pick the most optimal pages first. */
2605 iPage = gmmR0AllocatePagesAssociatedWithVM(pGMM, pGVM, &pGMM->PrivateX, iPage, cPages, paPages);
2606 if (iPage < cPages)
2607 {
2608 /* Maybe we should try getting pages from chunks "belonging" to
2609 other VMs before allocating more chunks? */
2610 bool fTriedOnSameAlready = false;
2611 if (gmmR0ShouldAllocatePagesInOtherChunksBecauseOfLimits(pGVM))
2612 {
2613 iPage = gmmR0AllocatePagesFromSameNode(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2614 fTriedOnSameAlready = true;
2615 }
2616
2617 /* Allocate memory from empty chunks. */
2618 if (iPage < cPages)
2619 iPage = gmmR0AllocatePagesFromEmptyChunksOnSameNode(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2620
2621 /* Grab empty shared chunks. */
2622 if (iPage < cPages)
2623 iPage = gmmR0AllocatePagesFromEmptyChunksOnSameNode(&pGMM->Shared, pGVM, iPage, cPages, paPages);
2624
2625 /* If there is a lof of free pages spread around, try not waste
2626 system memory on more chunks. (Should trigger defragmentation.) */
2627 if ( !fTriedOnSameAlready
2628 && gmmR0ShouldAllocatePagesInOtherChunksBecauseOfLotsFree(pGMM))
2629 {
2630 iPage = gmmR0AllocatePagesFromSameNode(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2631 if (iPage < cPages)
2632 iPage = gmmR0AllocatePagesIndiscriminately(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2633 }
2634
2635 /*
2636 * Ok, try allocate new chunks.
2637 */
2638 if (iPage < cPages)
2639 {
2640 do
2641 rc = gmmR0AllocateChunkNew(pGMM, pGVM, &pGMM->PrivateX, cPages, paPages, &iPage);
2642 while (iPage < cPages && RT_SUCCESS(rc));
2643
2644 /* If the host is out of memory, take whatever we can get. */
2645 if ( (rc == VERR_NO_MEMORY || rc == VERR_NO_PHYS_MEMORY)
2646 && pGMM->PrivateX.cFreePages + pGMM->Shared.cFreePages >= cPages - iPage)
2647 {
2648 iPage = gmmR0AllocatePagesIndiscriminately(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2649 if (iPage < cPages)
2650 iPage = gmmR0AllocatePagesIndiscriminately(&pGMM->Shared, pGVM, iPage, cPages, paPages);
2651 AssertRelease(iPage == cPages);
2652 rc = VINF_SUCCESS;
2653 }
2654 }
2655 }
2656 }
2657
2658 /*
2659 * Clean up on failure. Since this is bound to be a low-memory condition
2660 * we will give back any empty chunks that might be hanging around.
2661 */
2662 if (RT_FAILURE(rc))
2663 {
2664 /* Update the statistics. */
2665 pGVM->gmm.s.Stats.cPrivatePages -= cPages;
2666 pGMM->cAllocatedPages -= cPages - iPage;
2667 switch (enmAccount)
2668 {
2669 case GMMACCOUNT_BASE: pGVM->gmm.s.Stats.Allocated.cBasePages -= cPages; break;
2670 case GMMACCOUNT_SHADOW: pGVM->gmm.s.Stats.Allocated.cShadowPages -= cPages; break;
2671 case GMMACCOUNT_FIXED: pGVM->gmm.s.Stats.Allocated.cFixedPages -= cPages; break;
2672 default: AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
2673 }
2674
2675 /* Release the pages. */
2676 while (iPage-- > 0)
2677 {
2678 uint32_t idPage = paPages[iPage].idPage;
2679 PGMMPAGE pPage = gmmR0GetPage(pGMM, idPage);
2680 if (RT_LIKELY(pPage))
2681 {
2682 Assert(GMM_PAGE_IS_PRIVATE(pPage));
2683 Assert(pPage->Private.hGVM == pGVM->hSelf);
2684 gmmR0FreePrivatePage(pGMM, pGVM, idPage, pPage);
2685 }
2686 else
2687 AssertMsgFailed(("idPage=%#x\n", idPage));
2688
2689 paPages[iPage].idPage = NIL_GMM_PAGEID;
2690 paPages[iPage].idSharedPage = NIL_GMM_PAGEID;
2691 paPages[iPage].HCPhysGCPhys = NIL_RTHCPHYS;
2692 }
2693
2694 /* Free empty chunks. */
2695 /** @todo */
2696
2697 /* return the fail status on failure */
2698 return rc;
2699 }
2700 return VINF_SUCCESS;
2701}
2702
2703
2704/**
2705 * Updates the previous allocations and allocates more pages.
2706 *
2707 * The handy pages are always taken from the 'base' memory account.
2708 * The allocated pages are not cleared and will contains random garbage.
2709 *
2710 * @returns VBox status code:
2711 * @retval VINF_SUCCESS on success.
2712 * @retval VERR_NOT_OWNER if the caller is not an EMT.
2713 * @retval VERR_GMM_PAGE_NOT_FOUND if one of the pages to update wasn't found.
2714 * @retval VERR_GMM_PAGE_NOT_PRIVATE if one of the pages to update wasn't a
2715 * private page.
2716 * @retval VERR_GMM_PAGE_NOT_SHARED if one of the pages to update wasn't a
2717 * shared page.
2718 * @retval VERR_GMM_NOT_PAGE_OWNER if one of the pages to be updated wasn't
2719 * owned by the VM.
2720 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk is necessary.
2721 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
2722 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
2723 * that is we're trying to allocate more than we've reserved.
2724 *
2725 * @param pVM The cross context VM structure.
2726 * @param idCpu The VCPU id.
2727 * @param cPagesToUpdate The number of pages to update (starting from the head).
2728 * @param cPagesToAlloc The number of pages to allocate (starting from the head).
2729 * @param paPages The array of page descriptors.
2730 * See GMMPAGEDESC for details on what is expected on input.
2731 * @thread EMT.
2732 */
2733GMMR0DECL(int) GMMR0AllocateHandyPages(PVM pVM, VMCPUID idCpu, uint32_t cPagesToUpdate, uint32_t cPagesToAlloc, PGMMPAGEDESC paPages)
2734{
2735 LogFlow(("GMMR0AllocateHandyPages: pVM=%p cPagesToUpdate=%#x cPagesToAlloc=%#x paPages=%p\n",
2736 pVM, cPagesToUpdate, cPagesToAlloc, paPages));
2737
2738 /*
2739 * Validate, get basics and take the semaphore.
2740 * (This is a relatively busy path, so make predictions where possible.)
2741 */
2742 PGMM pGMM;
2743 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
2744 PGVM pGVM;
2745 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
2746 if (RT_FAILURE(rc))
2747 return rc;
2748
2749 AssertPtrReturn(paPages, VERR_INVALID_PARAMETER);
2750 AssertMsgReturn( (cPagesToUpdate && cPagesToUpdate < 1024)
2751 || (cPagesToAlloc && cPagesToAlloc < 1024),
2752 ("cPagesToUpdate=%#x cPagesToAlloc=%#x\n", cPagesToUpdate, cPagesToAlloc),
2753 VERR_INVALID_PARAMETER);
2754
2755 unsigned iPage = 0;
2756 for (; iPage < cPagesToUpdate; iPage++)
2757 {
2758 AssertMsgReturn( ( paPages[iPage].HCPhysGCPhys <= GMM_GCPHYS_LAST
2759 && !(paPages[iPage].HCPhysGCPhys & PAGE_OFFSET_MASK))
2760 || paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS
2761 || paPages[iPage].HCPhysGCPhys == GMM_GCPHYS_UNSHAREABLE,
2762 ("#%#x: %RHp\n", iPage, paPages[iPage].HCPhysGCPhys),
2763 VERR_INVALID_PARAMETER);
2764 AssertMsgReturn( paPages[iPage].idPage <= GMM_PAGEID_LAST
2765 /*|| paPages[iPage].idPage == NIL_GMM_PAGEID*/,
2766 ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
2767 AssertMsgReturn( paPages[iPage].idPage <= GMM_PAGEID_LAST
2768 /*|| paPages[iPage].idSharedPage == NIL_GMM_PAGEID*/,
2769 ("#%#x: %#x\n", iPage, paPages[iPage].idSharedPage), VERR_INVALID_PARAMETER);
2770 }
2771
2772 for (; iPage < cPagesToAlloc; iPage++)
2773 {
2774 AssertMsgReturn(paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS, ("#%#x: %RHp\n", iPage, paPages[iPage].HCPhysGCPhys), VERR_INVALID_PARAMETER);
2775 AssertMsgReturn(paPages[iPage].idPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
2776 AssertMsgReturn(paPages[iPage].idSharedPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idSharedPage), VERR_INVALID_PARAMETER);
2777 }
2778
2779 gmmR0MutexAcquire(pGMM);
2780 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
2781 {
2782 /* No allocations before the initial reservation has been made! */
2783 if (RT_LIKELY( pGVM->gmm.s.Stats.Reserved.cBasePages
2784 && pGVM->gmm.s.Stats.Reserved.cFixedPages
2785 && pGVM->gmm.s.Stats.Reserved.cShadowPages))
2786 {
2787 /*
2788 * Perform the updates.
2789 * Stop on the first error.
2790 */
2791 for (iPage = 0; iPage < cPagesToUpdate; iPage++)
2792 {
2793 if (paPages[iPage].idPage != NIL_GMM_PAGEID)
2794 {
2795 PGMMPAGE pPage = gmmR0GetPage(pGMM, paPages[iPage].idPage);
2796 if (RT_LIKELY(pPage))
2797 {
2798 if (RT_LIKELY(GMM_PAGE_IS_PRIVATE(pPage)))
2799 {
2800 if (RT_LIKELY(pPage->Private.hGVM == pGVM->hSelf))
2801 {
2802 AssertCompile(NIL_RTHCPHYS > GMM_GCPHYS_LAST && GMM_GCPHYS_UNSHAREABLE > GMM_GCPHYS_LAST);
2803 if (RT_LIKELY(paPages[iPage].HCPhysGCPhys <= GMM_GCPHYS_LAST))
2804 pPage->Private.pfn = paPages[iPage].HCPhysGCPhys >> PAGE_SHIFT;
2805 else if (paPages[iPage].HCPhysGCPhys == GMM_GCPHYS_UNSHAREABLE)
2806 pPage->Private.pfn = GMM_PAGE_PFN_UNSHAREABLE;
2807 /* else: NIL_RTHCPHYS nothing */
2808
2809 paPages[iPage].idPage = NIL_GMM_PAGEID;
2810 paPages[iPage].HCPhysGCPhys = NIL_RTHCPHYS;
2811 }
2812 else
2813 {
2814 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not owner! hGVM=%#x hSelf=%#x\n",
2815 iPage, paPages[iPage].idPage, pPage->Private.hGVM, pGVM->hSelf));
2816 rc = VERR_GMM_NOT_PAGE_OWNER;
2817 break;
2818 }
2819 }
2820 else
2821 {
2822 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not private! %.*Rhxs (type %d)\n", iPage, paPages[iPage].idPage, sizeof(*pPage), pPage, pPage->Common.u2State));
2823 rc = VERR_GMM_PAGE_NOT_PRIVATE;
2824 break;
2825 }
2826 }
2827 else
2828 {
2829 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not found! (private)\n", iPage, paPages[iPage].idPage));
2830 rc = VERR_GMM_PAGE_NOT_FOUND;
2831 break;
2832 }
2833 }
2834
2835 if (paPages[iPage].idSharedPage != NIL_GMM_PAGEID)
2836 {
2837 PGMMPAGE pPage = gmmR0GetPage(pGMM, paPages[iPage].idSharedPage);
2838 if (RT_LIKELY(pPage))
2839 {
2840 if (RT_LIKELY(GMM_PAGE_IS_SHARED(pPage)))
2841 {
2842 AssertCompile(NIL_RTHCPHYS > GMM_GCPHYS_LAST && GMM_GCPHYS_UNSHAREABLE > GMM_GCPHYS_LAST);
2843 Assert(pPage->Shared.cRefs);
2844 Assert(pGVM->gmm.s.Stats.cSharedPages);
2845 Assert(pGVM->gmm.s.Stats.Allocated.cBasePages);
2846
2847 Log(("GMMR0AllocateHandyPages: free shared page %x cRefs=%d\n", paPages[iPage].idSharedPage, pPage->Shared.cRefs));
2848 pGVM->gmm.s.Stats.cSharedPages--;
2849 pGVM->gmm.s.Stats.Allocated.cBasePages--;
2850 if (!--pPage->Shared.cRefs)
2851 gmmR0FreeSharedPage(pGMM, pGVM, paPages[iPage].idSharedPage, pPage);
2852 else
2853 {
2854 Assert(pGMM->cDuplicatePages);
2855 pGMM->cDuplicatePages--;
2856 }
2857
2858 paPages[iPage].idSharedPage = NIL_GMM_PAGEID;
2859 }
2860 else
2861 {
2862 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not shared!\n", iPage, paPages[iPage].idSharedPage));
2863 rc = VERR_GMM_PAGE_NOT_SHARED;
2864 break;
2865 }
2866 }
2867 else
2868 {
2869 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not found! (shared)\n", iPage, paPages[iPage].idSharedPage));
2870 rc = VERR_GMM_PAGE_NOT_FOUND;
2871 break;
2872 }
2873 }
2874 } /* for each page to update */
2875
2876 if (RT_SUCCESS(rc) && cPagesToAlloc > 0)
2877 {
2878#if defined(VBOX_STRICT) && 0 /** @todo re-test this later. Appeared to be a PGM init bug. */
2879 for (iPage = 0; iPage < cPagesToAlloc; iPage++)
2880 {
2881 Assert(paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS);
2882 Assert(paPages[iPage].idPage == NIL_GMM_PAGEID);
2883 Assert(paPages[iPage].idSharedPage == NIL_GMM_PAGEID);
2884 }
2885#endif
2886
2887 /*
2888 * Join paths with GMMR0AllocatePages for the allocation.
2889 * Note! gmmR0AllocateMoreChunks may leave the protection of the mutex!
2890 */
2891 rc = gmmR0AllocatePagesNew(pGMM, pGVM, cPagesToAlloc, paPages, GMMACCOUNT_BASE);
2892 }
2893 }
2894 else
2895 rc = VERR_WRONG_ORDER;
2896 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
2897 }
2898 else
2899 rc = VERR_GMM_IS_NOT_SANE;
2900 gmmR0MutexRelease(pGMM);
2901 LogFlow(("GMMR0AllocateHandyPages: returns %Rrc\n", rc));
2902 return rc;
2903}
2904
2905
2906/**
2907 * Allocate one or more pages.
2908 *
2909 * This is typically used for ROMs and MMIO2 (VRAM) during VM creation.
2910 * The allocated pages are not cleared and will contain random garbage.
2911 *
2912 * @returns VBox status code:
2913 * @retval VINF_SUCCESS on success.
2914 * @retval VERR_NOT_OWNER if the caller is not an EMT.
2915 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk is necessary.
2916 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
2917 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
2918 * that is we're trying to allocate more than we've reserved.
2919 *
2920 * @param pVM The cross context VM structure.
2921 * @param idCpu The VCPU id.
2922 * @param cPages The number of pages to allocate.
2923 * @param paPages Pointer to the page descriptors.
2924 * See GMMPAGEDESC for details on what is expected on
2925 * input.
2926 * @param enmAccount The account to charge.
2927 *
2928 * @thread EMT.
2929 */
2930GMMR0DECL(int) GMMR0AllocatePages(PVM pVM, VMCPUID idCpu, uint32_t cPages, PGMMPAGEDESC paPages, GMMACCOUNT enmAccount)
2931{
2932 LogFlow(("GMMR0AllocatePages: pVM=%p cPages=%#x paPages=%p enmAccount=%d\n", pVM, cPages, paPages, enmAccount));
2933
2934 /*
2935 * Validate, get basics and take the semaphore.
2936 */
2937 PGMM pGMM;
2938 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
2939 PGVM pGVM;
2940 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
2941 if (RT_FAILURE(rc))
2942 return rc;
2943
2944 AssertPtrReturn(paPages, VERR_INVALID_PARAMETER);
2945 AssertMsgReturn(enmAccount > GMMACCOUNT_INVALID && enmAccount < GMMACCOUNT_END, ("%d\n", enmAccount), VERR_INVALID_PARAMETER);
2946 AssertMsgReturn(cPages > 0 && cPages < RT_BIT(32 - PAGE_SHIFT), ("%#x\n", cPages), VERR_INVALID_PARAMETER);
2947
2948 for (unsigned iPage = 0; iPage < cPages; iPage++)
2949 {
2950 AssertMsgReturn( paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS
2951 || paPages[iPage].HCPhysGCPhys == GMM_GCPHYS_UNSHAREABLE
2952 || ( enmAccount == GMMACCOUNT_BASE
2953 && paPages[iPage].HCPhysGCPhys <= GMM_GCPHYS_LAST
2954 && !(paPages[iPage].HCPhysGCPhys & PAGE_OFFSET_MASK)),
2955 ("#%#x: %RHp enmAccount=%d\n", iPage, paPages[iPage].HCPhysGCPhys, enmAccount),
2956 VERR_INVALID_PARAMETER);
2957 AssertMsgReturn(paPages[iPage].idPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
2958 AssertMsgReturn(paPages[iPage].idSharedPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idSharedPage), VERR_INVALID_PARAMETER);
2959 }
2960
2961 gmmR0MutexAcquire(pGMM);
2962 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
2963 {
2964
2965 /* No allocations before the initial reservation has been made! */
2966 if (RT_LIKELY( pGVM->gmm.s.Stats.Reserved.cBasePages
2967 && pGVM->gmm.s.Stats.Reserved.cFixedPages
2968 && pGVM->gmm.s.Stats.Reserved.cShadowPages))
2969 rc = gmmR0AllocatePagesNew(pGMM, pGVM, cPages, paPages, enmAccount);
2970 else
2971 rc = VERR_WRONG_ORDER;
2972 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
2973 }
2974 else
2975 rc = VERR_GMM_IS_NOT_SANE;
2976 gmmR0MutexRelease(pGMM);
2977 LogFlow(("GMMR0AllocatePages: returns %Rrc\n", rc));
2978 return rc;
2979}
2980
2981
2982/**
2983 * VMMR0 request wrapper for GMMR0AllocatePages.
2984 *
2985 * @returns see GMMR0AllocatePages.
2986 * @param pVM The cross context VM structure.
2987 * @param idCpu The VCPU id.
2988 * @param pReq Pointer to the request packet.
2989 */
2990GMMR0DECL(int) GMMR0AllocatePagesReq(PVM pVM, VMCPUID idCpu, PGMMALLOCATEPAGESREQ pReq)
2991{
2992 /*
2993 * Validate input and pass it on.
2994 */
2995 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
2996 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
2997 AssertMsgReturn(pReq->Hdr.cbReq >= RT_UOFFSETOF(GMMALLOCATEPAGESREQ, aPages[0]),
2998 ("%#x < %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF(GMMALLOCATEPAGESREQ, aPages[0])),
2999 VERR_INVALID_PARAMETER);
3000 AssertMsgReturn(pReq->Hdr.cbReq == RT_UOFFSETOF(GMMALLOCATEPAGESREQ, aPages[pReq->cPages]),
3001 ("%#x != %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF(GMMALLOCATEPAGESREQ, aPages[pReq->cPages])),
3002 VERR_INVALID_PARAMETER);
3003
3004 return GMMR0AllocatePages(pVM, idCpu, pReq->cPages, &pReq->aPages[0], pReq->enmAccount);
3005}
3006
3007
3008/**
3009 * Allocate a large page to represent guest RAM
3010 *
3011 * The allocated pages are not cleared and will contains random garbage.
3012 *
3013 * @returns VBox status code:
3014 * @retval VINF_SUCCESS on success.
3015 * @retval VERR_NOT_OWNER if the caller is not an EMT.
3016 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk is necessary.
3017 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
3018 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
3019 * that is we're trying to allocate more than we've reserved.
3020 * @returns see GMMR0AllocatePages.
3021 *
3022 * @param pVM The cross context VM structure.
3023 * @param idCpu The VCPU id.
3024 * @param cbPage Large page size.
3025 * @param pIdPage Where to return the GMM page ID of the page.
3026 * @param pHCPhys Where to return the host physical address of the page.
3027 */
3028GMMR0DECL(int) GMMR0AllocateLargePage(PVM pVM, VMCPUID idCpu, uint32_t cbPage, uint32_t *pIdPage, RTHCPHYS *pHCPhys)
3029{
3030 LogFlow(("GMMR0AllocateLargePage: pVM=%p cbPage=%x\n", pVM, cbPage));
3031
3032 AssertReturn(cbPage == GMM_CHUNK_SIZE, VERR_INVALID_PARAMETER);
3033 AssertPtrReturn(pIdPage, VERR_INVALID_PARAMETER);
3034 AssertPtrReturn(pHCPhys, VERR_INVALID_PARAMETER);
3035
3036 /*
3037 * Validate, get basics and take the semaphore.
3038 */
3039 PGMM pGMM;
3040 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3041 PGVM pGVM;
3042 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
3043 if (RT_FAILURE(rc))
3044 return rc;
3045
3046 /* Not supported in legacy mode where we allocate the memory in ring 3 and lock it in ring 0. */
3047 if (pGMM->fLegacyAllocationMode)
3048 return VERR_NOT_SUPPORTED;
3049
3050 *pHCPhys = NIL_RTHCPHYS;
3051 *pIdPage = NIL_GMM_PAGEID;
3052
3053 gmmR0MutexAcquire(pGMM);
3054 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3055 {
3056 const unsigned cPages = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
3057 if (RT_UNLIKELY( pGVM->gmm.s.Stats.Allocated.cBasePages + pGVM->gmm.s.Stats.cBalloonedPages + cPages
3058 > pGVM->gmm.s.Stats.Reserved.cBasePages))
3059 {
3060 Log(("GMMR0AllocateLargePage: Reserved=%#llx Allocated+Requested=%#llx+%#x!\n",
3061 pGVM->gmm.s.Stats.Reserved.cBasePages, pGVM->gmm.s.Stats.Allocated.cBasePages, cPages));
3062 gmmR0MutexRelease(pGMM);
3063 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
3064 }
3065
3066 /*
3067 * Allocate a new large page chunk.
3068 *
3069 * Note! We leave the giant GMM lock temporarily as the allocation might
3070 * take a long time. gmmR0RegisterChunk will retake it (ugly).
3071 */
3072 AssertCompile(GMM_CHUNK_SIZE == _2M);
3073 gmmR0MutexRelease(pGMM);
3074
3075 RTR0MEMOBJ hMemObj;
3076 rc = RTR0MemObjAllocPhysEx(&hMemObj, GMM_CHUNK_SIZE, NIL_RTHCPHYS, GMM_CHUNK_SIZE);
3077 if (RT_SUCCESS(rc))
3078 {
3079 PGMMCHUNKFREESET pSet = pGMM->fBoundMemoryMode ? &pGVM->gmm.s.Private : &pGMM->PrivateX;
3080 PGMMCHUNK pChunk;
3081 rc = gmmR0RegisterChunk(pGMM, pSet, hMemObj, pGVM->hSelf, GMM_CHUNK_FLAGS_LARGE_PAGE, &pChunk);
3082 if (RT_SUCCESS(rc))
3083 {
3084 /*
3085 * Allocate all the pages in the chunk.
3086 */
3087 /* Unlink the new chunk from the free list. */
3088 gmmR0UnlinkChunk(pChunk);
3089
3090 /** @todo rewrite this to skip the looping. */
3091 /* Allocate all pages. */
3092 GMMPAGEDESC PageDesc;
3093 gmmR0AllocatePage(pChunk, pGVM->hSelf, &PageDesc);
3094
3095 /* Return the first page as we'll use the whole chunk as one big page. */
3096 *pIdPage = PageDesc.idPage;
3097 *pHCPhys = PageDesc.HCPhysGCPhys;
3098
3099 for (unsigned i = 1; i < cPages; i++)
3100 gmmR0AllocatePage(pChunk, pGVM->hSelf, &PageDesc);
3101
3102 /* Update accounting. */
3103 pGVM->gmm.s.Stats.Allocated.cBasePages += cPages;
3104 pGVM->gmm.s.Stats.cPrivatePages += cPages;
3105 pGMM->cAllocatedPages += cPages;
3106
3107 gmmR0LinkChunk(pChunk, pSet);
3108 gmmR0MutexRelease(pGMM);
3109 }
3110 else
3111 RTR0MemObjFree(hMemObj, false /* fFreeMappings */);
3112 }
3113 }
3114 else
3115 {
3116 gmmR0MutexRelease(pGMM);
3117 rc = VERR_GMM_IS_NOT_SANE;
3118 }
3119
3120 LogFlow(("GMMR0AllocateLargePage: returns %Rrc\n", rc));
3121 return rc;
3122}
3123
3124
3125/**
3126 * Free a large page.
3127 *
3128 * @returns VBox status code:
3129 * @param pVM The cross context VM structure.
3130 * @param idCpu The VCPU id.
3131 * @param idPage The large page id.
3132 */
3133GMMR0DECL(int) GMMR0FreeLargePage(PVM pVM, VMCPUID idCpu, uint32_t idPage)
3134{
3135 LogFlow(("GMMR0FreeLargePage: pVM=%p idPage=%x\n", pVM, idPage));
3136
3137 /*
3138 * Validate, get basics and take the semaphore.
3139 */
3140 PGMM pGMM;
3141 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3142 PGVM pGVM;
3143 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
3144 if (RT_FAILURE(rc))
3145 return rc;
3146
3147 /* Not supported in legacy mode where we allocate the memory in ring 3 and lock it in ring 0. */
3148 if (pGMM->fLegacyAllocationMode)
3149 return VERR_NOT_SUPPORTED;
3150
3151 gmmR0MutexAcquire(pGMM);
3152 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3153 {
3154 const unsigned cPages = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
3155
3156 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cBasePages < cPages))
3157 {
3158 Log(("GMMR0FreeLargePage: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cBasePages, cPages));
3159 gmmR0MutexRelease(pGMM);
3160 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3161 }
3162
3163 PGMMPAGE pPage = gmmR0GetPage(pGMM, idPage);
3164 if (RT_LIKELY( pPage
3165 && GMM_PAGE_IS_PRIVATE(pPage)))
3166 {
3167 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
3168 Assert(pChunk);
3169 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
3170 Assert(pChunk->cPrivate > 0);
3171
3172 /* Release the memory immediately. */
3173 gmmR0FreeChunk(pGMM, NULL, pChunk, false /*fRelaxedSem*/); /** @todo this can be relaxed too! */
3174
3175 /* Update accounting. */
3176 pGVM->gmm.s.Stats.Allocated.cBasePages -= cPages;
3177 pGVM->gmm.s.Stats.cPrivatePages -= cPages;
3178 pGMM->cAllocatedPages -= cPages;
3179 }
3180 else
3181 rc = VERR_GMM_PAGE_NOT_FOUND;
3182 }
3183 else
3184 rc = VERR_GMM_IS_NOT_SANE;
3185
3186 gmmR0MutexRelease(pGMM);
3187 LogFlow(("GMMR0FreeLargePage: returns %Rrc\n", rc));
3188 return rc;
3189}
3190
3191
3192/**
3193 * VMMR0 request wrapper for GMMR0FreeLargePage.
3194 *
3195 * @returns see GMMR0FreeLargePage.
3196 * @param pVM The cross context VM structure.
3197 * @param idCpu The VCPU id.
3198 * @param pReq Pointer to the request packet.
3199 */
3200GMMR0DECL(int) GMMR0FreeLargePageReq(PVM pVM, VMCPUID idCpu, PGMMFREELARGEPAGEREQ pReq)
3201{
3202 /*
3203 * Validate input and pass it on.
3204 */
3205 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
3206 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3207 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMFREEPAGESREQ),
3208 ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(GMMFREEPAGESREQ)),
3209 VERR_INVALID_PARAMETER);
3210
3211 return GMMR0FreeLargePage(pVM, idCpu, pReq->idPage);
3212}
3213
3214
3215/**
3216 * Frees a chunk, giving it back to the host OS.
3217 *
3218 * @param pGMM Pointer to the GMM instance.
3219 * @param pGVM This is set when called from GMMR0CleanupVM so we can
3220 * unmap and free the chunk in one go.
3221 * @param pChunk The chunk to free.
3222 * @param fRelaxedSem Whether we can release the semaphore while doing the
3223 * freeing (@c true) or not.
3224 */
3225static bool gmmR0FreeChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem)
3226{
3227 Assert(pChunk->Core.Key != NIL_GMM_CHUNKID);
3228
3229 GMMR0CHUNKMTXSTATE MtxState;
3230 gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk, GMMR0CHUNK_MTX_KEEP_GIANT);
3231
3232 /*
3233 * Cleanup hack! Unmap the chunk from the callers address space.
3234 * This shouldn't happen, so screw lock contention...
3235 */
3236 if ( pChunk->cMappingsX
3237 && !pGMM->fLegacyAllocationMode
3238 && pGVM)
3239 gmmR0UnmapChunkLocked(pGMM, pGVM, pChunk);
3240
3241 /*
3242 * If there are current mappings of the chunk, then request the
3243 * VMs to unmap them. Reposition the chunk in the free list so
3244 * it won't be a likely candidate for allocations.
3245 */
3246 if (pChunk->cMappingsX)
3247 {
3248 /** @todo R0 -> VM request */
3249 /* The chunk can be mapped by more than one VM if fBoundMemoryMode is false! */
3250 Log(("gmmR0FreeChunk: chunk still has %d mappings; don't free!\n", pChunk->cMappingsX));
3251 gmmR0ChunkMutexRelease(&MtxState, pChunk);
3252 return false;
3253 }
3254
3255
3256 /*
3257 * Save and trash the handle.
3258 */
3259 RTR0MEMOBJ const hMemObj = pChunk->hMemObj;
3260 pChunk->hMemObj = NIL_RTR0MEMOBJ;
3261
3262 /*
3263 * Unlink it from everywhere.
3264 */
3265 gmmR0UnlinkChunk(pChunk);
3266
3267 RTListNodeRemove(&pChunk->ListNode);
3268
3269 PAVLU32NODECORE pCore = RTAvlU32Remove(&pGMM->pChunks, pChunk->Core.Key);
3270 Assert(pCore == &pChunk->Core); NOREF(pCore);
3271
3272 PGMMCHUNKTLBE pTlbe = &pGMM->ChunkTLB.aEntries[GMM_CHUNKTLB_IDX(pChunk->Core.Key)];
3273 if (pTlbe->pChunk == pChunk)
3274 {
3275 pTlbe->idChunk = NIL_GMM_CHUNKID;
3276 pTlbe->pChunk = NULL;
3277 }
3278
3279 Assert(pGMM->cChunks > 0);
3280 pGMM->cChunks--;
3281
3282 /*
3283 * Free the Chunk ID before dropping the locks and freeing the rest.
3284 */
3285 gmmR0FreeChunkId(pGMM, pChunk->Core.Key);
3286 pChunk->Core.Key = NIL_GMM_CHUNKID;
3287
3288 pGMM->cFreedChunks++;
3289
3290 gmmR0ChunkMutexRelease(&MtxState, NULL);
3291 if (fRelaxedSem)
3292 gmmR0MutexRelease(pGMM);
3293
3294 RTMemFree(pChunk->paMappingsX);
3295 pChunk->paMappingsX = NULL;
3296
3297 RTMemFree(pChunk);
3298
3299 int rc = RTR0MemObjFree(hMemObj, false /* fFreeMappings */);
3300 AssertLogRelRC(rc);
3301
3302 if (fRelaxedSem)
3303 gmmR0MutexAcquire(pGMM);
3304 return fRelaxedSem;
3305}
3306
3307
3308/**
3309 * Free page worker.
3310 *
3311 * The caller does all the statistic decrementing, we do all the incrementing.
3312 *
3313 * @param pGMM Pointer to the GMM instance data.
3314 * @param pGVM Pointer to the GVM instance.
3315 * @param pChunk Pointer to the chunk this page belongs to.
3316 * @param idPage The Page ID.
3317 * @param pPage Pointer to the page.
3318 */
3319static void gmmR0FreePageWorker(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, uint32_t idPage, PGMMPAGE pPage)
3320{
3321 Log3(("F pPage=%p iPage=%#x/%#x u2State=%d iFreeHead=%#x\n",
3322 pPage, pPage - &pChunk->aPages[0], idPage, pPage->Common.u2State, pChunk->iFreeHead)); NOREF(idPage);
3323
3324 /*
3325 * Put the page on the free list.
3326 */
3327 pPage->u = 0;
3328 pPage->Free.u2State = GMM_PAGE_STATE_FREE;
3329 Assert(pChunk->iFreeHead < RT_ELEMENTS(pChunk->aPages) || pChunk->iFreeHead == UINT16_MAX);
3330 pPage->Free.iNext = pChunk->iFreeHead;
3331 pChunk->iFreeHead = pPage - &pChunk->aPages[0];
3332
3333 /*
3334 * Update statistics (the cShared/cPrivate stats are up to date already),
3335 * and relink the chunk if necessary.
3336 */
3337 unsigned const cFree = pChunk->cFree;
3338 if ( !cFree
3339 || gmmR0SelectFreeSetList(cFree) != gmmR0SelectFreeSetList(cFree + 1))
3340 {
3341 gmmR0UnlinkChunk(pChunk);
3342 pChunk->cFree++;
3343 gmmR0SelectSetAndLinkChunk(pGMM, pGVM, pChunk);
3344 }
3345 else
3346 {
3347 pChunk->cFree = cFree + 1;
3348 pChunk->pSet->cFreePages++;
3349 }
3350
3351 /*
3352 * If the chunk becomes empty, consider giving memory back to the host OS.
3353 *
3354 * The current strategy is to try give it back if there are other chunks
3355 * in this free list, meaning if there are at least 240 free pages in this
3356 * category. Note that since there are probably mappings of the chunk,
3357 * it won't be freed up instantly, which probably screws up this logic
3358 * a bit...
3359 */
3360 /** @todo Do this on the way out. */
3361 if (RT_UNLIKELY( pChunk->cFree == GMM_CHUNK_NUM_PAGES
3362 && pChunk->pFreeNext
3363 && pChunk->pFreePrev /** @todo this is probably misfiring, see reset... */
3364 && !pGMM->fLegacyAllocationMode))
3365 gmmR0FreeChunk(pGMM, NULL, pChunk, false);
3366
3367}
3368
3369
3370/**
3371 * Frees a shared page, the page is known to exist and be valid and such.
3372 *
3373 * @param pGMM Pointer to the GMM instance.
3374 * @param pGVM Pointer to the GVM instance.
3375 * @param idPage The page id.
3376 * @param pPage The page structure.
3377 */
3378DECLINLINE(void) gmmR0FreeSharedPage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage)
3379{
3380 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
3381 Assert(pChunk);
3382 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
3383 Assert(pChunk->cShared > 0);
3384 Assert(pGMM->cSharedPages > 0);
3385 Assert(pGMM->cAllocatedPages > 0);
3386 Assert(!pPage->Shared.cRefs);
3387
3388 pChunk->cShared--;
3389 pGMM->cAllocatedPages--;
3390 pGMM->cSharedPages--;
3391 gmmR0FreePageWorker(pGMM, pGVM, pChunk, idPage, pPage);
3392}
3393
3394
3395/**
3396 * Frees a private page, the page is known to exist and be valid and such.
3397 *
3398 * @param pGMM Pointer to the GMM instance.
3399 * @param pGVM Pointer to the GVM instance.
3400 * @param idPage The page id.
3401 * @param pPage The page structure.
3402 */
3403DECLINLINE(void) gmmR0FreePrivatePage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage)
3404{
3405 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
3406 Assert(pChunk);
3407 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
3408 Assert(pChunk->cPrivate > 0);
3409 Assert(pGMM->cAllocatedPages > 0);
3410
3411 pChunk->cPrivate--;
3412 pGMM->cAllocatedPages--;
3413 gmmR0FreePageWorker(pGMM, pGVM, pChunk, idPage, pPage);
3414}
3415
3416
3417/**
3418 * Common worker for GMMR0FreePages and GMMR0BalloonedPages.
3419 *
3420 * @returns VBox status code:
3421 * @retval xxx
3422 *
3423 * @param pGMM Pointer to the GMM instance data.
3424 * @param pGVM Pointer to the VM.
3425 * @param cPages The number of pages to free.
3426 * @param paPages Pointer to the page descriptors.
3427 * @param enmAccount The account this relates to.
3428 */
3429static int gmmR0FreePages(PGMM pGMM, PGVM pGVM, uint32_t cPages, PGMMFREEPAGEDESC paPages, GMMACCOUNT enmAccount)
3430{
3431 /*
3432 * Check that the request isn't impossible wrt to the account status.
3433 */
3434 switch (enmAccount)
3435 {
3436 case GMMACCOUNT_BASE:
3437 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cBasePages < cPages))
3438 {
3439 Log(("gmmR0FreePages: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cBasePages, cPages));
3440 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3441 }
3442 break;
3443 case GMMACCOUNT_SHADOW:
3444 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cShadowPages < cPages))
3445 {
3446 Log(("gmmR0FreePages: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cShadowPages, cPages));
3447 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3448 }
3449 break;
3450 case GMMACCOUNT_FIXED:
3451 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cFixedPages < cPages))
3452 {
3453 Log(("gmmR0FreePages: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cFixedPages, cPages));
3454 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3455 }
3456 break;
3457 default:
3458 AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
3459 }
3460
3461 /*
3462 * Walk the descriptors and free the pages.
3463 *
3464 * Statistics (except the account) are being updated as we go along,
3465 * unlike the alloc code. Also, stop on the first error.
3466 */
3467 int rc = VINF_SUCCESS;
3468 uint32_t iPage;
3469 for (iPage = 0; iPage < cPages; iPage++)
3470 {
3471 uint32_t idPage = paPages[iPage].idPage;
3472 PGMMPAGE pPage = gmmR0GetPage(pGMM, idPage);
3473 if (RT_LIKELY(pPage))
3474 {
3475 if (RT_LIKELY(GMM_PAGE_IS_PRIVATE(pPage)))
3476 {
3477 if (RT_LIKELY(pPage->Private.hGVM == pGVM->hSelf))
3478 {
3479 Assert(pGVM->gmm.s.Stats.cPrivatePages);
3480 pGVM->gmm.s.Stats.cPrivatePages--;
3481 gmmR0FreePrivatePage(pGMM, pGVM, idPage, pPage);
3482 }
3483 else
3484 {
3485 Log(("gmmR0AllocatePages: #%#x/%#x: not owner! hGVM=%#x hSelf=%#x\n", iPage, idPage,
3486 pPage->Private.hGVM, pGVM->hSelf));
3487 rc = VERR_GMM_NOT_PAGE_OWNER;
3488 break;
3489 }
3490 }
3491 else if (RT_LIKELY(GMM_PAGE_IS_SHARED(pPage)))
3492 {
3493 Assert(pGVM->gmm.s.Stats.cSharedPages);
3494 Assert(pPage->Shared.cRefs);
3495#if defined(VBOX_WITH_PAGE_SHARING) && defined(VBOX_STRICT) && HC_ARCH_BITS == 64
3496 if (pPage->Shared.u14Checksum)
3497 {
3498 uint32_t uChecksum = gmmR0StrictPageChecksum(pGMM, pGVM, idPage);
3499 uChecksum &= UINT32_C(0x00003fff);
3500 AssertMsg(!uChecksum || uChecksum == pPage->Shared.u14Checksum,
3501 ("%#x vs %#x - idPage=%#x\n", uChecksum, pPage->Shared.u14Checksum, idPage));
3502 }
3503#endif
3504 pGVM->gmm.s.Stats.cSharedPages--;
3505 if (!--pPage->Shared.cRefs)
3506 gmmR0FreeSharedPage(pGMM, pGVM, idPage, pPage);
3507 else
3508 {
3509 Assert(pGMM->cDuplicatePages);
3510 pGMM->cDuplicatePages--;
3511 }
3512 }
3513 else
3514 {
3515 Log(("gmmR0AllocatePages: #%#x/%#x: already free!\n", iPage, idPage));
3516 rc = VERR_GMM_PAGE_ALREADY_FREE;
3517 break;
3518 }
3519 }
3520 else
3521 {
3522 Log(("gmmR0AllocatePages: #%#x/%#x: not found!\n", iPage, idPage));
3523 rc = VERR_GMM_PAGE_NOT_FOUND;
3524 break;
3525 }
3526 paPages[iPage].idPage = NIL_GMM_PAGEID;
3527 }
3528
3529 /*
3530 * Update the account.
3531 */
3532 switch (enmAccount)
3533 {
3534 case GMMACCOUNT_BASE: pGVM->gmm.s.Stats.Allocated.cBasePages -= iPage; break;
3535 case GMMACCOUNT_SHADOW: pGVM->gmm.s.Stats.Allocated.cShadowPages -= iPage; break;
3536 case GMMACCOUNT_FIXED: pGVM->gmm.s.Stats.Allocated.cFixedPages -= iPage; break;
3537 default:
3538 AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
3539 }
3540
3541 /*
3542 * Any threshold stuff to be done here?
3543 */
3544
3545 return rc;
3546}
3547
3548
3549/**
3550 * Free one or more pages.
3551 *
3552 * This is typically used at reset time or power off.
3553 *
3554 * @returns VBox status code:
3555 * @retval xxx
3556 *
3557 * @param pVM The cross context VM structure.
3558 * @param idCpu The VCPU id.
3559 * @param cPages The number of pages to allocate.
3560 * @param paPages Pointer to the page descriptors containing the page IDs
3561 * for each page.
3562 * @param enmAccount The account this relates to.
3563 * @thread EMT.
3564 */
3565GMMR0DECL(int) GMMR0FreePages(PVM pVM, VMCPUID idCpu, uint32_t cPages, PGMMFREEPAGEDESC paPages, GMMACCOUNT enmAccount)
3566{
3567 LogFlow(("GMMR0FreePages: pVM=%p cPages=%#x paPages=%p enmAccount=%d\n", pVM, cPages, paPages, enmAccount));
3568
3569 /*
3570 * Validate input and get the basics.
3571 */
3572 PGMM pGMM;
3573 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3574 PGVM pGVM;
3575 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
3576 if (RT_FAILURE(rc))
3577 return rc;
3578
3579 AssertPtrReturn(paPages, VERR_INVALID_PARAMETER);
3580 AssertMsgReturn(enmAccount > GMMACCOUNT_INVALID && enmAccount < GMMACCOUNT_END, ("%d\n", enmAccount), VERR_INVALID_PARAMETER);
3581 AssertMsgReturn(cPages > 0 && cPages < RT_BIT(32 - PAGE_SHIFT), ("%#x\n", cPages), VERR_INVALID_PARAMETER);
3582
3583 for (unsigned iPage = 0; iPage < cPages; iPage++)
3584 AssertMsgReturn( paPages[iPage].idPage <= GMM_PAGEID_LAST
3585 /*|| paPages[iPage].idPage == NIL_GMM_PAGEID*/,
3586 ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
3587
3588 /*
3589 * Take the semaphore and call the worker function.
3590 */
3591 gmmR0MutexAcquire(pGMM);
3592 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3593 {
3594 rc = gmmR0FreePages(pGMM, pGVM, cPages, paPages, enmAccount);
3595 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
3596 }
3597 else
3598 rc = VERR_GMM_IS_NOT_SANE;
3599 gmmR0MutexRelease(pGMM);
3600 LogFlow(("GMMR0FreePages: returns %Rrc\n", rc));
3601 return rc;
3602}
3603
3604
3605/**
3606 * VMMR0 request wrapper for GMMR0FreePages.
3607 *
3608 * @returns see GMMR0FreePages.
3609 * @param pVM The cross context VM structure.
3610 * @param idCpu The VCPU id.
3611 * @param pReq Pointer to the request packet.
3612 */
3613GMMR0DECL(int) GMMR0FreePagesReq(PVM pVM, VMCPUID idCpu, PGMMFREEPAGESREQ pReq)
3614{
3615 /*
3616 * Validate input and pass it on.
3617 */
3618 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
3619 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3620 AssertMsgReturn(pReq->Hdr.cbReq >= RT_UOFFSETOF(GMMFREEPAGESREQ, aPages[0]),
3621 ("%#x < %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF(GMMFREEPAGESREQ, aPages[0])),
3622 VERR_INVALID_PARAMETER);
3623 AssertMsgReturn(pReq->Hdr.cbReq == RT_UOFFSETOF(GMMFREEPAGESREQ, aPages[pReq->cPages]),
3624 ("%#x != %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF(GMMFREEPAGESREQ, aPages[pReq->cPages])),
3625 VERR_INVALID_PARAMETER);
3626
3627 return GMMR0FreePages(pVM, idCpu, pReq->cPages, &pReq->aPages[0], pReq->enmAccount);
3628}
3629
3630
3631/**
3632 * Report back on a memory ballooning request.
3633 *
3634 * The request may or may not have been initiated by the GMM. If it was initiated
3635 * by the GMM it is important that this function is called even if no pages were
3636 * ballooned.
3637 *
3638 * @returns VBox status code:
3639 * @retval VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH
3640 * @retval VERR_GMM_ATTEMPT_TO_DEFLATE_TOO_MUCH
3641 * @retval VERR_GMM_OVERCOMMITTED_TRY_AGAIN_IN_A_BIT - reset condition
3642 * indicating that we won't necessarily have sufficient RAM to boot
3643 * the VM again and that it should pause until this changes (we'll try
3644 * balloon some other VM). (For standard deflate we have little choice
3645 * but to hope the VM won't use the memory that was returned to it.)
3646 *
3647 * @param pVM The cross context VM structure.
3648 * @param idCpu The VCPU id.
3649 * @param enmAction Inflate/deflate/reset.
3650 * @param cBalloonedPages The number of pages that was ballooned.
3651 *
3652 * @thread EMT.
3653 */
3654GMMR0DECL(int) GMMR0BalloonedPages(PVM pVM, VMCPUID idCpu, GMMBALLOONACTION enmAction, uint32_t cBalloonedPages)
3655{
3656 LogFlow(("GMMR0BalloonedPages: pVM=%p enmAction=%d cBalloonedPages=%#x\n",
3657 pVM, enmAction, cBalloonedPages));
3658
3659 AssertMsgReturn(cBalloonedPages < RT_BIT(32 - PAGE_SHIFT), ("%#x\n", cBalloonedPages), VERR_INVALID_PARAMETER);
3660
3661 /*
3662 * Validate input and get the basics.
3663 */
3664 PGMM pGMM;
3665 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3666 PGVM pGVM;
3667 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
3668 if (RT_FAILURE(rc))
3669 return rc;
3670
3671 /*
3672 * Take the semaphore and do some more validations.
3673 */
3674 gmmR0MutexAcquire(pGMM);
3675 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3676 {
3677 switch (enmAction)
3678 {
3679 case GMMBALLOONACTION_INFLATE:
3680 {
3681 if (RT_LIKELY(pGVM->gmm.s.Stats.Allocated.cBasePages + pGVM->gmm.s.Stats.cBalloonedPages + cBalloonedPages
3682 <= pGVM->gmm.s.Stats.Reserved.cBasePages))
3683 {
3684 /*
3685 * Record the ballooned memory.
3686 */
3687 pGMM->cBalloonedPages += cBalloonedPages;
3688 if (pGVM->gmm.s.Stats.cReqBalloonedPages)
3689 {
3690 /* Codepath never taken. Might be interesting in the future to request ballooned memory from guests in low memory conditions.. */
3691 AssertFailed();
3692
3693 pGVM->gmm.s.Stats.cBalloonedPages += cBalloonedPages;
3694 pGVM->gmm.s.Stats.cReqActuallyBalloonedPages += cBalloonedPages;
3695 Log(("GMMR0BalloonedPages: +%#x - Global=%#llx / VM: Total=%#llx Req=%#llx Actual=%#llx (pending)\n",
3696 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages,
3697 pGVM->gmm.s.Stats.cReqBalloonedPages, pGVM->gmm.s.Stats.cReqActuallyBalloonedPages));
3698 }
3699 else
3700 {
3701 pGVM->gmm.s.Stats.cBalloonedPages += cBalloonedPages;
3702 Log(("GMMR0BalloonedPages: +%#x - Global=%#llx / VM: Total=%#llx (user)\n",
3703 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages));
3704 }
3705 }
3706 else
3707 {
3708 Log(("GMMR0BalloonedPages: cBasePages=%#llx Total=%#llx cBalloonedPages=%#llx Reserved=%#llx\n",
3709 pGVM->gmm.s.Stats.Allocated.cBasePages, pGVM->gmm.s.Stats.cBalloonedPages, cBalloonedPages,
3710 pGVM->gmm.s.Stats.Reserved.cBasePages));
3711 rc = VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3712 }
3713 break;
3714 }
3715
3716 case GMMBALLOONACTION_DEFLATE:
3717 {
3718 /* Deflate. */
3719 if (pGVM->gmm.s.Stats.cBalloonedPages >= cBalloonedPages)
3720 {
3721 /*
3722 * Record the ballooned memory.
3723 */
3724 Assert(pGMM->cBalloonedPages >= cBalloonedPages);
3725 pGMM->cBalloonedPages -= cBalloonedPages;
3726 pGVM->gmm.s.Stats.cBalloonedPages -= cBalloonedPages;
3727 if (pGVM->gmm.s.Stats.cReqDeflatePages)
3728 {
3729 AssertFailed(); /* This is path is for later. */
3730 Log(("GMMR0BalloonedPages: -%#x - Global=%#llx / VM: Total=%#llx Req=%#llx\n",
3731 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages, pGVM->gmm.s.Stats.cReqDeflatePages));
3732
3733 /*
3734 * Anything we need to do here now when the request has been completed?
3735 */
3736 pGVM->gmm.s.Stats.cReqDeflatePages = 0;
3737 }
3738 else
3739 Log(("GMMR0BalloonedPages: -%#x - Global=%#llx / VM: Total=%#llx (user)\n",
3740 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages));
3741 }
3742 else
3743 {
3744 Log(("GMMR0BalloonedPages: Total=%#llx cBalloonedPages=%#llx\n", pGVM->gmm.s.Stats.cBalloonedPages, cBalloonedPages));
3745 rc = VERR_GMM_ATTEMPT_TO_DEFLATE_TOO_MUCH;
3746 }
3747 break;
3748 }
3749
3750 case GMMBALLOONACTION_RESET:
3751 {
3752 /* Reset to an empty balloon. */
3753 Assert(pGMM->cBalloonedPages >= pGVM->gmm.s.Stats.cBalloonedPages);
3754
3755 pGMM->cBalloonedPages -= pGVM->gmm.s.Stats.cBalloonedPages;
3756 pGVM->gmm.s.Stats.cBalloonedPages = 0;
3757 break;
3758 }
3759
3760 default:
3761 rc = VERR_INVALID_PARAMETER;
3762 break;
3763 }
3764 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
3765 }
3766 else
3767 rc = VERR_GMM_IS_NOT_SANE;
3768
3769 gmmR0MutexRelease(pGMM);
3770 LogFlow(("GMMR0BalloonedPages: returns %Rrc\n", rc));
3771 return rc;
3772}
3773
3774
3775/**
3776 * VMMR0 request wrapper for GMMR0BalloonedPages.
3777 *
3778 * @returns see GMMR0BalloonedPages.
3779 * @param pVM The cross context VM structure.
3780 * @param idCpu The VCPU id.
3781 * @param pReq Pointer to the request packet.
3782 */
3783GMMR0DECL(int) GMMR0BalloonedPagesReq(PVM pVM, VMCPUID idCpu, PGMMBALLOONEDPAGESREQ pReq)
3784{
3785 /*
3786 * Validate input and pass it on.
3787 */
3788 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
3789 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3790 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMBALLOONEDPAGESREQ),
3791 ("%#x < %#x\n", pReq->Hdr.cbReq, sizeof(GMMBALLOONEDPAGESREQ)),
3792 VERR_INVALID_PARAMETER);
3793
3794 return GMMR0BalloonedPages(pVM, idCpu, pReq->enmAction, pReq->cBalloonedPages);
3795}
3796
3797/**
3798 * Return memory statistics for the hypervisor
3799 *
3800 * @returns VBox status code:
3801 * @param pVM The cross context VM structure.
3802 * @param pReq Pointer to the request packet.
3803 */
3804GMMR0DECL(int) GMMR0QueryHypervisorMemoryStatsReq(PVM pVM, PGMMMEMSTATSREQ pReq)
3805{
3806 /*
3807 * Validate input and pass it on.
3808 */
3809 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
3810 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3811 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMMEMSTATSREQ),
3812 ("%#x < %#x\n", pReq->Hdr.cbReq, sizeof(GMMMEMSTATSREQ)),
3813 VERR_INVALID_PARAMETER);
3814
3815 /*
3816 * Validate input and get the basics.
3817 */
3818 PGMM pGMM;
3819 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3820 pReq->cAllocPages = pGMM->cAllocatedPages;
3821 pReq->cFreePages = (pGMM->cChunks << (GMM_CHUNK_SHIFT- PAGE_SHIFT)) - pGMM->cAllocatedPages;
3822 pReq->cBalloonedPages = pGMM->cBalloonedPages;
3823 pReq->cMaxPages = pGMM->cMaxPages;
3824 pReq->cSharedPages = pGMM->cDuplicatePages;
3825 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
3826
3827 return VINF_SUCCESS;
3828}
3829
3830/**
3831 * Return memory statistics for the VM
3832 *
3833 * @returns VBox status code:
3834 * @param pVM The cross context VM structure.
3835 * @param idCpu Cpu id.
3836 * @param pReq Pointer to the request packet.
3837 */
3838GMMR0DECL(int) GMMR0QueryMemoryStatsReq(PVM pVM, VMCPUID idCpu, PGMMMEMSTATSREQ pReq)
3839{
3840 /*
3841 * Validate input and pass it on.
3842 */
3843 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
3844 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3845 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMMEMSTATSREQ),
3846 ("%#x < %#x\n", pReq->Hdr.cbReq, sizeof(GMMMEMSTATSREQ)),
3847 VERR_INVALID_PARAMETER);
3848
3849 /*
3850 * Validate input and get the basics.
3851 */
3852 PGMM pGMM;
3853 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3854 PGVM pGVM;
3855 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
3856 if (RT_FAILURE(rc))
3857 return rc;
3858
3859 /*
3860 * Take the semaphore and do some more validations.
3861 */
3862 gmmR0MutexAcquire(pGMM);
3863 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3864 {
3865 pReq->cAllocPages = pGVM->gmm.s.Stats.Allocated.cBasePages;
3866 pReq->cBalloonedPages = pGVM->gmm.s.Stats.cBalloonedPages;
3867 pReq->cMaxPages = pGVM->gmm.s.Stats.Reserved.cBasePages;
3868 pReq->cFreePages = pReq->cMaxPages - pReq->cAllocPages;
3869 }
3870 else
3871 rc = VERR_GMM_IS_NOT_SANE;
3872
3873 gmmR0MutexRelease(pGMM);
3874 LogFlow(("GMMR3QueryVMMemoryStats: returns %Rrc\n", rc));
3875 return rc;
3876}
3877
3878
3879/**
3880 * Worker for gmmR0UnmapChunk and gmmr0FreeChunk.
3881 *
3882 * Don't call this in legacy allocation mode!
3883 *
3884 * @returns VBox status code.
3885 * @param pGMM Pointer to the GMM instance data.
3886 * @param pGVM Pointer to the Global VM structure.
3887 * @param pChunk Pointer to the chunk to be unmapped.
3888 */
3889static int gmmR0UnmapChunkLocked(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk)
3890{
3891 Assert(!pGMM->fLegacyAllocationMode); NOREF(pGMM);
3892
3893 /*
3894 * Find the mapping and try unmapping it.
3895 */
3896 uint32_t cMappings = pChunk->cMappingsX;
3897 for (uint32_t i = 0; i < cMappings; i++)
3898 {
3899 Assert(pChunk->paMappingsX[i].pGVM && pChunk->paMappingsX[i].hMapObj != NIL_RTR0MEMOBJ);
3900 if (pChunk->paMappingsX[i].pGVM == pGVM)
3901 {
3902 /* unmap */
3903 int rc = RTR0MemObjFree(pChunk->paMappingsX[i].hMapObj, false /* fFreeMappings (NA) */);
3904 if (RT_SUCCESS(rc))
3905 {
3906 /* update the record. */
3907 cMappings--;
3908 if (i < cMappings)
3909 pChunk->paMappingsX[i] = pChunk->paMappingsX[cMappings];
3910 pChunk->paMappingsX[cMappings].hMapObj = NIL_RTR0MEMOBJ;
3911 pChunk->paMappingsX[cMappings].pGVM = NULL;
3912 Assert(pChunk->cMappingsX - 1U == cMappings);
3913 pChunk->cMappingsX = cMappings;
3914 }
3915
3916 return rc;
3917 }
3918 }
3919
3920 Log(("gmmR0UnmapChunk: Chunk %#x is not mapped into pGVM=%p/%#x\n", pChunk->Core.Key, pGVM, pGVM->hSelf));
3921 return VERR_GMM_CHUNK_NOT_MAPPED;
3922}
3923
3924
3925/**
3926 * Unmaps a chunk previously mapped into the address space of the current process.
3927 *
3928 * @returns VBox status code.
3929 * @param pGMM Pointer to the GMM instance data.
3930 * @param pGVM Pointer to the Global VM structure.
3931 * @param pChunk Pointer to the chunk to be unmapped.
3932 * @param fRelaxedSem Whether we can release the semaphore while doing the
3933 * mapping (@c true) or not.
3934 */
3935static int gmmR0UnmapChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem)
3936{
3937 if (!pGMM->fLegacyAllocationMode)
3938 {
3939 /*
3940 * Lock the chunk and if possible leave the giant GMM lock.
3941 */
3942 GMMR0CHUNKMTXSTATE MtxState;
3943 int rc = gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk,
3944 fRelaxedSem ? GMMR0CHUNK_MTX_RETAKE_GIANT : GMMR0CHUNK_MTX_KEEP_GIANT);
3945 if (RT_SUCCESS(rc))
3946 {
3947 rc = gmmR0UnmapChunkLocked(pGMM, pGVM, pChunk);
3948 gmmR0ChunkMutexRelease(&MtxState, pChunk);
3949 }
3950 return rc;
3951 }
3952
3953 if (pChunk->hGVM == pGVM->hSelf)
3954 return VINF_SUCCESS;
3955
3956 Log(("gmmR0UnmapChunk: Chunk %#x is not mapped into pGVM=%p/%#x (legacy)\n", pChunk->Core.Key, pGVM, pGVM->hSelf));
3957 return VERR_GMM_CHUNK_NOT_MAPPED;
3958}
3959
3960
3961/**
3962 * Worker for gmmR0MapChunk.
3963 *
3964 * @returns VBox status code.
3965 * @param pGMM Pointer to the GMM instance data.
3966 * @param pGVM Pointer to the Global VM structure.
3967 * @param pChunk Pointer to the chunk to be mapped.
3968 * @param ppvR3 Where to store the ring-3 address of the mapping.
3969 * In the VERR_GMM_CHUNK_ALREADY_MAPPED case, this will be
3970 * contain the address of the existing mapping.
3971 */
3972static int gmmR0MapChunkLocked(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, PRTR3PTR ppvR3)
3973{
3974 /*
3975 * If we're in legacy mode this is simple.
3976 */
3977 if (pGMM->fLegacyAllocationMode)
3978 {
3979 if (pChunk->hGVM != pGVM->hSelf)
3980 {
3981 Log(("gmmR0MapChunk: chunk %#x is already mapped at %p!\n", pChunk->Core.Key, *ppvR3));
3982 return VERR_GMM_CHUNK_NOT_FOUND;
3983 }
3984
3985 *ppvR3 = RTR0MemObjAddressR3(pChunk->hMemObj);
3986 return VINF_SUCCESS;
3987 }
3988
3989 /*
3990 * Check to see if the chunk is already mapped.
3991 */
3992 for (uint32_t i = 0; i < pChunk->cMappingsX; i++)
3993 {
3994 Assert(pChunk->paMappingsX[i].pGVM && pChunk->paMappingsX[i].hMapObj != NIL_RTR0MEMOBJ);
3995 if (pChunk->paMappingsX[i].pGVM == pGVM)
3996 {
3997 *ppvR3 = RTR0MemObjAddressR3(pChunk->paMappingsX[i].hMapObj);
3998 Log(("gmmR0MapChunk: chunk %#x is already mapped at %p!\n", pChunk->Core.Key, *ppvR3));
3999#ifdef VBOX_WITH_PAGE_SHARING
4000 /* The ring-3 chunk cache can be out of sync; don't fail. */
4001 return VINF_SUCCESS;
4002#else
4003 return VERR_GMM_CHUNK_ALREADY_MAPPED;
4004#endif
4005 }
4006 }
4007
4008 /*
4009 * Do the mapping.
4010 */
4011 RTR0MEMOBJ hMapObj;
4012 int rc = RTR0MemObjMapUser(&hMapObj, pChunk->hMemObj, (RTR3PTR)-1, 0, RTMEM_PROT_READ | RTMEM_PROT_WRITE, NIL_RTR0PROCESS);
4013 if (RT_SUCCESS(rc))
4014 {
4015 /* reallocate the array? assumes few users per chunk (usually one). */
4016 unsigned iMapping = pChunk->cMappingsX;
4017 if ( iMapping <= 3
4018 || (iMapping & 3) == 0)
4019 {
4020 unsigned cNewSize = iMapping <= 3
4021 ? iMapping + 1
4022 : iMapping + 4;
4023 Assert(cNewSize < 4 || RT_ALIGN_32(cNewSize, 4) == cNewSize);
4024 if (RT_UNLIKELY(cNewSize > UINT16_MAX))
4025 {
4026 rc = RTR0MemObjFree(hMapObj, false /* fFreeMappings (NA) */); AssertRC(rc);
4027 return VERR_GMM_TOO_MANY_CHUNK_MAPPINGS;
4028 }
4029
4030 void *pvMappings = RTMemRealloc(pChunk->paMappingsX, cNewSize * sizeof(pChunk->paMappingsX[0]));
4031 if (RT_UNLIKELY(!pvMappings))
4032 {
4033 rc = RTR0MemObjFree(hMapObj, false /* fFreeMappings (NA) */); AssertRC(rc);
4034 return VERR_NO_MEMORY;
4035 }
4036 pChunk->paMappingsX = (PGMMCHUNKMAP)pvMappings;
4037 }
4038
4039 /* insert new entry */
4040 pChunk->paMappingsX[iMapping].hMapObj = hMapObj;
4041 pChunk->paMappingsX[iMapping].pGVM = pGVM;
4042 Assert(pChunk->cMappingsX == iMapping);
4043 pChunk->cMappingsX = iMapping + 1;
4044
4045 *ppvR3 = RTR0MemObjAddressR3(hMapObj);
4046 }
4047
4048 return rc;
4049}
4050
4051
4052/**
4053 * Maps a chunk into the user address space of the current process.
4054 *
4055 * @returns VBox status code.
4056 * @param pGMM Pointer to the GMM instance data.
4057 * @param pGVM Pointer to the Global VM structure.
4058 * @param pChunk Pointer to the chunk to be mapped.
4059 * @param fRelaxedSem Whether we can release the semaphore while doing the
4060 * mapping (@c true) or not.
4061 * @param ppvR3 Where to store the ring-3 address of the mapping.
4062 * In the VERR_GMM_CHUNK_ALREADY_MAPPED case, this will be
4063 * contain the address of the existing mapping.
4064 */
4065static int gmmR0MapChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem, PRTR3PTR ppvR3)
4066{
4067 /*
4068 * Take the chunk lock and leave the giant GMM lock when possible, then
4069 * call the worker function.
4070 */
4071 GMMR0CHUNKMTXSTATE MtxState;
4072 int rc = gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk,
4073 fRelaxedSem ? GMMR0CHUNK_MTX_RETAKE_GIANT : GMMR0CHUNK_MTX_KEEP_GIANT);
4074 if (RT_SUCCESS(rc))
4075 {
4076 rc = gmmR0MapChunkLocked(pGMM, pGVM, pChunk, ppvR3);
4077 gmmR0ChunkMutexRelease(&MtxState, pChunk);
4078 }
4079
4080 return rc;
4081}
4082
4083
4084
4085#if defined(VBOX_WITH_PAGE_SHARING) || (defined(VBOX_STRICT) && HC_ARCH_BITS == 64)
4086/**
4087 * Check if a chunk is mapped into the specified VM
4088 *
4089 * @returns mapped yes/no
4090 * @param pGMM Pointer to the GMM instance.
4091 * @param pGVM Pointer to the Global VM structure.
4092 * @param pChunk Pointer to the chunk to be mapped.
4093 * @param ppvR3 Where to store the ring-3 address of the mapping.
4094 */
4095static bool gmmR0IsChunkMapped(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, PRTR3PTR ppvR3)
4096{
4097 GMMR0CHUNKMTXSTATE MtxState;
4098 gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk, GMMR0CHUNK_MTX_KEEP_GIANT);
4099 for (uint32_t i = 0; i < pChunk->cMappingsX; i++)
4100 {
4101 Assert(pChunk->paMappingsX[i].pGVM && pChunk->paMappingsX[i].hMapObj != NIL_RTR0MEMOBJ);
4102 if (pChunk->paMappingsX[i].pGVM == pGVM)
4103 {
4104 *ppvR3 = RTR0MemObjAddressR3(pChunk->paMappingsX[i].hMapObj);
4105 gmmR0ChunkMutexRelease(&MtxState, pChunk);
4106 return true;
4107 }
4108 }
4109 *ppvR3 = NULL;
4110 gmmR0ChunkMutexRelease(&MtxState, pChunk);
4111 return false;
4112}
4113#endif /* VBOX_WITH_PAGE_SHARING || (VBOX_STRICT && 64-BIT) */
4114
4115
4116/**
4117 * Map a chunk and/or unmap another chunk.
4118 *
4119 * The mapping and unmapping applies to the current process.
4120 *
4121 * This API does two things because it saves a kernel call per mapping when
4122 * when the ring-3 mapping cache is full.
4123 *
4124 * @returns VBox status code.
4125 * @param pVM The cross context VM structure.
4126 * @param idChunkMap The chunk to map. NIL_GMM_CHUNKID if nothing to map.
4127 * @param idChunkUnmap The chunk to unmap. NIL_GMM_CHUNKID if nothing to unmap.
4128 * @param ppvR3 Where to store the address of the mapped chunk. NULL is ok if nothing to map.
4129 * @thread EMT
4130 */
4131GMMR0DECL(int) GMMR0MapUnmapChunk(PVM pVM, uint32_t idChunkMap, uint32_t idChunkUnmap, PRTR3PTR ppvR3)
4132{
4133 LogFlow(("GMMR0MapUnmapChunk: pVM=%p idChunkMap=%#x idChunkUnmap=%#x ppvR3=%p\n",
4134 pVM, idChunkMap, idChunkUnmap, ppvR3));
4135
4136 /*
4137 * Validate input and get the basics.
4138 */
4139 PGMM pGMM;
4140 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4141 PGVM pGVM;
4142 int rc = GVMMR0ByVM(pVM, &pGVM);
4143 if (RT_FAILURE(rc))
4144 return rc;
4145
4146 AssertCompile(NIL_GMM_CHUNKID == 0);
4147 AssertMsgReturn(idChunkMap <= GMM_CHUNKID_LAST, ("%#x\n", idChunkMap), VERR_INVALID_PARAMETER);
4148 AssertMsgReturn(idChunkUnmap <= GMM_CHUNKID_LAST, ("%#x\n", idChunkUnmap), VERR_INVALID_PARAMETER);
4149
4150 if ( idChunkMap == NIL_GMM_CHUNKID
4151 && idChunkUnmap == NIL_GMM_CHUNKID)
4152 return VERR_INVALID_PARAMETER;
4153
4154 if (idChunkMap != NIL_GMM_CHUNKID)
4155 {
4156 AssertPtrReturn(ppvR3, VERR_INVALID_POINTER);
4157 *ppvR3 = NIL_RTR3PTR;
4158 }
4159
4160 /*
4161 * Take the semaphore and do the work.
4162 *
4163 * The unmapping is done last since it's easier to undo a mapping than
4164 * undoing an unmapping. The ring-3 mapping cache cannot not be so big
4165 * that it pushes the user virtual address space to within a chunk of
4166 * it it's limits, so, no problem here.
4167 */
4168 gmmR0MutexAcquire(pGMM);
4169 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
4170 {
4171 PGMMCHUNK pMap = NULL;
4172 if (idChunkMap != NIL_GVM_HANDLE)
4173 {
4174 pMap = gmmR0GetChunk(pGMM, idChunkMap);
4175 if (RT_LIKELY(pMap))
4176 rc = gmmR0MapChunk(pGMM, pGVM, pMap, true /*fRelaxedSem*/, ppvR3);
4177 else
4178 {
4179 Log(("GMMR0MapUnmapChunk: idChunkMap=%#x\n", idChunkMap));
4180 rc = VERR_GMM_CHUNK_NOT_FOUND;
4181 }
4182 }
4183/** @todo split this operation, the bail out might (theoretcially) not be
4184 * entirely safe. */
4185
4186 if ( idChunkUnmap != NIL_GMM_CHUNKID
4187 && RT_SUCCESS(rc))
4188 {
4189 PGMMCHUNK pUnmap = gmmR0GetChunk(pGMM, idChunkUnmap);
4190 if (RT_LIKELY(pUnmap))
4191 rc = gmmR0UnmapChunk(pGMM, pGVM, pUnmap, true /*fRelaxedSem*/);
4192 else
4193 {
4194 Log(("GMMR0MapUnmapChunk: idChunkUnmap=%#x\n", idChunkUnmap));
4195 rc = VERR_GMM_CHUNK_NOT_FOUND;
4196 }
4197
4198 if (RT_FAILURE(rc) && pMap)
4199 gmmR0UnmapChunk(pGMM, pGVM, pMap, false /*fRelaxedSem*/);
4200 }
4201
4202 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
4203 }
4204 else
4205 rc = VERR_GMM_IS_NOT_SANE;
4206 gmmR0MutexRelease(pGMM);
4207
4208 LogFlow(("GMMR0MapUnmapChunk: returns %Rrc\n", rc));
4209 return rc;
4210}
4211
4212
4213/**
4214 * VMMR0 request wrapper for GMMR0MapUnmapChunk.
4215 *
4216 * @returns see GMMR0MapUnmapChunk.
4217 * @param pVM The cross context VM structure.
4218 * @param pReq Pointer to the request packet.
4219 */
4220GMMR0DECL(int) GMMR0MapUnmapChunkReq(PVM pVM, PGMMMAPUNMAPCHUNKREQ pReq)
4221{
4222 /*
4223 * Validate input and pass it on.
4224 */
4225 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
4226 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
4227 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
4228
4229 return GMMR0MapUnmapChunk(pVM, pReq->idChunkMap, pReq->idChunkUnmap, &pReq->pvR3);
4230}
4231
4232
4233/**
4234 * Legacy mode API for supplying pages.
4235 *
4236 * The specified user address points to a allocation chunk sized block that
4237 * will be locked down and used by the GMM when the GM asks for pages.
4238 *
4239 * @returns VBox status code.
4240 * @param pVM The cross context VM structure.
4241 * @param idCpu The VCPU id.
4242 * @param pvR3 Pointer to the chunk size memory block to lock down.
4243 */
4244GMMR0DECL(int) GMMR0SeedChunk(PVM pVM, VMCPUID idCpu, RTR3PTR pvR3)
4245{
4246 /*
4247 * Validate input and get the basics.
4248 */
4249 PGMM pGMM;
4250 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4251 PGVM pGVM;
4252 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
4253 if (RT_FAILURE(rc))
4254 return rc;
4255
4256 AssertPtrReturn(pvR3, VERR_INVALID_POINTER);
4257 AssertReturn(!(PAGE_OFFSET_MASK & pvR3), VERR_INVALID_POINTER);
4258
4259 if (!pGMM->fLegacyAllocationMode)
4260 {
4261 Log(("GMMR0SeedChunk: not in legacy allocation mode!\n"));
4262 return VERR_NOT_SUPPORTED;
4263 }
4264
4265 /*
4266 * Lock the memory and add it as new chunk with our hGVM.
4267 * (The GMM locking is done inside gmmR0RegisterChunk.)
4268 */
4269 RTR0MEMOBJ MemObj;
4270 rc = RTR0MemObjLockUser(&MemObj, pvR3, GMM_CHUNK_SIZE, RTMEM_PROT_READ | RTMEM_PROT_WRITE, NIL_RTR0PROCESS);
4271 if (RT_SUCCESS(rc))
4272 {
4273 rc = gmmR0RegisterChunk(pGMM, &pGVM->gmm.s.Private, MemObj, pGVM->hSelf, 0 /*fChunkFlags*/, NULL);
4274 if (RT_SUCCESS(rc))
4275 gmmR0MutexRelease(pGMM);
4276 else
4277 RTR0MemObjFree(MemObj, false /* fFreeMappings */);
4278 }
4279
4280 LogFlow(("GMMR0SeedChunk: rc=%d (pvR3=%p)\n", rc, pvR3));
4281 return rc;
4282}
4283
4284#ifdef VBOX_WITH_PAGE_SHARING
4285
4286# ifdef VBOX_STRICT
4287/**
4288 * For checksumming shared pages in strict builds.
4289 *
4290 * The purpose is making sure that a page doesn't change.
4291 *
4292 * @returns Checksum, 0 on failure.
4293 * @param pGMM The GMM instance data.
4294 * @param pGVM Pointer to the kernel-only VM instace data.
4295 * @param idPage The page ID.
4296 */
4297static uint32_t gmmR0StrictPageChecksum(PGMM pGMM, PGVM pGVM, uint32_t idPage)
4298{
4299 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
4300 AssertMsgReturn(pChunk, ("idPage=%#x\n", idPage), 0);
4301
4302 uint8_t *pbChunk;
4303 if (!gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
4304 return 0;
4305 uint8_t const *pbPage = pbChunk + ((idPage & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
4306
4307 return RTCrc32(pbPage, PAGE_SIZE);
4308}
4309# endif /* VBOX_STRICT */
4310
4311
4312/**
4313 * Calculates the module hash value.
4314 *
4315 * @returns Hash value.
4316 * @param pszModuleName The module name.
4317 * @param pszVersion The module version string.
4318 */
4319static uint32_t gmmR0ShModCalcHash(const char *pszModuleName, const char *pszVersion)
4320{
4321 return RTStrHash1ExN(3, pszModuleName, RTSTR_MAX, "::", (size_t)2, pszVersion, RTSTR_MAX);
4322}
4323
4324
4325/**
4326 * Finds a global module.
4327 *
4328 * @returns Pointer to the global module on success, NULL if not found.
4329 * @param pGMM The GMM instance data.
4330 * @param uHash The hash as calculated by gmmR0ShModCalcHash.
4331 * @param cbModule The module size.
4332 * @param enmGuestOS The guest OS type.
4333 * @param cRegions The number of regions.
4334 * @param pszModuleName The module name.
4335 * @param pszVersion The module version.
4336 * @param paRegions The region descriptions.
4337 */
4338static PGMMSHAREDMODULE gmmR0ShModFindGlobal(PGMM pGMM, uint32_t uHash, uint32_t cbModule, VBOXOSFAMILY enmGuestOS,
4339 uint32_t cRegions, const char *pszModuleName, const char *pszVersion,
4340 struct VMMDEVSHAREDREGIONDESC const *paRegions)
4341{
4342 for (PGMMSHAREDMODULE pGblMod = (PGMMSHAREDMODULE)RTAvllU32Get(&pGMM->pGlobalSharedModuleTree, uHash);
4343 pGblMod;
4344 pGblMod = (PGMMSHAREDMODULE)pGblMod->Core.pList)
4345 {
4346 if (pGblMod->cbModule != cbModule)
4347 continue;
4348 if (pGblMod->enmGuestOS != enmGuestOS)
4349 continue;
4350 if (pGblMod->cRegions != cRegions)
4351 continue;
4352 if (strcmp(pGblMod->szName, pszModuleName))
4353 continue;
4354 if (strcmp(pGblMod->szVersion, pszVersion))
4355 continue;
4356
4357 uint32_t i;
4358 for (i = 0; i < cRegions; i++)
4359 {
4360 uint32_t off = paRegions[i].GCRegionAddr & PAGE_OFFSET_MASK;
4361 if (pGblMod->aRegions[i].off != off)
4362 break;
4363
4364 uint32_t cb = RT_ALIGN_32(paRegions[i].cbRegion + off, PAGE_SIZE);
4365 if (pGblMod->aRegions[i].cb != cb)
4366 break;
4367 }
4368
4369 if (i == cRegions)
4370 return pGblMod;
4371 }
4372
4373 return NULL;
4374}
4375
4376
4377/**
4378 * Creates a new global module.
4379 *
4380 * @returns VBox status code.
4381 * @param pGMM The GMM instance data.
4382 * @param uHash The hash as calculated by gmmR0ShModCalcHash.
4383 * @param cbModule The module size.
4384 * @param enmGuestOS The guest OS type.
4385 * @param cRegions The number of regions.
4386 * @param pszModuleName The module name.
4387 * @param pszVersion The module version.
4388 * @param paRegions The region descriptions.
4389 * @param ppGblMod Where to return the new module on success.
4390 */
4391static int gmmR0ShModNewGlobal(PGMM pGMM, uint32_t uHash, uint32_t cbModule, VBOXOSFAMILY enmGuestOS,
4392 uint32_t cRegions, const char *pszModuleName, const char *pszVersion,
4393 struct VMMDEVSHAREDREGIONDESC const *paRegions, PGMMSHAREDMODULE *ppGblMod)
4394{
4395 Log(("gmmR0ShModNewGlobal: %s %s size %#x os %u rgn %u\n", pszModuleName, pszVersion, cbModule, enmGuestOS, cRegions));
4396 if (pGMM->cShareableModules >= GMM_MAX_SHARED_GLOBAL_MODULES)
4397 {
4398 Log(("gmmR0ShModNewGlobal: Too many modules\n"));
4399 return VERR_GMM_TOO_MANY_GLOBAL_MODULES;
4400 }
4401
4402 PGMMSHAREDMODULE pGblMod = (PGMMSHAREDMODULE)RTMemAllocZ(RT_OFFSETOF(GMMSHAREDMODULE, aRegions[cRegions]));
4403 if (!pGblMod)
4404 {
4405 Log(("gmmR0ShModNewGlobal: No memory\n"));
4406 return VERR_NO_MEMORY;
4407 }
4408
4409 pGblMod->Core.Key = uHash;
4410 pGblMod->cbModule = cbModule;
4411 pGblMod->cRegions = cRegions;
4412 pGblMod->cUsers = 1;
4413 pGblMod->enmGuestOS = enmGuestOS;
4414 strcpy(pGblMod->szName, pszModuleName);
4415 strcpy(pGblMod->szVersion, pszVersion);
4416
4417 for (uint32_t i = 0; i < cRegions; i++)
4418 {
4419 Log(("gmmR0ShModNewGlobal: rgn[%u]=%RGvLB%#x\n", i, paRegions[i].GCRegionAddr, paRegions[i].cbRegion));
4420 pGblMod->aRegions[i].off = paRegions[i].GCRegionAddr & PAGE_OFFSET_MASK;
4421 pGblMod->aRegions[i].cb = paRegions[i].cbRegion + pGblMod->aRegions[i].off;
4422 pGblMod->aRegions[i].cb = RT_ALIGN_32(pGblMod->aRegions[i].cb, PAGE_SIZE);
4423 pGblMod->aRegions[i].paidPages = NULL; /* allocated when needed. */
4424 }
4425
4426 bool fInsert = RTAvllU32Insert(&pGMM->pGlobalSharedModuleTree, &pGblMod->Core);
4427 Assert(fInsert); NOREF(fInsert);
4428 pGMM->cShareableModules++;
4429
4430 *ppGblMod = pGblMod;
4431 return VINF_SUCCESS;
4432}
4433
4434
4435/**
4436 * Deletes a global module which is no longer referenced by anyone.
4437 *
4438 * @param pGMM The GMM instance data.
4439 * @param pGblMod The module to delete.
4440 */
4441static void gmmR0ShModDeleteGlobal(PGMM pGMM, PGMMSHAREDMODULE pGblMod)
4442{
4443 Assert(pGblMod->cUsers == 0);
4444 Assert(pGMM->cShareableModules > 0 && pGMM->cShareableModules <= GMM_MAX_SHARED_GLOBAL_MODULES);
4445
4446 void *pvTest = RTAvllU32RemoveNode(&pGMM->pGlobalSharedModuleTree, &pGblMod->Core);
4447 Assert(pvTest == pGblMod); NOREF(pvTest);
4448 pGMM->cShareableModules--;
4449
4450 uint32_t i = pGblMod->cRegions;
4451 while (i-- > 0)
4452 {
4453 if (pGblMod->aRegions[i].paidPages)
4454 {
4455 /* We don't doing anything to the pages as they are handled by the
4456 copy-on-write mechanism in PGM. */
4457 RTMemFree(pGblMod->aRegions[i].paidPages);
4458 pGblMod->aRegions[i].paidPages = NULL;
4459 }
4460 }
4461 RTMemFree(pGblMod);
4462}
4463
4464
4465static int gmmR0ShModNewPerVM(PGVM pGVM, RTGCPTR GCBaseAddr, uint32_t cRegions, const VMMDEVSHAREDREGIONDESC *paRegions,
4466 PGMMSHAREDMODULEPERVM *ppRecVM)
4467{
4468 if (pGVM->gmm.s.Stats.cShareableModules >= GMM_MAX_SHARED_PER_VM_MODULES)
4469 return VERR_GMM_TOO_MANY_PER_VM_MODULES;
4470
4471 PGMMSHAREDMODULEPERVM pRecVM;
4472 pRecVM = (PGMMSHAREDMODULEPERVM)RTMemAllocZ(RT_OFFSETOF(GMMSHAREDMODULEPERVM, aRegionsGCPtrs[cRegions]));
4473 if (!pRecVM)
4474 return VERR_NO_MEMORY;
4475
4476 pRecVM->Core.Key = GCBaseAddr;
4477 for (uint32_t i = 0; i < cRegions; i++)
4478 pRecVM->aRegionsGCPtrs[i] = paRegions[i].GCRegionAddr;
4479
4480 bool fInsert = RTAvlGCPtrInsert(&pGVM->gmm.s.pSharedModuleTree, &pRecVM->Core);
4481 Assert(fInsert); NOREF(fInsert);
4482 pGVM->gmm.s.Stats.cShareableModules++;
4483
4484 *ppRecVM = pRecVM;
4485 return VINF_SUCCESS;
4486}
4487
4488
4489static void gmmR0ShModDeletePerVM(PGMM pGMM, PGVM pGVM, PGMMSHAREDMODULEPERVM pRecVM, bool fRemove)
4490{
4491 /*
4492 * Free the per-VM module.
4493 */
4494 PGMMSHAREDMODULE pGblMod = pRecVM->pGlobalModule;
4495 pRecVM->pGlobalModule = NULL;
4496
4497 if (fRemove)
4498 {
4499 void *pvTest = RTAvlGCPtrRemove(&pGVM->gmm.s.pSharedModuleTree, pRecVM->Core.Key);
4500 Assert(pvTest == &pRecVM->Core); NOREF(pvTest);
4501 }
4502
4503 RTMemFree(pRecVM);
4504
4505 /*
4506 * Release the global module.
4507 * (In the registration bailout case, it might not be.)
4508 */
4509 if (pGblMod)
4510 {
4511 Assert(pGblMod->cUsers > 0);
4512 pGblMod->cUsers--;
4513 if (pGblMod->cUsers == 0)
4514 gmmR0ShModDeleteGlobal(pGMM, pGblMod);
4515 }
4516}
4517
4518#endif /* VBOX_WITH_PAGE_SHARING */
4519
4520/**
4521 * Registers a new shared module for the VM.
4522 *
4523 * @returns VBox status code.
4524 * @param pVM The cross context VM structure.
4525 * @param idCpu The VCPU id.
4526 * @param enmGuestOS The guest OS type.
4527 * @param pszModuleName The module name.
4528 * @param pszVersion The module version.
4529 * @param GCPtrModBase The module base address.
4530 * @param cbModule The module size.
4531 * @param cRegions The mumber of shared region descriptors.
4532 * @param paRegions Pointer to an array of shared region(s).
4533 */
4534GMMR0DECL(int) GMMR0RegisterSharedModule(PVM pVM, VMCPUID idCpu, VBOXOSFAMILY enmGuestOS, char *pszModuleName,
4535 char *pszVersion, RTGCPTR GCPtrModBase, uint32_t cbModule,
4536 uint32_t cRegions, struct VMMDEVSHAREDREGIONDESC const *paRegions)
4537{
4538#ifdef VBOX_WITH_PAGE_SHARING
4539 /*
4540 * Validate input and get the basics.
4541 *
4542 * Note! Turns out the module size does necessarily match the size of the
4543 * regions. (iTunes on XP)
4544 */
4545 PGMM pGMM;
4546 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4547 PGVM pGVM;
4548 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
4549 if (RT_FAILURE(rc))
4550 return rc;
4551
4552 if (RT_UNLIKELY(cRegions > VMMDEVSHAREDREGIONDESC_MAX))
4553 return VERR_GMM_TOO_MANY_REGIONS;
4554
4555 if (RT_UNLIKELY(cbModule == 0 || cbModule > _1G))
4556 return VERR_GMM_BAD_SHARED_MODULE_SIZE;
4557
4558 uint32_t cbTotal = 0;
4559 for (uint32_t i = 0; i < cRegions; i++)
4560 {
4561 if (RT_UNLIKELY(paRegions[i].cbRegion == 0 || paRegions[i].cbRegion > _1G))
4562 return VERR_GMM_SHARED_MODULE_BAD_REGIONS_SIZE;
4563
4564 cbTotal += paRegions[i].cbRegion;
4565 if (RT_UNLIKELY(cbTotal > _1G))
4566 return VERR_GMM_SHARED_MODULE_BAD_REGIONS_SIZE;
4567 }
4568
4569 AssertPtrReturn(pszModuleName, VERR_INVALID_POINTER);
4570 if (RT_UNLIKELY(!memchr(pszModuleName, '\0', GMM_SHARED_MODULE_MAX_NAME_STRING)))
4571 return VERR_GMM_MODULE_NAME_TOO_LONG;
4572
4573 AssertPtrReturn(pszVersion, VERR_INVALID_POINTER);
4574 if (RT_UNLIKELY(!memchr(pszVersion, '\0', GMM_SHARED_MODULE_MAX_VERSION_STRING)))
4575 return VERR_GMM_MODULE_NAME_TOO_LONG;
4576
4577 uint32_t const uHash = gmmR0ShModCalcHash(pszModuleName, pszVersion);
4578 Log(("GMMR0RegisterSharedModule %s %s base %RGv size %x hash %x\n", pszModuleName, pszVersion, GCPtrModBase, cbModule, uHash));
4579
4580 /*
4581 * Take the semaphore and do some more validations.
4582 */
4583 gmmR0MutexAcquire(pGMM);
4584 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
4585 {
4586 /*
4587 * Check if this module is already locally registered and register
4588 * it if it isn't. The base address is a unique module identifier
4589 * locally.
4590 */
4591 PGMMSHAREDMODULEPERVM pRecVM = (PGMMSHAREDMODULEPERVM)RTAvlGCPtrGet(&pGVM->gmm.s.pSharedModuleTree, GCPtrModBase);
4592 bool fNewModule = pRecVM == NULL;
4593 if (fNewModule)
4594 {
4595 rc = gmmR0ShModNewPerVM(pGVM, GCPtrModBase, cRegions, paRegions, &pRecVM);
4596 if (RT_SUCCESS(rc))
4597 {
4598 /*
4599 * Find a matching global module, register a new one if needed.
4600 */
4601 PGMMSHAREDMODULE pGblMod = gmmR0ShModFindGlobal(pGMM, uHash, cbModule, enmGuestOS, cRegions,
4602 pszModuleName, pszVersion, paRegions);
4603 if (!pGblMod)
4604 {
4605 Assert(fNewModule);
4606 rc = gmmR0ShModNewGlobal(pGMM, uHash, cbModule, enmGuestOS, cRegions,
4607 pszModuleName, pszVersion, paRegions, &pGblMod);
4608 if (RT_SUCCESS(rc))
4609 {
4610 pRecVM->pGlobalModule = pGblMod; /* (One referenced returned by gmmR0ShModNewGlobal.) */
4611 Log(("GMMR0RegisterSharedModule: new module %s %s\n", pszModuleName, pszVersion));
4612 }
4613 else
4614 gmmR0ShModDeletePerVM(pGMM, pGVM, pRecVM, true /*fRemove*/);
4615 }
4616 else
4617 {
4618 Assert(pGblMod->cUsers > 0 && pGblMod->cUsers < UINT32_MAX / 2);
4619 pGblMod->cUsers++;
4620 pRecVM->pGlobalModule = pGblMod;
4621
4622 Log(("GMMR0RegisterSharedModule: new per vm module %s %s, gbl users %d\n", pszModuleName, pszVersion, pGblMod->cUsers));
4623 }
4624 }
4625 }
4626 else
4627 {
4628 /*
4629 * Attempt to re-register an existing module.
4630 */
4631 PGMMSHAREDMODULE pGblMod = gmmR0ShModFindGlobal(pGMM, uHash, cbModule, enmGuestOS, cRegions,
4632 pszModuleName, pszVersion, paRegions);
4633 if (pRecVM->pGlobalModule == pGblMod)
4634 {
4635 Log(("GMMR0RegisterSharedModule: already registered %s %s, gbl users %d\n", pszModuleName, pszVersion, pGblMod->cUsers));
4636 rc = VINF_GMM_SHARED_MODULE_ALREADY_REGISTERED;
4637 }
4638 else
4639 {
4640 /** @todo may have to unregister+register when this happens in case it's caused
4641 * by VBoxService crashing and being restarted... */
4642 Log(("GMMR0RegisterSharedModule: Address clash!\n"
4643 " incoming at %RGvLB%#x %s %s rgns %u\n"
4644 " existing at %RGvLB%#x %s %s rgns %u\n",
4645 GCPtrModBase, cbModule, pszModuleName, pszVersion, cRegions,
4646 pRecVM->Core.Key, pRecVM->pGlobalModule->cbModule, pRecVM->pGlobalModule->szName,
4647 pRecVM->pGlobalModule->szVersion, pRecVM->pGlobalModule->cRegions));
4648 rc = VERR_GMM_SHARED_MODULE_ADDRESS_CLASH;
4649 }
4650 }
4651 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
4652 }
4653 else
4654 rc = VERR_GMM_IS_NOT_SANE;
4655
4656 gmmR0MutexRelease(pGMM);
4657 return rc;
4658#else
4659
4660 NOREF(pVM); NOREF(idCpu); NOREF(enmGuestOS); NOREF(pszModuleName); NOREF(pszVersion);
4661 NOREF(GCPtrModBase); NOREF(cbModule); NOREF(cRegions); NOREF(paRegions);
4662 return VERR_NOT_IMPLEMENTED;
4663#endif
4664}
4665
4666
4667/**
4668 * VMMR0 request wrapper for GMMR0RegisterSharedModule.
4669 *
4670 * @returns see GMMR0RegisterSharedModule.
4671 * @param pVM The cross context VM structure.
4672 * @param idCpu The VCPU id.
4673 * @param pReq Pointer to the request packet.
4674 */
4675GMMR0DECL(int) GMMR0RegisterSharedModuleReq(PVM pVM, VMCPUID idCpu, PGMMREGISTERSHAREDMODULEREQ pReq)
4676{
4677 /*
4678 * Validate input and pass it on.
4679 */
4680 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
4681 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
4682 AssertMsgReturn(pReq->Hdr.cbReq >= sizeof(*pReq) && pReq->Hdr.cbReq == RT_UOFFSETOF(GMMREGISTERSHAREDMODULEREQ, aRegions[pReq->cRegions]), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
4683
4684 /* Pass back return code in the request packet to preserve informational codes. (VMMR3CallR0 chokes on them) */
4685 pReq->rc = GMMR0RegisterSharedModule(pVM, idCpu, pReq->enmGuestOS, pReq->szName, pReq->szVersion,
4686 pReq->GCBaseAddr, pReq->cbModule, pReq->cRegions, pReq->aRegions);
4687 return VINF_SUCCESS;
4688}
4689
4690
4691/**
4692 * Unregisters a shared module for the VM
4693 *
4694 * @returns VBox status code.
4695 * @param pVM The cross context VM structure.
4696 * @param idCpu The VCPU id.
4697 * @param pszModuleName The module name.
4698 * @param pszVersion The module version.
4699 * @param GCPtrModBase The module base address.
4700 * @param cbModule The module size.
4701 */
4702GMMR0DECL(int) GMMR0UnregisterSharedModule(PVM pVM, VMCPUID idCpu, char *pszModuleName, char *pszVersion,
4703 RTGCPTR GCPtrModBase, uint32_t cbModule)
4704{
4705#ifdef VBOX_WITH_PAGE_SHARING
4706 /*
4707 * Validate input and get the basics.
4708 */
4709 PGMM pGMM;
4710 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4711 PGVM pGVM;
4712 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
4713 if (RT_FAILURE(rc))
4714 return rc;
4715
4716 AssertPtrReturn(pszModuleName, VERR_INVALID_POINTER);
4717 AssertPtrReturn(pszVersion, VERR_INVALID_POINTER);
4718 if (RT_UNLIKELY(!memchr(pszModuleName, '\0', GMM_SHARED_MODULE_MAX_NAME_STRING)))
4719 return VERR_GMM_MODULE_NAME_TOO_LONG;
4720 if (RT_UNLIKELY(!memchr(pszVersion, '\0', GMM_SHARED_MODULE_MAX_VERSION_STRING)))
4721 return VERR_GMM_MODULE_NAME_TOO_LONG;
4722
4723 Log(("GMMR0UnregisterSharedModule %s %s base=%RGv size %x\n", pszModuleName, pszVersion, GCPtrModBase, cbModule));
4724
4725 /*
4726 * Take the semaphore and do some more validations.
4727 */
4728 gmmR0MutexAcquire(pGMM);
4729 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
4730 {
4731 /*
4732 * Locate and remove the specified module.
4733 */
4734 PGMMSHAREDMODULEPERVM pRecVM = (PGMMSHAREDMODULEPERVM)RTAvlGCPtrGet(&pGVM->gmm.s.pSharedModuleTree, GCPtrModBase);
4735 if (pRecVM)
4736 {
4737 /** @todo Do we need to do more validations here, like that the
4738 * name + version + cbModule matches? */
4739 NOREF(cbModule);
4740 Assert(pRecVM->pGlobalModule);
4741 gmmR0ShModDeletePerVM(pGMM, pGVM, pRecVM, true /*fRemove*/);
4742 }
4743 else
4744 rc = VERR_GMM_SHARED_MODULE_NOT_FOUND;
4745
4746 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
4747 }
4748 else
4749 rc = VERR_GMM_IS_NOT_SANE;
4750
4751 gmmR0MutexRelease(pGMM);
4752 return rc;
4753#else
4754
4755 NOREF(pVM); NOREF(idCpu); NOREF(pszModuleName); NOREF(pszVersion); NOREF(GCPtrModBase); NOREF(cbModule);
4756 return VERR_NOT_IMPLEMENTED;
4757#endif
4758}
4759
4760
4761/**
4762 * VMMR0 request wrapper for GMMR0UnregisterSharedModule.
4763 *
4764 * @returns see GMMR0UnregisterSharedModule.
4765 * @param pVM The cross context VM structure.
4766 * @param idCpu The VCPU id.
4767 * @param pReq Pointer to the request packet.
4768 */
4769GMMR0DECL(int) GMMR0UnregisterSharedModuleReq(PVM pVM, VMCPUID idCpu, PGMMUNREGISTERSHAREDMODULEREQ pReq)
4770{
4771 /*
4772 * Validate input and pass it on.
4773 */
4774 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
4775 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
4776 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
4777
4778 return GMMR0UnregisterSharedModule(pVM, idCpu, pReq->szName, pReq->szVersion, pReq->GCBaseAddr, pReq->cbModule);
4779}
4780
4781#ifdef VBOX_WITH_PAGE_SHARING
4782
4783/**
4784 * Increase the use count of a shared page, the page is known to exist and be valid and such.
4785 *
4786 * @param pGMM Pointer to the GMM instance.
4787 * @param pGVM Pointer to the GVM instance.
4788 * @param pPage The page structure.
4789 */
4790DECLINLINE(void) gmmR0UseSharedPage(PGMM pGMM, PGVM pGVM, PGMMPAGE pPage)
4791{
4792 Assert(pGMM->cSharedPages > 0);
4793 Assert(pGMM->cAllocatedPages > 0);
4794
4795 pGMM->cDuplicatePages++;
4796
4797 pPage->Shared.cRefs++;
4798 pGVM->gmm.s.Stats.cSharedPages++;
4799 pGVM->gmm.s.Stats.Allocated.cBasePages++;
4800}
4801
4802
4803/**
4804 * Converts a private page to a shared page, the page is known to exist and be valid and such.
4805 *
4806 * @param pGMM Pointer to the GMM instance.
4807 * @param pGVM Pointer to the GVM instance.
4808 * @param HCPhys Host physical address
4809 * @param idPage The Page ID
4810 * @param pPage The page structure.
4811 * @param pPageDesc Shared page descriptor
4812 */
4813DECLINLINE(void) gmmR0ConvertToSharedPage(PGMM pGMM, PGVM pGVM, RTHCPHYS HCPhys, uint32_t idPage, PGMMPAGE pPage,
4814 PGMMSHAREDPAGEDESC pPageDesc)
4815{
4816 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
4817 Assert(pChunk);
4818 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
4819 Assert(GMM_PAGE_IS_PRIVATE(pPage));
4820
4821 pChunk->cPrivate--;
4822 pChunk->cShared++;
4823
4824 pGMM->cSharedPages++;
4825
4826 pGVM->gmm.s.Stats.cSharedPages++;
4827 pGVM->gmm.s.Stats.cPrivatePages--;
4828
4829 /* Modify the page structure. */
4830 pPage->Shared.pfn = (uint32_t)(uint64_t)(HCPhys >> PAGE_SHIFT);
4831 pPage->Shared.cRefs = 1;
4832#ifdef VBOX_STRICT
4833 pPageDesc->u32StrictChecksum = gmmR0StrictPageChecksum(pGMM, pGVM, idPage);
4834 pPage->Shared.u14Checksum = pPageDesc->u32StrictChecksum;
4835#else
4836 NOREF(pPageDesc);
4837 pPage->Shared.u14Checksum = 0;
4838#endif
4839 pPage->Shared.u2State = GMM_PAGE_STATE_SHARED;
4840}
4841
4842
4843static int gmmR0SharedModuleCheckPageFirstTime(PGMM pGMM, PGVM pGVM, PGMMSHAREDMODULE pModule,
4844 unsigned idxRegion, unsigned idxPage,
4845 PGMMSHAREDPAGEDESC pPageDesc, PGMMSHAREDREGIONDESC pGlobalRegion)
4846{
4847 NOREF(pModule);
4848
4849 /* Easy case: just change the internal page type. */
4850 PGMMPAGE pPage = gmmR0GetPage(pGMM, pPageDesc->idPage);
4851 AssertMsgReturn(pPage, ("idPage=%#x (GCPhys=%RGp HCPhys=%RHp idxRegion=%#x idxPage=%#x) #1\n",
4852 pPageDesc->idPage, pPageDesc->GCPhys, pPageDesc->HCPhys, idxRegion, idxPage),
4853 VERR_PGM_PHYS_INVALID_PAGE_ID);
4854 NOREF(idxRegion);
4855
4856 AssertMsg(pPageDesc->GCPhys == (pPage->Private.pfn << 12), ("desc %RGp gmm %RGp\n", pPageDesc->HCPhys, (pPage->Private.pfn << 12)));
4857
4858 gmmR0ConvertToSharedPage(pGMM, pGVM, pPageDesc->HCPhys, pPageDesc->idPage, pPage, pPageDesc);
4859
4860 /* Keep track of these references. */
4861 pGlobalRegion->paidPages[idxPage] = pPageDesc->idPage;
4862
4863 return VINF_SUCCESS;
4864}
4865
4866/**
4867 * Checks specified shared module range for changes
4868 *
4869 * Performs the following tasks:
4870 * - If a shared page is new, then it changes the GMM page type to shared and
4871 * returns it in the pPageDesc descriptor.
4872 * - If a shared page already exists, then it checks if the VM page is
4873 * identical and if so frees the VM page and returns the shared page in
4874 * pPageDesc descriptor.
4875 *
4876 * @remarks ASSUMES the caller has acquired the GMM semaphore!!
4877 *
4878 * @returns VBox status code.
4879 * @param pGVM Pointer to the GVM instance data.
4880 * @param pModule Module description
4881 * @param idxRegion Region index
4882 * @param idxPage Page index
4883 * @param pPageDesc Page descriptor
4884 */
4885GMMR0DECL(int) GMMR0SharedModuleCheckPage(PGVM pGVM, PGMMSHAREDMODULE pModule, uint32_t idxRegion, uint32_t idxPage,
4886 PGMMSHAREDPAGEDESC pPageDesc)
4887{
4888 int rc;
4889 PGMM pGMM;
4890 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4891 pPageDesc->u32StrictChecksum = 0;
4892
4893 AssertMsgReturn(idxRegion < pModule->cRegions,
4894 ("idxRegion=%#x cRegions=%#x %s %s\n", idxRegion, pModule->cRegions, pModule->szName, pModule->szVersion),
4895 VERR_INVALID_PARAMETER);
4896
4897 uint32_t const cPages = pModule->aRegions[idxRegion].cb >> PAGE_SHIFT;
4898 AssertMsgReturn(idxPage < cPages,
4899 ("idxRegion=%#x cRegions=%#x %s %s\n", idxRegion, pModule->cRegions, pModule->szName, pModule->szVersion),
4900 VERR_INVALID_PARAMETER);
4901
4902 LogFlow(("GMMR0SharedModuleCheckRange %s base %RGv region %d idxPage %d\n", pModule->szName, pModule->Core.Key, idxRegion, idxPage));
4903
4904 /*
4905 * First time; create a page descriptor array.
4906 */
4907 PGMMSHAREDREGIONDESC pGlobalRegion = &pModule->aRegions[idxRegion];
4908 if (!pGlobalRegion->paidPages)
4909 {
4910 Log(("Allocate page descriptor array for %d pages\n", cPages));
4911 pGlobalRegion->paidPages = (uint32_t *)RTMemAlloc(cPages * sizeof(pGlobalRegion->paidPages[0]));
4912 AssertReturn(pGlobalRegion->paidPages, VERR_NO_MEMORY);
4913
4914 /* Invalidate all descriptors. */
4915 uint32_t i = cPages;
4916 while (i-- > 0)
4917 pGlobalRegion->paidPages[i] = NIL_GMM_PAGEID;
4918 }
4919
4920 /*
4921 * We've seen this shared page for the first time?
4922 */
4923 if (pGlobalRegion->paidPages[idxPage] == NIL_GMM_PAGEID)
4924 {
4925 Log(("New shared page guest %RGp host %RHp\n", pPageDesc->GCPhys, pPageDesc->HCPhys));
4926 return gmmR0SharedModuleCheckPageFirstTime(pGMM, pGVM, pModule, idxRegion, idxPage, pPageDesc, pGlobalRegion);
4927 }
4928
4929 /*
4930 * We've seen it before...
4931 */
4932 Log(("Replace existing page guest %RGp host %RHp id %#x -> id %#x\n",
4933 pPageDesc->GCPhys, pPageDesc->HCPhys, pPageDesc->idPage, pGlobalRegion->paidPages[idxPage]));
4934 Assert(pPageDesc->idPage != pGlobalRegion->paidPages[idxPage]);
4935
4936 /*
4937 * Get the shared page source.
4938 */
4939 PGMMPAGE pPage = gmmR0GetPage(pGMM, pGlobalRegion->paidPages[idxPage]);
4940 AssertMsgReturn(pPage, ("idPage=%#x (idxRegion=%#x idxPage=%#x) #2\n", pPageDesc->idPage, idxRegion, idxPage),
4941 VERR_PGM_PHYS_INVALID_PAGE_ID);
4942
4943 if (pPage->Common.u2State != GMM_PAGE_STATE_SHARED)
4944 {
4945 /*
4946 * Page was freed at some point; invalidate this entry.
4947 */
4948 /** @todo this isn't really bullet proof. */
4949 Log(("Old shared page was freed -> create a new one\n"));
4950 pGlobalRegion->paidPages[idxPage] = NIL_GMM_PAGEID;
4951 return gmmR0SharedModuleCheckPageFirstTime(pGMM, pGVM, pModule, idxRegion, idxPage, pPageDesc, pGlobalRegion);
4952 }
4953
4954 Log(("Replace existing page guest host %RHp -> %RHp\n", pPageDesc->HCPhys, ((uint64_t)pPage->Shared.pfn) << PAGE_SHIFT));
4955
4956 /*
4957 * Calculate the virtual address of the local page.
4958 */
4959 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, pPageDesc->idPage >> GMM_CHUNKID_SHIFT);
4960 AssertMsgReturn(pChunk, ("idPage=%#x (idxRegion=%#x idxPage=%#x) #4\n", pPageDesc->idPage, idxRegion, idxPage),
4961 VERR_PGM_PHYS_INVALID_PAGE_ID);
4962
4963 uint8_t *pbChunk;
4964 AssertMsgReturn(gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk),
4965 ("idPage=%#x (idxRegion=%#x idxPage=%#x) #3\n", pPageDesc->idPage, idxRegion, idxPage),
4966 VERR_PGM_PHYS_INVALID_PAGE_ID);
4967 uint8_t *pbLocalPage = pbChunk + ((pPageDesc->idPage & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
4968
4969 /*
4970 * Calculate the virtual address of the shared page.
4971 */
4972 pChunk = gmmR0GetChunk(pGMM, pGlobalRegion->paidPages[idxPage] >> GMM_CHUNKID_SHIFT);
4973 Assert(pChunk); /* can't fail as gmmR0GetPage succeeded. */
4974
4975 /*
4976 * Get the virtual address of the physical page; map the chunk into the VM
4977 * process if not already done.
4978 */
4979 if (!gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
4980 {
4981 Log(("Map chunk into process!\n"));
4982 rc = gmmR0MapChunk(pGMM, pGVM, pChunk, false /*fRelaxedSem*/, (PRTR3PTR)&pbChunk);
4983 AssertRCReturn(rc, rc);
4984 }
4985 uint8_t *pbSharedPage = pbChunk + ((pGlobalRegion->paidPages[idxPage] & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
4986
4987#ifdef VBOX_STRICT
4988 pPageDesc->u32StrictChecksum = RTCrc32(pbSharedPage, PAGE_SIZE);
4989 uint32_t uChecksum = pPageDesc->u32StrictChecksum & UINT32_C(0x00003fff);
4990 AssertMsg(!uChecksum || uChecksum == pPage->Shared.u14Checksum || !pPage->Shared.u14Checksum,
4991 ("%#x vs %#x - idPage=%#x - %s %s\n", uChecksum, pPage->Shared.u14Checksum,
4992 pGlobalRegion->paidPages[idxPage], pModule->szName, pModule->szVersion));
4993#endif
4994
4995 /** @todo write ASMMemComparePage. */
4996 if (memcmp(pbSharedPage, pbLocalPage, PAGE_SIZE))
4997 {
4998 Log(("Unexpected differences found between local and shared page; skip\n"));
4999 /* Signal to the caller that this one hasn't changed. */
5000 pPageDesc->idPage = NIL_GMM_PAGEID;
5001 return VINF_SUCCESS;
5002 }
5003
5004 /*
5005 * Free the old local page.
5006 */
5007 GMMFREEPAGEDESC PageDesc;
5008 PageDesc.idPage = pPageDesc->idPage;
5009 rc = gmmR0FreePages(pGMM, pGVM, 1, &PageDesc, GMMACCOUNT_BASE);
5010 AssertRCReturn(rc, rc);
5011
5012 gmmR0UseSharedPage(pGMM, pGVM, pPage);
5013
5014 /*
5015 * Pass along the new physical address & page id.
5016 */
5017 pPageDesc->HCPhys = ((uint64_t)pPage->Shared.pfn) << PAGE_SHIFT;
5018 pPageDesc->idPage = pGlobalRegion->paidPages[idxPage];
5019
5020 return VINF_SUCCESS;
5021}
5022
5023
5024/**
5025 * RTAvlGCPtrDestroy callback.
5026 *
5027 * @returns 0 or VERR_GMM_INSTANCE.
5028 * @param pNode The node to destroy.
5029 * @param pvArgs Pointer to an argument packet.
5030 */
5031static DECLCALLBACK(int) gmmR0CleanupSharedModule(PAVLGCPTRNODECORE pNode, void *pvArgs)
5032{
5033 gmmR0ShModDeletePerVM(((GMMR0SHMODPERVMDTORARGS *)pvArgs)->pGMM,
5034 ((GMMR0SHMODPERVMDTORARGS *)pvArgs)->pGVM,
5035 (PGMMSHAREDMODULEPERVM)pNode,
5036 false /*fRemove*/);
5037 return VINF_SUCCESS;
5038}
5039
5040
5041/**
5042 * Used by GMMR0CleanupVM to clean up shared modules.
5043 *
5044 * This is called without taking the GMM lock so that it can be yielded as
5045 * needed here.
5046 *
5047 * @param pGMM The GMM handle.
5048 * @param pGVM The global VM handle.
5049 */
5050static void gmmR0SharedModuleCleanup(PGMM pGMM, PGVM pGVM)
5051{
5052 gmmR0MutexAcquire(pGMM);
5053 GMM_CHECK_SANITY_UPON_ENTERING(pGMM);
5054
5055 GMMR0SHMODPERVMDTORARGS Args;
5056 Args.pGVM = pGVM;
5057 Args.pGMM = pGMM;
5058 RTAvlGCPtrDestroy(&pGVM->gmm.s.pSharedModuleTree, gmmR0CleanupSharedModule, &Args);
5059
5060 AssertMsg(pGVM->gmm.s.Stats.cShareableModules == 0, ("%d\n", pGVM->gmm.s.Stats.cShareableModules));
5061 pGVM->gmm.s.Stats.cShareableModules = 0;
5062
5063 gmmR0MutexRelease(pGMM);
5064}
5065
5066#endif /* VBOX_WITH_PAGE_SHARING */
5067
5068/**
5069 * Removes all shared modules for the specified VM
5070 *
5071 * @returns VBox status code.
5072 * @param pVM The cross context VM structure.
5073 * @param idCpu The VCPU id.
5074 */
5075GMMR0DECL(int) GMMR0ResetSharedModules(PVM pVM, VMCPUID idCpu)
5076{
5077#ifdef VBOX_WITH_PAGE_SHARING
5078 /*
5079 * Validate input and get the basics.
5080 */
5081 PGMM pGMM;
5082 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5083 PGVM pGVM;
5084 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
5085 if (RT_FAILURE(rc))
5086 return rc;
5087
5088 /*
5089 * Take the semaphore and do some more validations.
5090 */
5091 gmmR0MutexAcquire(pGMM);
5092 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
5093 {
5094 Log(("GMMR0ResetSharedModules\n"));
5095 GMMR0SHMODPERVMDTORARGS Args;
5096 Args.pGVM = pGVM;
5097 Args.pGMM = pGMM;
5098 RTAvlGCPtrDestroy(&pGVM->gmm.s.pSharedModuleTree, gmmR0CleanupSharedModule, &Args);
5099 pGVM->gmm.s.Stats.cShareableModules = 0;
5100
5101 rc = VINF_SUCCESS;
5102 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
5103 }
5104 else
5105 rc = VERR_GMM_IS_NOT_SANE;
5106
5107 gmmR0MutexRelease(pGMM);
5108 return rc;
5109#else
5110 NOREF(pVM); NOREF(idCpu);
5111 return VERR_NOT_IMPLEMENTED;
5112#endif
5113}
5114
5115#ifdef VBOX_WITH_PAGE_SHARING
5116
5117/**
5118 * Tree enumeration callback for checking a shared module.
5119 */
5120static DECLCALLBACK(int) gmmR0CheckSharedModule(PAVLGCPTRNODECORE pNode, void *pvUser)
5121{
5122 GMMCHECKSHAREDMODULEINFO *pArgs = (GMMCHECKSHAREDMODULEINFO*)pvUser;
5123 PGMMSHAREDMODULEPERVM pRecVM = (PGMMSHAREDMODULEPERVM)pNode;
5124 PGMMSHAREDMODULE pGblMod = pRecVM->pGlobalModule;
5125
5126 Log(("gmmR0CheckSharedModule: check %s %s base=%RGv size=%x\n",
5127 pGblMod->szName, pGblMod->szVersion, pGblMod->Core.Key, pGblMod->cbModule));
5128
5129 int rc = PGMR0SharedModuleCheck(pArgs->pGVM->pVM, pArgs->pGVM, pArgs->idCpu, pGblMod, pRecVM->aRegionsGCPtrs);
5130 if (RT_FAILURE(rc))
5131 return rc;
5132 return VINF_SUCCESS;
5133}
5134
5135#endif /* VBOX_WITH_PAGE_SHARING */
5136#ifdef DEBUG_sandervl
5137
5138/**
5139 * Setup for a GMMR0CheckSharedModules call (to allow log flush jumps back to ring 3)
5140 *
5141 * @returns VBox status code.
5142 * @param pVM The cross context VM structure.
5143 */
5144GMMR0DECL(int) GMMR0CheckSharedModulesStart(PVM pVM)
5145{
5146 /*
5147 * Validate input and get the basics.
5148 */
5149 PGMM pGMM;
5150 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5151
5152 /*
5153 * Take the semaphore and do some more validations.
5154 */
5155 gmmR0MutexAcquire(pGMM);
5156 if (!GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
5157 rc = VERR_GMM_IS_NOT_SANE;
5158 else
5159 rc = VINF_SUCCESS;
5160
5161 return rc;
5162}
5163
5164/**
5165 * Clean up after a GMMR0CheckSharedModules call (to allow log flush jumps back to ring 3)
5166 *
5167 * @returns VBox status code.
5168 * @param pVM The cross context VM structure.
5169 */
5170GMMR0DECL(int) GMMR0CheckSharedModulesEnd(PVM pVM)
5171{
5172 /*
5173 * Validate input and get the basics.
5174 */
5175 PGMM pGMM;
5176 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5177
5178 gmmR0MutexRelease(pGMM);
5179 return VINF_SUCCESS;
5180}
5181
5182#endif /* DEBUG_sandervl */
5183
5184/**
5185 * Check all shared modules for the specified VM.
5186 *
5187 * @returns VBox status code.
5188 * @param pVM The cross context VM structure.
5189 * @param pVCpu The cross context virtual CPU structure.
5190 */
5191GMMR0DECL(int) GMMR0CheckSharedModules(PVM pVM, PVMCPU pVCpu)
5192{
5193#ifdef VBOX_WITH_PAGE_SHARING
5194 /*
5195 * Validate input and get the basics.
5196 */
5197 PGMM pGMM;
5198 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5199 PGVM pGVM;
5200 int rc = GVMMR0ByVMAndEMT(pVM, pVCpu->idCpu, &pGVM);
5201 if (RT_FAILURE(rc))
5202 return rc;
5203
5204# ifndef DEBUG_sandervl
5205 /*
5206 * Take the semaphore and do some more validations.
5207 */
5208 gmmR0MutexAcquire(pGMM);
5209# endif
5210 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
5211 {
5212 /*
5213 * Walk the tree, checking each module.
5214 */
5215 Log(("GMMR0CheckSharedModules\n"));
5216
5217 GMMCHECKSHAREDMODULEINFO Args;
5218 Args.pGVM = pGVM;
5219 Args.idCpu = pVCpu->idCpu;
5220 rc = RTAvlGCPtrDoWithAll(&pGVM->gmm.s.pSharedModuleTree, true /* fFromLeft */, gmmR0CheckSharedModule, &Args);
5221
5222 Log(("GMMR0CheckSharedModules done (rc=%Rrc)!\n", rc));
5223 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
5224 }
5225 else
5226 rc = VERR_GMM_IS_NOT_SANE;
5227
5228# ifndef DEBUG_sandervl
5229 gmmR0MutexRelease(pGMM);
5230# endif
5231 return rc;
5232#else
5233 NOREF(pVM); NOREF(pVCpu);
5234 return VERR_NOT_IMPLEMENTED;
5235#endif
5236}
5237
5238#if defined(VBOX_STRICT) && HC_ARCH_BITS == 64
5239
5240/**
5241 * RTAvlU32DoWithAll callback.
5242 *
5243 * @returns 0
5244 * @param pNode The node to search.
5245 * @param pvUser Pointer to the input argument packet.
5246 */
5247static DECLCALLBACK(int) gmmR0FindDupPageInChunk(PAVLU32NODECORE pNode, void *pvUser)
5248{
5249 PGMMCHUNK pChunk = (PGMMCHUNK)pNode;
5250 GMMFINDDUPPAGEINFO *pArgs = (GMMFINDDUPPAGEINFO *)pvUser;
5251 PGVM pGVM = pArgs->pGVM;
5252 PGMM pGMM = pArgs->pGMM;
5253 uint8_t *pbChunk;
5254
5255 /* Only take chunks not mapped into this VM process; not entirely correct. */
5256 if (!gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
5257 {
5258 int rc = gmmR0MapChunk(pGMM, pGVM, pChunk, false /*fRelaxedSem*/, (PRTR3PTR)&pbChunk);
5259 if (RT_SUCCESS(rc))
5260 {
5261 /*
5262 * Look for duplicate pages
5263 */
5264 unsigned iPage = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
5265 while (iPage-- > 0)
5266 {
5267 if (GMM_PAGE_IS_PRIVATE(&pChunk->aPages[iPage]))
5268 {
5269 uint8_t *pbDestPage = pbChunk + (iPage << PAGE_SHIFT);
5270
5271 if (!memcmp(pArgs->pSourcePage, pbDestPage, PAGE_SIZE))
5272 {
5273 pArgs->fFoundDuplicate = true;
5274 break;
5275 }
5276 }
5277 }
5278 gmmR0UnmapChunk(pGMM, pGVM, pChunk, false /*fRelaxedSem*/);
5279 }
5280 }
5281 return pArgs->fFoundDuplicate; /* (stops search if true) */
5282}
5283
5284
5285/**
5286 * Find a duplicate of the specified page in other active VMs
5287 *
5288 * @returns VBox status code.
5289 * @param pVM The cross context VM structure.
5290 * @param pReq Pointer to the request packet.
5291 */
5292GMMR0DECL(int) GMMR0FindDuplicatePageReq(PVM pVM, PGMMFINDDUPLICATEPAGEREQ pReq)
5293{
5294 /*
5295 * Validate input and pass it on.
5296 */
5297 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
5298 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
5299 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
5300
5301 PGMM pGMM;
5302 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5303
5304 PGVM pGVM;
5305 int rc = GVMMR0ByVM(pVM, &pGVM);
5306 if (RT_FAILURE(rc))
5307 return rc;
5308
5309 /*
5310 * Take the semaphore and do some more validations.
5311 */
5312 rc = gmmR0MutexAcquire(pGMM);
5313 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
5314 {
5315 uint8_t *pbChunk;
5316 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, pReq->idPage >> GMM_CHUNKID_SHIFT);
5317 if (pChunk)
5318 {
5319 if (gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
5320 {
5321 uint8_t *pbSourcePage = pbChunk + ((pReq->idPage & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
5322 PGMMPAGE pPage = gmmR0GetPage(pGMM, pReq->idPage);
5323 if (pPage)
5324 {
5325 GMMFINDDUPPAGEINFO Args;
5326 Args.pGVM = pGVM;
5327 Args.pGMM = pGMM;
5328 Args.pSourcePage = pbSourcePage;
5329 Args.fFoundDuplicate = false;
5330 RTAvlU32DoWithAll(&pGMM->pChunks, true /* fFromLeft */, gmmR0FindDupPageInChunk, &Args);
5331
5332 pReq->fDuplicate = Args.fFoundDuplicate;
5333 }
5334 else
5335 {
5336 AssertFailed();
5337 rc = VERR_PGM_PHYS_INVALID_PAGE_ID;
5338 }
5339 }
5340 else
5341 AssertFailed();
5342 }
5343 else
5344 AssertFailed();
5345 }
5346 else
5347 rc = VERR_GMM_IS_NOT_SANE;
5348
5349 gmmR0MutexRelease(pGMM);
5350 return rc;
5351}
5352
5353#endif /* VBOX_STRICT && HC_ARCH_BITS == 64 */
5354
5355
5356/**
5357 * Retrieves the GMM statistics visible to the caller.
5358 *
5359 * @returns VBox status code.
5360 *
5361 * @param pStats Where to put the statistics.
5362 * @param pSession The current session.
5363 * @param pVM The VM to obtain statistics for. Optional.
5364 */
5365GMMR0DECL(int) GMMR0QueryStatistics(PGMMSTATS pStats, PSUPDRVSESSION pSession, PVM pVM)
5366{
5367 LogFlow(("GVMMR0QueryStatistics: pStats=%p pSession=%p pVM=%p\n", pStats, pSession, pVM));
5368
5369 /*
5370 * Validate input.
5371 */
5372 AssertPtrReturn(pSession, VERR_INVALID_POINTER);
5373 AssertPtrReturn(pStats, VERR_INVALID_POINTER);
5374 pStats->cMaxPages = 0; /* (crash before taking the mutex...) */
5375
5376 PGMM pGMM;
5377 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5378
5379 /*
5380 * Resolve the VM handle, if not NULL, and lock the GMM.
5381 */
5382 int rc;
5383 PGVM pGVM;
5384 if (pVM)
5385 {
5386 rc = GVMMR0ByVM(pVM, &pGVM);
5387 if (RT_FAILURE(rc))
5388 return rc;
5389 }
5390 else
5391 pGVM = NULL;
5392
5393 rc = gmmR0MutexAcquire(pGMM);
5394 if (RT_FAILURE(rc))
5395 return rc;
5396
5397 /*
5398 * Copy out the GMM statistics.
5399 */
5400 pStats->cMaxPages = pGMM->cMaxPages;
5401 pStats->cReservedPages = pGMM->cReservedPages;
5402 pStats->cOverCommittedPages = pGMM->cOverCommittedPages;
5403 pStats->cAllocatedPages = pGMM->cAllocatedPages;
5404 pStats->cSharedPages = pGMM->cSharedPages;
5405 pStats->cDuplicatePages = pGMM->cDuplicatePages;
5406 pStats->cLeftBehindSharedPages = pGMM->cLeftBehindSharedPages;
5407 pStats->cBalloonedPages = pGMM->cBalloonedPages;
5408 pStats->cChunks = pGMM->cChunks;
5409 pStats->cFreedChunks = pGMM->cFreedChunks;
5410 pStats->cShareableModules = pGMM->cShareableModules;
5411 RT_ZERO(pStats->au64Reserved);
5412
5413 /*
5414 * Copy out the VM statistics.
5415 */
5416 if (pGVM)
5417 pStats->VMStats = pGVM->gmm.s.Stats;
5418 else
5419 RT_ZERO(pStats->VMStats);
5420
5421 gmmR0MutexRelease(pGMM);
5422 return rc;
5423}
5424
5425
5426/**
5427 * VMMR0 request wrapper for GMMR0QueryStatistics.
5428 *
5429 * @returns see GMMR0QueryStatistics.
5430 * @param pVM The cross context VM structure. Optional.
5431 * @param pReq Pointer to the request packet.
5432 */
5433GMMR0DECL(int) GMMR0QueryStatisticsReq(PVM pVM, PGMMQUERYSTATISTICSSREQ pReq)
5434{
5435 /*
5436 * Validate input and pass it on.
5437 */
5438 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
5439 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
5440
5441 return GMMR0QueryStatistics(&pReq->Stats, pReq->pSession, pVM);
5442}
5443
5444
5445/**
5446 * Resets the specified GMM statistics.
5447 *
5448 * @returns VBox status code.
5449 *
5450 * @param pStats Which statistics to reset, that is, non-zero fields
5451 * indicates which to reset.
5452 * @param pSession The current session.
5453 * @param pVM The VM to reset statistics for. Optional.
5454 */
5455GMMR0DECL(int) GMMR0ResetStatistics(PCGMMSTATS pStats, PSUPDRVSESSION pSession, PVM pVM)
5456{
5457 NOREF(pStats); NOREF(pSession); NOREF(pVM);
5458 /* Currently nothing we can reset at the moment. */
5459 return VINF_SUCCESS;
5460}
5461
5462
5463/**
5464 * VMMR0 request wrapper for GMMR0ResetStatistics.
5465 *
5466 * @returns see GMMR0ResetStatistics.
5467 * @param pVM The cross context VM structure. Optional.
5468 * @param pReq Pointer to the request packet.
5469 */
5470GMMR0DECL(int) GMMR0ResetStatisticsReq(PVM pVM, PGMMRESETSTATISTICSSREQ pReq)
5471{
5472 /*
5473 * Validate input and pass it on.
5474 */
5475 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
5476 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
5477
5478 return GMMR0ResetStatistics(&pReq->Stats, pReq->pSession, pVM);
5479}
5480
Note: See TracBrowser for help on using the repository browser.

© 2024 Oracle Support Privacy / Do Not Sell My Info Terms of Use Trademark Policy Automated Access Etiquette