VirtualBox

source: vbox/trunk/src/VBox/VMM/VMMR0/GMMR0.cpp@ 75649

Last change on this file since 75649 was 73097, checked in by vboxsync, 6 years ago

*: Made RT_UOFFSETOF, RT_OFFSETOF, RT_UOFFSETOF_ADD and RT_OFFSETOF_ADD work like builtin_offsetof() and require compile time resolvable requests, adding RT_UOFFSETOF_DYN for the dynamic questions that can only be answered at runtime.
  • Property svn:eol-style set to native
  • Property svn:keywords set to Id Revision
File size: 191.9 KB
Line 
1/* $Id: GMMR0.cpp 73097 2018-07-12 21:06:33Z vboxsync $ */
2/** @file
3 * GMM - Global Memory Manager.
4 */
5
6/*
7 * Copyright (C) 2007-2017 Oracle Corporation
8 *
9 * This file is part of VirtualBox Open Source Edition (OSE), as
10 * available from http://www.virtualbox.org. This file is free software;
11 * you can redistribute it and/or modify it under the terms of the GNU
12 * General Public License (GPL) as published by the Free Software
13 * Foundation, in version 2 as it comes in the "COPYING" file of the
14 * VirtualBox OSE distribution. VirtualBox OSE is distributed in the
15 * hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
16 */
17
18
19/** @page pg_gmm GMM - The Global Memory Manager
20 *
21 * As the name indicates, this component is responsible for global memory
22 * management. Currently only guest RAM is allocated from the GMM, but this
23 * may change to include shadow page tables and other bits later.
24 *
25 * Guest RAM is managed as individual pages, but allocated from the host OS
26 * in chunks for reasons of portability / efficiency. To minimize the memory
27 * footprint all tracking structure must be as small as possible without
28 * unnecessary performance penalties.
29 *
30 * The allocation chunks has fixed sized, the size defined at compile time
31 * by the #GMM_CHUNK_SIZE \#define.
32 *
33 * Each chunk is given an unique ID. Each page also has a unique ID. The
34 * relationship between the two IDs is:
35 * @code
36 * GMM_CHUNK_SHIFT = log2(GMM_CHUNK_SIZE / PAGE_SIZE);
37 * idPage = (idChunk << GMM_CHUNK_SHIFT) | iPage;
38 * @endcode
39 * Where iPage is the index of the page within the chunk. This ID scheme
40 * permits for efficient chunk and page lookup, but it relies on the chunk size
41 * to be set at compile time. The chunks are organized in an AVL tree with their
42 * IDs being the keys.
43 *
44 * The physical address of each page in an allocation chunk is maintained by
45 * the #RTR0MEMOBJ and obtained using #RTR0MemObjGetPagePhysAddr. There is no
46 * need to duplicate this information (it'll cost 8-bytes per page if we did).
47 *
48 * So what do we need to track per page? Most importantly we need to know
49 * which state the page is in:
50 * - Private - Allocated for (eventually) backing one particular VM page.
51 * - Shared - Readonly page that is used by one or more VMs and treated
52 * as COW by PGM.
53 * - Free - Not used by anyone.
54 *
55 * For the page replacement operations (sharing, defragmenting and freeing)
56 * to be somewhat efficient, private pages needs to be associated with a
57 * particular page in a particular VM.
58 *
59 * Tracking the usage of shared pages is impractical and expensive, so we'll
60 * settle for a reference counting system instead.
61 *
62 * Free pages will be chained on LIFOs
63 *
64 * On 64-bit systems we will use a 64-bit bitfield per page, while on 32-bit
65 * systems a 32-bit bitfield will have to suffice because of address space
66 * limitations. The #GMMPAGE structure shows the details.
67 *
68 *
69 * @section sec_gmm_alloc_strat Page Allocation Strategy
70 *
71 * The strategy for allocating pages has to take fragmentation and shared
72 * pages into account, or we may end up with with 2000 chunks with only
73 * a few pages in each. Shared pages cannot easily be reallocated because
74 * of the inaccurate usage accounting (see above). Private pages can be
75 * reallocated by a defragmentation thread in the same manner that sharing
76 * is done.
77 *
78 * The first approach is to manage the free pages in two sets depending on
79 * whether they are mainly for the allocation of shared or private pages.
80 * In the initial implementation there will be almost no possibility for
81 * mixing shared and private pages in the same chunk (only if we're really
82 * stressed on memory), but when we implement forking of VMs and have to
83 * deal with lots of COW pages it'll start getting kind of interesting.
84 *
85 * The sets are lists of chunks with approximately the same number of
86 * free pages. Say the chunk size is 1MB, meaning 256 pages, and a set
87 * consists of 16 lists. So, the first list will contain the chunks with
88 * 1-7 free pages, the second covers 8-15, and so on. The chunks will be
89 * moved between the lists as pages are freed up or allocated.
90 *
91 *
92 * @section sec_gmm_costs Costs
93 *
94 * The per page cost in kernel space is 32-bit plus whatever RTR0MEMOBJ
95 * entails. In addition there is the chunk cost of approximately
96 * (sizeof(RT0MEMOBJ) + sizeof(CHUNK)) / 2^CHUNK_SHIFT bytes per page.
97 *
98 * On Windows the per page #RTR0MEMOBJ cost is 32-bit on 32-bit windows
99 * and 64-bit on 64-bit windows (a PFN_NUMBER in the MDL). So, 64-bit per page.
100 * The cost on Linux is identical, but here it's because of sizeof(struct page *).
101 *
102 *
103 * @section sec_gmm_legacy Legacy Mode for Non-Tier-1 Platforms
104 *
105 * In legacy mode the page source is locked user pages and not
106 * #RTR0MemObjAllocPhysNC, this means that a page can only be allocated
107 * by the VM that locked it. We will make no attempt at implementing
108 * page sharing on these systems, just do enough to make it all work.
109 *
110 *
111 * @subsection sub_gmm_locking Serializing
112 *
113 * One simple fast mutex will be employed in the initial implementation, not
114 * two as mentioned in @ref sec_pgmPhys_Serializing.
115 *
116 * @see @ref sec_pgmPhys_Serializing
117 *
118 *
119 * @section sec_gmm_overcommit Memory Over-Commitment Management
120 *
121 * The GVM will have to do the system wide memory over-commitment
122 * management. My current ideas are:
123 * - Per VM oc policy that indicates how much to initially commit
124 * to it and what to do in a out-of-memory situation.
125 * - Prevent overtaxing the host.
126 *
127 * There are some challenges here, the main ones are configurability and
128 * security. Should we for instance permit anyone to request 100% memory
129 * commitment? Who should be allowed to do runtime adjustments of the
130 * config. And how to prevent these settings from being lost when the last
131 * VM process exits? The solution is probably to have an optional root
132 * daemon the will keep VMMR0.r0 in memory and enable the security measures.
133 *
134 *
135 *
136 * @section sec_gmm_numa NUMA
137 *
138 * NUMA considerations will be designed and implemented a bit later.
139 *
140 * The preliminary guesses is that we will have to try allocate memory as
141 * close as possible to the CPUs the VM is executed on (EMT and additional CPU
142 * threads). Which means it's mostly about allocation and sharing policies.
143 * Both the scheduler and allocator interface will to supply some NUMA info
144 * and we'll need to have a way to calc access costs.
145 *
146 */
147
148
149/*********************************************************************************************************************************
150* Header Files *
151*********************************************************************************************************************************/
152#define LOG_GROUP LOG_GROUP_GMM
153#include <VBox/rawpci.h>
154#include <VBox/vmm/vm.h>
155#include <VBox/vmm/gmm.h>
156#include "GMMR0Internal.h"
157#include <VBox/vmm/gvm.h>
158#include <VBox/vmm/pgm.h>
159#include <VBox/log.h>
160#include <VBox/param.h>
161#include <VBox/err.h>
162#include <iprt/asm.h>
163#include <iprt/avl.h>
164#ifdef VBOX_STRICT
165# include <iprt/crc.h>
166#endif
167#include <iprt/critsect.h>
168#include <iprt/list.h>
169#include <iprt/mem.h>
170#include <iprt/memobj.h>
171#include <iprt/mp.h>
172#include <iprt/semaphore.h>
173#include <iprt/string.h>
174#include <iprt/time.h>
175
176
177/*********************************************************************************************************************************
178* Defined Constants And Macros *
179*********************************************************************************************************************************/
180/** @def VBOX_USE_CRIT_SECT_FOR_GIANT
181 * Use a critical section instead of a fast mutex for the giant GMM lock.
182 *
183 * @remarks This is primarily a way of avoiding the deadlock checks in the
184 * windows driver verifier. */
185#if defined(RT_OS_WINDOWS) || defined(DOXYGEN_RUNNING)
186# define VBOX_USE_CRIT_SECT_FOR_GIANT
187#endif
188
189
190/*********************************************************************************************************************************
191* Structures and Typedefs *
192*********************************************************************************************************************************/
193/** Pointer to set of free chunks. */
194typedef struct GMMCHUNKFREESET *PGMMCHUNKFREESET;
195
196/**
197 * The per-page tracking structure employed by the GMM.
198 *
199 * On 32-bit hosts we'll some trickery is necessary to compress all
200 * the information into 32-bits. When the fSharedFree member is set,
201 * the 30th bit decides whether it's a free page or not.
202 *
203 * Because of the different layout on 32-bit and 64-bit hosts, macros
204 * are used to get and set some of the data.
205 */
206typedef union GMMPAGE
207{
208#if HC_ARCH_BITS == 64
209 /** Unsigned integer view. */
210 uint64_t u;
211
212 /** The common view. */
213 struct GMMPAGECOMMON
214 {
215 uint32_t uStuff1 : 32;
216 uint32_t uStuff2 : 30;
217 /** The page state. */
218 uint32_t u2State : 2;
219 } Common;
220
221 /** The view of a private page. */
222 struct GMMPAGEPRIVATE
223 {
224 /** The guest page frame number. (Max addressable: 2 ^ 44 - 16) */
225 uint32_t pfn;
226 /** The GVM handle. (64K VMs) */
227 uint32_t hGVM : 16;
228 /** Reserved. */
229 uint32_t u16Reserved : 14;
230 /** The page state. */
231 uint32_t u2State : 2;
232 } Private;
233
234 /** The view of a shared page. */
235 struct GMMPAGESHARED
236 {
237 /** The host page frame number. (Max addressable: 2 ^ 44 - 16) */
238 uint32_t pfn;
239 /** The reference count (64K VMs). */
240 uint32_t cRefs : 16;
241 /** Used for debug checksumming. */
242 uint32_t u14Checksum : 14;
243 /** The page state. */
244 uint32_t u2State : 2;
245 } Shared;
246
247 /** The view of a free page. */
248 struct GMMPAGEFREE
249 {
250 /** The index of the next page in the free list. UINT16_MAX is NIL. */
251 uint16_t iNext;
252 /** Reserved. Checksum or something? */
253 uint16_t u16Reserved0;
254 /** Reserved. Checksum or something? */
255 uint32_t u30Reserved1 : 30;
256 /** The page state. */
257 uint32_t u2State : 2;
258 } Free;
259
260#else /* 32-bit */
261 /** Unsigned integer view. */
262 uint32_t u;
263
264 /** The common view. */
265 struct GMMPAGECOMMON
266 {
267 uint32_t uStuff : 30;
268 /** The page state. */
269 uint32_t u2State : 2;
270 } Common;
271
272 /** The view of a private page. */
273 struct GMMPAGEPRIVATE
274 {
275 /** The guest page frame number. (Max addressable: 2 ^ 36) */
276 uint32_t pfn : 24;
277 /** The GVM handle. (127 VMs) */
278 uint32_t hGVM : 7;
279 /** The top page state bit, MBZ. */
280 uint32_t fZero : 1;
281 } Private;
282
283 /** The view of a shared page. */
284 struct GMMPAGESHARED
285 {
286 /** The reference count. */
287 uint32_t cRefs : 30;
288 /** The page state. */
289 uint32_t u2State : 2;
290 } Shared;
291
292 /** The view of a free page. */
293 struct GMMPAGEFREE
294 {
295 /** The index of the next page in the free list. UINT16_MAX is NIL. */
296 uint32_t iNext : 16;
297 /** Reserved. Checksum or something? */
298 uint32_t u14Reserved : 14;
299 /** The page state. */
300 uint32_t u2State : 2;
301 } Free;
302#endif
303} GMMPAGE;
304AssertCompileSize(GMMPAGE, sizeof(RTHCUINTPTR));
305/** Pointer to a GMMPAGE. */
306typedef GMMPAGE *PGMMPAGE;
307
308
309/** @name The Page States.
310 * @{ */
311/** A private page. */
312#define GMM_PAGE_STATE_PRIVATE 0
313/** A private page - alternative value used on the 32-bit implementation.
314 * This will never be used on 64-bit hosts. */
315#define GMM_PAGE_STATE_PRIVATE_32 1
316/** A shared page. */
317#define GMM_PAGE_STATE_SHARED 2
318/** A free page. */
319#define GMM_PAGE_STATE_FREE 3
320/** @} */
321
322
323/** @def GMM_PAGE_IS_PRIVATE
324 *
325 * @returns true if private, false if not.
326 * @param pPage The GMM page.
327 */
328#if HC_ARCH_BITS == 64
329# define GMM_PAGE_IS_PRIVATE(pPage) ( (pPage)->Common.u2State == GMM_PAGE_STATE_PRIVATE )
330#else
331# define GMM_PAGE_IS_PRIVATE(pPage) ( (pPage)->Private.fZero == 0 )
332#endif
333
334/** @def GMM_PAGE_IS_SHARED
335 *
336 * @returns true if shared, false if not.
337 * @param pPage The GMM page.
338 */
339#define GMM_PAGE_IS_SHARED(pPage) ( (pPage)->Common.u2State == GMM_PAGE_STATE_SHARED )
340
341/** @def GMM_PAGE_IS_FREE
342 *
343 * @returns true if free, false if not.
344 * @param pPage The GMM page.
345 */
346#define GMM_PAGE_IS_FREE(pPage) ( (pPage)->Common.u2State == GMM_PAGE_STATE_FREE )
347
348/** @def GMM_PAGE_PFN_LAST
349 * The last valid guest pfn range.
350 * @remark Some of the values outside the range has special meaning,
351 * see GMM_PAGE_PFN_UNSHAREABLE.
352 */
353#if HC_ARCH_BITS == 64
354# define GMM_PAGE_PFN_LAST UINT32_C(0xfffffff0)
355#else
356# define GMM_PAGE_PFN_LAST UINT32_C(0x00fffff0)
357#endif
358AssertCompile(GMM_PAGE_PFN_LAST == (GMM_GCPHYS_LAST >> PAGE_SHIFT));
359
360/** @def GMM_PAGE_PFN_UNSHAREABLE
361 * Indicates that this page isn't used for normal guest memory and thus isn't shareable.
362 */
363#if HC_ARCH_BITS == 64
364# define GMM_PAGE_PFN_UNSHAREABLE UINT32_C(0xfffffff1)
365#else
366# define GMM_PAGE_PFN_UNSHAREABLE UINT32_C(0x00fffff1)
367#endif
368AssertCompile(GMM_PAGE_PFN_UNSHAREABLE == (GMM_GCPHYS_UNSHAREABLE >> PAGE_SHIFT));
369
370
371/**
372 * A GMM allocation chunk ring-3 mapping record.
373 *
374 * This should really be associated with a session and not a VM, but
375 * it's simpler to associated with a VM and cleanup with the VM object
376 * is destroyed.
377 */
378typedef struct GMMCHUNKMAP
379{
380 /** The mapping object. */
381 RTR0MEMOBJ hMapObj;
382 /** The VM owning the mapping. */
383 PGVM pGVM;
384} GMMCHUNKMAP;
385/** Pointer to a GMM allocation chunk mapping. */
386typedef struct GMMCHUNKMAP *PGMMCHUNKMAP;
387
388
389/**
390 * A GMM allocation chunk.
391 */
392typedef struct GMMCHUNK
393{
394 /** The AVL node core.
395 * The Key is the chunk ID. (Giant mtx.) */
396 AVLU32NODECORE Core;
397 /** The memory object.
398 * Either from RTR0MemObjAllocPhysNC or RTR0MemObjLockUser depending on
399 * what the host can dish up with. (Chunk mtx protects mapping accesses
400 * and related frees.) */
401 RTR0MEMOBJ hMemObj;
402 /** Pointer to the next chunk in the free list. (Giant mtx.) */
403 PGMMCHUNK pFreeNext;
404 /** Pointer to the previous chunk in the free list. (Giant mtx.) */
405 PGMMCHUNK pFreePrev;
406 /** Pointer to the free set this chunk belongs to. NULL for
407 * chunks with no free pages. (Giant mtx.) */
408 PGMMCHUNKFREESET pSet;
409 /** List node in the chunk list (GMM::ChunkList). (Giant mtx.) */
410 RTLISTNODE ListNode;
411 /** Pointer to an array of mappings. (Chunk mtx.) */
412 PGMMCHUNKMAP paMappingsX;
413 /** The number of mappings. (Chunk mtx.) */
414 uint16_t cMappingsX;
415 /** The mapping lock this chunk is using using. UINT16_MAX if nobody is
416 * mapping or freeing anything. (Giant mtx.) */
417 uint8_t volatile iChunkMtx;
418 /** Flags field reserved for future use (like eliminating enmType).
419 * (Giant mtx.) */
420 uint8_t fFlags;
421 /** The head of the list of free pages. UINT16_MAX is the NIL value.
422 * (Giant mtx.) */
423 uint16_t iFreeHead;
424 /** The number of free pages. (Giant mtx.) */
425 uint16_t cFree;
426 /** The GVM handle of the VM that first allocated pages from this chunk, this
427 * is used as a preference when there are several chunks to choose from.
428 * When in bound memory mode this isn't a preference any longer. (Giant
429 * mtx.) */
430 uint16_t hGVM;
431 /** The ID of the NUMA node the memory mostly resides on. (Reserved for
432 * future use.) (Giant mtx.) */
433 uint16_t idNumaNode;
434 /** The number of private pages. (Giant mtx.) */
435 uint16_t cPrivate;
436 /** The number of shared pages. (Giant mtx.) */
437 uint16_t cShared;
438 /** The pages. (Giant mtx.) */
439 GMMPAGE aPages[GMM_CHUNK_SIZE >> PAGE_SHIFT];
440} GMMCHUNK;
441
442/** Indicates that the NUMA properies of the memory is unknown. */
443#define GMM_CHUNK_NUMA_ID_UNKNOWN UINT16_C(0xfffe)
444
445/** @name GMM_CHUNK_FLAGS_XXX - chunk flags.
446 * @{ */
447/** Indicates that the chunk is a large page (2MB). */
448#define GMM_CHUNK_FLAGS_LARGE_PAGE UINT16_C(0x0001)
449/** @} */
450
451
452/**
453 * An allocation chunk TLB entry.
454 */
455typedef struct GMMCHUNKTLBE
456{
457 /** The chunk id. */
458 uint32_t idChunk;
459 /** Pointer to the chunk. */
460 PGMMCHUNK pChunk;
461} GMMCHUNKTLBE;
462/** Pointer to an allocation chunk TLB entry. */
463typedef GMMCHUNKTLBE *PGMMCHUNKTLBE;
464
465
466/** The number of entries tin the allocation chunk TLB. */
467#define GMM_CHUNKTLB_ENTRIES 32
468/** Gets the TLB entry index for the given Chunk ID. */
469#define GMM_CHUNKTLB_IDX(idChunk) ( (idChunk) & (GMM_CHUNKTLB_ENTRIES - 1) )
470
471/**
472 * An allocation chunk TLB.
473 */
474typedef struct GMMCHUNKTLB
475{
476 /** The TLB entries. */
477 GMMCHUNKTLBE aEntries[GMM_CHUNKTLB_ENTRIES];
478} GMMCHUNKTLB;
479/** Pointer to an allocation chunk TLB. */
480typedef GMMCHUNKTLB *PGMMCHUNKTLB;
481
482
483/**
484 * The GMM instance data.
485 */
486typedef struct GMM
487{
488 /** Magic / eye catcher. GMM_MAGIC */
489 uint32_t u32Magic;
490 /** The number of threads waiting on the mutex. */
491 uint32_t cMtxContenders;
492#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
493 /** The critical section protecting the GMM.
494 * More fine grained locking can be implemented later if necessary. */
495 RTCRITSECT GiantCritSect;
496#else
497 /** The fast mutex protecting the GMM.
498 * More fine grained locking can be implemented later if necessary. */
499 RTSEMFASTMUTEX hMtx;
500#endif
501#ifdef VBOX_STRICT
502 /** The current mutex owner. */
503 RTNATIVETHREAD hMtxOwner;
504#endif
505 /** The chunk tree. */
506 PAVLU32NODECORE pChunks;
507 /** The chunk TLB. */
508 GMMCHUNKTLB ChunkTLB;
509 /** The private free set. */
510 GMMCHUNKFREESET PrivateX;
511 /** The shared free set. */
512 GMMCHUNKFREESET Shared;
513
514 /** Shared module tree (global).
515 * @todo separate trees for distinctly different guest OSes. */
516 PAVLLU32NODECORE pGlobalSharedModuleTree;
517 /** Sharable modules (count of nodes in pGlobalSharedModuleTree). */
518 uint32_t cShareableModules;
519
520 /** The chunk list. For simplifying the cleanup process. */
521 RTLISTANCHOR ChunkList;
522
523 /** The maximum number of pages we're allowed to allocate.
524 * @gcfgm{GMM/MaxPages,64-bit, Direct.}
525 * @gcfgm{GMM/PctPages,32-bit, Relative to the number of host pages.} */
526 uint64_t cMaxPages;
527 /** The number of pages that has been reserved.
528 * The deal is that cReservedPages - cOverCommittedPages <= cMaxPages. */
529 uint64_t cReservedPages;
530 /** The number of pages that we have over-committed in reservations. */
531 uint64_t cOverCommittedPages;
532 /** The number of actually allocated (committed if you like) pages. */
533 uint64_t cAllocatedPages;
534 /** The number of pages that are shared. A subset of cAllocatedPages. */
535 uint64_t cSharedPages;
536 /** The number of pages that are actually shared between VMs. */
537 uint64_t cDuplicatePages;
538 /** The number of pages that are shared that has been left behind by
539 * VMs not doing proper cleanups. */
540 uint64_t cLeftBehindSharedPages;
541 /** The number of allocation chunks.
542 * (The number of pages we've allocated from the host can be derived from this.) */
543 uint32_t cChunks;
544 /** The number of current ballooned pages. */
545 uint64_t cBalloonedPages;
546
547 /** The legacy allocation mode indicator.
548 * This is determined at initialization time. */
549 bool fLegacyAllocationMode;
550 /** The bound memory mode indicator.
551 * When set, the memory will be bound to a specific VM and never
552 * shared. This is always set if fLegacyAllocationMode is set.
553 * (Also determined at initialization time.) */
554 bool fBoundMemoryMode;
555 /** The number of registered VMs. */
556 uint16_t cRegisteredVMs;
557
558 /** The number of freed chunks ever. This is used a list generation to
559 * avoid restarting the cleanup scanning when the list wasn't modified. */
560 uint32_t cFreedChunks;
561 /** The previous allocated Chunk ID.
562 * Used as a hint to avoid scanning the whole bitmap. */
563 uint32_t idChunkPrev;
564 /** Chunk ID allocation bitmap.
565 * Bits of allocated IDs are set, free ones are clear.
566 * The NIL id (0) is marked allocated. */
567 uint32_t bmChunkId[(GMM_CHUNKID_LAST + 1 + 31) / 32];
568
569 /** The index of the next mutex to use. */
570 uint32_t iNextChunkMtx;
571 /** Chunk locks for reducing lock contention without having to allocate
572 * one lock per chunk. */
573 struct
574 {
575 /** The mutex */
576 RTSEMFASTMUTEX hMtx;
577 /** The number of threads currently using this mutex. */
578 uint32_t volatile cUsers;
579 } aChunkMtx[64];
580} GMM;
581/** Pointer to the GMM instance. */
582typedef GMM *PGMM;
583
584/** The value of GMM::u32Magic (Katsuhiro Otomo). */
585#define GMM_MAGIC UINT32_C(0x19540414)
586
587
588/**
589 * GMM chunk mutex state.
590 *
591 * This is returned by gmmR0ChunkMutexAcquire and is used by the other
592 * gmmR0ChunkMutex* methods.
593 */
594typedef struct GMMR0CHUNKMTXSTATE
595{
596 PGMM pGMM;
597 /** The index of the chunk mutex. */
598 uint8_t iChunkMtx;
599 /** The relevant flags (GMMR0CHUNK_MTX_XXX). */
600 uint8_t fFlags;
601} GMMR0CHUNKMTXSTATE;
602/** Pointer to a chunk mutex state. */
603typedef GMMR0CHUNKMTXSTATE *PGMMR0CHUNKMTXSTATE;
604
605/** @name GMMR0CHUNK_MTX_XXX
606 * @{ */
607#define GMMR0CHUNK_MTX_INVALID UINT32_C(0)
608#define GMMR0CHUNK_MTX_KEEP_GIANT UINT32_C(1)
609#define GMMR0CHUNK_MTX_RETAKE_GIANT UINT32_C(2)
610#define GMMR0CHUNK_MTX_DROP_GIANT UINT32_C(3)
611#define GMMR0CHUNK_MTX_END UINT32_C(4)
612/** @} */
613
614
615/** The maximum number of shared modules per-vm. */
616#define GMM_MAX_SHARED_PER_VM_MODULES 2048
617/** The maximum number of shared modules GMM is allowed to track. */
618#define GMM_MAX_SHARED_GLOBAL_MODULES 16834
619
620
621/**
622 * Argument packet for gmmR0SharedModuleCleanup.
623 */
624typedef struct GMMR0SHMODPERVMDTORARGS
625{
626 PGVM pGVM;
627 PGMM pGMM;
628} GMMR0SHMODPERVMDTORARGS;
629
630/**
631 * Argument packet for gmmR0CheckSharedModule.
632 */
633typedef struct GMMCHECKSHAREDMODULEINFO
634{
635 PGVM pGVM;
636 VMCPUID idCpu;
637} GMMCHECKSHAREDMODULEINFO;
638
639/**
640 * Argument packet for gmmR0FindDupPageInChunk by GMMR0FindDuplicatePage.
641 */
642typedef struct GMMFINDDUPPAGEINFO
643{
644 PGVM pGVM;
645 PGMM pGMM;
646 uint8_t *pSourcePage;
647 bool fFoundDuplicate;
648} GMMFINDDUPPAGEINFO;
649
650
651/*********************************************************************************************************************************
652* Global Variables *
653*********************************************************************************************************************************/
654/** Pointer to the GMM instance data. */
655static PGMM g_pGMM = NULL;
656
657/** Macro for obtaining and validating the g_pGMM pointer.
658 *
659 * On failure it will return from the invoking function with the specified
660 * return value.
661 *
662 * @param pGMM The name of the pGMM variable.
663 * @param rc The return value on failure. Use VERR_GMM_INSTANCE for VBox
664 * status codes.
665 */
666#define GMM_GET_VALID_INSTANCE(pGMM, rc) \
667 do { \
668 (pGMM) = g_pGMM; \
669 AssertPtrReturn((pGMM), (rc)); \
670 AssertMsgReturn((pGMM)->u32Magic == GMM_MAGIC, ("%p - %#x\n", (pGMM), (pGMM)->u32Magic), (rc)); \
671 } while (0)
672
673/** Macro for obtaining and validating the g_pGMM pointer, void function
674 * variant.
675 *
676 * On failure it will return from the invoking function.
677 *
678 * @param pGMM The name of the pGMM variable.
679 */
680#define GMM_GET_VALID_INSTANCE_VOID(pGMM) \
681 do { \
682 (pGMM) = g_pGMM; \
683 AssertPtrReturnVoid((pGMM)); \
684 AssertMsgReturnVoid((pGMM)->u32Magic == GMM_MAGIC, ("%p - %#x\n", (pGMM), (pGMM)->u32Magic)); \
685 } while (0)
686
687
688/** @def GMM_CHECK_SANITY_UPON_ENTERING
689 * Checks the sanity of the GMM instance data before making changes.
690 *
691 * This is macro is a stub by default and must be enabled manually in the code.
692 *
693 * @returns true if sane, false if not.
694 * @param pGMM The name of the pGMM variable.
695 */
696#if defined(VBOX_STRICT) && defined(GMMR0_WITH_SANITY_CHECK) && 0
697# define GMM_CHECK_SANITY_UPON_ENTERING(pGMM) (gmmR0SanityCheck((pGMM), __PRETTY_FUNCTION__, __LINE__) == 0)
698#else
699# define GMM_CHECK_SANITY_UPON_ENTERING(pGMM) (true)
700#endif
701
702/** @def GMM_CHECK_SANITY_UPON_LEAVING
703 * Checks the sanity of the GMM instance data after making changes.
704 *
705 * This is macro is a stub by default and must be enabled manually in the code.
706 *
707 * @returns true if sane, false if not.
708 * @param pGMM The name of the pGMM variable.
709 */
710#if defined(VBOX_STRICT) && defined(GMMR0_WITH_SANITY_CHECK) && 0
711# define GMM_CHECK_SANITY_UPON_LEAVING(pGMM) (gmmR0SanityCheck((pGMM), __PRETTY_FUNCTION__, __LINE__) == 0)
712#else
713# define GMM_CHECK_SANITY_UPON_LEAVING(pGMM) (true)
714#endif
715
716/** @def GMM_CHECK_SANITY_IN_LOOPS
717 * Checks the sanity of the GMM instance in the allocation loops.
718 *
719 * This is macro is a stub by default and must be enabled manually in the code.
720 *
721 * @returns true if sane, false if not.
722 * @param pGMM The name of the pGMM variable.
723 */
724#if defined(VBOX_STRICT) && defined(GMMR0_WITH_SANITY_CHECK) && 0
725# define GMM_CHECK_SANITY_IN_LOOPS(pGMM) (gmmR0SanityCheck((pGMM), __PRETTY_FUNCTION__, __LINE__) == 0)
726#else
727# define GMM_CHECK_SANITY_IN_LOOPS(pGMM) (true)
728#endif
729
730
731/*********************************************************************************************************************************
732* Internal Functions *
733*********************************************************************************************************************************/
734static DECLCALLBACK(int) gmmR0TermDestroyChunk(PAVLU32NODECORE pNode, void *pvGMM);
735static bool gmmR0CleanupVMScanChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk);
736DECLINLINE(void) gmmR0UnlinkChunk(PGMMCHUNK pChunk);
737DECLINLINE(void) gmmR0LinkChunk(PGMMCHUNK pChunk, PGMMCHUNKFREESET pSet);
738DECLINLINE(void) gmmR0SelectSetAndLinkChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk);
739#ifdef GMMR0_WITH_SANITY_CHECK
740static uint32_t gmmR0SanityCheck(PGMM pGMM, const char *pszFunction, unsigned uLineNo);
741#endif
742static bool gmmR0FreeChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem);
743DECLINLINE(void) gmmR0FreePrivatePage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage);
744DECLINLINE(void) gmmR0FreeSharedPage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage);
745static int gmmR0UnmapChunkLocked(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk);
746#ifdef VBOX_WITH_PAGE_SHARING
747static void gmmR0SharedModuleCleanup(PGMM pGMM, PGVM pGVM);
748# ifdef VBOX_STRICT
749static uint32_t gmmR0StrictPageChecksum(PGMM pGMM, PGVM pGVM, uint32_t idPage);
750# endif
751#endif
752
753
754
755/**
756 * Initializes the GMM component.
757 *
758 * This is called when the VMMR0.r0 module is loaded and protected by the
759 * loader semaphore.
760 *
761 * @returns VBox status code.
762 */
763GMMR0DECL(int) GMMR0Init(void)
764{
765 LogFlow(("GMMInit:\n"));
766
767 /*
768 * Allocate the instance data and the locks.
769 */
770 PGMM pGMM = (PGMM)RTMemAllocZ(sizeof(*pGMM));
771 if (!pGMM)
772 return VERR_NO_MEMORY;
773
774 pGMM->u32Magic = GMM_MAGIC;
775 for (unsigned i = 0; i < RT_ELEMENTS(pGMM->ChunkTLB.aEntries); i++)
776 pGMM->ChunkTLB.aEntries[i].idChunk = NIL_GMM_CHUNKID;
777 RTListInit(&pGMM->ChunkList);
778 ASMBitSet(&pGMM->bmChunkId[0], NIL_GMM_CHUNKID);
779
780#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
781 int rc = RTCritSectInit(&pGMM->GiantCritSect);
782#else
783 int rc = RTSemFastMutexCreate(&pGMM->hMtx);
784#endif
785 if (RT_SUCCESS(rc))
786 {
787 unsigned iMtx;
788 for (iMtx = 0; iMtx < RT_ELEMENTS(pGMM->aChunkMtx); iMtx++)
789 {
790 rc = RTSemFastMutexCreate(&pGMM->aChunkMtx[iMtx].hMtx);
791 if (RT_FAILURE(rc))
792 break;
793 }
794 if (RT_SUCCESS(rc))
795 {
796 /*
797 * Check and see if RTR0MemObjAllocPhysNC works.
798 */
799#if 0 /* later, see @bufref{3170}. */
800 RTR0MEMOBJ MemObj;
801 rc = RTR0MemObjAllocPhysNC(&MemObj, _64K, NIL_RTHCPHYS);
802 if (RT_SUCCESS(rc))
803 {
804 rc = RTR0MemObjFree(MemObj, true);
805 AssertRC(rc);
806 }
807 else if (rc == VERR_NOT_SUPPORTED)
808 pGMM->fLegacyAllocationMode = pGMM->fBoundMemoryMode = true;
809 else
810 SUPR0Printf("GMMR0Init: RTR0MemObjAllocPhysNC(,64K,Any) -> %d!\n", rc);
811#else
812# if defined(RT_OS_WINDOWS) || (defined(RT_OS_SOLARIS) && ARCH_BITS == 64) || defined(RT_OS_LINUX) || defined(RT_OS_FREEBSD)
813 pGMM->fLegacyAllocationMode = false;
814# if ARCH_BITS == 32
815 /* Don't reuse possibly partial chunks because of the virtual
816 address space limitation. */
817 pGMM->fBoundMemoryMode = true;
818# else
819 pGMM->fBoundMemoryMode = false;
820# endif
821# else
822 pGMM->fLegacyAllocationMode = true;
823 pGMM->fBoundMemoryMode = true;
824# endif
825#endif
826
827 /*
828 * Query system page count and guess a reasonable cMaxPages value.
829 */
830 pGMM->cMaxPages = UINT32_MAX; /** @todo IPRT function for query ram size and such. */
831
832 g_pGMM = pGMM;
833 LogFlow(("GMMInit: pGMM=%p fLegacyAllocationMode=%RTbool fBoundMemoryMode=%RTbool\n", pGMM, pGMM->fLegacyAllocationMode, pGMM->fBoundMemoryMode));
834 return VINF_SUCCESS;
835 }
836
837 /*
838 * Bail out.
839 */
840 while (iMtx-- > 0)
841 RTSemFastMutexDestroy(pGMM->aChunkMtx[iMtx].hMtx);
842#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
843 RTCritSectDelete(&pGMM->GiantCritSect);
844#else
845 RTSemFastMutexDestroy(pGMM->hMtx);
846#endif
847 }
848
849 pGMM->u32Magic = 0;
850 RTMemFree(pGMM);
851 SUPR0Printf("GMMR0Init: failed! rc=%d\n", rc);
852 return rc;
853}
854
855
856/**
857 * Terminates the GMM component.
858 */
859GMMR0DECL(void) GMMR0Term(void)
860{
861 LogFlow(("GMMTerm:\n"));
862
863 /*
864 * Take care / be paranoid...
865 */
866 PGMM pGMM = g_pGMM;
867 if (!VALID_PTR(pGMM))
868 return;
869 if (pGMM->u32Magic != GMM_MAGIC)
870 {
871 SUPR0Printf("GMMR0Term: u32Magic=%#x\n", pGMM->u32Magic);
872 return;
873 }
874
875 /*
876 * Undo what init did and free all the resources we've acquired.
877 */
878 /* Destroy the fundamentals. */
879 g_pGMM = NULL;
880 pGMM->u32Magic = ~GMM_MAGIC;
881#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
882 RTCritSectDelete(&pGMM->GiantCritSect);
883#else
884 RTSemFastMutexDestroy(pGMM->hMtx);
885 pGMM->hMtx = NIL_RTSEMFASTMUTEX;
886#endif
887
888 /* Free any chunks still hanging around. */
889 RTAvlU32Destroy(&pGMM->pChunks, gmmR0TermDestroyChunk, pGMM);
890
891 /* Destroy the chunk locks. */
892 for (unsigned iMtx = 0; iMtx < RT_ELEMENTS(pGMM->aChunkMtx); iMtx++)
893 {
894 Assert(pGMM->aChunkMtx[iMtx].cUsers == 0);
895 RTSemFastMutexDestroy(pGMM->aChunkMtx[iMtx].hMtx);
896 pGMM->aChunkMtx[iMtx].hMtx = NIL_RTSEMFASTMUTEX;
897 }
898
899 /* Finally the instance data itself. */
900 RTMemFree(pGMM);
901 LogFlow(("GMMTerm: done\n"));
902}
903
904
905/**
906 * RTAvlU32Destroy callback.
907 *
908 * @returns 0
909 * @param pNode The node to destroy.
910 * @param pvGMM The GMM handle.
911 */
912static DECLCALLBACK(int) gmmR0TermDestroyChunk(PAVLU32NODECORE pNode, void *pvGMM)
913{
914 PGMMCHUNK pChunk = (PGMMCHUNK)pNode;
915
916 if (pChunk->cFree != (GMM_CHUNK_SIZE >> PAGE_SHIFT))
917 SUPR0Printf("GMMR0Term: %RKv/%#x: cFree=%d cPrivate=%d cShared=%d cMappings=%d\n", pChunk,
918 pChunk->Core.Key, pChunk->cFree, pChunk->cPrivate, pChunk->cShared, pChunk->cMappingsX);
919
920 int rc = RTR0MemObjFree(pChunk->hMemObj, true /* fFreeMappings */);
921 if (RT_FAILURE(rc))
922 {
923 SUPR0Printf("GMMR0Term: %RKv/%#x: RTRMemObjFree(%RKv,true) -> %d (cMappings=%d)\n", pChunk,
924 pChunk->Core.Key, pChunk->hMemObj, rc, pChunk->cMappingsX);
925 AssertRC(rc);
926 }
927 pChunk->hMemObj = NIL_RTR0MEMOBJ;
928
929 RTMemFree(pChunk->paMappingsX);
930 pChunk->paMappingsX = NULL;
931
932 RTMemFree(pChunk);
933 NOREF(pvGMM);
934 return 0;
935}
936
937
938/**
939 * Initializes the per-VM data for the GMM.
940 *
941 * This is called from within the GVMM lock (from GVMMR0CreateVM)
942 * and should only initialize the data members so GMMR0CleanupVM
943 * can deal with them. We reserve no memory or anything here,
944 * that's done later in GMMR0InitVM.
945 *
946 * @param pGVM Pointer to the Global VM structure.
947 */
948GMMR0DECL(void) GMMR0InitPerVMData(PGVM pGVM)
949{
950 AssertCompile(RT_SIZEOFMEMB(GVM,gmm.s) <= RT_SIZEOFMEMB(GVM,gmm.padding));
951
952 pGVM->gmm.s.Stats.enmPolicy = GMMOCPOLICY_INVALID;
953 pGVM->gmm.s.Stats.enmPriority = GMMPRIORITY_INVALID;
954 pGVM->gmm.s.Stats.fMayAllocate = false;
955}
956
957
958/**
959 * Acquires the GMM giant lock.
960 *
961 * @returns Assert status code from RTSemFastMutexRequest.
962 * @param pGMM Pointer to the GMM instance.
963 */
964static int gmmR0MutexAcquire(PGMM pGMM)
965{
966 ASMAtomicIncU32(&pGMM->cMtxContenders);
967#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
968 int rc = RTCritSectEnter(&pGMM->GiantCritSect);
969#else
970 int rc = RTSemFastMutexRequest(pGMM->hMtx);
971#endif
972 ASMAtomicDecU32(&pGMM->cMtxContenders);
973 AssertRC(rc);
974#ifdef VBOX_STRICT
975 pGMM->hMtxOwner = RTThreadNativeSelf();
976#endif
977 return rc;
978}
979
980
981/**
982 * Releases the GMM giant lock.
983 *
984 * @returns Assert status code from RTSemFastMutexRequest.
985 * @param pGMM Pointer to the GMM instance.
986 */
987static int gmmR0MutexRelease(PGMM pGMM)
988{
989#ifdef VBOX_STRICT
990 pGMM->hMtxOwner = NIL_RTNATIVETHREAD;
991#endif
992#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
993 int rc = RTCritSectLeave(&pGMM->GiantCritSect);
994#else
995 int rc = RTSemFastMutexRelease(pGMM->hMtx);
996 AssertRC(rc);
997#endif
998 return rc;
999}
1000
1001
1002/**
1003 * Yields the GMM giant lock if there is contention and a certain minimum time
1004 * has elapsed since we took it.
1005 *
1006 * @returns @c true if the mutex was yielded, @c false if not.
1007 * @param pGMM Pointer to the GMM instance.
1008 * @param puLockNanoTS Where the lock acquisition time stamp is kept
1009 * (in/out).
1010 */
1011static bool gmmR0MutexYield(PGMM pGMM, uint64_t *puLockNanoTS)
1012{
1013 /*
1014 * If nobody is contending the mutex, don't bother checking the time.
1015 */
1016 if (ASMAtomicReadU32(&pGMM->cMtxContenders) == 0)
1017 return false;
1018
1019 /*
1020 * Don't yield if we haven't executed for at least 2 milliseconds.
1021 */
1022 uint64_t uNanoNow = RTTimeSystemNanoTS();
1023 if (uNanoNow - *puLockNanoTS < UINT32_C(2000000))
1024 return false;
1025
1026 /*
1027 * Yield the mutex.
1028 */
1029#ifdef VBOX_STRICT
1030 pGMM->hMtxOwner = NIL_RTNATIVETHREAD;
1031#endif
1032 ASMAtomicIncU32(&pGMM->cMtxContenders);
1033#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
1034 int rc1 = RTCritSectLeave(&pGMM->GiantCritSect); AssertRC(rc1);
1035#else
1036 int rc1 = RTSemFastMutexRelease(pGMM->hMtx); AssertRC(rc1);
1037#endif
1038
1039 RTThreadYield();
1040
1041#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
1042 int rc2 = RTCritSectEnter(&pGMM->GiantCritSect); AssertRC(rc2);
1043#else
1044 int rc2 = RTSemFastMutexRequest(pGMM->hMtx); AssertRC(rc2);
1045#endif
1046 *puLockNanoTS = RTTimeSystemNanoTS();
1047 ASMAtomicDecU32(&pGMM->cMtxContenders);
1048#ifdef VBOX_STRICT
1049 pGMM->hMtxOwner = RTThreadNativeSelf();
1050#endif
1051
1052 return true;
1053}
1054
1055
1056/**
1057 * Acquires a chunk lock.
1058 *
1059 * The caller must own the giant lock.
1060 *
1061 * @returns Assert status code from RTSemFastMutexRequest.
1062 * @param pMtxState The chunk mutex state info. (Avoids
1063 * passing the same flags and stuff around
1064 * for subsequent release and drop-giant
1065 * calls.)
1066 * @param pGMM Pointer to the GMM instance.
1067 * @param pChunk Pointer to the chunk.
1068 * @param fFlags Flags regarding the giant lock, GMMR0CHUNK_MTX_XXX.
1069 */
1070static int gmmR0ChunkMutexAcquire(PGMMR0CHUNKMTXSTATE pMtxState, PGMM pGMM, PGMMCHUNK pChunk, uint32_t fFlags)
1071{
1072 Assert(fFlags > GMMR0CHUNK_MTX_INVALID && fFlags < GMMR0CHUNK_MTX_END);
1073 Assert(pGMM->hMtxOwner == RTThreadNativeSelf());
1074
1075 pMtxState->pGMM = pGMM;
1076 pMtxState->fFlags = (uint8_t)fFlags;
1077
1078 /*
1079 * Get the lock index and reference the lock.
1080 */
1081 Assert(pGMM->hMtxOwner == RTThreadNativeSelf());
1082 uint32_t iChunkMtx = pChunk->iChunkMtx;
1083 if (iChunkMtx == UINT8_MAX)
1084 {
1085 iChunkMtx = pGMM->iNextChunkMtx++;
1086 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1087
1088 /* Try get an unused one... */
1089 if (pGMM->aChunkMtx[iChunkMtx].cUsers)
1090 {
1091 iChunkMtx = pGMM->iNextChunkMtx++;
1092 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1093 if (pGMM->aChunkMtx[iChunkMtx].cUsers)
1094 {
1095 iChunkMtx = pGMM->iNextChunkMtx++;
1096 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1097 if (pGMM->aChunkMtx[iChunkMtx].cUsers)
1098 {
1099 iChunkMtx = pGMM->iNextChunkMtx++;
1100 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1101 }
1102 }
1103 }
1104
1105 pChunk->iChunkMtx = iChunkMtx;
1106 }
1107 AssertCompile(RT_ELEMENTS(pGMM->aChunkMtx) < UINT8_MAX);
1108 pMtxState->iChunkMtx = (uint8_t)iChunkMtx;
1109 ASMAtomicIncU32(&pGMM->aChunkMtx[iChunkMtx].cUsers);
1110
1111 /*
1112 * Drop the giant?
1113 */
1114 if (fFlags != GMMR0CHUNK_MTX_KEEP_GIANT)
1115 {
1116 /** @todo GMM life cycle cleanup (we may race someone
1117 * destroying and cleaning up GMM)? */
1118 gmmR0MutexRelease(pGMM);
1119 }
1120
1121 /*
1122 * Take the chunk mutex.
1123 */
1124 int rc = RTSemFastMutexRequest(pGMM->aChunkMtx[iChunkMtx].hMtx);
1125 AssertRC(rc);
1126 return rc;
1127}
1128
1129
1130/**
1131 * Releases the GMM giant lock.
1132 *
1133 * @returns Assert status code from RTSemFastMutexRequest.
1134 * @param pMtxState Pointer to the chunk mutex state.
1135 * @param pChunk Pointer to the chunk if it's still
1136 * alive, NULL if it isn't. This is used to deassociate
1137 * the chunk from the mutex on the way out so a new one
1138 * can be selected next time, thus avoiding contented
1139 * mutexes.
1140 */
1141static int gmmR0ChunkMutexRelease(PGMMR0CHUNKMTXSTATE pMtxState, PGMMCHUNK pChunk)
1142{
1143 PGMM pGMM = pMtxState->pGMM;
1144
1145 /*
1146 * Release the chunk mutex and reacquire the giant if requested.
1147 */
1148 int rc = RTSemFastMutexRelease(pGMM->aChunkMtx[pMtxState->iChunkMtx].hMtx);
1149 AssertRC(rc);
1150 if (pMtxState->fFlags == GMMR0CHUNK_MTX_RETAKE_GIANT)
1151 rc = gmmR0MutexAcquire(pGMM);
1152 else
1153 Assert((pMtxState->fFlags != GMMR0CHUNK_MTX_DROP_GIANT) == (pGMM->hMtxOwner == RTThreadNativeSelf()));
1154
1155 /*
1156 * Drop the chunk mutex user reference and deassociate it from the chunk
1157 * when possible.
1158 */
1159 if ( ASMAtomicDecU32(&pGMM->aChunkMtx[pMtxState->iChunkMtx].cUsers) == 0
1160 && pChunk
1161 && RT_SUCCESS(rc) )
1162 {
1163 if (pMtxState->fFlags != GMMR0CHUNK_MTX_DROP_GIANT)
1164 pChunk->iChunkMtx = UINT8_MAX;
1165 else
1166 {
1167 rc = gmmR0MutexAcquire(pGMM);
1168 if (RT_SUCCESS(rc))
1169 {
1170 if (pGMM->aChunkMtx[pMtxState->iChunkMtx].cUsers == 0)
1171 pChunk->iChunkMtx = UINT8_MAX;
1172 rc = gmmR0MutexRelease(pGMM);
1173 }
1174 }
1175 }
1176
1177 pMtxState->pGMM = NULL;
1178 return rc;
1179}
1180
1181
1182/**
1183 * Drops the giant GMM lock we kept in gmmR0ChunkMutexAcquire while keeping the
1184 * chunk locked.
1185 *
1186 * This only works if gmmR0ChunkMutexAcquire was called with
1187 * GMMR0CHUNK_MTX_KEEP_GIANT. gmmR0ChunkMutexRelease will retake the giant
1188 * mutex, i.e. behave as if GMMR0CHUNK_MTX_RETAKE_GIANT was used.
1189 *
1190 * @returns VBox status code (assuming success is ok).
1191 * @param pMtxState Pointer to the chunk mutex state.
1192 */
1193static int gmmR0ChunkMutexDropGiant(PGMMR0CHUNKMTXSTATE pMtxState)
1194{
1195 AssertReturn(pMtxState->fFlags == GMMR0CHUNK_MTX_KEEP_GIANT, VERR_GMM_MTX_FLAGS);
1196 Assert(pMtxState->pGMM->hMtxOwner == RTThreadNativeSelf());
1197 pMtxState->fFlags = GMMR0CHUNK_MTX_RETAKE_GIANT;
1198 /** @todo GMM life cycle cleanup (we may race someone
1199 * destroying and cleaning up GMM)? */
1200 return gmmR0MutexRelease(pMtxState->pGMM);
1201}
1202
1203
1204/**
1205 * For experimenting with NUMA affinity and such.
1206 *
1207 * @returns The current NUMA Node ID.
1208 */
1209static uint16_t gmmR0GetCurrentNumaNodeId(void)
1210{
1211#if 1
1212 return GMM_CHUNK_NUMA_ID_UNKNOWN;
1213#else
1214 return RTMpCpuId() / 16;
1215#endif
1216}
1217
1218
1219
1220/**
1221 * Cleans up when a VM is terminating.
1222 *
1223 * @param pGVM Pointer to the Global VM structure.
1224 */
1225GMMR0DECL(void) GMMR0CleanupVM(PGVM pGVM)
1226{
1227 LogFlow(("GMMR0CleanupVM: pGVM=%p:{.pVM=%p, .hSelf=%#x}\n", pGVM, pGVM->pVM, pGVM->hSelf));
1228
1229 PGMM pGMM;
1230 GMM_GET_VALID_INSTANCE_VOID(pGMM);
1231
1232#ifdef VBOX_WITH_PAGE_SHARING
1233 /*
1234 * Clean up all registered shared modules first.
1235 */
1236 gmmR0SharedModuleCleanup(pGMM, pGVM);
1237#endif
1238
1239 gmmR0MutexAcquire(pGMM);
1240 uint64_t uLockNanoTS = RTTimeSystemNanoTS();
1241 GMM_CHECK_SANITY_UPON_ENTERING(pGMM);
1242
1243 /*
1244 * The policy is 'INVALID' until the initial reservation
1245 * request has been serviced.
1246 */
1247 if ( pGVM->gmm.s.Stats.enmPolicy > GMMOCPOLICY_INVALID
1248 && pGVM->gmm.s.Stats.enmPolicy < GMMOCPOLICY_END)
1249 {
1250 /*
1251 * If it's the last VM around, we can skip walking all the chunk looking
1252 * for the pages owned by this VM and instead flush the whole shebang.
1253 *
1254 * This takes care of the eventuality that a VM has left shared page
1255 * references behind (shouldn't happen of course, but you never know).
1256 */
1257 Assert(pGMM->cRegisteredVMs);
1258 pGMM->cRegisteredVMs--;
1259
1260 /*
1261 * Walk the entire pool looking for pages that belong to this VM
1262 * and leftover mappings. (This'll only catch private pages,
1263 * shared pages will be 'left behind'.)
1264 */
1265 /** @todo r=bird: This scanning+freeing could be optimized in bound mode! */
1266 uint64_t cPrivatePages = pGVM->gmm.s.Stats.cPrivatePages; /* save */
1267
1268 unsigned iCountDown = 64;
1269 bool fRedoFromStart;
1270 PGMMCHUNK pChunk;
1271 do
1272 {
1273 fRedoFromStart = false;
1274 RTListForEachReverse(&pGMM->ChunkList, pChunk, GMMCHUNK, ListNode)
1275 {
1276 uint32_t const cFreeChunksOld = pGMM->cFreedChunks;
1277 if ( ( !pGMM->fBoundMemoryMode
1278 || pChunk->hGVM == pGVM->hSelf)
1279 && gmmR0CleanupVMScanChunk(pGMM, pGVM, pChunk))
1280 {
1281 /* We left the giant mutex, so reset the yield counters. */
1282 uLockNanoTS = RTTimeSystemNanoTS();
1283 iCountDown = 64;
1284 }
1285 else
1286 {
1287 /* Didn't leave it, so do normal yielding. */
1288 if (!iCountDown)
1289 gmmR0MutexYield(pGMM, &uLockNanoTS);
1290 else
1291 iCountDown--;
1292 }
1293 if (pGMM->cFreedChunks != cFreeChunksOld)
1294 {
1295 fRedoFromStart = true;
1296 break;
1297 }
1298 }
1299 } while (fRedoFromStart);
1300
1301 if (pGVM->gmm.s.Stats.cPrivatePages)
1302 SUPR0Printf("GMMR0CleanupVM: hGVM=%#x has %#x private pages that cannot be found!\n", pGVM->hSelf, pGVM->gmm.s.Stats.cPrivatePages);
1303
1304 pGMM->cAllocatedPages -= cPrivatePages;
1305
1306 /*
1307 * Free empty chunks.
1308 */
1309 PGMMCHUNKFREESET pPrivateSet = pGMM->fBoundMemoryMode ? &pGVM->gmm.s.Private : &pGMM->PrivateX;
1310 do
1311 {
1312 fRedoFromStart = false;
1313 iCountDown = 10240;
1314 pChunk = pPrivateSet->apLists[GMM_CHUNK_FREE_SET_UNUSED_LIST];
1315 while (pChunk)
1316 {
1317 PGMMCHUNK pNext = pChunk->pFreeNext;
1318 Assert(pChunk->cFree == GMM_CHUNK_NUM_PAGES);
1319 if ( !pGMM->fBoundMemoryMode
1320 || pChunk->hGVM == pGVM->hSelf)
1321 {
1322 uint64_t const idGenerationOld = pPrivateSet->idGeneration;
1323 if (gmmR0FreeChunk(pGMM, pGVM, pChunk, true /*fRelaxedSem*/))
1324 {
1325 /* We've left the giant mutex, restart? (+1 for our unlink) */
1326 fRedoFromStart = pPrivateSet->idGeneration != idGenerationOld + 1;
1327 if (fRedoFromStart)
1328 break;
1329 uLockNanoTS = RTTimeSystemNanoTS();
1330 iCountDown = 10240;
1331 }
1332 }
1333
1334 /* Advance and maybe yield the lock. */
1335 pChunk = pNext;
1336 if (--iCountDown == 0)
1337 {
1338 uint64_t const idGenerationOld = pPrivateSet->idGeneration;
1339 fRedoFromStart = gmmR0MutexYield(pGMM, &uLockNanoTS)
1340 && pPrivateSet->idGeneration != idGenerationOld;
1341 if (fRedoFromStart)
1342 break;
1343 iCountDown = 10240;
1344 }
1345 }
1346 } while (fRedoFromStart);
1347
1348 /*
1349 * Account for shared pages that weren't freed.
1350 */
1351 if (pGVM->gmm.s.Stats.cSharedPages)
1352 {
1353 Assert(pGMM->cSharedPages >= pGVM->gmm.s.Stats.cSharedPages);
1354 SUPR0Printf("GMMR0CleanupVM: hGVM=%#x left %#x shared pages behind!\n", pGVM->hSelf, pGVM->gmm.s.Stats.cSharedPages);
1355 pGMM->cLeftBehindSharedPages += pGVM->gmm.s.Stats.cSharedPages;
1356 }
1357
1358 /*
1359 * Clean up balloon statistics in case the VM process crashed.
1360 */
1361 Assert(pGMM->cBalloonedPages >= pGVM->gmm.s.Stats.cBalloonedPages);
1362 pGMM->cBalloonedPages -= pGVM->gmm.s.Stats.cBalloonedPages;
1363
1364 /*
1365 * Update the over-commitment management statistics.
1366 */
1367 pGMM->cReservedPages -= pGVM->gmm.s.Stats.Reserved.cBasePages
1368 + pGVM->gmm.s.Stats.Reserved.cFixedPages
1369 + pGVM->gmm.s.Stats.Reserved.cShadowPages;
1370 switch (pGVM->gmm.s.Stats.enmPolicy)
1371 {
1372 case GMMOCPOLICY_NO_OC:
1373 break;
1374 default:
1375 /** @todo Update GMM->cOverCommittedPages */
1376 break;
1377 }
1378 }
1379
1380 /* zap the GVM data. */
1381 pGVM->gmm.s.Stats.enmPolicy = GMMOCPOLICY_INVALID;
1382 pGVM->gmm.s.Stats.enmPriority = GMMPRIORITY_INVALID;
1383 pGVM->gmm.s.Stats.fMayAllocate = false;
1384
1385 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
1386 gmmR0MutexRelease(pGMM);
1387
1388 LogFlow(("GMMR0CleanupVM: returns\n"));
1389}
1390
1391
1392/**
1393 * Scan one chunk for private pages belonging to the specified VM.
1394 *
1395 * @note This function may drop the giant mutex!
1396 *
1397 * @returns @c true if we've temporarily dropped the giant mutex, @c false if
1398 * we didn't.
1399 * @param pGMM Pointer to the GMM instance.
1400 * @param pGVM The global VM handle.
1401 * @param pChunk The chunk to scan.
1402 */
1403static bool gmmR0CleanupVMScanChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk)
1404{
1405 Assert(!pGMM->fBoundMemoryMode || pChunk->hGVM == pGVM->hSelf);
1406
1407 /*
1408 * Look for pages belonging to the VM.
1409 * (Perform some internal checks while we're scanning.)
1410 */
1411#ifndef VBOX_STRICT
1412 if (pChunk->cFree != (GMM_CHUNK_SIZE >> PAGE_SHIFT))
1413#endif
1414 {
1415 unsigned cPrivate = 0;
1416 unsigned cShared = 0;
1417 unsigned cFree = 0;
1418
1419 gmmR0UnlinkChunk(pChunk); /* avoiding cFreePages updates. */
1420
1421 uint16_t hGVM = pGVM->hSelf;
1422 unsigned iPage = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
1423 while (iPage-- > 0)
1424 if (GMM_PAGE_IS_PRIVATE(&pChunk->aPages[iPage]))
1425 {
1426 if (pChunk->aPages[iPage].Private.hGVM == hGVM)
1427 {
1428 /*
1429 * Free the page.
1430 *
1431 * The reason for not using gmmR0FreePrivatePage here is that we
1432 * must *not* cause the chunk to be freed from under us - we're in
1433 * an AVL tree walk here.
1434 */
1435 pChunk->aPages[iPage].u = 0;
1436 pChunk->aPages[iPage].Free.iNext = pChunk->iFreeHead;
1437 pChunk->aPages[iPage].Free.u2State = GMM_PAGE_STATE_FREE;
1438 pChunk->iFreeHead = iPage;
1439 pChunk->cPrivate--;
1440 pChunk->cFree++;
1441 pGVM->gmm.s.Stats.cPrivatePages--;
1442 cFree++;
1443 }
1444 else
1445 cPrivate++;
1446 }
1447 else if (GMM_PAGE_IS_FREE(&pChunk->aPages[iPage]))
1448 cFree++;
1449 else
1450 cShared++;
1451
1452 gmmR0SelectSetAndLinkChunk(pGMM, pGVM, pChunk);
1453
1454 /*
1455 * Did it add up?
1456 */
1457 if (RT_UNLIKELY( pChunk->cFree != cFree
1458 || pChunk->cPrivate != cPrivate
1459 || pChunk->cShared != cShared))
1460 {
1461 SUPR0Printf("gmmR0CleanupVMScanChunk: Chunk %RKv/%#x has bogus stats - free=%d/%d private=%d/%d shared=%d/%d\n",
1462 pChunk, pChunk->Core.Key, pChunk->cFree, cFree, pChunk->cPrivate, cPrivate, pChunk->cShared, cShared);
1463 pChunk->cFree = cFree;
1464 pChunk->cPrivate = cPrivate;
1465 pChunk->cShared = cShared;
1466 }
1467 }
1468
1469 /*
1470 * If not in bound memory mode, we should reset the hGVM field
1471 * if it has our handle in it.
1472 */
1473 if (pChunk->hGVM == pGVM->hSelf)
1474 {
1475 if (!g_pGMM->fBoundMemoryMode)
1476 pChunk->hGVM = NIL_GVM_HANDLE;
1477 else if (pChunk->cFree != GMM_CHUNK_NUM_PAGES)
1478 {
1479 SUPR0Printf("gmmR0CleanupVMScanChunk: %RKv/%#x: cFree=%#x - it should be 0 in bound mode!\n",
1480 pChunk, pChunk->Core.Key, pChunk->cFree);
1481 AssertMsgFailed(("%p/%#x: cFree=%#x - it should be 0 in bound mode!\n", pChunk, pChunk->Core.Key, pChunk->cFree));
1482
1483 gmmR0UnlinkChunk(pChunk);
1484 pChunk->cFree = GMM_CHUNK_NUM_PAGES;
1485 gmmR0SelectSetAndLinkChunk(pGMM, pGVM, pChunk);
1486 }
1487 }
1488
1489 /*
1490 * Look for a mapping belonging to the terminating VM.
1491 */
1492 GMMR0CHUNKMTXSTATE MtxState;
1493 gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk, GMMR0CHUNK_MTX_KEEP_GIANT);
1494 unsigned cMappings = pChunk->cMappingsX;
1495 for (unsigned i = 0; i < cMappings; i++)
1496 if (pChunk->paMappingsX[i].pGVM == pGVM)
1497 {
1498 gmmR0ChunkMutexDropGiant(&MtxState);
1499
1500 RTR0MEMOBJ hMemObj = pChunk->paMappingsX[i].hMapObj;
1501
1502 cMappings--;
1503 if (i < cMappings)
1504 pChunk->paMappingsX[i] = pChunk->paMappingsX[cMappings];
1505 pChunk->paMappingsX[cMappings].pGVM = NULL;
1506 pChunk->paMappingsX[cMappings].hMapObj = NIL_RTR0MEMOBJ;
1507 Assert(pChunk->cMappingsX - 1U == cMappings);
1508 pChunk->cMappingsX = cMappings;
1509
1510 int rc = RTR0MemObjFree(hMemObj, false /* fFreeMappings (NA) */);
1511 if (RT_FAILURE(rc))
1512 {
1513 SUPR0Printf("gmmR0CleanupVMScanChunk: %RKv/%#x: mapping #%x: RTRMemObjFree(%RKv,false) -> %d \n",
1514 pChunk, pChunk->Core.Key, i, hMemObj, rc);
1515 AssertRC(rc);
1516 }
1517
1518 gmmR0ChunkMutexRelease(&MtxState, pChunk);
1519 return true;
1520 }
1521
1522 gmmR0ChunkMutexRelease(&MtxState, pChunk);
1523 return false;
1524}
1525
1526
1527/**
1528 * The initial resource reservations.
1529 *
1530 * This will make memory reservations according to policy and priority. If there aren't
1531 * sufficient resources available to sustain the VM this function will fail and all
1532 * future allocations requests will fail as well.
1533 *
1534 * These are just the initial reservations made very very early during the VM creation
1535 * process and will be adjusted later in the GMMR0UpdateReservation call after the
1536 * ring-3 init has completed.
1537 *
1538 * @returns VBox status code.
1539 * @retval VERR_GMM_MEMORY_RESERVATION_DECLINED
1540 * @retval VERR_GMM_
1541 *
1542 * @param pGVM The global (ring-0) VM structure.
1543 * @param pVM The cross context VM structure.
1544 * @param idCpu The VCPU id - must be zero.
1545 * @param cBasePages The number of pages that may be allocated for the base RAM and ROMs.
1546 * This does not include MMIO2 and similar.
1547 * @param cShadowPages The number of pages that may be allocated for shadow paging structures.
1548 * @param cFixedPages The number of pages that may be allocated for fixed objects like the
1549 * hyper heap, MMIO2 and similar.
1550 * @param enmPolicy The OC policy to use on this VM.
1551 * @param enmPriority The priority in an out-of-memory situation.
1552 *
1553 * @thread The creator thread / EMT(0).
1554 */
1555GMMR0DECL(int) GMMR0InitialReservation(PGVM pGVM, PVM pVM, VMCPUID idCpu, uint64_t cBasePages, uint32_t cShadowPages,
1556 uint32_t cFixedPages, GMMOCPOLICY enmPolicy, GMMPRIORITY enmPriority)
1557{
1558 LogFlow(("GMMR0InitialReservation: pGVM=%p pVM=%p cBasePages=%#llx cShadowPages=%#x cFixedPages=%#x enmPolicy=%d enmPriority=%d\n",
1559 pGVM, pVM, cBasePages, cShadowPages, cFixedPages, enmPolicy, enmPriority));
1560
1561 /*
1562 * Validate, get basics and take the semaphore.
1563 */
1564 AssertReturn(idCpu == 0, VERR_INVALID_CPU_ID);
1565 PGMM pGMM;
1566 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
1567 int rc = GVMMR0ValidateGVMandVMandEMT(pGVM, pVM, idCpu);
1568 if (RT_FAILURE(rc))
1569 return rc;
1570
1571 AssertReturn(cBasePages, VERR_INVALID_PARAMETER);
1572 AssertReturn(cShadowPages, VERR_INVALID_PARAMETER);
1573 AssertReturn(cFixedPages, VERR_INVALID_PARAMETER);
1574 AssertReturn(enmPolicy > GMMOCPOLICY_INVALID && enmPolicy < GMMOCPOLICY_END, VERR_INVALID_PARAMETER);
1575 AssertReturn(enmPriority > GMMPRIORITY_INVALID && enmPriority < GMMPRIORITY_END, VERR_INVALID_PARAMETER);
1576
1577 gmmR0MutexAcquire(pGMM);
1578 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
1579 {
1580 if ( !pGVM->gmm.s.Stats.Reserved.cBasePages
1581 && !pGVM->gmm.s.Stats.Reserved.cFixedPages
1582 && !pGVM->gmm.s.Stats.Reserved.cShadowPages)
1583 {
1584 /*
1585 * Check if we can accommodate this.
1586 */
1587 /* ... later ... */
1588 if (RT_SUCCESS(rc))
1589 {
1590 /*
1591 * Update the records.
1592 */
1593 pGVM->gmm.s.Stats.Reserved.cBasePages = cBasePages;
1594 pGVM->gmm.s.Stats.Reserved.cFixedPages = cFixedPages;
1595 pGVM->gmm.s.Stats.Reserved.cShadowPages = cShadowPages;
1596 pGVM->gmm.s.Stats.enmPolicy = enmPolicy;
1597 pGVM->gmm.s.Stats.enmPriority = enmPriority;
1598 pGVM->gmm.s.Stats.fMayAllocate = true;
1599
1600 pGMM->cReservedPages += cBasePages + cFixedPages + cShadowPages;
1601 pGMM->cRegisteredVMs++;
1602 }
1603 }
1604 else
1605 rc = VERR_WRONG_ORDER;
1606 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
1607 }
1608 else
1609 rc = VERR_GMM_IS_NOT_SANE;
1610 gmmR0MutexRelease(pGMM);
1611 LogFlow(("GMMR0InitialReservation: returns %Rrc\n", rc));
1612 return rc;
1613}
1614
1615
1616/**
1617 * VMMR0 request wrapper for GMMR0InitialReservation.
1618 *
1619 * @returns see GMMR0InitialReservation.
1620 * @param pGVM The global (ring-0) VM structure.
1621 * @param pVM The cross context VM structure.
1622 * @param idCpu The VCPU id.
1623 * @param pReq Pointer to the request packet.
1624 */
1625GMMR0DECL(int) GMMR0InitialReservationReq(PGVM pGVM, PVM pVM, VMCPUID idCpu, PGMMINITIALRESERVATIONREQ pReq)
1626{
1627 /*
1628 * Validate input and pass it on.
1629 */
1630 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
1631 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
1632 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
1633
1634 return GMMR0InitialReservation(pGVM, pVM, idCpu, pReq->cBasePages, pReq->cShadowPages,
1635 pReq->cFixedPages, pReq->enmPolicy, pReq->enmPriority);
1636}
1637
1638
1639/**
1640 * This updates the memory reservation with the additional MMIO2 and ROM pages.
1641 *
1642 * @returns VBox status code.
1643 * @retval VERR_GMM_MEMORY_RESERVATION_DECLINED
1644 *
1645 * @param pGVM The global (ring-0) VM structure.
1646 * @param pVM The cross context VM structure.
1647 * @param idCpu The VCPU id.
1648 * @param cBasePages The number of pages that may be allocated for the base RAM and ROMs.
1649 * This does not include MMIO2 and similar.
1650 * @param cShadowPages The number of pages that may be allocated for shadow paging structures.
1651 * @param cFixedPages The number of pages that may be allocated for fixed objects like the
1652 * hyper heap, MMIO2 and similar.
1653 *
1654 * @thread EMT(idCpu)
1655 */
1656GMMR0DECL(int) GMMR0UpdateReservation(PGVM pGVM, PVM pVM, VMCPUID idCpu, uint64_t cBasePages,
1657 uint32_t cShadowPages, uint32_t cFixedPages)
1658{
1659 LogFlow(("GMMR0UpdateReservation: pGVM=%p pVM=%p cBasePages=%#llx cShadowPages=%#x cFixedPages=%#x\n",
1660 pGVM, pVM, cBasePages, cShadowPages, cFixedPages));
1661
1662 /*
1663 * Validate, get basics and take the semaphore.
1664 */
1665 PGMM pGMM;
1666 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
1667 int rc = GVMMR0ValidateGVMandVMandEMT(pGVM, pVM, idCpu);
1668 if (RT_FAILURE(rc))
1669 return rc;
1670
1671 AssertReturn(cBasePages, VERR_INVALID_PARAMETER);
1672 AssertReturn(cShadowPages, VERR_INVALID_PARAMETER);
1673 AssertReturn(cFixedPages, VERR_INVALID_PARAMETER);
1674
1675 gmmR0MutexAcquire(pGMM);
1676 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
1677 {
1678 if ( pGVM->gmm.s.Stats.Reserved.cBasePages
1679 && pGVM->gmm.s.Stats.Reserved.cFixedPages
1680 && pGVM->gmm.s.Stats.Reserved.cShadowPages)
1681 {
1682 /*
1683 * Check if we can accommodate this.
1684 */
1685 /* ... later ... */
1686 if (RT_SUCCESS(rc))
1687 {
1688 /*
1689 * Update the records.
1690 */
1691 pGMM->cReservedPages -= pGVM->gmm.s.Stats.Reserved.cBasePages
1692 + pGVM->gmm.s.Stats.Reserved.cFixedPages
1693 + pGVM->gmm.s.Stats.Reserved.cShadowPages;
1694 pGMM->cReservedPages += cBasePages + cFixedPages + cShadowPages;
1695
1696 pGVM->gmm.s.Stats.Reserved.cBasePages = cBasePages;
1697 pGVM->gmm.s.Stats.Reserved.cFixedPages = cFixedPages;
1698 pGVM->gmm.s.Stats.Reserved.cShadowPages = cShadowPages;
1699 }
1700 }
1701 else
1702 rc = VERR_WRONG_ORDER;
1703 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
1704 }
1705 else
1706 rc = VERR_GMM_IS_NOT_SANE;
1707 gmmR0MutexRelease(pGMM);
1708 LogFlow(("GMMR0UpdateReservation: returns %Rrc\n", rc));
1709 return rc;
1710}
1711
1712
1713/**
1714 * VMMR0 request wrapper for GMMR0UpdateReservation.
1715 *
1716 * @returns see GMMR0UpdateReservation.
1717 * @param pGVM The global (ring-0) VM structure.
1718 * @param pVM The cross context VM structure.
1719 * @param idCpu The VCPU id.
1720 * @param pReq Pointer to the request packet.
1721 */
1722GMMR0DECL(int) GMMR0UpdateReservationReq(PGVM pGVM, PVM pVM, VMCPUID idCpu, PGMMUPDATERESERVATIONREQ pReq)
1723{
1724 /*
1725 * Validate input and pass it on.
1726 */
1727 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
1728 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
1729 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
1730
1731 return GMMR0UpdateReservation(pGVM, pVM, idCpu, pReq->cBasePages, pReq->cShadowPages, pReq->cFixedPages);
1732}
1733
1734#ifdef GMMR0_WITH_SANITY_CHECK
1735
1736/**
1737 * Performs sanity checks on a free set.
1738 *
1739 * @returns Error count.
1740 *
1741 * @param pGMM Pointer to the GMM instance.
1742 * @param pSet Pointer to the set.
1743 * @param pszSetName The set name.
1744 * @param pszFunction The function from which it was called.
1745 * @param uLine The line number.
1746 */
1747static uint32_t gmmR0SanityCheckSet(PGMM pGMM, PGMMCHUNKFREESET pSet, const char *pszSetName,
1748 const char *pszFunction, unsigned uLineNo)
1749{
1750 uint32_t cErrors = 0;
1751
1752 /*
1753 * Count the free pages in all the chunks and match it against pSet->cFreePages.
1754 */
1755 uint32_t cPages = 0;
1756 for (unsigned i = 0; i < RT_ELEMENTS(pSet->apLists); i++)
1757 {
1758 for (PGMMCHUNK pCur = pSet->apLists[i]; pCur; pCur = pCur->pFreeNext)
1759 {
1760 /** @todo check that the chunk is hash into the right set. */
1761 cPages += pCur->cFree;
1762 }
1763 }
1764 if (RT_UNLIKELY(cPages != pSet->cFreePages))
1765 {
1766 SUPR0Printf("GMM insanity: found %#x pages in the %s set, expected %#x. (%s, line %u)\n",
1767 cPages, pszSetName, pSet->cFreePages, pszFunction, uLineNo);
1768 cErrors++;
1769 }
1770
1771 return cErrors;
1772}
1773
1774
1775/**
1776 * Performs some sanity checks on the GMM while owning lock.
1777 *
1778 * @returns Error count.
1779 *
1780 * @param pGMM Pointer to the GMM instance.
1781 * @param pszFunction The function from which it is called.
1782 * @param uLineNo The line number.
1783 */
1784static uint32_t gmmR0SanityCheck(PGMM pGMM, const char *pszFunction, unsigned uLineNo)
1785{
1786 uint32_t cErrors = 0;
1787
1788 cErrors += gmmR0SanityCheckSet(pGMM, &pGMM->PrivateX, "private", pszFunction, uLineNo);
1789 cErrors += gmmR0SanityCheckSet(pGMM, &pGMM->Shared, "shared", pszFunction, uLineNo);
1790 /** @todo add more sanity checks. */
1791
1792 return cErrors;
1793}
1794
1795#endif /* GMMR0_WITH_SANITY_CHECK */
1796
1797/**
1798 * Looks up a chunk in the tree and fill in the TLB entry for it.
1799 *
1800 * This is not expected to fail and will bitch if it does.
1801 *
1802 * @returns Pointer to the allocation chunk, NULL if not found.
1803 * @param pGMM Pointer to the GMM instance.
1804 * @param idChunk The ID of the chunk to find.
1805 * @param pTlbe Pointer to the TLB entry.
1806 */
1807static PGMMCHUNK gmmR0GetChunkSlow(PGMM pGMM, uint32_t idChunk, PGMMCHUNKTLBE pTlbe)
1808{
1809 PGMMCHUNK pChunk = (PGMMCHUNK)RTAvlU32Get(&pGMM->pChunks, idChunk);
1810 AssertMsgReturn(pChunk, ("Chunk %#x not found!\n", idChunk), NULL);
1811 pTlbe->idChunk = idChunk;
1812 pTlbe->pChunk = pChunk;
1813 return pChunk;
1814}
1815
1816
1817/**
1818 * Finds a allocation chunk.
1819 *
1820 * This is not expected to fail and will bitch if it does.
1821 *
1822 * @returns Pointer to the allocation chunk, NULL if not found.
1823 * @param pGMM Pointer to the GMM instance.
1824 * @param idChunk The ID of the chunk to find.
1825 */
1826DECLINLINE(PGMMCHUNK) gmmR0GetChunk(PGMM pGMM, uint32_t idChunk)
1827{
1828 /*
1829 * Do a TLB lookup, branch if not in the TLB.
1830 */
1831 PGMMCHUNKTLBE pTlbe = &pGMM->ChunkTLB.aEntries[GMM_CHUNKTLB_IDX(idChunk)];
1832 if ( pTlbe->idChunk != idChunk
1833 || !pTlbe->pChunk)
1834 return gmmR0GetChunkSlow(pGMM, idChunk, pTlbe);
1835 return pTlbe->pChunk;
1836}
1837
1838
1839/**
1840 * Finds a page.
1841 *
1842 * This is not expected to fail and will bitch if it does.
1843 *
1844 * @returns Pointer to the page, NULL if not found.
1845 * @param pGMM Pointer to the GMM instance.
1846 * @param idPage The ID of the page to find.
1847 */
1848DECLINLINE(PGMMPAGE) gmmR0GetPage(PGMM pGMM, uint32_t idPage)
1849{
1850 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
1851 if (RT_LIKELY(pChunk))
1852 return &pChunk->aPages[idPage & GMM_PAGEID_IDX_MASK];
1853 return NULL;
1854}
1855
1856
1857#if 0 /* unused */
1858/**
1859 * Gets the host physical address for a page given by it's ID.
1860 *
1861 * @returns The host physical address or NIL_RTHCPHYS.
1862 * @param pGMM Pointer to the GMM instance.
1863 * @param idPage The ID of the page to find.
1864 */
1865DECLINLINE(RTHCPHYS) gmmR0GetPageHCPhys(PGMM pGMM, uint32_t idPage)
1866{
1867 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
1868 if (RT_LIKELY(pChunk))
1869 return RTR0MemObjGetPagePhysAddr(pChunk->hMemObj, idPage & GMM_PAGEID_IDX_MASK);
1870 return NIL_RTHCPHYS;
1871}
1872#endif /* unused */
1873
1874
1875/**
1876 * Selects the appropriate free list given the number of free pages.
1877 *
1878 * @returns Free list index.
1879 * @param cFree The number of free pages in the chunk.
1880 */
1881DECLINLINE(unsigned) gmmR0SelectFreeSetList(unsigned cFree)
1882{
1883 unsigned iList = cFree >> GMM_CHUNK_FREE_SET_SHIFT;
1884 AssertMsg(iList < RT_SIZEOFMEMB(GMMCHUNKFREESET, apLists) / RT_SIZEOFMEMB(GMMCHUNKFREESET, apLists[0]),
1885 ("%d (%u)\n", iList, cFree));
1886 return iList;
1887}
1888
1889
1890/**
1891 * Unlinks the chunk from the free list it's currently on (if any).
1892 *
1893 * @param pChunk The allocation chunk.
1894 */
1895DECLINLINE(void) gmmR0UnlinkChunk(PGMMCHUNK pChunk)
1896{
1897 PGMMCHUNKFREESET pSet = pChunk->pSet;
1898 if (RT_LIKELY(pSet))
1899 {
1900 pSet->cFreePages -= pChunk->cFree;
1901 pSet->idGeneration++;
1902
1903 PGMMCHUNK pPrev = pChunk->pFreePrev;
1904 PGMMCHUNK pNext = pChunk->pFreeNext;
1905 if (pPrev)
1906 pPrev->pFreeNext = pNext;
1907 else
1908 pSet->apLists[gmmR0SelectFreeSetList(pChunk->cFree)] = pNext;
1909 if (pNext)
1910 pNext->pFreePrev = pPrev;
1911
1912 pChunk->pSet = NULL;
1913 pChunk->pFreeNext = NULL;
1914 pChunk->pFreePrev = NULL;
1915 }
1916 else
1917 {
1918 Assert(!pChunk->pFreeNext);
1919 Assert(!pChunk->pFreePrev);
1920 Assert(!pChunk->cFree);
1921 }
1922}
1923
1924
1925/**
1926 * Links the chunk onto the appropriate free list in the specified free set.
1927 *
1928 * If no free entries, it's not linked into any list.
1929 *
1930 * @param pChunk The allocation chunk.
1931 * @param pSet The free set.
1932 */
1933DECLINLINE(void) gmmR0LinkChunk(PGMMCHUNK pChunk, PGMMCHUNKFREESET pSet)
1934{
1935 Assert(!pChunk->pSet);
1936 Assert(!pChunk->pFreeNext);
1937 Assert(!pChunk->pFreePrev);
1938
1939 if (pChunk->cFree > 0)
1940 {
1941 pChunk->pSet = pSet;
1942 pChunk->pFreePrev = NULL;
1943 unsigned const iList = gmmR0SelectFreeSetList(pChunk->cFree);
1944 pChunk->pFreeNext = pSet->apLists[iList];
1945 if (pChunk->pFreeNext)
1946 pChunk->pFreeNext->pFreePrev = pChunk;
1947 pSet->apLists[iList] = pChunk;
1948
1949 pSet->cFreePages += pChunk->cFree;
1950 pSet->idGeneration++;
1951 }
1952}
1953
1954
1955/**
1956 * Links the chunk onto the appropriate free list in the specified free set.
1957 *
1958 * If no free entries, it's not linked into any list.
1959 *
1960 * @param pGMM Pointer to the GMM instance.
1961 * @param pGVM Pointer to the kernel-only VM instace data.
1962 * @param pChunk The allocation chunk.
1963 */
1964DECLINLINE(void) gmmR0SelectSetAndLinkChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk)
1965{
1966 PGMMCHUNKFREESET pSet;
1967 if (pGMM->fBoundMemoryMode)
1968 pSet = &pGVM->gmm.s.Private;
1969 else if (pChunk->cShared)
1970 pSet = &pGMM->Shared;
1971 else
1972 pSet = &pGMM->PrivateX;
1973 gmmR0LinkChunk(pChunk, pSet);
1974}
1975
1976
1977/**
1978 * Frees a Chunk ID.
1979 *
1980 * @param pGMM Pointer to the GMM instance.
1981 * @param idChunk The Chunk ID to free.
1982 */
1983static void gmmR0FreeChunkId(PGMM pGMM, uint32_t idChunk)
1984{
1985 AssertReturnVoid(idChunk != NIL_GMM_CHUNKID);
1986 AssertMsg(ASMBitTest(&pGMM->bmChunkId[0], idChunk), ("%#x\n", idChunk));
1987 ASMAtomicBitClear(&pGMM->bmChunkId[0], idChunk);
1988}
1989
1990
1991/**
1992 * Allocates a new Chunk ID.
1993 *
1994 * @returns The Chunk ID.
1995 * @param pGMM Pointer to the GMM instance.
1996 */
1997static uint32_t gmmR0AllocateChunkId(PGMM pGMM)
1998{
1999 AssertCompile(!((GMM_CHUNKID_LAST + 1) & 31)); /* must be a multiple of 32 */
2000 AssertCompile(NIL_GMM_CHUNKID == 0);
2001
2002 /*
2003 * Try the next sequential one.
2004 */
2005 int32_t idChunk = ++pGMM->idChunkPrev;
2006#if 0 /** @todo enable this code */
2007 if ( idChunk <= GMM_CHUNKID_LAST
2008 && idChunk > NIL_GMM_CHUNKID
2009 && !ASMAtomicBitTestAndSet(&pVMM->bmChunkId[0], idChunk))
2010 return idChunk;
2011#endif
2012
2013 /*
2014 * Scan sequentially from the last one.
2015 */
2016 if ( (uint32_t)idChunk < GMM_CHUNKID_LAST
2017 && idChunk > NIL_GMM_CHUNKID)
2018 {
2019 idChunk = ASMBitNextClear(&pGMM->bmChunkId[0], GMM_CHUNKID_LAST + 1, idChunk - 1);
2020 if (idChunk > NIL_GMM_CHUNKID)
2021 {
2022 AssertMsgReturn(!ASMAtomicBitTestAndSet(&pGMM->bmChunkId[0], idChunk), ("%#x\n", idChunk), NIL_GMM_CHUNKID);
2023 return pGMM->idChunkPrev = idChunk;
2024 }
2025 }
2026
2027 /*
2028 * Ok, scan from the start.
2029 * We're not racing anyone, so there is no need to expect failures or have restart loops.
2030 */
2031 idChunk = ASMBitFirstClear(&pGMM->bmChunkId[0], GMM_CHUNKID_LAST + 1);
2032 AssertMsgReturn(idChunk > NIL_GMM_CHUNKID, ("%#x\n", idChunk), NIL_GVM_HANDLE);
2033 AssertMsgReturn(!ASMAtomicBitTestAndSet(&pGMM->bmChunkId[0], idChunk), ("%#x\n", idChunk), NIL_GMM_CHUNKID);
2034
2035 return pGMM->idChunkPrev = idChunk;
2036}
2037
2038
2039/**
2040 * Allocates one private page.
2041 *
2042 * Worker for gmmR0AllocatePages.
2043 *
2044 * @param pChunk The chunk to allocate it from.
2045 * @param hGVM The GVM handle of the VM requesting memory.
2046 * @param pPageDesc The page descriptor.
2047 */
2048static void gmmR0AllocatePage(PGMMCHUNK pChunk, uint32_t hGVM, PGMMPAGEDESC pPageDesc)
2049{
2050 /* update the chunk stats. */
2051 if (pChunk->hGVM == NIL_GVM_HANDLE)
2052 pChunk->hGVM = hGVM;
2053 Assert(pChunk->cFree);
2054 pChunk->cFree--;
2055 pChunk->cPrivate++;
2056
2057 /* unlink the first free page. */
2058 const uint32_t iPage = pChunk->iFreeHead;
2059 AssertReleaseMsg(iPage < RT_ELEMENTS(pChunk->aPages), ("%d\n", iPage));
2060 PGMMPAGE pPage = &pChunk->aPages[iPage];
2061 Assert(GMM_PAGE_IS_FREE(pPage));
2062 pChunk->iFreeHead = pPage->Free.iNext;
2063 Log3(("A pPage=%p iPage=%#x/%#x u2State=%d iFreeHead=%#x iNext=%#x\n",
2064 pPage, iPage, (pChunk->Core.Key << GMM_CHUNKID_SHIFT) | iPage,
2065 pPage->Common.u2State, pChunk->iFreeHead, pPage->Free.iNext));
2066
2067 /* make the page private. */
2068 pPage->u = 0;
2069 AssertCompile(GMM_PAGE_STATE_PRIVATE == 0);
2070 pPage->Private.hGVM = hGVM;
2071 AssertCompile(NIL_RTHCPHYS >= GMM_GCPHYS_LAST);
2072 AssertCompile(GMM_GCPHYS_UNSHAREABLE >= GMM_GCPHYS_LAST);
2073 if (pPageDesc->HCPhysGCPhys <= GMM_GCPHYS_LAST)
2074 pPage->Private.pfn = pPageDesc->HCPhysGCPhys >> PAGE_SHIFT;
2075 else
2076 pPage->Private.pfn = GMM_PAGE_PFN_UNSHAREABLE; /* unshareable / unassigned - same thing. */
2077
2078 /* update the page descriptor. */
2079 pPageDesc->HCPhysGCPhys = RTR0MemObjGetPagePhysAddr(pChunk->hMemObj, iPage);
2080 Assert(pPageDesc->HCPhysGCPhys != NIL_RTHCPHYS);
2081 pPageDesc->idPage = (pChunk->Core.Key << GMM_CHUNKID_SHIFT) | iPage;
2082 pPageDesc->idSharedPage = NIL_GMM_PAGEID;
2083}
2084
2085
2086/**
2087 * Picks the free pages from a chunk.
2088 *
2089 * @returns The new page descriptor table index.
2090 * @param pChunk The chunk.
2091 * @param hGVM The affinity of the chunk. NIL_GVM_HANDLE for no
2092 * affinity.
2093 * @param iPage The current page descriptor table index.
2094 * @param cPages The total number of pages to allocate.
2095 * @param paPages The page descriptor table (input + ouput).
2096 */
2097static uint32_t gmmR0AllocatePagesFromChunk(PGMMCHUNK pChunk, uint16_t const hGVM, uint32_t iPage, uint32_t cPages,
2098 PGMMPAGEDESC paPages)
2099{
2100 PGMMCHUNKFREESET pSet = pChunk->pSet; Assert(pSet);
2101 gmmR0UnlinkChunk(pChunk);
2102
2103 for (; pChunk->cFree && iPage < cPages; iPage++)
2104 gmmR0AllocatePage(pChunk, hGVM, &paPages[iPage]);
2105
2106 gmmR0LinkChunk(pChunk, pSet);
2107 return iPage;
2108}
2109
2110
2111/**
2112 * Registers a new chunk of memory.
2113 *
2114 * This is called by both gmmR0AllocateOneChunk and GMMR0SeedChunk.
2115 *
2116 * @returns VBox status code. On success, the giant GMM lock will be held, the
2117 * caller must release it (ugly).
2118 * @param pGMM Pointer to the GMM instance.
2119 * @param pSet Pointer to the set.
2120 * @param MemObj The memory object for the chunk.
2121 * @param hGVM The affinity of the chunk. NIL_GVM_HANDLE for no
2122 * affinity.
2123 * @param fChunkFlags The chunk flags, GMM_CHUNK_FLAGS_XXX.
2124 * @param ppChunk Chunk address (out). Optional.
2125 *
2126 * @remarks The caller must not own the giant GMM mutex.
2127 * The giant GMM mutex will be acquired and returned acquired in
2128 * the success path. On failure, no locks will be held.
2129 */
2130static int gmmR0RegisterChunk(PGMM pGMM, PGMMCHUNKFREESET pSet, RTR0MEMOBJ MemObj, uint16_t hGVM, uint16_t fChunkFlags,
2131 PGMMCHUNK *ppChunk)
2132{
2133 Assert(pGMM->hMtxOwner != RTThreadNativeSelf());
2134 Assert(hGVM != NIL_GVM_HANDLE || pGMM->fBoundMemoryMode);
2135 Assert(fChunkFlags == 0 || fChunkFlags == GMM_CHUNK_FLAGS_LARGE_PAGE);
2136
2137 int rc;
2138 PGMMCHUNK pChunk = (PGMMCHUNK)RTMemAllocZ(sizeof(*pChunk));
2139 if (pChunk)
2140 {
2141 /*
2142 * Initialize it.
2143 */
2144 pChunk->hMemObj = MemObj;
2145 pChunk->cFree = GMM_CHUNK_NUM_PAGES;
2146 pChunk->hGVM = hGVM;
2147 /*pChunk->iFreeHead = 0;*/
2148 pChunk->idNumaNode = gmmR0GetCurrentNumaNodeId();
2149 pChunk->iChunkMtx = UINT8_MAX;
2150 pChunk->fFlags = fChunkFlags;
2151 for (unsigned iPage = 0; iPage < RT_ELEMENTS(pChunk->aPages) - 1; iPage++)
2152 {
2153 pChunk->aPages[iPage].Free.u2State = GMM_PAGE_STATE_FREE;
2154 pChunk->aPages[iPage].Free.iNext = iPage + 1;
2155 }
2156 pChunk->aPages[RT_ELEMENTS(pChunk->aPages) - 1].Free.u2State = GMM_PAGE_STATE_FREE;
2157 pChunk->aPages[RT_ELEMENTS(pChunk->aPages) - 1].Free.iNext = UINT16_MAX;
2158
2159 /*
2160 * Allocate a Chunk ID and insert it into the tree.
2161 * This has to be done behind the mutex of course.
2162 */
2163 rc = gmmR0MutexAcquire(pGMM);
2164 if (RT_SUCCESS(rc))
2165 {
2166 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
2167 {
2168 pChunk->Core.Key = gmmR0AllocateChunkId(pGMM);
2169 if ( pChunk->Core.Key != NIL_GMM_CHUNKID
2170 && pChunk->Core.Key <= GMM_CHUNKID_LAST
2171 && RTAvlU32Insert(&pGMM->pChunks, &pChunk->Core))
2172 {
2173 pGMM->cChunks++;
2174 RTListAppend(&pGMM->ChunkList, &pChunk->ListNode);
2175 gmmR0LinkChunk(pChunk, pSet);
2176 LogFlow(("gmmR0RegisterChunk: pChunk=%p id=%#x cChunks=%d\n", pChunk, pChunk->Core.Key, pGMM->cChunks));
2177
2178 if (ppChunk)
2179 *ppChunk = pChunk;
2180 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
2181 return VINF_SUCCESS;
2182 }
2183
2184 /* bail out */
2185 rc = VERR_GMM_CHUNK_INSERT;
2186 }
2187 else
2188 rc = VERR_GMM_IS_NOT_SANE;
2189 gmmR0MutexRelease(pGMM);
2190 }
2191
2192 RTMemFree(pChunk);
2193 }
2194 else
2195 rc = VERR_NO_MEMORY;
2196 return rc;
2197}
2198
2199
2200/**
2201 * Allocate a new chunk, immediately pick the requested pages from it, and adds
2202 * what's remaining to the specified free set.
2203 *
2204 * @note This will leave the giant mutex while allocating the new chunk!
2205 *
2206 * @returns VBox status code.
2207 * @param pGMM Pointer to the GMM instance data.
2208 * @param pGVM Pointer to the kernel-only VM instace data.
2209 * @param pSet Pointer to the free set.
2210 * @param cPages The number of pages requested.
2211 * @param paPages The page descriptor table (input + output).
2212 * @param piPage The pointer to the page descriptor table index variable.
2213 * This will be updated.
2214 */
2215static int gmmR0AllocateChunkNew(PGMM pGMM, PGVM pGVM, PGMMCHUNKFREESET pSet, uint32_t cPages,
2216 PGMMPAGEDESC paPages, uint32_t *piPage)
2217{
2218 gmmR0MutexRelease(pGMM);
2219
2220 RTR0MEMOBJ hMemObj;
2221 int rc = RTR0MemObjAllocPhysNC(&hMemObj, GMM_CHUNK_SIZE, NIL_RTHCPHYS);
2222 if (RT_SUCCESS(rc))
2223 {
2224/** @todo Duplicate gmmR0RegisterChunk here so we can avoid chaining up the
2225 * free pages first and then unchaining them right afterwards. Instead
2226 * do as much work as possible without holding the giant lock. */
2227 PGMMCHUNK pChunk;
2228 rc = gmmR0RegisterChunk(pGMM, pSet, hMemObj, pGVM->hSelf, 0 /*fChunkFlags*/, &pChunk);
2229 if (RT_SUCCESS(rc))
2230 {
2231 *piPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, *piPage, cPages, paPages);
2232 return VINF_SUCCESS;
2233 }
2234
2235 /* bail out */
2236 RTR0MemObjFree(hMemObj, false /* fFreeMappings */);
2237 }
2238
2239 int rc2 = gmmR0MutexAcquire(pGMM);
2240 AssertRCReturn(rc2, RT_FAILURE(rc) ? rc : rc2);
2241 return rc;
2242
2243}
2244
2245
2246/**
2247 * As a last restort we'll pick any page we can get.
2248 *
2249 * @returns The new page descriptor table index.
2250 * @param pSet The set to pick from.
2251 * @param pGVM Pointer to the global VM structure.
2252 * @param iPage The current page descriptor table index.
2253 * @param cPages The total number of pages to allocate.
2254 * @param paPages The page descriptor table (input + ouput).
2255 */
2256static uint32_t gmmR0AllocatePagesIndiscriminately(PGMMCHUNKFREESET pSet, PGVM pGVM,
2257 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2258{
2259 unsigned iList = RT_ELEMENTS(pSet->apLists);
2260 while (iList-- > 0)
2261 {
2262 PGMMCHUNK pChunk = pSet->apLists[iList];
2263 while (pChunk)
2264 {
2265 PGMMCHUNK pNext = pChunk->pFreeNext;
2266
2267 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2268 if (iPage >= cPages)
2269 return iPage;
2270
2271 pChunk = pNext;
2272 }
2273 }
2274 return iPage;
2275}
2276
2277
2278/**
2279 * Pick pages from empty chunks on the same NUMA node.
2280 *
2281 * @returns The new page descriptor table index.
2282 * @param pSet The set to pick from.
2283 * @param pGVM Pointer to the global VM structure.
2284 * @param iPage The current page descriptor table index.
2285 * @param cPages The total number of pages to allocate.
2286 * @param paPages The page descriptor table (input + ouput).
2287 */
2288static uint32_t gmmR0AllocatePagesFromEmptyChunksOnSameNode(PGMMCHUNKFREESET pSet, PGVM pGVM,
2289 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2290{
2291 PGMMCHUNK pChunk = pSet->apLists[GMM_CHUNK_FREE_SET_UNUSED_LIST];
2292 if (pChunk)
2293 {
2294 uint16_t const idNumaNode = gmmR0GetCurrentNumaNodeId();
2295 while (pChunk)
2296 {
2297 PGMMCHUNK pNext = pChunk->pFreeNext;
2298
2299 if (pChunk->idNumaNode == idNumaNode)
2300 {
2301 pChunk->hGVM = pGVM->hSelf;
2302 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2303 if (iPage >= cPages)
2304 {
2305 pGVM->gmm.s.idLastChunkHint = pChunk->cFree ? pChunk->Core.Key : NIL_GMM_CHUNKID;
2306 return iPage;
2307 }
2308 }
2309
2310 pChunk = pNext;
2311 }
2312 }
2313 return iPage;
2314}
2315
2316
2317/**
2318 * Pick pages from non-empty chunks on the same NUMA node.
2319 *
2320 * @returns The new page descriptor table index.
2321 * @param pSet The set to pick from.
2322 * @param pGVM Pointer to the global VM structure.
2323 * @param iPage The current page descriptor table index.
2324 * @param cPages The total number of pages to allocate.
2325 * @param paPages The page descriptor table (input + ouput).
2326 */
2327static uint32_t gmmR0AllocatePagesFromSameNode(PGMMCHUNKFREESET pSet, PGVM pGVM,
2328 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2329{
2330 /** @todo start by picking from chunks with about the right size first? */
2331 uint16_t const idNumaNode = gmmR0GetCurrentNumaNodeId();
2332 unsigned iList = GMM_CHUNK_FREE_SET_UNUSED_LIST;
2333 while (iList-- > 0)
2334 {
2335 PGMMCHUNK pChunk = pSet->apLists[iList];
2336 while (pChunk)
2337 {
2338 PGMMCHUNK pNext = pChunk->pFreeNext;
2339
2340 if (pChunk->idNumaNode == idNumaNode)
2341 {
2342 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2343 if (iPage >= cPages)
2344 {
2345 pGVM->gmm.s.idLastChunkHint = pChunk->cFree ? pChunk->Core.Key : NIL_GMM_CHUNKID;
2346 return iPage;
2347 }
2348 }
2349
2350 pChunk = pNext;
2351 }
2352 }
2353 return iPage;
2354}
2355
2356
2357/**
2358 * Pick pages that are in chunks already associated with the VM.
2359 *
2360 * @returns The new page descriptor table index.
2361 * @param pGMM Pointer to the GMM instance data.
2362 * @param pGVM Pointer to the global VM structure.
2363 * @param pSet The set to pick from.
2364 * @param iPage The current page descriptor table index.
2365 * @param cPages The total number of pages to allocate.
2366 * @param paPages The page descriptor table (input + ouput).
2367 */
2368static uint32_t gmmR0AllocatePagesAssociatedWithVM(PGMM pGMM, PGVM pGVM, PGMMCHUNKFREESET pSet,
2369 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2370{
2371 uint16_t const hGVM = pGVM->hSelf;
2372
2373 /* Hint. */
2374 if (pGVM->gmm.s.idLastChunkHint != NIL_GMM_CHUNKID)
2375 {
2376 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, pGVM->gmm.s.idLastChunkHint);
2377 if (pChunk && pChunk->cFree)
2378 {
2379 iPage = gmmR0AllocatePagesFromChunk(pChunk, hGVM, iPage, cPages, paPages);
2380 if (iPage >= cPages)
2381 return iPage;
2382 }
2383 }
2384
2385 /* Scan. */
2386 for (unsigned iList = 0; iList < RT_ELEMENTS(pSet->apLists); iList++)
2387 {
2388 PGMMCHUNK pChunk = pSet->apLists[iList];
2389 while (pChunk)
2390 {
2391 PGMMCHUNK pNext = pChunk->pFreeNext;
2392
2393 if (pChunk->hGVM == hGVM)
2394 {
2395 iPage = gmmR0AllocatePagesFromChunk(pChunk, hGVM, iPage, cPages, paPages);
2396 if (iPage >= cPages)
2397 {
2398 pGVM->gmm.s.idLastChunkHint = pChunk->cFree ? pChunk->Core.Key : NIL_GMM_CHUNKID;
2399 return iPage;
2400 }
2401 }
2402
2403 pChunk = pNext;
2404 }
2405 }
2406 return iPage;
2407}
2408
2409
2410
2411/**
2412 * Pick pages in bound memory mode.
2413 *
2414 * @returns The new page descriptor table index.
2415 * @param pGVM Pointer to the global VM structure.
2416 * @param iPage The current page descriptor table index.
2417 * @param cPages The total number of pages to allocate.
2418 * @param paPages The page descriptor table (input + ouput).
2419 */
2420static uint32_t gmmR0AllocatePagesInBoundMode(PGVM pGVM, uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2421{
2422 for (unsigned iList = 0; iList < RT_ELEMENTS(pGVM->gmm.s.Private.apLists); iList++)
2423 {
2424 PGMMCHUNK pChunk = pGVM->gmm.s.Private.apLists[iList];
2425 while (pChunk)
2426 {
2427 Assert(pChunk->hGVM == pGVM->hSelf);
2428 PGMMCHUNK pNext = pChunk->pFreeNext;
2429 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2430 if (iPage >= cPages)
2431 return iPage;
2432 pChunk = pNext;
2433 }
2434 }
2435 return iPage;
2436}
2437
2438
2439/**
2440 * Checks if we should start picking pages from chunks of other VMs because
2441 * we're getting close to the system memory or reserved limit.
2442 *
2443 * @returns @c true if we should, @c false if we should first try allocate more
2444 * chunks.
2445 */
2446static bool gmmR0ShouldAllocatePagesInOtherChunksBecauseOfLimits(PGVM pGVM)
2447{
2448 /*
2449 * Don't allocate a new chunk if we're
2450 */
2451 uint64_t cPgReserved = pGVM->gmm.s.Stats.Reserved.cBasePages
2452 + pGVM->gmm.s.Stats.Reserved.cFixedPages
2453 - pGVM->gmm.s.Stats.cBalloonedPages
2454 /** @todo what about shared pages? */;
2455 uint64_t cPgAllocated = pGVM->gmm.s.Stats.Allocated.cBasePages
2456 + pGVM->gmm.s.Stats.Allocated.cFixedPages;
2457 uint64_t cPgDelta = cPgReserved - cPgAllocated;
2458 if (cPgDelta < GMM_CHUNK_NUM_PAGES * 4)
2459 return true;
2460 /** @todo make the threshold configurable, also test the code to see if
2461 * this ever kicks in (we might be reserving too much or smth). */
2462
2463 /*
2464 * Check how close we're to the max memory limit and how many fragments
2465 * there are?...
2466 */
2467 /** @todo. */
2468
2469 return false;
2470}
2471
2472
2473/**
2474 * Checks if we should start picking pages from chunks of other VMs because
2475 * there is a lot of free pages around.
2476 *
2477 * @returns @c true if we should, @c false if we should first try allocate more
2478 * chunks.
2479 */
2480static bool gmmR0ShouldAllocatePagesInOtherChunksBecauseOfLotsFree(PGMM pGMM)
2481{
2482 /*
2483 * Setting the limit at 16 chunks (32 MB) at the moment.
2484 */
2485 if (pGMM->PrivateX.cFreePages >= GMM_CHUNK_NUM_PAGES * 16)
2486 return true;
2487 return false;
2488}
2489
2490
2491/**
2492 * Common worker for GMMR0AllocateHandyPages and GMMR0AllocatePages.
2493 *
2494 * @returns VBox status code:
2495 * @retval VINF_SUCCESS on success.
2496 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk or
2497 * gmmR0AllocateMoreChunks is necessary.
2498 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
2499 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
2500 * that is we're trying to allocate more than we've reserved.
2501 *
2502 * @param pGMM Pointer to the GMM instance data.
2503 * @param pGVM Pointer to the VM.
2504 * @param cPages The number of pages to allocate.
2505 * @param paPages Pointer to the page descriptors. See GMMPAGEDESC for
2506 * details on what is expected on input.
2507 * @param enmAccount The account to charge.
2508 *
2509 * @remarks Call takes the giant GMM lock.
2510 */
2511static int gmmR0AllocatePagesNew(PGMM pGMM, PGVM pGVM, uint32_t cPages, PGMMPAGEDESC paPages, GMMACCOUNT enmAccount)
2512{
2513 Assert(pGMM->hMtxOwner == RTThreadNativeSelf());
2514
2515 /*
2516 * Check allocation limits.
2517 */
2518 if (RT_UNLIKELY(pGMM->cAllocatedPages + cPages > pGMM->cMaxPages))
2519 return VERR_GMM_HIT_GLOBAL_LIMIT;
2520
2521 switch (enmAccount)
2522 {
2523 case GMMACCOUNT_BASE:
2524 if (RT_UNLIKELY( pGVM->gmm.s.Stats.Allocated.cBasePages + pGVM->gmm.s.Stats.cBalloonedPages + cPages
2525 > pGVM->gmm.s.Stats.Reserved.cBasePages))
2526 {
2527 Log(("gmmR0AllocatePages:Base: Reserved=%#llx Allocated+Ballooned+Requested=%#llx+%#llx+%#x!\n",
2528 pGVM->gmm.s.Stats.Reserved.cBasePages, pGVM->gmm.s.Stats.Allocated.cBasePages,
2529 pGVM->gmm.s.Stats.cBalloonedPages, cPages));
2530 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
2531 }
2532 break;
2533 case GMMACCOUNT_SHADOW:
2534 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cShadowPages + cPages > pGVM->gmm.s.Stats.Reserved.cShadowPages))
2535 {
2536 Log(("gmmR0AllocatePages:Shadow: Reserved=%#x Allocated+Requested=%#x+%#x!\n",
2537 pGVM->gmm.s.Stats.Reserved.cShadowPages, pGVM->gmm.s.Stats.Allocated.cShadowPages, cPages));
2538 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
2539 }
2540 break;
2541 case GMMACCOUNT_FIXED:
2542 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cFixedPages + cPages > pGVM->gmm.s.Stats.Reserved.cFixedPages))
2543 {
2544 Log(("gmmR0AllocatePages:Fixed: Reserved=%#x Allocated+Requested=%#x+%#x!\n",
2545 pGVM->gmm.s.Stats.Reserved.cFixedPages, pGVM->gmm.s.Stats.Allocated.cFixedPages, cPages));
2546 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
2547 }
2548 break;
2549 default:
2550 AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
2551 }
2552
2553 /*
2554 * If we're in legacy memory mode, it's easy to figure if we have
2555 * sufficient number of pages up-front.
2556 */
2557 if ( pGMM->fLegacyAllocationMode
2558 && pGVM->gmm.s.Private.cFreePages < cPages)
2559 {
2560 Assert(pGMM->fBoundMemoryMode);
2561 return VERR_GMM_SEED_ME;
2562 }
2563
2564 /*
2565 * Update the accounts before we proceed because we might be leaving the
2566 * protection of the global mutex and thus run the risk of permitting
2567 * too much memory to be allocated.
2568 */
2569 switch (enmAccount)
2570 {
2571 case GMMACCOUNT_BASE: pGVM->gmm.s.Stats.Allocated.cBasePages += cPages; break;
2572 case GMMACCOUNT_SHADOW: pGVM->gmm.s.Stats.Allocated.cShadowPages += cPages; break;
2573 case GMMACCOUNT_FIXED: pGVM->gmm.s.Stats.Allocated.cFixedPages += cPages; break;
2574 default: AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
2575 }
2576 pGVM->gmm.s.Stats.cPrivatePages += cPages;
2577 pGMM->cAllocatedPages += cPages;
2578
2579 /*
2580 * Part two of it's-easy-in-legacy-memory-mode.
2581 */
2582 uint32_t iPage = 0;
2583 if (pGMM->fLegacyAllocationMode)
2584 {
2585 iPage = gmmR0AllocatePagesInBoundMode(pGVM, iPage, cPages, paPages);
2586 AssertReleaseReturn(iPage == cPages, VERR_GMM_ALLOC_PAGES_IPE);
2587 return VINF_SUCCESS;
2588 }
2589
2590 /*
2591 * Bound mode is also relatively straightforward.
2592 */
2593 int rc = VINF_SUCCESS;
2594 if (pGMM->fBoundMemoryMode)
2595 {
2596 iPage = gmmR0AllocatePagesInBoundMode(pGVM, iPage, cPages, paPages);
2597 if (iPage < cPages)
2598 do
2599 rc = gmmR0AllocateChunkNew(pGMM, pGVM, &pGVM->gmm.s.Private, cPages, paPages, &iPage);
2600 while (iPage < cPages && RT_SUCCESS(rc));
2601 }
2602 /*
2603 * Shared mode is trickier as we should try archive the same locality as
2604 * in bound mode, but smartly make use of non-full chunks allocated by
2605 * other VMs if we're low on memory.
2606 */
2607 else
2608 {
2609 /* Pick the most optimal pages first. */
2610 iPage = gmmR0AllocatePagesAssociatedWithVM(pGMM, pGVM, &pGMM->PrivateX, iPage, cPages, paPages);
2611 if (iPage < cPages)
2612 {
2613 /* Maybe we should try getting pages from chunks "belonging" to
2614 other VMs before allocating more chunks? */
2615 bool fTriedOnSameAlready = false;
2616 if (gmmR0ShouldAllocatePagesInOtherChunksBecauseOfLimits(pGVM))
2617 {
2618 iPage = gmmR0AllocatePagesFromSameNode(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2619 fTriedOnSameAlready = true;
2620 }
2621
2622 /* Allocate memory from empty chunks. */
2623 if (iPage < cPages)
2624 iPage = gmmR0AllocatePagesFromEmptyChunksOnSameNode(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2625
2626 /* Grab empty shared chunks. */
2627 if (iPage < cPages)
2628 iPage = gmmR0AllocatePagesFromEmptyChunksOnSameNode(&pGMM->Shared, pGVM, iPage, cPages, paPages);
2629
2630 /* If there is a lof of free pages spread around, try not waste
2631 system memory on more chunks. (Should trigger defragmentation.) */
2632 if ( !fTriedOnSameAlready
2633 && gmmR0ShouldAllocatePagesInOtherChunksBecauseOfLotsFree(pGMM))
2634 {
2635 iPage = gmmR0AllocatePagesFromSameNode(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2636 if (iPage < cPages)
2637 iPage = gmmR0AllocatePagesIndiscriminately(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2638 }
2639
2640 /*
2641 * Ok, try allocate new chunks.
2642 */
2643 if (iPage < cPages)
2644 {
2645 do
2646 rc = gmmR0AllocateChunkNew(pGMM, pGVM, &pGMM->PrivateX, cPages, paPages, &iPage);
2647 while (iPage < cPages && RT_SUCCESS(rc));
2648
2649 /* If the host is out of memory, take whatever we can get. */
2650 if ( (rc == VERR_NO_MEMORY || rc == VERR_NO_PHYS_MEMORY)
2651 && pGMM->PrivateX.cFreePages + pGMM->Shared.cFreePages >= cPages - iPage)
2652 {
2653 iPage = gmmR0AllocatePagesIndiscriminately(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2654 if (iPage < cPages)
2655 iPage = gmmR0AllocatePagesIndiscriminately(&pGMM->Shared, pGVM, iPage, cPages, paPages);
2656 AssertRelease(iPage == cPages);
2657 rc = VINF_SUCCESS;
2658 }
2659 }
2660 }
2661 }
2662
2663 /*
2664 * Clean up on failure. Since this is bound to be a low-memory condition
2665 * we will give back any empty chunks that might be hanging around.
2666 */
2667 if (RT_FAILURE(rc))
2668 {
2669 /* Update the statistics. */
2670 pGVM->gmm.s.Stats.cPrivatePages -= cPages;
2671 pGMM->cAllocatedPages -= cPages - iPage;
2672 switch (enmAccount)
2673 {
2674 case GMMACCOUNT_BASE: pGVM->gmm.s.Stats.Allocated.cBasePages -= cPages; break;
2675 case GMMACCOUNT_SHADOW: pGVM->gmm.s.Stats.Allocated.cShadowPages -= cPages; break;
2676 case GMMACCOUNT_FIXED: pGVM->gmm.s.Stats.Allocated.cFixedPages -= cPages; break;
2677 default: AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
2678 }
2679
2680 /* Release the pages. */
2681 while (iPage-- > 0)
2682 {
2683 uint32_t idPage = paPages[iPage].idPage;
2684 PGMMPAGE pPage = gmmR0GetPage(pGMM, idPage);
2685 if (RT_LIKELY(pPage))
2686 {
2687 Assert(GMM_PAGE_IS_PRIVATE(pPage));
2688 Assert(pPage->Private.hGVM == pGVM->hSelf);
2689 gmmR0FreePrivatePage(pGMM, pGVM, idPage, pPage);
2690 }
2691 else
2692 AssertMsgFailed(("idPage=%#x\n", idPage));
2693
2694 paPages[iPage].idPage = NIL_GMM_PAGEID;
2695 paPages[iPage].idSharedPage = NIL_GMM_PAGEID;
2696 paPages[iPage].HCPhysGCPhys = NIL_RTHCPHYS;
2697 }
2698
2699 /* Free empty chunks. */
2700 /** @todo */
2701
2702 /* return the fail status on failure */
2703 return rc;
2704 }
2705 return VINF_SUCCESS;
2706}
2707
2708
2709/**
2710 * Updates the previous allocations and allocates more pages.
2711 *
2712 * The handy pages are always taken from the 'base' memory account.
2713 * The allocated pages are not cleared and will contains random garbage.
2714 *
2715 * @returns VBox status code:
2716 * @retval VINF_SUCCESS on success.
2717 * @retval VERR_NOT_OWNER if the caller is not an EMT.
2718 * @retval VERR_GMM_PAGE_NOT_FOUND if one of the pages to update wasn't found.
2719 * @retval VERR_GMM_PAGE_NOT_PRIVATE if one of the pages to update wasn't a
2720 * private page.
2721 * @retval VERR_GMM_PAGE_NOT_SHARED if one of the pages to update wasn't a
2722 * shared page.
2723 * @retval VERR_GMM_NOT_PAGE_OWNER if one of the pages to be updated wasn't
2724 * owned by the VM.
2725 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk is necessary.
2726 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
2727 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
2728 * that is we're trying to allocate more than we've reserved.
2729 *
2730 * @param pGVM The global (ring-0) VM structure.
2731 * @param pVM The cross context VM structure.
2732 * @param idCpu The VCPU id.
2733 * @param cPagesToUpdate The number of pages to update (starting from the head).
2734 * @param cPagesToAlloc The number of pages to allocate (starting from the head).
2735 * @param paPages The array of page descriptors.
2736 * See GMMPAGEDESC for details on what is expected on input.
2737 * @thread EMT(idCpu)
2738 */
2739GMMR0DECL(int) GMMR0AllocateHandyPages(PGVM pGVM, PVM pVM, VMCPUID idCpu, uint32_t cPagesToUpdate,
2740 uint32_t cPagesToAlloc, PGMMPAGEDESC paPages)
2741{
2742 LogFlow(("GMMR0AllocateHandyPages: pGVM=%p pVM=%p cPagesToUpdate=%#x cPagesToAlloc=%#x paPages=%p\n",
2743 pGVM, pVM, cPagesToUpdate, cPagesToAlloc, paPages));
2744
2745 /*
2746 * Validate, get basics and take the semaphore.
2747 * (This is a relatively busy path, so make predictions where possible.)
2748 */
2749 PGMM pGMM;
2750 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
2751 int rc = GVMMR0ValidateGVMandVMandEMT(pGVM, pVM, idCpu);
2752 if (RT_FAILURE(rc))
2753 return rc;
2754
2755 AssertPtrReturn(paPages, VERR_INVALID_PARAMETER);
2756 AssertMsgReturn( (cPagesToUpdate && cPagesToUpdate < 1024)
2757 || (cPagesToAlloc && cPagesToAlloc < 1024),
2758 ("cPagesToUpdate=%#x cPagesToAlloc=%#x\n", cPagesToUpdate, cPagesToAlloc),
2759 VERR_INVALID_PARAMETER);
2760
2761 unsigned iPage = 0;
2762 for (; iPage < cPagesToUpdate; iPage++)
2763 {
2764 AssertMsgReturn( ( paPages[iPage].HCPhysGCPhys <= GMM_GCPHYS_LAST
2765 && !(paPages[iPage].HCPhysGCPhys & PAGE_OFFSET_MASK))
2766 || paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS
2767 || paPages[iPage].HCPhysGCPhys == GMM_GCPHYS_UNSHAREABLE,
2768 ("#%#x: %RHp\n", iPage, paPages[iPage].HCPhysGCPhys),
2769 VERR_INVALID_PARAMETER);
2770 AssertMsgReturn( paPages[iPage].idPage <= GMM_PAGEID_LAST
2771 /*|| paPages[iPage].idPage == NIL_GMM_PAGEID*/,
2772 ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
2773 AssertMsgReturn( paPages[iPage].idPage <= GMM_PAGEID_LAST
2774 /*|| paPages[iPage].idSharedPage == NIL_GMM_PAGEID*/,
2775 ("#%#x: %#x\n", iPage, paPages[iPage].idSharedPage), VERR_INVALID_PARAMETER);
2776 }
2777
2778 for (; iPage < cPagesToAlloc; iPage++)
2779 {
2780 AssertMsgReturn(paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS, ("#%#x: %RHp\n", iPage, paPages[iPage].HCPhysGCPhys), VERR_INVALID_PARAMETER);
2781 AssertMsgReturn(paPages[iPage].idPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
2782 AssertMsgReturn(paPages[iPage].idSharedPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idSharedPage), VERR_INVALID_PARAMETER);
2783 }
2784
2785 gmmR0MutexAcquire(pGMM);
2786 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
2787 {
2788 /* No allocations before the initial reservation has been made! */
2789 if (RT_LIKELY( pGVM->gmm.s.Stats.Reserved.cBasePages
2790 && pGVM->gmm.s.Stats.Reserved.cFixedPages
2791 && pGVM->gmm.s.Stats.Reserved.cShadowPages))
2792 {
2793 /*
2794 * Perform the updates.
2795 * Stop on the first error.
2796 */
2797 for (iPage = 0; iPage < cPagesToUpdate; iPage++)
2798 {
2799 if (paPages[iPage].idPage != NIL_GMM_PAGEID)
2800 {
2801 PGMMPAGE pPage = gmmR0GetPage(pGMM, paPages[iPage].idPage);
2802 if (RT_LIKELY(pPage))
2803 {
2804 if (RT_LIKELY(GMM_PAGE_IS_PRIVATE(pPage)))
2805 {
2806 if (RT_LIKELY(pPage->Private.hGVM == pGVM->hSelf))
2807 {
2808 AssertCompile(NIL_RTHCPHYS > GMM_GCPHYS_LAST && GMM_GCPHYS_UNSHAREABLE > GMM_GCPHYS_LAST);
2809 if (RT_LIKELY(paPages[iPage].HCPhysGCPhys <= GMM_GCPHYS_LAST))
2810 pPage->Private.pfn = paPages[iPage].HCPhysGCPhys >> PAGE_SHIFT;
2811 else if (paPages[iPage].HCPhysGCPhys == GMM_GCPHYS_UNSHAREABLE)
2812 pPage->Private.pfn = GMM_PAGE_PFN_UNSHAREABLE;
2813 /* else: NIL_RTHCPHYS nothing */
2814
2815 paPages[iPage].idPage = NIL_GMM_PAGEID;
2816 paPages[iPage].HCPhysGCPhys = NIL_RTHCPHYS;
2817 }
2818 else
2819 {
2820 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not owner! hGVM=%#x hSelf=%#x\n",
2821 iPage, paPages[iPage].idPage, pPage->Private.hGVM, pGVM->hSelf));
2822 rc = VERR_GMM_NOT_PAGE_OWNER;
2823 break;
2824 }
2825 }
2826 else
2827 {
2828 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not private! %.*Rhxs (type %d)\n", iPage, paPages[iPage].idPage, sizeof(*pPage), pPage, pPage->Common.u2State));
2829 rc = VERR_GMM_PAGE_NOT_PRIVATE;
2830 break;
2831 }
2832 }
2833 else
2834 {
2835 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not found! (private)\n", iPage, paPages[iPage].idPage));
2836 rc = VERR_GMM_PAGE_NOT_FOUND;
2837 break;
2838 }
2839 }
2840
2841 if (paPages[iPage].idSharedPage != NIL_GMM_PAGEID)
2842 {
2843 PGMMPAGE pPage = gmmR0GetPage(pGMM, paPages[iPage].idSharedPage);
2844 if (RT_LIKELY(pPage))
2845 {
2846 if (RT_LIKELY(GMM_PAGE_IS_SHARED(pPage)))
2847 {
2848 AssertCompile(NIL_RTHCPHYS > GMM_GCPHYS_LAST && GMM_GCPHYS_UNSHAREABLE > GMM_GCPHYS_LAST);
2849 Assert(pPage->Shared.cRefs);
2850 Assert(pGVM->gmm.s.Stats.cSharedPages);
2851 Assert(pGVM->gmm.s.Stats.Allocated.cBasePages);
2852
2853 Log(("GMMR0AllocateHandyPages: free shared page %x cRefs=%d\n", paPages[iPage].idSharedPage, pPage->Shared.cRefs));
2854 pGVM->gmm.s.Stats.cSharedPages--;
2855 pGVM->gmm.s.Stats.Allocated.cBasePages--;
2856 if (!--pPage->Shared.cRefs)
2857 gmmR0FreeSharedPage(pGMM, pGVM, paPages[iPage].idSharedPage, pPage);
2858 else
2859 {
2860 Assert(pGMM->cDuplicatePages);
2861 pGMM->cDuplicatePages--;
2862 }
2863
2864 paPages[iPage].idSharedPage = NIL_GMM_PAGEID;
2865 }
2866 else
2867 {
2868 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not shared!\n", iPage, paPages[iPage].idSharedPage));
2869 rc = VERR_GMM_PAGE_NOT_SHARED;
2870 break;
2871 }
2872 }
2873 else
2874 {
2875 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not found! (shared)\n", iPage, paPages[iPage].idSharedPage));
2876 rc = VERR_GMM_PAGE_NOT_FOUND;
2877 break;
2878 }
2879 }
2880 } /* for each page to update */
2881
2882 if (RT_SUCCESS(rc) && cPagesToAlloc > 0)
2883 {
2884#if defined(VBOX_STRICT) && 0 /** @todo re-test this later. Appeared to be a PGM init bug. */
2885 for (iPage = 0; iPage < cPagesToAlloc; iPage++)
2886 {
2887 Assert(paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS);
2888 Assert(paPages[iPage].idPage == NIL_GMM_PAGEID);
2889 Assert(paPages[iPage].idSharedPage == NIL_GMM_PAGEID);
2890 }
2891#endif
2892
2893 /*
2894 * Join paths with GMMR0AllocatePages for the allocation.
2895 * Note! gmmR0AllocateMoreChunks may leave the protection of the mutex!
2896 */
2897 rc = gmmR0AllocatePagesNew(pGMM, pGVM, cPagesToAlloc, paPages, GMMACCOUNT_BASE);
2898 }
2899 }
2900 else
2901 rc = VERR_WRONG_ORDER;
2902 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
2903 }
2904 else
2905 rc = VERR_GMM_IS_NOT_SANE;
2906 gmmR0MutexRelease(pGMM);
2907 LogFlow(("GMMR0AllocateHandyPages: returns %Rrc\n", rc));
2908 return rc;
2909}
2910
2911
2912/**
2913 * Allocate one or more pages.
2914 *
2915 * This is typically used for ROMs and MMIO2 (VRAM) during VM creation.
2916 * The allocated pages are not cleared and will contain random garbage.
2917 *
2918 * @returns VBox status code:
2919 * @retval VINF_SUCCESS on success.
2920 * @retval VERR_NOT_OWNER if the caller is not an EMT.
2921 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk is necessary.
2922 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
2923 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
2924 * that is we're trying to allocate more than we've reserved.
2925 *
2926 * @param pGVM The global (ring-0) VM structure.
2927 * @param pVM The cross context VM structure.
2928 * @param idCpu The VCPU id.
2929 * @param cPages The number of pages to allocate.
2930 * @param paPages Pointer to the page descriptors.
2931 * See GMMPAGEDESC for details on what is expected on
2932 * input.
2933 * @param enmAccount The account to charge.
2934 *
2935 * @thread EMT.
2936 */
2937GMMR0DECL(int) GMMR0AllocatePages(PGVM pGVM, PVM pVM, VMCPUID idCpu, uint32_t cPages, PGMMPAGEDESC paPages, GMMACCOUNT enmAccount)
2938{
2939 LogFlow(("GMMR0AllocatePages: pGVM=%p pVM=%p cPages=%#x paPages=%p enmAccount=%d\n", pGVM, pVM, cPages, paPages, enmAccount));
2940
2941 /*
2942 * Validate, get basics and take the semaphore.
2943 */
2944 PGMM pGMM;
2945 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
2946 int rc = GVMMR0ValidateGVMandVMandEMT(pGVM, pVM, idCpu);
2947 if (RT_FAILURE(rc))
2948 return rc;
2949
2950 AssertPtrReturn(paPages, VERR_INVALID_PARAMETER);
2951 AssertMsgReturn(enmAccount > GMMACCOUNT_INVALID && enmAccount < GMMACCOUNT_END, ("%d\n", enmAccount), VERR_INVALID_PARAMETER);
2952 AssertMsgReturn(cPages > 0 && cPages < RT_BIT(32 - PAGE_SHIFT), ("%#x\n", cPages), VERR_INVALID_PARAMETER);
2953
2954 for (unsigned iPage = 0; iPage < cPages; iPage++)
2955 {
2956 AssertMsgReturn( paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS
2957 || paPages[iPage].HCPhysGCPhys == GMM_GCPHYS_UNSHAREABLE
2958 || ( enmAccount == GMMACCOUNT_BASE
2959 && paPages[iPage].HCPhysGCPhys <= GMM_GCPHYS_LAST
2960 && !(paPages[iPage].HCPhysGCPhys & PAGE_OFFSET_MASK)),
2961 ("#%#x: %RHp enmAccount=%d\n", iPage, paPages[iPage].HCPhysGCPhys, enmAccount),
2962 VERR_INVALID_PARAMETER);
2963 AssertMsgReturn(paPages[iPage].idPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
2964 AssertMsgReturn(paPages[iPage].idSharedPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idSharedPage), VERR_INVALID_PARAMETER);
2965 }
2966
2967 gmmR0MutexAcquire(pGMM);
2968 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
2969 {
2970
2971 /* No allocations before the initial reservation has been made! */
2972 if (RT_LIKELY( pGVM->gmm.s.Stats.Reserved.cBasePages
2973 && pGVM->gmm.s.Stats.Reserved.cFixedPages
2974 && pGVM->gmm.s.Stats.Reserved.cShadowPages))
2975 rc = gmmR0AllocatePagesNew(pGMM, pGVM, cPages, paPages, enmAccount);
2976 else
2977 rc = VERR_WRONG_ORDER;
2978 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
2979 }
2980 else
2981 rc = VERR_GMM_IS_NOT_SANE;
2982 gmmR0MutexRelease(pGMM);
2983 LogFlow(("GMMR0AllocatePages: returns %Rrc\n", rc));
2984 return rc;
2985}
2986
2987
2988/**
2989 * VMMR0 request wrapper for GMMR0AllocatePages.
2990 *
2991 * @returns see GMMR0AllocatePages.
2992 * @param pGVM The global (ring-0) VM structure.
2993 * @param pVM The cross context VM structure.
2994 * @param idCpu The VCPU id.
2995 * @param pReq Pointer to the request packet.
2996 */
2997GMMR0DECL(int) GMMR0AllocatePagesReq(PGVM pGVM, PVM pVM, VMCPUID idCpu, PGMMALLOCATEPAGESREQ pReq)
2998{
2999 /*
3000 * Validate input and pass it on.
3001 */
3002 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3003 AssertMsgReturn(pReq->Hdr.cbReq >= RT_UOFFSETOF(GMMALLOCATEPAGESREQ, aPages[0]),
3004 ("%#x < %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF(GMMALLOCATEPAGESREQ, aPages[0])),
3005 VERR_INVALID_PARAMETER);
3006 AssertMsgReturn(pReq->Hdr.cbReq == RT_UOFFSETOF_DYN(GMMALLOCATEPAGESREQ, aPages[pReq->cPages]),
3007 ("%#x != %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF_DYN(GMMALLOCATEPAGESREQ, aPages[pReq->cPages])),
3008 VERR_INVALID_PARAMETER);
3009
3010 return GMMR0AllocatePages(pGVM, pVM, idCpu, pReq->cPages, &pReq->aPages[0], pReq->enmAccount);
3011}
3012
3013
3014/**
3015 * Allocate a large page to represent guest RAM
3016 *
3017 * The allocated pages are not cleared and will contains random garbage.
3018 *
3019 * @returns VBox status code:
3020 * @retval VINF_SUCCESS on success.
3021 * @retval VERR_NOT_OWNER if the caller is not an EMT.
3022 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk is necessary.
3023 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
3024 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
3025 * that is we're trying to allocate more than we've reserved.
3026 * @returns see GMMR0AllocatePages.
3027 *
3028 * @param pGVM The global (ring-0) VM structure.
3029 * @param pVM The cross context VM structure.
3030 * @param idCpu The VCPU id.
3031 * @param cbPage Large page size.
3032 * @param pIdPage Where to return the GMM page ID of the page.
3033 * @param pHCPhys Where to return the host physical address of the page.
3034 */
3035GMMR0DECL(int) GMMR0AllocateLargePage(PGVM pGVM, PVM pVM, VMCPUID idCpu, uint32_t cbPage, uint32_t *pIdPage, RTHCPHYS *pHCPhys)
3036{
3037 LogFlow(("GMMR0AllocateLargePage: pGVM=%p pVM=%p cbPage=%x\n", pGVM, pVM, cbPage));
3038
3039 AssertReturn(cbPage == GMM_CHUNK_SIZE, VERR_INVALID_PARAMETER);
3040 AssertPtrReturn(pIdPage, VERR_INVALID_PARAMETER);
3041 AssertPtrReturn(pHCPhys, VERR_INVALID_PARAMETER);
3042
3043 /*
3044 * Validate, get basics and take the semaphore.
3045 */
3046 PGMM pGMM;
3047 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3048 int rc = GVMMR0ValidateGVMandVMandEMT(pGVM, pVM, idCpu);
3049 if (RT_FAILURE(rc))
3050 return rc;
3051
3052 /* Not supported in legacy mode where we allocate the memory in ring 3 and lock it in ring 0. */
3053 if (pGMM->fLegacyAllocationMode)
3054 return VERR_NOT_SUPPORTED;
3055
3056 *pHCPhys = NIL_RTHCPHYS;
3057 *pIdPage = NIL_GMM_PAGEID;
3058
3059 gmmR0MutexAcquire(pGMM);
3060 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3061 {
3062 const unsigned cPages = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
3063 if (RT_UNLIKELY( pGVM->gmm.s.Stats.Allocated.cBasePages + pGVM->gmm.s.Stats.cBalloonedPages + cPages
3064 > pGVM->gmm.s.Stats.Reserved.cBasePages))
3065 {
3066 Log(("GMMR0AllocateLargePage: Reserved=%#llx Allocated+Requested=%#llx+%#x!\n",
3067 pGVM->gmm.s.Stats.Reserved.cBasePages, pGVM->gmm.s.Stats.Allocated.cBasePages, cPages));
3068 gmmR0MutexRelease(pGMM);
3069 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
3070 }
3071
3072 /*
3073 * Allocate a new large page chunk.
3074 *
3075 * Note! We leave the giant GMM lock temporarily as the allocation might
3076 * take a long time. gmmR0RegisterChunk will retake it (ugly).
3077 */
3078 AssertCompile(GMM_CHUNK_SIZE == _2M);
3079 gmmR0MutexRelease(pGMM);
3080
3081 RTR0MEMOBJ hMemObj;
3082 rc = RTR0MemObjAllocPhysEx(&hMemObj, GMM_CHUNK_SIZE, NIL_RTHCPHYS, GMM_CHUNK_SIZE);
3083 if (RT_SUCCESS(rc))
3084 {
3085 PGMMCHUNKFREESET pSet = pGMM->fBoundMemoryMode ? &pGVM->gmm.s.Private : &pGMM->PrivateX;
3086 PGMMCHUNK pChunk;
3087 rc = gmmR0RegisterChunk(pGMM, pSet, hMemObj, pGVM->hSelf, GMM_CHUNK_FLAGS_LARGE_PAGE, &pChunk);
3088 if (RT_SUCCESS(rc))
3089 {
3090 /*
3091 * Allocate all the pages in the chunk.
3092 */
3093 /* Unlink the new chunk from the free list. */
3094 gmmR0UnlinkChunk(pChunk);
3095
3096 /** @todo rewrite this to skip the looping. */
3097 /* Allocate all pages. */
3098 GMMPAGEDESC PageDesc;
3099 gmmR0AllocatePage(pChunk, pGVM->hSelf, &PageDesc);
3100
3101 /* Return the first page as we'll use the whole chunk as one big page. */
3102 *pIdPage = PageDesc.idPage;
3103 *pHCPhys = PageDesc.HCPhysGCPhys;
3104
3105 for (unsigned i = 1; i < cPages; i++)
3106 gmmR0AllocatePage(pChunk, pGVM->hSelf, &PageDesc);
3107
3108 /* Update accounting. */
3109 pGVM->gmm.s.Stats.Allocated.cBasePages += cPages;
3110 pGVM->gmm.s.Stats.cPrivatePages += cPages;
3111 pGMM->cAllocatedPages += cPages;
3112
3113 gmmR0LinkChunk(pChunk, pSet);
3114 gmmR0MutexRelease(pGMM);
3115 }
3116 else
3117 RTR0MemObjFree(hMemObj, false /* fFreeMappings */);
3118 }
3119 }
3120 else
3121 {
3122 gmmR0MutexRelease(pGMM);
3123 rc = VERR_GMM_IS_NOT_SANE;
3124 }
3125
3126 LogFlow(("GMMR0AllocateLargePage: returns %Rrc\n", rc));
3127 return rc;
3128}
3129
3130
3131/**
3132 * Free a large page.
3133 *
3134 * @returns VBox status code:
3135 * @param pGVM The global (ring-0) VM structure.
3136 * @param pVM The cross context VM structure.
3137 * @param idCpu The VCPU id.
3138 * @param idPage The large page id.
3139 */
3140GMMR0DECL(int) GMMR0FreeLargePage(PGVM pGVM, PVM pVM, VMCPUID idCpu, uint32_t idPage)
3141{
3142 LogFlow(("GMMR0FreeLargePage: pGVM=%p pVM=%p idPage=%x\n", pGVM, pVM, idPage));
3143
3144 /*
3145 * Validate, get basics and take the semaphore.
3146 */
3147 PGMM pGMM;
3148 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3149 int rc = GVMMR0ValidateGVMandVMandEMT(pGVM, pVM, idCpu);
3150 if (RT_FAILURE(rc))
3151 return rc;
3152
3153 /* Not supported in legacy mode where we allocate the memory in ring 3 and lock it in ring 0. */
3154 if (pGMM->fLegacyAllocationMode)
3155 return VERR_NOT_SUPPORTED;
3156
3157 gmmR0MutexAcquire(pGMM);
3158 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3159 {
3160 const unsigned cPages = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
3161
3162 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cBasePages < cPages))
3163 {
3164 Log(("GMMR0FreeLargePage: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cBasePages, cPages));
3165 gmmR0MutexRelease(pGMM);
3166 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3167 }
3168
3169 PGMMPAGE pPage = gmmR0GetPage(pGMM, idPage);
3170 if (RT_LIKELY( pPage
3171 && GMM_PAGE_IS_PRIVATE(pPage)))
3172 {
3173 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
3174 Assert(pChunk);
3175 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
3176 Assert(pChunk->cPrivate > 0);
3177
3178 /* Release the memory immediately. */
3179 gmmR0FreeChunk(pGMM, NULL, pChunk, false /*fRelaxedSem*/); /** @todo this can be relaxed too! */
3180
3181 /* Update accounting. */
3182 pGVM->gmm.s.Stats.Allocated.cBasePages -= cPages;
3183 pGVM->gmm.s.Stats.cPrivatePages -= cPages;
3184 pGMM->cAllocatedPages -= cPages;
3185 }
3186 else
3187 rc = VERR_GMM_PAGE_NOT_FOUND;
3188 }
3189 else
3190 rc = VERR_GMM_IS_NOT_SANE;
3191
3192 gmmR0MutexRelease(pGMM);
3193 LogFlow(("GMMR0FreeLargePage: returns %Rrc\n", rc));
3194 return rc;
3195}
3196
3197
3198/**
3199 * VMMR0 request wrapper for GMMR0FreeLargePage.
3200 *
3201 * @returns see GMMR0FreeLargePage.
3202 * @param pGVM The global (ring-0) VM structure.
3203 * @param pVM The cross context VM structure.
3204 * @param idCpu The VCPU id.
3205 * @param pReq Pointer to the request packet.
3206 */
3207GMMR0DECL(int) GMMR0FreeLargePageReq(PGVM pGVM, PVM pVM, VMCPUID idCpu, PGMMFREELARGEPAGEREQ pReq)
3208{
3209 /*
3210 * Validate input and pass it on.
3211 */
3212 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3213 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMFREEPAGESREQ),
3214 ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(GMMFREEPAGESREQ)),
3215 VERR_INVALID_PARAMETER);
3216
3217 return GMMR0FreeLargePage(pGVM, pVM, idCpu, pReq->idPage);
3218}
3219
3220
3221/**
3222 * Frees a chunk, giving it back to the host OS.
3223 *
3224 * @param pGMM Pointer to the GMM instance.
3225 * @param pGVM This is set when called from GMMR0CleanupVM so we can
3226 * unmap and free the chunk in one go.
3227 * @param pChunk The chunk to free.
3228 * @param fRelaxedSem Whether we can release the semaphore while doing the
3229 * freeing (@c true) or not.
3230 */
3231static bool gmmR0FreeChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem)
3232{
3233 Assert(pChunk->Core.Key != NIL_GMM_CHUNKID);
3234
3235 GMMR0CHUNKMTXSTATE MtxState;
3236 gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk, GMMR0CHUNK_MTX_KEEP_GIANT);
3237
3238 /*
3239 * Cleanup hack! Unmap the chunk from the callers address space.
3240 * This shouldn't happen, so screw lock contention...
3241 */
3242 if ( pChunk->cMappingsX
3243 && !pGMM->fLegacyAllocationMode
3244 && pGVM)
3245 gmmR0UnmapChunkLocked(pGMM, pGVM, pChunk);
3246
3247 /*
3248 * If there are current mappings of the chunk, then request the
3249 * VMs to unmap them. Reposition the chunk in the free list so
3250 * it won't be a likely candidate for allocations.
3251 */
3252 if (pChunk->cMappingsX)
3253 {
3254 /** @todo R0 -> VM request */
3255 /* The chunk can be mapped by more than one VM if fBoundMemoryMode is false! */
3256 Log(("gmmR0FreeChunk: chunk still has %d mappings; don't free!\n", pChunk->cMappingsX));
3257 gmmR0ChunkMutexRelease(&MtxState, pChunk);
3258 return false;
3259 }
3260
3261
3262 /*
3263 * Save and trash the handle.
3264 */
3265 RTR0MEMOBJ const hMemObj = pChunk->hMemObj;
3266 pChunk->hMemObj = NIL_RTR0MEMOBJ;
3267
3268 /*
3269 * Unlink it from everywhere.
3270 */
3271 gmmR0UnlinkChunk(pChunk);
3272
3273 RTListNodeRemove(&pChunk->ListNode);
3274
3275 PAVLU32NODECORE pCore = RTAvlU32Remove(&pGMM->pChunks, pChunk->Core.Key);
3276 Assert(pCore == &pChunk->Core); NOREF(pCore);
3277
3278 PGMMCHUNKTLBE pTlbe = &pGMM->ChunkTLB.aEntries[GMM_CHUNKTLB_IDX(pChunk->Core.Key)];
3279 if (pTlbe->pChunk == pChunk)
3280 {
3281 pTlbe->idChunk = NIL_GMM_CHUNKID;
3282 pTlbe->pChunk = NULL;
3283 }
3284
3285 Assert(pGMM->cChunks > 0);
3286 pGMM->cChunks--;
3287
3288 /*
3289 * Free the Chunk ID before dropping the locks and freeing the rest.
3290 */
3291 gmmR0FreeChunkId(pGMM, pChunk->Core.Key);
3292 pChunk->Core.Key = NIL_GMM_CHUNKID;
3293
3294 pGMM->cFreedChunks++;
3295
3296 gmmR0ChunkMutexRelease(&MtxState, NULL);
3297 if (fRelaxedSem)
3298 gmmR0MutexRelease(pGMM);
3299
3300 RTMemFree(pChunk->paMappingsX);
3301 pChunk->paMappingsX = NULL;
3302
3303 RTMemFree(pChunk);
3304
3305 int rc = RTR0MemObjFree(hMemObj, false /* fFreeMappings */);
3306 AssertLogRelRC(rc);
3307
3308 if (fRelaxedSem)
3309 gmmR0MutexAcquire(pGMM);
3310 return fRelaxedSem;
3311}
3312
3313
3314/**
3315 * Free page worker.
3316 *
3317 * The caller does all the statistic decrementing, we do all the incrementing.
3318 *
3319 * @param pGMM Pointer to the GMM instance data.
3320 * @param pGVM Pointer to the GVM instance.
3321 * @param pChunk Pointer to the chunk this page belongs to.
3322 * @param idPage The Page ID.
3323 * @param pPage Pointer to the page.
3324 */
3325static void gmmR0FreePageWorker(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, uint32_t idPage, PGMMPAGE pPage)
3326{
3327 Log3(("F pPage=%p iPage=%#x/%#x u2State=%d iFreeHead=%#x\n",
3328 pPage, pPage - &pChunk->aPages[0], idPage, pPage->Common.u2State, pChunk->iFreeHead)); NOREF(idPage);
3329
3330 /*
3331 * Put the page on the free list.
3332 */
3333 pPage->u = 0;
3334 pPage->Free.u2State = GMM_PAGE_STATE_FREE;
3335 Assert(pChunk->iFreeHead < RT_ELEMENTS(pChunk->aPages) || pChunk->iFreeHead == UINT16_MAX);
3336 pPage->Free.iNext = pChunk->iFreeHead;
3337 pChunk->iFreeHead = pPage - &pChunk->aPages[0];
3338
3339 /*
3340 * Update statistics (the cShared/cPrivate stats are up to date already),
3341 * and relink the chunk if necessary.
3342 */
3343 unsigned const cFree = pChunk->cFree;
3344 if ( !cFree
3345 || gmmR0SelectFreeSetList(cFree) != gmmR0SelectFreeSetList(cFree + 1))
3346 {
3347 gmmR0UnlinkChunk(pChunk);
3348 pChunk->cFree++;
3349 gmmR0SelectSetAndLinkChunk(pGMM, pGVM, pChunk);
3350 }
3351 else
3352 {
3353 pChunk->cFree = cFree + 1;
3354 pChunk->pSet->cFreePages++;
3355 }
3356
3357 /*
3358 * If the chunk becomes empty, consider giving memory back to the host OS.
3359 *
3360 * The current strategy is to try give it back if there are other chunks
3361 * in this free list, meaning if there are at least 240 free pages in this
3362 * category. Note that since there are probably mappings of the chunk,
3363 * it won't be freed up instantly, which probably screws up this logic
3364 * a bit...
3365 */
3366 /** @todo Do this on the way out. */
3367 if (RT_UNLIKELY( pChunk->cFree == GMM_CHUNK_NUM_PAGES
3368 && pChunk->pFreeNext
3369 && pChunk->pFreePrev /** @todo this is probably misfiring, see reset... */
3370 && !pGMM->fLegacyAllocationMode))
3371 gmmR0FreeChunk(pGMM, NULL, pChunk, false);
3372
3373}
3374
3375
3376/**
3377 * Frees a shared page, the page is known to exist and be valid and such.
3378 *
3379 * @param pGMM Pointer to the GMM instance.
3380 * @param pGVM Pointer to the GVM instance.
3381 * @param idPage The page id.
3382 * @param pPage The page structure.
3383 */
3384DECLINLINE(void) gmmR0FreeSharedPage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage)
3385{
3386 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
3387 Assert(pChunk);
3388 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
3389 Assert(pChunk->cShared > 0);
3390 Assert(pGMM->cSharedPages > 0);
3391 Assert(pGMM->cAllocatedPages > 0);
3392 Assert(!pPage->Shared.cRefs);
3393
3394 pChunk->cShared--;
3395 pGMM->cAllocatedPages--;
3396 pGMM->cSharedPages--;
3397 gmmR0FreePageWorker(pGMM, pGVM, pChunk, idPage, pPage);
3398}
3399
3400
3401/**
3402 * Frees a private page, the page is known to exist and be valid and such.
3403 *
3404 * @param pGMM Pointer to the GMM instance.
3405 * @param pGVM Pointer to the GVM instance.
3406 * @param idPage The page id.
3407 * @param pPage The page structure.
3408 */
3409DECLINLINE(void) gmmR0FreePrivatePage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage)
3410{
3411 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
3412 Assert(pChunk);
3413 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
3414 Assert(pChunk->cPrivate > 0);
3415 Assert(pGMM->cAllocatedPages > 0);
3416
3417 pChunk->cPrivate--;
3418 pGMM->cAllocatedPages--;
3419 gmmR0FreePageWorker(pGMM, pGVM, pChunk, idPage, pPage);
3420}
3421
3422
3423/**
3424 * Common worker for GMMR0FreePages and GMMR0BalloonedPages.
3425 *
3426 * @returns VBox status code:
3427 * @retval xxx
3428 *
3429 * @param pGMM Pointer to the GMM instance data.
3430 * @param pGVM Pointer to the VM.
3431 * @param cPages The number of pages to free.
3432 * @param paPages Pointer to the page descriptors.
3433 * @param enmAccount The account this relates to.
3434 */
3435static int gmmR0FreePages(PGMM pGMM, PGVM pGVM, uint32_t cPages, PGMMFREEPAGEDESC paPages, GMMACCOUNT enmAccount)
3436{
3437 /*
3438 * Check that the request isn't impossible wrt to the account status.
3439 */
3440 switch (enmAccount)
3441 {
3442 case GMMACCOUNT_BASE:
3443 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cBasePages < cPages))
3444 {
3445 Log(("gmmR0FreePages: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cBasePages, cPages));
3446 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3447 }
3448 break;
3449 case GMMACCOUNT_SHADOW:
3450 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cShadowPages < cPages))
3451 {
3452 Log(("gmmR0FreePages: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cShadowPages, cPages));
3453 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3454 }
3455 break;
3456 case GMMACCOUNT_FIXED:
3457 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cFixedPages < cPages))
3458 {
3459 Log(("gmmR0FreePages: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cFixedPages, cPages));
3460 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3461 }
3462 break;
3463 default:
3464 AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
3465 }
3466
3467 /*
3468 * Walk the descriptors and free the pages.
3469 *
3470 * Statistics (except the account) are being updated as we go along,
3471 * unlike the alloc code. Also, stop on the first error.
3472 */
3473 int rc = VINF_SUCCESS;
3474 uint32_t iPage;
3475 for (iPage = 0; iPage < cPages; iPage++)
3476 {
3477 uint32_t idPage = paPages[iPage].idPage;
3478 PGMMPAGE pPage = gmmR0GetPage(pGMM, idPage);
3479 if (RT_LIKELY(pPage))
3480 {
3481 if (RT_LIKELY(GMM_PAGE_IS_PRIVATE(pPage)))
3482 {
3483 if (RT_LIKELY(pPage->Private.hGVM == pGVM->hSelf))
3484 {
3485 Assert(pGVM->gmm.s.Stats.cPrivatePages);
3486 pGVM->gmm.s.Stats.cPrivatePages--;
3487 gmmR0FreePrivatePage(pGMM, pGVM, idPage, pPage);
3488 }
3489 else
3490 {
3491 Log(("gmmR0AllocatePages: #%#x/%#x: not owner! hGVM=%#x hSelf=%#x\n", iPage, idPage,
3492 pPage->Private.hGVM, pGVM->hSelf));
3493 rc = VERR_GMM_NOT_PAGE_OWNER;
3494 break;
3495 }
3496 }
3497 else if (RT_LIKELY(GMM_PAGE_IS_SHARED(pPage)))
3498 {
3499 Assert(pGVM->gmm.s.Stats.cSharedPages);
3500 Assert(pPage->Shared.cRefs);
3501#if defined(VBOX_WITH_PAGE_SHARING) && defined(VBOX_STRICT) && HC_ARCH_BITS == 64
3502 if (pPage->Shared.u14Checksum)
3503 {
3504 uint32_t uChecksum = gmmR0StrictPageChecksum(pGMM, pGVM, idPage);
3505 uChecksum &= UINT32_C(0x00003fff);
3506 AssertMsg(!uChecksum || uChecksum == pPage->Shared.u14Checksum,
3507 ("%#x vs %#x - idPage=%#x\n", uChecksum, pPage->Shared.u14Checksum, idPage));
3508 }
3509#endif
3510 pGVM->gmm.s.Stats.cSharedPages--;
3511 if (!--pPage->Shared.cRefs)
3512 gmmR0FreeSharedPage(pGMM, pGVM, idPage, pPage);
3513 else
3514 {
3515 Assert(pGMM->cDuplicatePages);
3516 pGMM->cDuplicatePages--;
3517 }
3518 }
3519 else
3520 {
3521 Log(("gmmR0AllocatePages: #%#x/%#x: already free!\n", iPage, idPage));
3522 rc = VERR_GMM_PAGE_ALREADY_FREE;
3523 break;
3524 }
3525 }
3526 else
3527 {
3528 Log(("gmmR0AllocatePages: #%#x/%#x: not found!\n", iPage, idPage));
3529 rc = VERR_GMM_PAGE_NOT_FOUND;
3530 break;
3531 }
3532 paPages[iPage].idPage = NIL_GMM_PAGEID;
3533 }
3534
3535 /*
3536 * Update the account.
3537 */
3538 switch (enmAccount)
3539 {
3540 case GMMACCOUNT_BASE: pGVM->gmm.s.Stats.Allocated.cBasePages -= iPage; break;
3541 case GMMACCOUNT_SHADOW: pGVM->gmm.s.Stats.Allocated.cShadowPages -= iPage; break;
3542 case GMMACCOUNT_FIXED: pGVM->gmm.s.Stats.Allocated.cFixedPages -= iPage; break;
3543 default:
3544 AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
3545 }
3546
3547 /*
3548 * Any threshold stuff to be done here?
3549 */
3550
3551 return rc;
3552}
3553
3554
3555/**
3556 * Free one or more pages.
3557 *
3558 * This is typically used at reset time or power off.
3559 *
3560 * @returns VBox status code:
3561 * @retval xxx
3562 *
3563 * @param pGVM The global (ring-0) VM structure.
3564 * @param pVM The cross context VM structure.
3565 * @param idCpu The VCPU id.
3566 * @param cPages The number of pages to allocate.
3567 * @param paPages Pointer to the page descriptors containing the page IDs
3568 * for each page.
3569 * @param enmAccount The account this relates to.
3570 * @thread EMT.
3571 */
3572GMMR0DECL(int) GMMR0FreePages(PGVM pGVM, PVM pVM, VMCPUID idCpu, uint32_t cPages, PGMMFREEPAGEDESC paPages, GMMACCOUNT enmAccount)
3573{
3574 LogFlow(("GMMR0FreePages: pGVM=%p pVM=%p cPages=%#x paPages=%p enmAccount=%d\n", pGVM, pVM, cPages, paPages, enmAccount));
3575
3576 /*
3577 * Validate input and get the basics.
3578 */
3579 PGMM pGMM;
3580 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3581 int rc = GVMMR0ValidateGVMandVMandEMT(pGVM, pVM, idCpu);
3582 if (RT_FAILURE(rc))
3583 return rc;
3584
3585 AssertPtrReturn(paPages, VERR_INVALID_PARAMETER);
3586 AssertMsgReturn(enmAccount > GMMACCOUNT_INVALID && enmAccount < GMMACCOUNT_END, ("%d\n", enmAccount), VERR_INVALID_PARAMETER);
3587 AssertMsgReturn(cPages > 0 && cPages < RT_BIT(32 - PAGE_SHIFT), ("%#x\n", cPages), VERR_INVALID_PARAMETER);
3588
3589 for (unsigned iPage = 0; iPage < cPages; iPage++)
3590 AssertMsgReturn( paPages[iPage].idPage <= GMM_PAGEID_LAST
3591 /*|| paPages[iPage].idPage == NIL_GMM_PAGEID*/,
3592 ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
3593
3594 /*
3595 * Take the semaphore and call the worker function.
3596 */
3597 gmmR0MutexAcquire(pGMM);
3598 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3599 {
3600 rc = gmmR0FreePages(pGMM, pGVM, cPages, paPages, enmAccount);
3601 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
3602 }
3603 else
3604 rc = VERR_GMM_IS_NOT_SANE;
3605 gmmR0MutexRelease(pGMM);
3606 LogFlow(("GMMR0FreePages: returns %Rrc\n", rc));
3607 return rc;
3608}
3609
3610
3611/**
3612 * VMMR0 request wrapper for GMMR0FreePages.
3613 *
3614 * @returns see GMMR0FreePages.
3615 * @param pGVM The global (ring-0) VM structure.
3616 * @param pVM The cross context VM structure.
3617 * @param idCpu The VCPU id.
3618 * @param pReq Pointer to the request packet.
3619 */
3620GMMR0DECL(int) GMMR0FreePagesReq(PGVM pGVM, PVM pVM, VMCPUID idCpu, PGMMFREEPAGESREQ pReq)
3621{
3622 /*
3623 * Validate input and pass it on.
3624 */
3625 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3626 AssertMsgReturn(pReq->Hdr.cbReq >= RT_UOFFSETOF(GMMFREEPAGESREQ, aPages[0]),
3627 ("%#x < %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF(GMMFREEPAGESREQ, aPages[0])),
3628 VERR_INVALID_PARAMETER);
3629 AssertMsgReturn(pReq->Hdr.cbReq == RT_UOFFSETOF_DYN(GMMFREEPAGESREQ, aPages[pReq->cPages]),
3630 ("%#x != %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF_DYN(GMMFREEPAGESREQ, aPages[pReq->cPages])),
3631 VERR_INVALID_PARAMETER);
3632
3633 return GMMR0FreePages(pGVM, pVM, idCpu, pReq->cPages, &pReq->aPages[0], pReq->enmAccount);
3634}
3635
3636
3637/**
3638 * Report back on a memory ballooning request.
3639 *
3640 * The request may or may not have been initiated by the GMM. If it was initiated
3641 * by the GMM it is important that this function is called even if no pages were
3642 * ballooned.
3643 *
3644 * @returns VBox status code:
3645 * @retval VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH
3646 * @retval VERR_GMM_ATTEMPT_TO_DEFLATE_TOO_MUCH
3647 * @retval VERR_GMM_OVERCOMMITTED_TRY_AGAIN_IN_A_BIT - reset condition
3648 * indicating that we won't necessarily have sufficient RAM to boot
3649 * the VM again and that it should pause until this changes (we'll try
3650 * balloon some other VM). (For standard deflate we have little choice
3651 * but to hope the VM won't use the memory that was returned to it.)
3652 *
3653 * @param pGVM The global (ring-0) VM structure.
3654 * @param pVM The cross context VM structure.
3655 * @param idCpu The VCPU id.
3656 * @param enmAction Inflate/deflate/reset.
3657 * @param cBalloonedPages The number of pages that was ballooned.
3658 *
3659 * @thread EMT(idCpu)
3660 */
3661GMMR0DECL(int) GMMR0BalloonedPages(PGVM pGVM, PVM pVM, VMCPUID idCpu, GMMBALLOONACTION enmAction, uint32_t cBalloonedPages)
3662{
3663 LogFlow(("GMMR0BalloonedPages: pGVM=%p pVM=%p enmAction=%d cBalloonedPages=%#x\n",
3664 pGVM, pVM, enmAction, cBalloonedPages));
3665
3666 AssertMsgReturn(cBalloonedPages < RT_BIT(32 - PAGE_SHIFT), ("%#x\n", cBalloonedPages), VERR_INVALID_PARAMETER);
3667
3668 /*
3669 * Validate input and get the basics.
3670 */
3671 PGMM pGMM;
3672 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3673 int rc = GVMMR0ValidateGVMandVMandEMT(pGVM, pVM, idCpu);
3674 if (RT_FAILURE(rc))
3675 return rc;
3676
3677 /*
3678 * Take the semaphore and do some more validations.
3679 */
3680 gmmR0MutexAcquire(pGMM);
3681 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3682 {
3683 switch (enmAction)
3684 {
3685 case GMMBALLOONACTION_INFLATE:
3686 {
3687 if (RT_LIKELY(pGVM->gmm.s.Stats.Allocated.cBasePages + pGVM->gmm.s.Stats.cBalloonedPages + cBalloonedPages
3688 <= pGVM->gmm.s.Stats.Reserved.cBasePages))
3689 {
3690 /*
3691 * Record the ballooned memory.
3692 */
3693 pGMM->cBalloonedPages += cBalloonedPages;
3694 if (pGVM->gmm.s.Stats.cReqBalloonedPages)
3695 {
3696 /* Codepath never taken. Might be interesting in the future to request ballooned memory from guests in low memory conditions.. */
3697 AssertFailed();
3698
3699 pGVM->gmm.s.Stats.cBalloonedPages += cBalloonedPages;
3700 pGVM->gmm.s.Stats.cReqActuallyBalloonedPages += cBalloonedPages;
3701 Log(("GMMR0BalloonedPages: +%#x - Global=%#llx / VM: Total=%#llx Req=%#llx Actual=%#llx (pending)\n",
3702 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages,
3703 pGVM->gmm.s.Stats.cReqBalloonedPages, pGVM->gmm.s.Stats.cReqActuallyBalloonedPages));
3704 }
3705 else
3706 {
3707 pGVM->gmm.s.Stats.cBalloonedPages += cBalloonedPages;
3708 Log(("GMMR0BalloonedPages: +%#x - Global=%#llx / VM: Total=%#llx (user)\n",
3709 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages));
3710 }
3711 }
3712 else
3713 {
3714 Log(("GMMR0BalloonedPages: cBasePages=%#llx Total=%#llx cBalloonedPages=%#llx Reserved=%#llx\n",
3715 pGVM->gmm.s.Stats.Allocated.cBasePages, pGVM->gmm.s.Stats.cBalloonedPages, cBalloonedPages,
3716 pGVM->gmm.s.Stats.Reserved.cBasePages));
3717 rc = VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3718 }
3719 break;
3720 }
3721
3722 case GMMBALLOONACTION_DEFLATE:
3723 {
3724 /* Deflate. */
3725 if (pGVM->gmm.s.Stats.cBalloonedPages >= cBalloonedPages)
3726 {
3727 /*
3728 * Record the ballooned memory.
3729 */
3730 Assert(pGMM->cBalloonedPages >= cBalloonedPages);
3731 pGMM->cBalloonedPages -= cBalloonedPages;
3732 pGVM->gmm.s.Stats.cBalloonedPages -= cBalloonedPages;
3733 if (pGVM->gmm.s.Stats.cReqDeflatePages)
3734 {
3735 AssertFailed(); /* This is path is for later. */
3736 Log(("GMMR0BalloonedPages: -%#x - Global=%#llx / VM: Total=%#llx Req=%#llx\n",
3737 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages, pGVM->gmm.s.Stats.cReqDeflatePages));
3738
3739 /*
3740 * Anything we need to do here now when the request has been completed?
3741 */
3742 pGVM->gmm.s.Stats.cReqDeflatePages = 0;
3743 }
3744 else
3745 Log(("GMMR0BalloonedPages: -%#x - Global=%#llx / VM: Total=%#llx (user)\n",
3746 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages));
3747 }
3748 else
3749 {
3750 Log(("GMMR0BalloonedPages: Total=%#llx cBalloonedPages=%#llx\n", pGVM->gmm.s.Stats.cBalloonedPages, cBalloonedPages));
3751 rc = VERR_GMM_ATTEMPT_TO_DEFLATE_TOO_MUCH;
3752 }
3753 break;
3754 }
3755
3756 case GMMBALLOONACTION_RESET:
3757 {
3758 /* Reset to an empty balloon. */
3759 Assert(pGMM->cBalloonedPages >= pGVM->gmm.s.Stats.cBalloonedPages);
3760
3761 pGMM->cBalloonedPages -= pGVM->gmm.s.Stats.cBalloonedPages;
3762 pGVM->gmm.s.Stats.cBalloonedPages = 0;
3763 break;
3764 }
3765
3766 default:
3767 rc = VERR_INVALID_PARAMETER;
3768 break;
3769 }
3770 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
3771 }
3772 else
3773 rc = VERR_GMM_IS_NOT_SANE;
3774
3775 gmmR0MutexRelease(pGMM);
3776 LogFlow(("GMMR0BalloonedPages: returns %Rrc\n", rc));
3777 return rc;
3778}
3779
3780
3781/**
3782 * VMMR0 request wrapper for GMMR0BalloonedPages.
3783 *
3784 * @returns see GMMR0BalloonedPages.
3785 * @param pGVM The global (ring-0) VM structure.
3786 * @param pVM The cross context VM structure.
3787 * @param idCpu The VCPU id.
3788 * @param pReq Pointer to the request packet.
3789 */
3790GMMR0DECL(int) GMMR0BalloonedPagesReq(PGVM pGVM, PVM pVM, VMCPUID idCpu, PGMMBALLOONEDPAGESREQ pReq)
3791{
3792 /*
3793 * Validate input and pass it on.
3794 */
3795 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3796 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMBALLOONEDPAGESREQ),
3797 ("%#x < %#x\n", pReq->Hdr.cbReq, sizeof(GMMBALLOONEDPAGESREQ)),
3798 VERR_INVALID_PARAMETER);
3799
3800 return GMMR0BalloonedPages(pGVM, pVM, idCpu, pReq->enmAction, pReq->cBalloonedPages);
3801}
3802
3803
3804/**
3805 * Return memory statistics for the hypervisor
3806 *
3807 * @returns VBox status code.
3808 * @param pReq Pointer to the request packet.
3809 */
3810GMMR0DECL(int) GMMR0QueryHypervisorMemoryStatsReq(PGMMMEMSTATSREQ pReq)
3811{
3812 /*
3813 * Validate input and pass it on.
3814 */
3815 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3816 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMMEMSTATSREQ),
3817 ("%#x < %#x\n", pReq->Hdr.cbReq, sizeof(GMMMEMSTATSREQ)),
3818 VERR_INVALID_PARAMETER);
3819
3820 /*
3821 * Validate input and get the basics.
3822 */
3823 PGMM pGMM;
3824 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3825 pReq->cAllocPages = pGMM->cAllocatedPages;
3826 pReq->cFreePages = (pGMM->cChunks << (GMM_CHUNK_SHIFT- PAGE_SHIFT)) - pGMM->cAllocatedPages;
3827 pReq->cBalloonedPages = pGMM->cBalloonedPages;
3828 pReq->cMaxPages = pGMM->cMaxPages;
3829 pReq->cSharedPages = pGMM->cDuplicatePages;
3830 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
3831
3832 return VINF_SUCCESS;
3833}
3834
3835
3836/**
3837 * Return memory statistics for the VM
3838 *
3839 * @returns VBox status code.
3840 * @param pGVM The global (ring-0) VM structure.
3841 * @param pVM The cross context VM structure.
3842 * @param idCpu Cpu id.
3843 * @param pReq Pointer to the request packet.
3844 *
3845 * @thread EMT(idCpu)
3846 */
3847GMMR0DECL(int) GMMR0QueryMemoryStatsReq(PGVM pGVM, PVM pVM, VMCPUID idCpu, PGMMMEMSTATSREQ pReq)
3848{
3849 /*
3850 * Validate input and pass it on.
3851 */
3852 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3853 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMMEMSTATSREQ),
3854 ("%#x < %#x\n", pReq->Hdr.cbReq, sizeof(GMMMEMSTATSREQ)),
3855 VERR_INVALID_PARAMETER);
3856
3857 /*
3858 * Validate input and get the basics.
3859 */
3860 PGMM pGMM;
3861 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3862 int rc = GVMMR0ValidateGVMandVMandEMT(pGVM, pVM, idCpu);
3863 if (RT_FAILURE(rc))
3864 return rc;
3865
3866 /*
3867 * Take the semaphore and do some more validations.
3868 */
3869 gmmR0MutexAcquire(pGMM);
3870 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3871 {
3872 pReq->cAllocPages = pGVM->gmm.s.Stats.Allocated.cBasePages;
3873 pReq->cBalloonedPages = pGVM->gmm.s.Stats.cBalloonedPages;
3874 pReq->cMaxPages = pGVM->gmm.s.Stats.Reserved.cBasePages;
3875 pReq->cFreePages = pReq->cMaxPages - pReq->cAllocPages;
3876 }
3877 else
3878 rc = VERR_GMM_IS_NOT_SANE;
3879
3880 gmmR0MutexRelease(pGMM);
3881 LogFlow(("GMMR3QueryVMMemoryStats: returns %Rrc\n", rc));
3882 return rc;
3883}
3884
3885
3886/**
3887 * Worker for gmmR0UnmapChunk and gmmr0FreeChunk.
3888 *
3889 * Don't call this in legacy allocation mode!
3890 *
3891 * @returns VBox status code.
3892 * @param pGMM Pointer to the GMM instance data.
3893 * @param pGVM Pointer to the Global VM structure.
3894 * @param pChunk Pointer to the chunk to be unmapped.
3895 */
3896static int gmmR0UnmapChunkLocked(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk)
3897{
3898 Assert(!pGMM->fLegacyAllocationMode); NOREF(pGMM);
3899
3900 /*
3901 * Find the mapping and try unmapping it.
3902 */
3903 uint32_t cMappings = pChunk->cMappingsX;
3904 for (uint32_t i = 0; i < cMappings; i++)
3905 {
3906 Assert(pChunk->paMappingsX[i].pGVM && pChunk->paMappingsX[i].hMapObj != NIL_RTR0MEMOBJ);
3907 if (pChunk->paMappingsX[i].pGVM == pGVM)
3908 {
3909 /* unmap */
3910 int rc = RTR0MemObjFree(pChunk->paMappingsX[i].hMapObj, false /* fFreeMappings (NA) */);
3911 if (RT_SUCCESS(rc))
3912 {
3913 /* update the record. */
3914 cMappings--;
3915 if (i < cMappings)
3916 pChunk->paMappingsX[i] = pChunk->paMappingsX[cMappings];
3917 pChunk->paMappingsX[cMappings].hMapObj = NIL_RTR0MEMOBJ;
3918 pChunk->paMappingsX[cMappings].pGVM = NULL;
3919 Assert(pChunk->cMappingsX - 1U == cMappings);
3920 pChunk->cMappingsX = cMappings;
3921 }
3922
3923 return rc;
3924 }
3925 }
3926
3927 Log(("gmmR0UnmapChunk: Chunk %#x is not mapped into pGVM=%p/%#x\n", pChunk->Core.Key, pGVM, pGVM->hSelf));
3928 return VERR_GMM_CHUNK_NOT_MAPPED;
3929}
3930
3931
3932/**
3933 * Unmaps a chunk previously mapped into the address space of the current process.
3934 *
3935 * @returns VBox status code.
3936 * @param pGMM Pointer to the GMM instance data.
3937 * @param pGVM Pointer to the Global VM structure.
3938 * @param pChunk Pointer to the chunk to be unmapped.
3939 * @param fRelaxedSem Whether we can release the semaphore while doing the
3940 * mapping (@c true) or not.
3941 */
3942static int gmmR0UnmapChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem)
3943{
3944 if (!pGMM->fLegacyAllocationMode)
3945 {
3946 /*
3947 * Lock the chunk and if possible leave the giant GMM lock.
3948 */
3949 GMMR0CHUNKMTXSTATE MtxState;
3950 int rc = gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk,
3951 fRelaxedSem ? GMMR0CHUNK_MTX_RETAKE_GIANT : GMMR0CHUNK_MTX_KEEP_GIANT);
3952 if (RT_SUCCESS(rc))
3953 {
3954 rc = gmmR0UnmapChunkLocked(pGMM, pGVM, pChunk);
3955 gmmR0ChunkMutexRelease(&MtxState, pChunk);
3956 }
3957 return rc;
3958 }
3959
3960 if (pChunk->hGVM == pGVM->hSelf)
3961 return VINF_SUCCESS;
3962
3963 Log(("gmmR0UnmapChunk: Chunk %#x is not mapped into pGVM=%p/%#x (legacy)\n", pChunk->Core.Key, pGVM, pGVM->hSelf));
3964 return VERR_GMM_CHUNK_NOT_MAPPED;
3965}
3966
3967
3968/**
3969 * Worker for gmmR0MapChunk.
3970 *
3971 * @returns VBox status code.
3972 * @param pGMM Pointer to the GMM instance data.
3973 * @param pGVM Pointer to the Global VM structure.
3974 * @param pChunk Pointer to the chunk to be mapped.
3975 * @param ppvR3 Where to store the ring-3 address of the mapping.
3976 * In the VERR_GMM_CHUNK_ALREADY_MAPPED case, this will be
3977 * contain the address of the existing mapping.
3978 */
3979static int gmmR0MapChunkLocked(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, PRTR3PTR ppvR3)
3980{
3981 /*
3982 * If we're in legacy mode this is simple.
3983 */
3984 if (pGMM->fLegacyAllocationMode)
3985 {
3986 if (pChunk->hGVM != pGVM->hSelf)
3987 {
3988 Log(("gmmR0MapChunk: chunk %#x is already mapped at %p!\n", pChunk->Core.Key, *ppvR3));
3989 return VERR_GMM_CHUNK_NOT_FOUND;
3990 }
3991
3992 *ppvR3 = RTR0MemObjAddressR3(pChunk->hMemObj);
3993 return VINF_SUCCESS;
3994 }
3995
3996 /*
3997 * Check to see if the chunk is already mapped.
3998 */
3999 for (uint32_t i = 0; i < pChunk->cMappingsX; i++)
4000 {
4001 Assert(pChunk->paMappingsX[i].pGVM && pChunk->paMappingsX[i].hMapObj != NIL_RTR0MEMOBJ);
4002 if (pChunk->paMappingsX[i].pGVM == pGVM)
4003 {
4004 *ppvR3 = RTR0MemObjAddressR3(pChunk->paMappingsX[i].hMapObj);
4005 Log(("gmmR0MapChunk: chunk %#x is already mapped at %p!\n", pChunk->Core.Key, *ppvR3));
4006#ifdef VBOX_WITH_PAGE_SHARING
4007 /* The ring-3 chunk cache can be out of sync; don't fail. */
4008 return VINF_SUCCESS;
4009#else
4010 return VERR_GMM_CHUNK_ALREADY_MAPPED;
4011#endif
4012 }
4013 }
4014
4015 /*
4016 * Do the mapping.
4017 */
4018 RTR0MEMOBJ hMapObj;
4019 int rc = RTR0MemObjMapUser(&hMapObj, pChunk->hMemObj, (RTR3PTR)-1, 0, RTMEM_PROT_READ | RTMEM_PROT_WRITE, NIL_RTR0PROCESS);
4020 if (RT_SUCCESS(rc))
4021 {
4022 /* reallocate the array? assumes few users per chunk (usually one). */
4023 unsigned iMapping = pChunk->cMappingsX;
4024 if ( iMapping <= 3
4025 || (iMapping & 3) == 0)
4026 {
4027 unsigned cNewSize = iMapping <= 3
4028 ? iMapping + 1
4029 : iMapping + 4;
4030 Assert(cNewSize < 4 || RT_ALIGN_32(cNewSize, 4) == cNewSize);
4031 if (RT_UNLIKELY(cNewSize > UINT16_MAX))
4032 {
4033 rc = RTR0MemObjFree(hMapObj, false /* fFreeMappings (NA) */); AssertRC(rc);
4034 return VERR_GMM_TOO_MANY_CHUNK_MAPPINGS;
4035 }
4036
4037 void *pvMappings = RTMemRealloc(pChunk->paMappingsX, cNewSize * sizeof(pChunk->paMappingsX[0]));
4038 if (RT_UNLIKELY(!pvMappings))
4039 {
4040 rc = RTR0MemObjFree(hMapObj, false /* fFreeMappings (NA) */); AssertRC(rc);
4041 return VERR_NO_MEMORY;
4042 }
4043 pChunk->paMappingsX = (PGMMCHUNKMAP)pvMappings;
4044 }
4045
4046 /* insert new entry */
4047 pChunk->paMappingsX[iMapping].hMapObj = hMapObj;
4048 pChunk->paMappingsX[iMapping].pGVM = pGVM;
4049 Assert(pChunk->cMappingsX == iMapping);
4050 pChunk->cMappingsX = iMapping + 1;
4051
4052 *ppvR3 = RTR0MemObjAddressR3(hMapObj);
4053 }
4054
4055 return rc;
4056}
4057
4058
4059/**
4060 * Maps a chunk into the user address space of the current process.
4061 *
4062 * @returns VBox status code.
4063 * @param pGMM Pointer to the GMM instance data.
4064 * @param pGVM Pointer to the Global VM structure.
4065 * @param pChunk Pointer to the chunk to be mapped.
4066 * @param fRelaxedSem Whether we can release the semaphore while doing the
4067 * mapping (@c true) or not.
4068 * @param ppvR3 Where to store the ring-3 address of the mapping.
4069 * In the VERR_GMM_CHUNK_ALREADY_MAPPED case, this will be
4070 * contain the address of the existing mapping.
4071 */
4072static int gmmR0MapChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem, PRTR3PTR ppvR3)
4073{
4074 /*
4075 * Take the chunk lock and leave the giant GMM lock when possible, then
4076 * call the worker function.
4077 */
4078 GMMR0CHUNKMTXSTATE MtxState;
4079 int rc = gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk,
4080 fRelaxedSem ? GMMR0CHUNK_MTX_RETAKE_GIANT : GMMR0CHUNK_MTX_KEEP_GIANT);
4081 if (RT_SUCCESS(rc))
4082 {
4083 rc = gmmR0MapChunkLocked(pGMM, pGVM, pChunk, ppvR3);
4084 gmmR0ChunkMutexRelease(&MtxState, pChunk);
4085 }
4086
4087 return rc;
4088}
4089
4090
4091
4092#if defined(VBOX_WITH_PAGE_SHARING) || (defined(VBOX_STRICT) && HC_ARCH_BITS == 64)
4093/**
4094 * Check if a chunk is mapped into the specified VM
4095 *
4096 * @returns mapped yes/no
4097 * @param pGMM Pointer to the GMM instance.
4098 * @param pGVM Pointer to the Global VM structure.
4099 * @param pChunk Pointer to the chunk to be mapped.
4100 * @param ppvR3 Where to store the ring-3 address of the mapping.
4101 */
4102static bool gmmR0IsChunkMapped(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, PRTR3PTR ppvR3)
4103{
4104 GMMR0CHUNKMTXSTATE MtxState;
4105 gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk, GMMR0CHUNK_MTX_KEEP_GIANT);
4106 for (uint32_t i = 0; i < pChunk->cMappingsX; i++)
4107 {
4108 Assert(pChunk->paMappingsX[i].pGVM && pChunk->paMappingsX[i].hMapObj != NIL_RTR0MEMOBJ);
4109 if (pChunk->paMappingsX[i].pGVM == pGVM)
4110 {
4111 *ppvR3 = RTR0MemObjAddressR3(pChunk->paMappingsX[i].hMapObj);
4112 gmmR0ChunkMutexRelease(&MtxState, pChunk);
4113 return true;
4114 }
4115 }
4116 *ppvR3 = NULL;
4117 gmmR0ChunkMutexRelease(&MtxState, pChunk);
4118 return false;
4119}
4120#endif /* VBOX_WITH_PAGE_SHARING || (VBOX_STRICT && 64-BIT) */
4121
4122
4123/**
4124 * Map a chunk and/or unmap another chunk.
4125 *
4126 * The mapping and unmapping applies to the current process.
4127 *
4128 * This API does two things because it saves a kernel call per mapping when
4129 * when the ring-3 mapping cache is full.
4130 *
4131 * @returns VBox status code.
4132 * @param pGVM The global (ring-0) VM structure.
4133 * @param pVM The cross context VM structure.
4134 * @param idChunkMap The chunk to map. NIL_GMM_CHUNKID if nothing to map.
4135 * @param idChunkUnmap The chunk to unmap. NIL_GMM_CHUNKID if nothing to unmap.
4136 * @param ppvR3 Where to store the address of the mapped chunk. NULL is ok if nothing to map.
4137 * @thread EMT ???
4138 */
4139GMMR0DECL(int) GMMR0MapUnmapChunk(PGVM pGVM, PVM pVM, uint32_t idChunkMap, uint32_t idChunkUnmap, PRTR3PTR ppvR3)
4140{
4141 LogFlow(("GMMR0MapUnmapChunk: pGVM=%p pVM=%p idChunkMap=%#x idChunkUnmap=%#x ppvR3=%p\n",
4142 pGVM, pVM, idChunkMap, idChunkUnmap, ppvR3));
4143
4144 /*
4145 * Validate input and get the basics.
4146 */
4147 PGMM pGMM;
4148 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4149 int rc = GVMMR0ValidateGVMandVM(pGVM, pVM);
4150 if (RT_FAILURE(rc))
4151 return rc;
4152
4153 AssertCompile(NIL_GMM_CHUNKID == 0);
4154 AssertMsgReturn(idChunkMap <= GMM_CHUNKID_LAST, ("%#x\n", idChunkMap), VERR_INVALID_PARAMETER);
4155 AssertMsgReturn(idChunkUnmap <= GMM_CHUNKID_LAST, ("%#x\n", idChunkUnmap), VERR_INVALID_PARAMETER);
4156
4157 if ( idChunkMap == NIL_GMM_CHUNKID
4158 && idChunkUnmap == NIL_GMM_CHUNKID)
4159 return VERR_INVALID_PARAMETER;
4160
4161 if (idChunkMap != NIL_GMM_CHUNKID)
4162 {
4163 AssertPtrReturn(ppvR3, VERR_INVALID_POINTER);
4164 *ppvR3 = NIL_RTR3PTR;
4165 }
4166
4167 /*
4168 * Take the semaphore and do the work.
4169 *
4170 * The unmapping is done last since it's easier to undo a mapping than
4171 * undoing an unmapping. The ring-3 mapping cache cannot not be so big
4172 * that it pushes the user virtual address space to within a chunk of
4173 * it it's limits, so, no problem here.
4174 */
4175 gmmR0MutexAcquire(pGMM);
4176 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
4177 {
4178 PGMMCHUNK pMap = NULL;
4179 if (idChunkMap != NIL_GVM_HANDLE)
4180 {
4181 pMap = gmmR0GetChunk(pGMM, idChunkMap);
4182 if (RT_LIKELY(pMap))
4183 rc = gmmR0MapChunk(pGMM, pGVM, pMap, true /*fRelaxedSem*/, ppvR3);
4184 else
4185 {
4186 Log(("GMMR0MapUnmapChunk: idChunkMap=%#x\n", idChunkMap));
4187 rc = VERR_GMM_CHUNK_NOT_FOUND;
4188 }
4189 }
4190/** @todo split this operation, the bail out might (theoretcially) not be
4191 * entirely safe. */
4192
4193 if ( idChunkUnmap != NIL_GMM_CHUNKID
4194 && RT_SUCCESS(rc))
4195 {
4196 PGMMCHUNK pUnmap = gmmR0GetChunk(pGMM, idChunkUnmap);
4197 if (RT_LIKELY(pUnmap))
4198 rc = gmmR0UnmapChunk(pGMM, pGVM, pUnmap, true /*fRelaxedSem*/);
4199 else
4200 {
4201 Log(("GMMR0MapUnmapChunk: idChunkUnmap=%#x\n", idChunkUnmap));
4202 rc = VERR_GMM_CHUNK_NOT_FOUND;
4203 }
4204
4205 if (RT_FAILURE(rc) && pMap)
4206 gmmR0UnmapChunk(pGMM, pGVM, pMap, false /*fRelaxedSem*/);
4207 }
4208
4209 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
4210 }
4211 else
4212 rc = VERR_GMM_IS_NOT_SANE;
4213 gmmR0MutexRelease(pGMM);
4214
4215 LogFlow(("GMMR0MapUnmapChunk: returns %Rrc\n", rc));
4216 return rc;
4217}
4218
4219
4220/**
4221 * VMMR0 request wrapper for GMMR0MapUnmapChunk.
4222 *
4223 * @returns see GMMR0MapUnmapChunk.
4224 * @param pGVM The global (ring-0) VM structure.
4225 * @param pVM The cross context VM structure.
4226 * @param pReq Pointer to the request packet.
4227 */
4228GMMR0DECL(int) GMMR0MapUnmapChunkReq(PGVM pGVM, PVM pVM, PGMMMAPUNMAPCHUNKREQ pReq)
4229{
4230 /*
4231 * Validate input and pass it on.
4232 */
4233 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
4234 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
4235
4236 return GMMR0MapUnmapChunk(pGVM, pVM, pReq->idChunkMap, pReq->idChunkUnmap, &pReq->pvR3);
4237}
4238
4239
4240/**
4241 * Legacy mode API for supplying pages.
4242 *
4243 * The specified user address points to a allocation chunk sized block that
4244 * will be locked down and used by the GMM when the GM asks for pages.
4245 *
4246 * @returns VBox status code.
4247 * @param pGVM The global (ring-0) VM structure.
4248 * @param pVM The cross context VM structure.
4249 * @param idCpu The VCPU id.
4250 * @param pvR3 Pointer to the chunk size memory block to lock down.
4251 */
4252GMMR0DECL(int) GMMR0SeedChunk(PGVM pGVM, PVM pVM, VMCPUID idCpu, RTR3PTR pvR3)
4253{
4254 /*
4255 * Validate input and get the basics.
4256 */
4257 PGMM pGMM;
4258 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4259 int rc = GVMMR0ValidateGVMandVMandEMT(pGVM, pVM, idCpu);
4260 if (RT_FAILURE(rc))
4261 return rc;
4262
4263 AssertPtrReturn(pvR3, VERR_INVALID_POINTER);
4264 AssertReturn(!(PAGE_OFFSET_MASK & pvR3), VERR_INVALID_POINTER);
4265
4266 if (!pGMM->fLegacyAllocationMode)
4267 {
4268 Log(("GMMR0SeedChunk: not in legacy allocation mode!\n"));
4269 return VERR_NOT_SUPPORTED;
4270 }
4271
4272 /*
4273 * Lock the memory and add it as new chunk with our hGVM.
4274 * (The GMM locking is done inside gmmR0RegisterChunk.)
4275 */
4276 RTR0MEMOBJ MemObj;
4277 rc = RTR0MemObjLockUser(&MemObj, pvR3, GMM_CHUNK_SIZE, RTMEM_PROT_READ | RTMEM_PROT_WRITE, NIL_RTR0PROCESS);
4278 if (RT_SUCCESS(rc))
4279 {
4280 rc = gmmR0RegisterChunk(pGMM, &pGVM->gmm.s.Private, MemObj, pGVM->hSelf, 0 /*fChunkFlags*/, NULL);
4281 if (RT_SUCCESS(rc))
4282 gmmR0MutexRelease(pGMM);
4283 else
4284 RTR0MemObjFree(MemObj, false /* fFreeMappings */);
4285 }
4286
4287 LogFlow(("GMMR0SeedChunk: rc=%d (pvR3=%p)\n", rc, pvR3));
4288 return rc;
4289}
4290
4291#ifdef VBOX_WITH_PAGE_SHARING
4292
4293# ifdef VBOX_STRICT
4294/**
4295 * For checksumming shared pages in strict builds.
4296 *
4297 * The purpose is making sure that a page doesn't change.
4298 *
4299 * @returns Checksum, 0 on failure.
4300 * @param pGMM The GMM instance data.
4301 * @param pGVM Pointer to the kernel-only VM instace data.
4302 * @param idPage The page ID.
4303 */
4304static uint32_t gmmR0StrictPageChecksum(PGMM pGMM, PGVM pGVM, uint32_t idPage)
4305{
4306 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
4307 AssertMsgReturn(pChunk, ("idPage=%#x\n", idPage), 0);
4308
4309 uint8_t *pbChunk;
4310 if (!gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
4311 return 0;
4312 uint8_t const *pbPage = pbChunk + ((idPage & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
4313
4314 return RTCrc32(pbPage, PAGE_SIZE);
4315}
4316# endif /* VBOX_STRICT */
4317
4318
4319/**
4320 * Calculates the module hash value.
4321 *
4322 * @returns Hash value.
4323 * @param pszModuleName The module name.
4324 * @param pszVersion The module version string.
4325 */
4326static uint32_t gmmR0ShModCalcHash(const char *pszModuleName, const char *pszVersion)
4327{
4328 return RTStrHash1ExN(3, pszModuleName, RTSTR_MAX, "::", (size_t)2, pszVersion, RTSTR_MAX);
4329}
4330
4331
4332/**
4333 * Finds a global module.
4334 *
4335 * @returns Pointer to the global module on success, NULL if not found.
4336 * @param pGMM The GMM instance data.
4337 * @param uHash The hash as calculated by gmmR0ShModCalcHash.
4338 * @param cbModule The module size.
4339 * @param enmGuestOS The guest OS type.
4340 * @param cRegions The number of regions.
4341 * @param pszModuleName The module name.
4342 * @param pszVersion The module version.
4343 * @param paRegions The region descriptions.
4344 */
4345static PGMMSHAREDMODULE gmmR0ShModFindGlobal(PGMM pGMM, uint32_t uHash, uint32_t cbModule, VBOXOSFAMILY enmGuestOS,
4346 uint32_t cRegions, const char *pszModuleName, const char *pszVersion,
4347 struct VMMDEVSHAREDREGIONDESC const *paRegions)
4348{
4349 for (PGMMSHAREDMODULE pGblMod = (PGMMSHAREDMODULE)RTAvllU32Get(&pGMM->pGlobalSharedModuleTree, uHash);
4350 pGblMod;
4351 pGblMod = (PGMMSHAREDMODULE)pGblMod->Core.pList)
4352 {
4353 if (pGblMod->cbModule != cbModule)
4354 continue;
4355 if (pGblMod->enmGuestOS != enmGuestOS)
4356 continue;
4357 if (pGblMod->cRegions != cRegions)
4358 continue;
4359 if (strcmp(pGblMod->szName, pszModuleName))
4360 continue;
4361 if (strcmp(pGblMod->szVersion, pszVersion))
4362 continue;
4363
4364 uint32_t i;
4365 for (i = 0; i < cRegions; i++)
4366 {
4367 uint32_t off = paRegions[i].GCRegionAddr & PAGE_OFFSET_MASK;
4368 if (pGblMod->aRegions[i].off != off)
4369 break;
4370
4371 uint32_t cb = RT_ALIGN_32(paRegions[i].cbRegion + off, PAGE_SIZE);
4372 if (pGblMod->aRegions[i].cb != cb)
4373 break;
4374 }
4375
4376 if (i == cRegions)
4377 return pGblMod;
4378 }
4379
4380 return NULL;
4381}
4382
4383
4384/**
4385 * Creates a new global module.
4386 *
4387 * @returns VBox status code.
4388 * @param pGMM The GMM instance data.
4389 * @param uHash The hash as calculated by gmmR0ShModCalcHash.
4390 * @param cbModule The module size.
4391 * @param enmGuestOS The guest OS type.
4392 * @param cRegions The number of regions.
4393 * @param pszModuleName The module name.
4394 * @param pszVersion The module version.
4395 * @param paRegions The region descriptions.
4396 * @param ppGblMod Where to return the new module on success.
4397 */
4398static int gmmR0ShModNewGlobal(PGMM pGMM, uint32_t uHash, uint32_t cbModule, VBOXOSFAMILY enmGuestOS,
4399 uint32_t cRegions, const char *pszModuleName, const char *pszVersion,
4400 struct VMMDEVSHAREDREGIONDESC const *paRegions, PGMMSHAREDMODULE *ppGblMod)
4401{
4402 Log(("gmmR0ShModNewGlobal: %s %s size %#x os %u rgn %u\n", pszModuleName, pszVersion, cbModule, enmGuestOS, cRegions));
4403 if (pGMM->cShareableModules >= GMM_MAX_SHARED_GLOBAL_MODULES)
4404 {
4405 Log(("gmmR0ShModNewGlobal: Too many modules\n"));
4406 return VERR_GMM_TOO_MANY_GLOBAL_MODULES;
4407 }
4408
4409 PGMMSHAREDMODULE pGblMod = (PGMMSHAREDMODULE)RTMemAllocZ(RT_UOFFSETOF_DYN(GMMSHAREDMODULE, aRegions[cRegions]));
4410 if (!pGblMod)
4411 {
4412 Log(("gmmR0ShModNewGlobal: No memory\n"));
4413 return VERR_NO_MEMORY;
4414 }
4415
4416 pGblMod->Core.Key = uHash;
4417 pGblMod->cbModule = cbModule;
4418 pGblMod->cRegions = cRegions;
4419 pGblMod->cUsers = 1;
4420 pGblMod->enmGuestOS = enmGuestOS;
4421 strcpy(pGblMod->szName, pszModuleName);
4422 strcpy(pGblMod->szVersion, pszVersion);
4423
4424 for (uint32_t i = 0; i < cRegions; i++)
4425 {
4426 Log(("gmmR0ShModNewGlobal: rgn[%u]=%RGvLB%#x\n", i, paRegions[i].GCRegionAddr, paRegions[i].cbRegion));
4427 pGblMod->aRegions[i].off = paRegions[i].GCRegionAddr & PAGE_OFFSET_MASK;
4428 pGblMod->aRegions[i].cb = paRegions[i].cbRegion + pGblMod->aRegions[i].off;
4429 pGblMod->aRegions[i].cb = RT_ALIGN_32(pGblMod->aRegions[i].cb, PAGE_SIZE);
4430 pGblMod->aRegions[i].paidPages = NULL; /* allocated when needed. */
4431 }
4432
4433 bool fInsert = RTAvllU32Insert(&pGMM->pGlobalSharedModuleTree, &pGblMod->Core);
4434 Assert(fInsert); NOREF(fInsert);
4435 pGMM->cShareableModules++;
4436
4437 *ppGblMod = pGblMod;
4438 return VINF_SUCCESS;
4439}
4440
4441
4442/**
4443 * Deletes a global module which is no longer referenced by anyone.
4444 *
4445 * @param pGMM The GMM instance data.
4446 * @param pGblMod The module to delete.
4447 */
4448static void gmmR0ShModDeleteGlobal(PGMM pGMM, PGMMSHAREDMODULE pGblMod)
4449{
4450 Assert(pGblMod->cUsers == 0);
4451 Assert(pGMM->cShareableModules > 0 && pGMM->cShareableModules <= GMM_MAX_SHARED_GLOBAL_MODULES);
4452
4453 void *pvTest = RTAvllU32RemoveNode(&pGMM->pGlobalSharedModuleTree, &pGblMod->Core);
4454 Assert(pvTest == pGblMod); NOREF(pvTest);
4455 pGMM->cShareableModules--;
4456
4457 uint32_t i = pGblMod->cRegions;
4458 while (i-- > 0)
4459 {
4460 if (pGblMod->aRegions[i].paidPages)
4461 {
4462 /* We don't doing anything to the pages as they are handled by the
4463 copy-on-write mechanism in PGM. */
4464 RTMemFree(pGblMod->aRegions[i].paidPages);
4465 pGblMod->aRegions[i].paidPages = NULL;
4466 }
4467 }
4468 RTMemFree(pGblMod);
4469}
4470
4471
4472static int gmmR0ShModNewPerVM(PGVM pGVM, RTGCPTR GCBaseAddr, uint32_t cRegions, const VMMDEVSHAREDREGIONDESC *paRegions,
4473 PGMMSHAREDMODULEPERVM *ppRecVM)
4474{
4475 if (pGVM->gmm.s.Stats.cShareableModules >= GMM_MAX_SHARED_PER_VM_MODULES)
4476 return VERR_GMM_TOO_MANY_PER_VM_MODULES;
4477
4478 PGMMSHAREDMODULEPERVM pRecVM;
4479 pRecVM = (PGMMSHAREDMODULEPERVM)RTMemAllocZ(RT_UOFFSETOF_DYN(GMMSHAREDMODULEPERVM, aRegionsGCPtrs[cRegions]));
4480 if (!pRecVM)
4481 return VERR_NO_MEMORY;
4482
4483 pRecVM->Core.Key = GCBaseAddr;
4484 for (uint32_t i = 0; i < cRegions; i++)
4485 pRecVM->aRegionsGCPtrs[i] = paRegions[i].GCRegionAddr;
4486
4487 bool fInsert = RTAvlGCPtrInsert(&pGVM->gmm.s.pSharedModuleTree, &pRecVM->Core);
4488 Assert(fInsert); NOREF(fInsert);
4489 pGVM->gmm.s.Stats.cShareableModules++;
4490
4491 *ppRecVM = pRecVM;
4492 return VINF_SUCCESS;
4493}
4494
4495
4496static void gmmR0ShModDeletePerVM(PGMM pGMM, PGVM pGVM, PGMMSHAREDMODULEPERVM pRecVM, bool fRemove)
4497{
4498 /*
4499 * Free the per-VM module.
4500 */
4501 PGMMSHAREDMODULE pGblMod = pRecVM->pGlobalModule;
4502 pRecVM->pGlobalModule = NULL;
4503
4504 if (fRemove)
4505 {
4506 void *pvTest = RTAvlGCPtrRemove(&pGVM->gmm.s.pSharedModuleTree, pRecVM->Core.Key);
4507 Assert(pvTest == &pRecVM->Core); NOREF(pvTest);
4508 }
4509
4510 RTMemFree(pRecVM);
4511
4512 /*
4513 * Release the global module.
4514 * (In the registration bailout case, it might not be.)
4515 */
4516 if (pGblMod)
4517 {
4518 Assert(pGblMod->cUsers > 0);
4519 pGblMod->cUsers--;
4520 if (pGblMod->cUsers == 0)
4521 gmmR0ShModDeleteGlobal(pGMM, pGblMod);
4522 }
4523}
4524
4525#endif /* VBOX_WITH_PAGE_SHARING */
4526
4527/**
4528 * Registers a new shared module for the VM.
4529 *
4530 * @returns VBox status code.
4531 * @param pGVM The global (ring-0) VM structure.
4532 * @param pVM The cross context VM structure.
4533 * @param idCpu The VCPU id.
4534 * @param enmGuestOS The guest OS type.
4535 * @param pszModuleName The module name.
4536 * @param pszVersion The module version.
4537 * @param GCPtrModBase The module base address.
4538 * @param cbModule The module size.
4539 * @param cRegions The mumber of shared region descriptors.
4540 * @param paRegions Pointer to an array of shared region(s).
4541 * @thread EMT(idCpu)
4542 */
4543GMMR0DECL(int) GMMR0RegisterSharedModule(PGVM pGVM, PVM pVM, VMCPUID idCpu, VBOXOSFAMILY enmGuestOS, char *pszModuleName,
4544 char *pszVersion, RTGCPTR GCPtrModBase, uint32_t cbModule,
4545 uint32_t cRegions, struct VMMDEVSHAREDREGIONDESC const *paRegions)
4546{
4547#ifdef VBOX_WITH_PAGE_SHARING
4548 /*
4549 * Validate input and get the basics.
4550 *
4551 * Note! Turns out the module size does necessarily match the size of the
4552 * regions. (iTunes on XP)
4553 */
4554 PGMM pGMM;
4555 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4556 int rc = GVMMR0ValidateGVMandVMandEMT(pGVM, pVM, idCpu);
4557 if (RT_FAILURE(rc))
4558 return rc;
4559
4560 if (RT_UNLIKELY(cRegions > VMMDEVSHAREDREGIONDESC_MAX))
4561 return VERR_GMM_TOO_MANY_REGIONS;
4562
4563 if (RT_UNLIKELY(cbModule == 0 || cbModule > _1G))
4564 return VERR_GMM_BAD_SHARED_MODULE_SIZE;
4565
4566 uint32_t cbTotal = 0;
4567 for (uint32_t i = 0; i < cRegions; i++)
4568 {
4569 if (RT_UNLIKELY(paRegions[i].cbRegion == 0 || paRegions[i].cbRegion > _1G))
4570 return VERR_GMM_SHARED_MODULE_BAD_REGIONS_SIZE;
4571
4572 cbTotal += paRegions[i].cbRegion;
4573 if (RT_UNLIKELY(cbTotal > _1G))
4574 return VERR_GMM_SHARED_MODULE_BAD_REGIONS_SIZE;
4575 }
4576
4577 AssertPtrReturn(pszModuleName, VERR_INVALID_POINTER);
4578 if (RT_UNLIKELY(!memchr(pszModuleName, '\0', GMM_SHARED_MODULE_MAX_NAME_STRING)))
4579 return VERR_GMM_MODULE_NAME_TOO_LONG;
4580
4581 AssertPtrReturn(pszVersion, VERR_INVALID_POINTER);
4582 if (RT_UNLIKELY(!memchr(pszVersion, '\0', GMM_SHARED_MODULE_MAX_VERSION_STRING)))
4583 return VERR_GMM_MODULE_NAME_TOO_LONG;
4584
4585 uint32_t const uHash = gmmR0ShModCalcHash(pszModuleName, pszVersion);
4586 Log(("GMMR0RegisterSharedModule %s %s base %RGv size %x hash %x\n", pszModuleName, pszVersion, GCPtrModBase, cbModule, uHash));
4587
4588 /*
4589 * Take the semaphore and do some more validations.
4590 */
4591 gmmR0MutexAcquire(pGMM);
4592 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
4593 {
4594 /*
4595 * Check if this module is already locally registered and register
4596 * it if it isn't. The base address is a unique module identifier
4597 * locally.
4598 */
4599 PGMMSHAREDMODULEPERVM pRecVM = (PGMMSHAREDMODULEPERVM)RTAvlGCPtrGet(&pGVM->gmm.s.pSharedModuleTree, GCPtrModBase);
4600 bool fNewModule = pRecVM == NULL;
4601 if (fNewModule)
4602 {
4603 rc = gmmR0ShModNewPerVM(pGVM, GCPtrModBase, cRegions, paRegions, &pRecVM);
4604 if (RT_SUCCESS(rc))
4605 {
4606 /*
4607 * Find a matching global module, register a new one if needed.
4608 */
4609 PGMMSHAREDMODULE pGblMod = gmmR0ShModFindGlobal(pGMM, uHash, cbModule, enmGuestOS, cRegions,
4610 pszModuleName, pszVersion, paRegions);
4611 if (!pGblMod)
4612 {
4613 Assert(fNewModule);
4614 rc = gmmR0ShModNewGlobal(pGMM, uHash, cbModule, enmGuestOS, cRegions,
4615 pszModuleName, pszVersion, paRegions, &pGblMod);
4616 if (RT_SUCCESS(rc))
4617 {
4618 pRecVM->pGlobalModule = pGblMod; /* (One referenced returned by gmmR0ShModNewGlobal.) */
4619 Log(("GMMR0RegisterSharedModule: new module %s %s\n", pszModuleName, pszVersion));
4620 }
4621 else
4622 gmmR0ShModDeletePerVM(pGMM, pGVM, pRecVM, true /*fRemove*/);
4623 }
4624 else
4625 {
4626 Assert(pGblMod->cUsers > 0 && pGblMod->cUsers < UINT32_MAX / 2);
4627 pGblMod->cUsers++;
4628 pRecVM->pGlobalModule = pGblMod;
4629
4630 Log(("GMMR0RegisterSharedModule: new per vm module %s %s, gbl users %d\n", pszModuleName, pszVersion, pGblMod->cUsers));
4631 }
4632 }
4633 }
4634 else
4635 {
4636 /*
4637 * Attempt to re-register an existing module.
4638 */
4639 PGMMSHAREDMODULE pGblMod = gmmR0ShModFindGlobal(pGMM, uHash, cbModule, enmGuestOS, cRegions,
4640 pszModuleName, pszVersion, paRegions);
4641 if (pRecVM->pGlobalModule == pGblMod)
4642 {
4643 Log(("GMMR0RegisterSharedModule: already registered %s %s, gbl users %d\n", pszModuleName, pszVersion, pGblMod->cUsers));
4644 rc = VINF_GMM_SHARED_MODULE_ALREADY_REGISTERED;
4645 }
4646 else
4647 {
4648 /** @todo may have to unregister+register when this happens in case it's caused
4649 * by VBoxService crashing and being restarted... */
4650 Log(("GMMR0RegisterSharedModule: Address clash!\n"
4651 " incoming at %RGvLB%#x %s %s rgns %u\n"
4652 " existing at %RGvLB%#x %s %s rgns %u\n",
4653 GCPtrModBase, cbModule, pszModuleName, pszVersion, cRegions,
4654 pRecVM->Core.Key, pRecVM->pGlobalModule->cbModule, pRecVM->pGlobalModule->szName,
4655 pRecVM->pGlobalModule->szVersion, pRecVM->pGlobalModule->cRegions));
4656 rc = VERR_GMM_SHARED_MODULE_ADDRESS_CLASH;
4657 }
4658 }
4659 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
4660 }
4661 else
4662 rc = VERR_GMM_IS_NOT_SANE;
4663
4664 gmmR0MutexRelease(pGMM);
4665 return rc;
4666#else
4667
4668 NOREF(pGVM); NOREF(pVM); NOREF(idCpu); NOREF(enmGuestOS); NOREF(pszModuleName); NOREF(pszVersion);
4669 NOREF(GCPtrModBase); NOREF(cbModule); NOREF(cRegions); NOREF(paRegions);
4670 return VERR_NOT_IMPLEMENTED;
4671#endif
4672}
4673
4674
4675/**
4676 * VMMR0 request wrapper for GMMR0RegisterSharedModule.
4677 *
4678 * @returns see GMMR0RegisterSharedModule.
4679 * @param pGVM The global (ring-0) VM structure.
4680 * @param pVM The cross context VM structure.
4681 * @param idCpu The VCPU id.
4682 * @param pReq Pointer to the request packet.
4683 */
4684GMMR0DECL(int) GMMR0RegisterSharedModuleReq(PGVM pGVM, PVM pVM, VMCPUID idCpu, PGMMREGISTERSHAREDMODULEREQ pReq)
4685{
4686 /*
4687 * Validate input and pass it on.
4688 */
4689 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
4690 AssertMsgReturn( pReq->Hdr.cbReq >= sizeof(*pReq)
4691 && pReq->Hdr.cbReq == RT_UOFFSETOF_DYN(GMMREGISTERSHAREDMODULEREQ, aRegions[pReq->cRegions]),
4692 ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
4693
4694 /* Pass back return code in the request packet to preserve informational codes. (VMMR3CallR0 chokes on them) */
4695 pReq->rc = GMMR0RegisterSharedModule(pGVM, pVM, idCpu, pReq->enmGuestOS, pReq->szName, pReq->szVersion,
4696 pReq->GCBaseAddr, pReq->cbModule, pReq->cRegions, pReq->aRegions);
4697 return VINF_SUCCESS;
4698}
4699
4700
4701/**
4702 * Unregisters a shared module for the VM
4703 *
4704 * @returns VBox status code.
4705 * @param pGVM The global (ring-0) VM structure.
4706 * @param pVM The cross context VM structure.
4707 * @param idCpu The VCPU id.
4708 * @param pszModuleName The module name.
4709 * @param pszVersion The module version.
4710 * @param GCPtrModBase The module base address.
4711 * @param cbModule The module size.
4712 */
4713GMMR0DECL(int) GMMR0UnregisterSharedModule(PGVM pGVM, PVM pVM, VMCPUID idCpu, char *pszModuleName, char *pszVersion,
4714 RTGCPTR GCPtrModBase, uint32_t cbModule)
4715{
4716#ifdef VBOX_WITH_PAGE_SHARING
4717 /*
4718 * Validate input and get the basics.
4719 */
4720 PGMM pGMM;
4721 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4722 int rc = GVMMR0ValidateGVMandVMandEMT(pGVM, pVM, idCpu);
4723 if (RT_FAILURE(rc))
4724 return rc;
4725
4726 AssertPtrReturn(pszModuleName, VERR_INVALID_POINTER);
4727 AssertPtrReturn(pszVersion, VERR_INVALID_POINTER);
4728 if (RT_UNLIKELY(!memchr(pszModuleName, '\0', GMM_SHARED_MODULE_MAX_NAME_STRING)))
4729 return VERR_GMM_MODULE_NAME_TOO_LONG;
4730 if (RT_UNLIKELY(!memchr(pszVersion, '\0', GMM_SHARED_MODULE_MAX_VERSION_STRING)))
4731 return VERR_GMM_MODULE_NAME_TOO_LONG;
4732
4733 Log(("GMMR0UnregisterSharedModule %s %s base=%RGv size %x\n", pszModuleName, pszVersion, GCPtrModBase, cbModule));
4734
4735 /*
4736 * Take the semaphore and do some more validations.
4737 */
4738 gmmR0MutexAcquire(pGMM);
4739 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
4740 {
4741 /*
4742 * Locate and remove the specified module.
4743 */
4744 PGMMSHAREDMODULEPERVM pRecVM = (PGMMSHAREDMODULEPERVM)RTAvlGCPtrGet(&pGVM->gmm.s.pSharedModuleTree, GCPtrModBase);
4745 if (pRecVM)
4746 {
4747 /** @todo Do we need to do more validations here, like that the
4748 * name + version + cbModule matches? */
4749 NOREF(cbModule);
4750 Assert(pRecVM->pGlobalModule);
4751 gmmR0ShModDeletePerVM(pGMM, pGVM, pRecVM, true /*fRemove*/);
4752 }
4753 else
4754 rc = VERR_GMM_SHARED_MODULE_NOT_FOUND;
4755
4756 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
4757 }
4758 else
4759 rc = VERR_GMM_IS_NOT_SANE;
4760
4761 gmmR0MutexRelease(pGMM);
4762 return rc;
4763#else
4764
4765 NOREF(pGVM); NOREF(pVM); NOREF(idCpu); NOREF(pszModuleName); NOREF(pszVersion); NOREF(GCPtrModBase); NOREF(cbModule);
4766 return VERR_NOT_IMPLEMENTED;
4767#endif
4768}
4769
4770
4771/**
4772 * VMMR0 request wrapper for GMMR0UnregisterSharedModule.
4773 *
4774 * @returns see GMMR0UnregisterSharedModule.
4775 * @param pGVM The global (ring-0) VM structure.
4776 * @param pVM The cross context VM structure.
4777 * @param idCpu The VCPU id.
4778 * @param pReq Pointer to the request packet.
4779 */
4780GMMR0DECL(int) GMMR0UnregisterSharedModuleReq(PGVM pGVM, PVM pVM, VMCPUID idCpu, PGMMUNREGISTERSHAREDMODULEREQ pReq)
4781{
4782 /*
4783 * Validate input and pass it on.
4784 */
4785 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
4786 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
4787
4788 return GMMR0UnregisterSharedModule(pGVM, pVM, idCpu, pReq->szName, pReq->szVersion, pReq->GCBaseAddr, pReq->cbModule);
4789}
4790
4791#ifdef VBOX_WITH_PAGE_SHARING
4792
4793/**
4794 * Increase the use count of a shared page, the page is known to exist and be valid and such.
4795 *
4796 * @param pGMM Pointer to the GMM instance.
4797 * @param pGVM Pointer to the GVM instance.
4798 * @param pPage The page structure.
4799 */
4800DECLINLINE(void) gmmR0UseSharedPage(PGMM pGMM, PGVM pGVM, PGMMPAGE pPage)
4801{
4802 Assert(pGMM->cSharedPages > 0);
4803 Assert(pGMM->cAllocatedPages > 0);
4804
4805 pGMM->cDuplicatePages++;
4806
4807 pPage->Shared.cRefs++;
4808 pGVM->gmm.s.Stats.cSharedPages++;
4809 pGVM->gmm.s.Stats.Allocated.cBasePages++;
4810}
4811
4812
4813/**
4814 * Converts a private page to a shared page, the page is known to exist and be valid and such.
4815 *
4816 * @param pGMM Pointer to the GMM instance.
4817 * @param pGVM Pointer to the GVM instance.
4818 * @param HCPhys Host physical address
4819 * @param idPage The Page ID
4820 * @param pPage The page structure.
4821 * @param pPageDesc Shared page descriptor
4822 */
4823DECLINLINE(void) gmmR0ConvertToSharedPage(PGMM pGMM, PGVM pGVM, RTHCPHYS HCPhys, uint32_t idPage, PGMMPAGE pPage,
4824 PGMMSHAREDPAGEDESC pPageDesc)
4825{
4826 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
4827 Assert(pChunk);
4828 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
4829 Assert(GMM_PAGE_IS_PRIVATE(pPage));
4830
4831 pChunk->cPrivate--;
4832 pChunk->cShared++;
4833
4834 pGMM->cSharedPages++;
4835
4836 pGVM->gmm.s.Stats.cSharedPages++;
4837 pGVM->gmm.s.Stats.cPrivatePages--;
4838
4839 /* Modify the page structure. */
4840 pPage->Shared.pfn = (uint32_t)(uint64_t)(HCPhys >> PAGE_SHIFT);
4841 pPage->Shared.cRefs = 1;
4842#ifdef VBOX_STRICT
4843 pPageDesc->u32StrictChecksum = gmmR0StrictPageChecksum(pGMM, pGVM, idPage);
4844 pPage->Shared.u14Checksum = pPageDesc->u32StrictChecksum;
4845#else
4846 NOREF(pPageDesc);
4847 pPage->Shared.u14Checksum = 0;
4848#endif
4849 pPage->Shared.u2State = GMM_PAGE_STATE_SHARED;
4850}
4851
4852
4853static int gmmR0SharedModuleCheckPageFirstTime(PGMM pGMM, PGVM pGVM, PGMMSHAREDMODULE pModule,
4854 unsigned idxRegion, unsigned idxPage,
4855 PGMMSHAREDPAGEDESC pPageDesc, PGMMSHAREDREGIONDESC pGlobalRegion)
4856{
4857 NOREF(pModule);
4858
4859 /* Easy case: just change the internal page type. */
4860 PGMMPAGE pPage = gmmR0GetPage(pGMM, pPageDesc->idPage);
4861 AssertMsgReturn(pPage, ("idPage=%#x (GCPhys=%RGp HCPhys=%RHp idxRegion=%#x idxPage=%#x) #1\n",
4862 pPageDesc->idPage, pPageDesc->GCPhys, pPageDesc->HCPhys, idxRegion, idxPage),
4863 VERR_PGM_PHYS_INVALID_PAGE_ID);
4864 NOREF(idxRegion);
4865
4866 AssertMsg(pPageDesc->GCPhys == (pPage->Private.pfn << 12), ("desc %RGp gmm %RGp\n", pPageDesc->HCPhys, (pPage->Private.pfn << 12)));
4867
4868 gmmR0ConvertToSharedPage(pGMM, pGVM, pPageDesc->HCPhys, pPageDesc->idPage, pPage, pPageDesc);
4869
4870 /* Keep track of these references. */
4871 pGlobalRegion->paidPages[idxPage] = pPageDesc->idPage;
4872
4873 return VINF_SUCCESS;
4874}
4875
4876/**
4877 * Checks specified shared module range for changes
4878 *
4879 * Performs the following tasks:
4880 * - If a shared page is new, then it changes the GMM page type to shared and
4881 * returns it in the pPageDesc descriptor.
4882 * - If a shared page already exists, then it checks if the VM page is
4883 * identical and if so frees the VM page and returns the shared page in
4884 * pPageDesc descriptor.
4885 *
4886 * @remarks ASSUMES the caller has acquired the GMM semaphore!!
4887 *
4888 * @returns VBox status code.
4889 * @param pGVM Pointer to the GVM instance data.
4890 * @param pModule Module description
4891 * @param idxRegion Region index
4892 * @param idxPage Page index
4893 * @param pPageDesc Page descriptor
4894 */
4895GMMR0DECL(int) GMMR0SharedModuleCheckPage(PGVM pGVM, PGMMSHAREDMODULE pModule, uint32_t idxRegion, uint32_t idxPage,
4896 PGMMSHAREDPAGEDESC pPageDesc)
4897{
4898 int rc;
4899 PGMM pGMM;
4900 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4901 pPageDesc->u32StrictChecksum = 0;
4902
4903 AssertMsgReturn(idxRegion < pModule->cRegions,
4904 ("idxRegion=%#x cRegions=%#x %s %s\n", idxRegion, pModule->cRegions, pModule->szName, pModule->szVersion),
4905 VERR_INVALID_PARAMETER);
4906
4907 uint32_t const cPages = pModule->aRegions[idxRegion].cb >> PAGE_SHIFT;
4908 AssertMsgReturn(idxPage < cPages,
4909 ("idxRegion=%#x cRegions=%#x %s %s\n", idxRegion, pModule->cRegions, pModule->szName, pModule->szVersion),
4910 VERR_INVALID_PARAMETER);
4911
4912 LogFlow(("GMMR0SharedModuleCheckRange %s base %RGv region %d idxPage %d\n", pModule->szName, pModule->Core.Key, idxRegion, idxPage));
4913
4914 /*
4915 * First time; create a page descriptor array.
4916 */
4917 PGMMSHAREDREGIONDESC pGlobalRegion = &pModule->aRegions[idxRegion];
4918 if (!pGlobalRegion->paidPages)
4919 {
4920 Log(("Allocate page descriptor array for %d pages\n", cPages));
4921 pGlobalRegion->paidPages = (uint32_t *)RTMemAlloc(cPages * sizeof(pGlobalRegion->paidPages[0]));
4922 AssertReturn(pGlobalRegion->paidPages, VERR_NO_MEMORY);
4923
4924 /* Invalidate all descriptors. */
4925 uint32_t i = cPages;
4926 while (i-- > 0)
4927 pGlobalRegion->paidPages[i] = NIL_GMM_PAGEID;
4928 }
4929
4930 /*
4931 * We've seen this shared page for the first time?
4932 */
4933 if (pGlobalRegion->paidPages[idxPage] == NIL_GMM_PAGEID)
4934 {
4935 Log(("New shared page guest %RGp host %RHp\n", pPageDesc->GCPhys, pPageDesc->HCPhys));
4936 return gmmR0SharedModuleCheckPageFirstTime(pGMM, pGVM, pModule, idxRegion, idxPage, pPageDesc, pGlobalRegion);
4937 }
4938
4939 /*
4940 * We've seen it before...
4941 */
4942 Log(("Replace existing page guest %RGp host %RHp id %#x -> id %#x\n",
4943 pPageDesc->GCPhys, pPageDesc->HCPhys, pPageDesc->idPage, pGlobalRegion->paidPages[idxPage]));
4944 Assert(pPageDesc->idPage != pGlobalRegion->paidPages[idxPage]);
4945
4946 /*
4947 * Get the shared page source.
4948 */
4949 PGMMPAGE pPage = gmmR0GetPage(pGMM, pGlobalRegion->paidPages[idxPage]);
4950 AssertMsgReturn(pPage, ("idPage=%#x (idxRegion=%#x idxPage=%#x) #2\n", pPageDesc->idPage, idxRegion, idxPage),
4951 VERR_PGM_PHYS_INVALID_PAGE_ID);
4952
4953 if (pPage->Common.u2State != GMM_PAGE_STATE_SHARED)
4954 {
4955 /*
4956 * Page was freed at some point; invalidate this entry.
4957 */
4958 /** @todo this isn't really bullet proof. */
4959 Log(("Old shared page was freed -> create a new one\n"));
4960 pGlobalRegion->paidPages[idxPage] = NIL_GMM_PAGEID;
4961 return gmmR0SharedModuleCheckPageFirstTime(pGMM, pGVM, pModule, idxRegion, idxPage, pPageDesc, pGlobalRegion);
4962 }
4963
4964 Log(("Replace existing page guest host %RHp -> %RHp\n", pPageDesc->HCPhys, ((uint64_t)pPage->Shared.pfn) << PAGE_SHIFT));
4965
4966 /*
4967 * Calculate the virtual address of the local page.
4968 */
4969 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, pPageDesc->idPage >> GMM_CHUNKID_SHIFT);
4970 AssertMsgReturn(pChunk, ("idPage=%#x (idxRegion=%#x idxPage=%#x) #4\n", pPageDesc->idPage, idxRegion, idxPage),
4971 VERR_PGM_PHYS_INVALID_PAGE_ID);
4972
4973 uint8_t *pbChunk;
4974 AssertMsgReturn(gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk),
4975 ("idPage=%#x (idxRegion=%#x idxPage=%#x) #3\n", pPageDesc->idPage, idxRegion, idxPage),
4976 VERR_PGM_PHYS_INVALID_PAGE_ID);
4977 uint8_t *pbLocalPage = pbChunk + ((pPageDesc->idPage & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
4978
4979 /*
4980 * Calculate the virtual address of the shared page.
4981 */
4982 pChunk = gmmR0GetChunk(pGMM, pGlobalRegion->paidPages[idxPage] >> GMM_CHUNKID_SHIFT);
4983 Assert(pChunk); /* can't fail as gmmR0GetPage succeeded. */
4984
4985 /*
4986 * Get the virtual address of the physical page; map the chunk into the VM
4987 * process if not already done.
4988 */
4989 if (!gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
4990 {
4991 Log(("Map chunk into process!\n"));
4992 rc = gmmR0MapChunk(pGMM, pGVM, pChunk, false /*fRelaxedSem*/, (PRTR3PTR)&pbChunk);
4993 AssertRCReturn(rc, rc);
4994 }
4995 uint8_t *pbSharedPage = pbChunk + ((pGlobalRegion->paidPages[idxPage] & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
4996
4997#ifdef VBOX_STRICT
4998 pPageDesc->u32StrictChecksum = RTCrc32(pbSharedPage, PAGE_SIZE);
4999 uint32_t uChecksum = pPageDesc->u32StrictChecksum & UINT32_C(0x00003fff);
5000 AssertMsg(!uChecksum || uChecksum == pPage->Shared.u14Checksum || !pPage->Shared.u14Checksum,
5001 ("%#x vs %#x - idPage=%#x - %s %s\n", uChecksum, pPage->Shared.u14Checksum,
5002 pGlobalRegion->paidPages[idxPage], pModule->szName, pModule->szVersion));
5003#endif
5004
5005 /** @todo write ASMMemComparePage. */
5006 if (memcmp(pbSharedPage, pbLocalPage, PAGE_SIZE))
5007 {
5008 Log(("Unexpected differences found between local and shared page; skip\n"));
5009 /* Signal to the caller that this one hasn't changed. */
5010 pPageDesc->idPage = NIL_GMM_PAGEID;
5011 return VINF_SUCCESS;
5012 }
5013
5014 /*
5015 * Free the old local page.
5016 */
5017 GMMFREEPAGEDESC PageDesc;
5018 PageDesc.idPage = pPageDesc->idPage;
5019 rc = gmmR0FreePages(pGMM, pGVM, 1, &PageDesc, GMMACCOUNT_BASE);
5020 AssertRCReturn(rc, rc);
5021
5022 gmmR0UseSharedPage(pGMM, pGVM, pPage);
5023
5024 /*
5025 * Pass along the new physical address & page id.
5026 */
5027 pPageDesc->HCPhys = ((uint64_t)pPage->Shared.pfn) << PAGE_SHIFT;
5028 pPageDesc->idPage = pGlobalRegion->paidPages[idxPage];
5029
5030 return VINF_SUCCESS;
5031}
5032
5033
5034/**
5035 * RTAvlGCPtrDestroy callback.
5036 *
5037 * @returns 0 or VERR_GMM_INSTANCE.
5038 * @param pNode The node to destroy.
5039 * @param pvArgs Pointer to an argument packet.
5040 */
5041static DECLCALLBACK(int) gmmR0CleanupSharedModule(PAVLGCPTRNODECORE pNode, void *pvArgs)
5042{
5043 gmmR0ShModDeletePerVM(((GMMR0SHMODPERVMDTORARGS *)pvArgs)->pGMM,
5044 ((GMMR0SHMODPERVMDTORARGS *)pvArgs)->pGVM,
5045 (PGMMSHAREDMODULEPERVM)pNode,
5046 false /*fRemove*/);
5047 return VINF_SUCCESS;
5048}
5049
5050
5051/**
5052 * Used by GMMR0CleanupVM to clean up shared modules.
5053 *
5054 * This is called without taking the GMM lock so that it can be yielded as
5055 * needed here.
5056 *
5057 * @param pGMM The GMM handle.
5058 * @param pGVM The global VM handle.
5059 */
5060static void gmmR0SharedModuleCleanup(PGMM pGMM, PGVM pGVM)
5061{
5062 gmmR0MutexAcquire(pGMM);
5063 GMM_CHECK_SANITY_UPON_ENTERING(pGMM);
5064
5065 GMMR0SHMODPERVMDTORARGS Args;
5066 Args.pGVM = pGVM;
5067 Args.pGMM = pGMM;
5068 RTAvlGCPtrDestroy(&pGVM->gmm.s.pSharedModuleTree, gmmR0CleanupSharedModule, &Args);
5069
5070 AssertMsg(pGVM->gmm.s.Stats.cShareableModules == 0, ("%d\n", pGVM->gmm.s.Stats.cShareableModules));
5071 pGVM->gmm.s.Stats.cShareableModules = 0;
5072
5073 gmmR0MutexRelease(pGMM);
5074}
5075
5076#endif /* VBOX_WITH_PAGE_SHARING */
5077
5078/**
5079 * Removes all shared modules for the specified VM
5080 *
5081 * @returns VBox status code.
5082 * @param pGVM The global (ring-0) VM structure.
5083 * @param pVM The cross context VM structure.
5084 * @param idCpu The VCPU id.
5085 */
5086GMMR0DECL(int) GMMR0ResetSharedModules(PGVM pGVM, PVM pVM, VMCPUID idCpu)
5087{
5088#ifdef VBOX_WITH_PAGE_SHARING
5089 /*
5090 * Validate input and get the basics.
5091 */
5092 PGMM pGMM;
5093 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5094 int rc = GVMMR0ValidateGVMandVMandEMT(pGVM, pVM, idCpu);
5095 if (RT_FAILURE(rc))
5096 return rc;
5097
5098 /*
5099 * Take the semaphore and do some more validations.
5100 */
5101 gmmR0MutexAcquire(pGMM);
5102 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
5103 {
5104 Log(("GMMR0ResetSharedModules\n"));
5105 GMMR0SHMODPERVMDTORARGS Args;
5106 Args.pGVM = pGVM;
5107 Args.pGMM = pGMM;
5108 RTAvlGCPtrDestroy(&pGVM->gmm.s.pSharedModuleTree, gmmR0CleanupSharedModule, &Args);
5109 pGVM->gmm.s.Stats.cShareableModules = 0;
5110
5111 rc = VINF_SUCCESS;
5112 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
5113 }
5114 else
5115 rc = VERR_GMM_IS_NOT_SANE;
5116
5117 gmmR0MutexRelease(pGMM);
5118 return rc;
5119#else
5120 RT_NOREF(pGVM, pVM, idCpu);
5121 return VERR_NOT_IMPLEMENTED;
5122#endif
5123}
5124
5125#ifdef VBOX_WITH_PAGE_SHARING
5126
5127/**
5128 * Tree enumeration callback for checking a shared module.
5129 */
5130static DECLCALLBACK(int) gmmR0CheckSharedModule(PAVLGCPTRNODECORE pNode, void *pvUser)
5131{
5132 GMMCHECKSHAREDMODULEINFO *pArgs = (GMMCHECKSHAREDMODULEINFO*)pvUser;
5133 PGMMSHAREDMODULEPERVM pRecVM = (PGMMSHAREDMODULEPERVM)pNode;
5134 PGMMSHAREDMODULE pGblMod = pRecVM->pGlobalModule;
5135
5136 Log(("gmmR0CheckSharedModule: check %s %s base=%RGv size=%x\n",
5137 pGblMod->szName, pGblMod->szVersion, pGblMod->Core.Key, pGblMod->cbModule));
5138
5139 int rc = PGMR0SharedModuleCheck(pArgs->pGVM->pVM, pArgs->pGVM, pArgs->idCpu, pGblMod, pRecVM->aRegionsGCPtrs);
5140 if (RT_FAILURE(rc))
5141 return rc;
5142 return VINF_SUCCESS;
5143}
5144
5145#endif /* VBOX_WITH_PAGE_SHARING */
5146
5147/**
5148 * Check all shared modules for the specified VM.
5149 *
5150 * @returns VBox status code.
5151 * @param pGVM The global (ring-0) VM structure.
5152 * @param pVM The cross context VM structure.
5153 * @param idCpu The calling EMT number.
5154 * @thread EMT(idCpu)
5155 */
5156GMMR0DECL(int) GMMR0CheckSharedModules(PGVM pGVM, PVM pVM, VMCPUID idCpu)
5157{
5158#ifdef VBOX_WITH_PAGE_SHARING
5159 /*
5160 * Validate input and get the basics.
5161 */
5162 PGMM pGMM;
5163 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5164 int rc = GVMMR0ValidateGVMandVMandEMT(pGVM, pVM, idCpu);
5165 if (RT_FAILURE(rc))
5166 return rc;
5167
5168# ifndef DEBUG_sandervl
5169 /*
5170 * Take the semaphore and do some more validations.
5171 */
5172 gmmR0MutexAcquire(pGMM);
5173# endif
5174 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
5175 {
5176 /*
5177 * Walk the tree, checking each module.
5178 */
5179 Log(("GMMR0CheckSharedModules\n"));
5180
5181 GMMCHECKSHAREDMODULEINFO Args;
5182 Args.pGVM = pGVM;
5183 Args.idCpu = idCpu;
5184 rc = RTAvlGCPtrDoWithAll(&pGVM->gmm.s.pSharedModuleTree, true /* fFromLeft */, gmmR0CheckSharedModule, &Args);
5185
5186 Log(("GMMR0CheckSharedModules done (rc=%Rrc)!\n", rc));
5187 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
5188 }
5189 else
5190 rc = VERR_GMM_IS_NOT_SANE;
5191
5192# ifndef DEBUG_sandervl
5193 gmmR0MutexRelease(pGMM);
5194# endif
5195 return rc;
5196#else
5197 RT_NOREF(pGVM, pVM, idCpu);
5198 return VERR_NOT_IMPLEMENTED;
5199#endif
5200}
5201
5202#if defined(VBOX_STRICT) && HC_ARCH_BITS == 64
5203
5204/**
5205 * RTAvlU32DoWithAll callback.
5206 *
5207 * @returns 0
5208 * @param pNode The node to search.
5209 * @param pvUser Pointer to the input argument packet.
5210 */
5211static DECLCALLBACK(int) gmmR0FindDupPageInChunk(PAVLU32NODECORE pNode, void *pvUser)
5212{
5213 PGMMCHUNK pChunk = (PGMMCHUNK)pNode;
5214 GMMFINDDUPPAGEINFO *pArgs = (GMMFINDDUPPAGEINFO *)pvUser;
5215 PGVM pGVM = pArgs->pGVM;
5216 PGMM pGMM = pArgs->pGMM;
5217 uint8_t *pbChunk;
5218
5219 /* Only take chunks not mapped into this VM process; not entirely correct. */
5220 if (!gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
5221 {
5222 int rc = gmmR0MapChunk(pGMM, pGVM, pChunk, false /*fRelaxedSem*/, (PRTR3PTR)&pbChunk);
5223 if (RT_SUCCESS(rc))
5224 {
5225 /*
5226 * Look for duplicate pages
5227 */
5228 unsigned iPage = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
5229 while (iPage-- > 0)
5230 {
5231 if (GMM_PAGE_IS_PRIVATE(&pChunk->aPages[iPage]))
5232 {
5233 uint8_t *pbDestPage = pbChunk + (iPage << PAGE_SHIFT);
5234
5235 if (!memcmp(pArgs->pSourcePage, pbDestPage, PAGE_SIZE))
5236 {
5237 pArgs->fFoundDuplicate = true;
5238 break;
5239 }
5240 }
5241 }
5242 gmmR0UnmapChunk(pGMM, pGVM, pChunk, false /*fRelaxedSem*/);
5243 }
5244 }
5245 return pArgs->fFoundDuplicate; /* (stops search if true) */
5246}
5247
5248
5249/**
5250 * Find a duplicate of the specified page in other active VMs
5251 *
5252 * @returns VBox status code.
5253 * @param pGVM The global (ring-0) VM structure.
5254 * @param pVM The cross context VM structure.
5255 * @param pReq Pointer to the request packet.
5256 */
5257GMMR0DECL(int) GMMR0FindDuplicatePageReq(PGVM pGVM, PVM pVM, PGMMFINDDUPLICATEPAGEREQ pReq)
5258{
5259 /*
5260 * Validate input and pass it on.
5261 */
5262 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
5263 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
5264
5265 PGMM pGMM;
5266 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5267
5268 int rc = GVMMR0ValidateGVMandVM(pGVM, pVM);
5269 if (RT_FAILURE(rc))
5270 return rc;
5271
5272 /*
5273 * Take the semaphore and do some more validations.
5274 */
5275 rc = gmmR0MutexAcquire(pGMM);
5276 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
5277 {
5278 uint8_t *pbChunk;
5279 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, pReq->idPage >> GMM_CHUNKID_SHIFT);
5280 if (pChunk)
5281 {
5282 if (gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
5283 {
5284 uint8_t *pbSourcePage = pbChunk + ((pReq->idPage & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
5285 PGMMPAGE pPage = gmmR0GetPage(pGMM, pReq->idPage);
5286 if (pPage)
5287 {
5288 GMMFINDDUPPAGEINFO Args;
5289 Args.pGVM = pGVM;
5290 Args.pGMM = pGMM;
5291 Args.pSourcePage = pbSourcePage;
5292 Args.fFoundDuplicate = false;
5293 RTAvlU32DoWithAll(&pGMM->pChunks, true /* fFromLeft */, gmmR0FindDupPageInChunk, &Args);
5294
5295 pReq->fDuplicate = Args.fFoundDuplicate;
5296 }
5297 else
5298 {
5299 AssertFailed();
5300 rc = VERR_PGM_PHYS_INVALID_PAGE_ID;
5301 }
5302 }
5303 else
5304 AssertFailed();
5305 }
5306 else
5307 AssertFailed();
5308 }
5309 else
5310 rc = VERR_GMM_IS_NOT_SANE;
5311
5312 gmmR0MutexRelease(pGMM);
5313 return rc;
5314}
5315
5316#endif /* VBOX_STRICT && HC_ARCH_BITS == 64 */
5317
5318
5319/**
5320 * Retrieves the GMM statistics visible to the caller.
5321 *
5322 * @returns VBox status code.
5323 *
5324 * @param pStats Where to put the statistics.
5325 * @param pSession The current session.
5326 * @param pGVM The GVM to obtain statistics for. Optional.
5327 * @param pVM The VM structure corresponding to @a pGVM.
5328 */
5329GMMR0DECL(int) GMMR0QueryStatistics(PGMMSTATS pStats, PSUPDRVSESSION pSession, PGVM pGVM, PVM pVM)
5330{
5331 LogFlow(("GVMMR0QueryStatistics: pStats=%p pSession=%p pGVM=%p pVM=%p\n", pStats, pSession, pGVM, pVM));
5332
5333 /*
5334 * Validate input.
5335 */
5336 AssertPtrReturn(pSession, VERR_INVALID_POINTER);
5337 AssertPtrReturn(pStats, VERR_INVALID_POINTER);
5338 pStats->cMaxPages = 0; /* (crash before taking the mutex...) */
5339
5340 PGMM pGMM;
5341 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5342
5343 /*
5344 * Validate the VM handle, if not NULL, and lock the GMM.
5345 */
5346 int rc;
5347 if (pGVM)
5348 {
5349 rc = GVMMR0ValidateGVMandVM(pGVM, pVM);
5350 if (RT_FAILURE(rc))
5351 return rc;
5352 }
5353
5354 rc = gmmR0MutexAcquire(pGMM);
5355 if (RT_FAILURE(rc))
5356 return rc;
5357
5358 /*
5359 * Copy out the GMM statistics.
5360 */
5361 pStats->cMaxPages = pGMM->cMaxPages;
5362 pStats->cReservedPages = pGMM->cReservedPages;
5363 pStats->cOverCommittedPages = pGMM->cOverCommittedPages;
5364 pStats->cAllocatedPages = pGMM->cAllocatedPages;
5365 pStats->cSharedPages = pGMM->cSharedPages;
5366 pStats->cDuplicatePages = pGMM->cDuplicatePages;
5367 pStats->cLeftBehindSharedPages = pGMM->cLeftBehindSharedPages;
5368 pStats->cBalloonedPages = pGMM->cBalloonedPages;
5369 pStats->cChunks = pGMM->cChunks;
5370 pStats->cFreedChunks = pGMM->cFreedChunks;
5371 pStats->cShareableModules = pGMM->cShareableModules;
5372 RT_ZERO(pStats->au64Reserved);
5373
5374 /*
5375 * Copy out the VM statistics.
5376 */
5377 if (pGVM)
5378 pStats->VMStats = pGVM->gmm.s.Stats;
5379 else
5380 RT_ZERO(pStats->VMStats);
5381
5382 gmmR0MutexRelease(pGMM);
5383 return rc;
5384}
5385
5386
5387/**
5388 * VMMR0 request wrapper for GMMR0QueryStatistics.
5389 *
5390 * @returns see GMMR0QueryStatistics.
5391 * @param pGVM The global (ring-0) VM structure. Optional.
5392 * @param pVM The cross context VM structure. Optional.
5393 * @param pReq Pointer to the request packet.
5394 */
5395GMMR0DECL(int) GMMR0QueryStatisticsReq(PGVM pGVM, PVM pVM, PGMMQUERYSTATISTICSSREQ pReq)
5396{
5397 /*
5398 * Validate input and pass it on.
5399 */
5400 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
5401 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
5402
5403 return GMMR0QueryStatistics(&pReq->Stats, pReq->pSession, pGVM, pVM);
5404}
5405
5406
5407/**
5408 * Resets the specified GMM statistics.
5409 *
5410 * @returns VBox status code.
5411 *
5412 * @param pStats Which statistics to reset, that is, non-zero fields
5413 * indicates which to reset.
5414 * @param pSession The current session.
5415 * @param pGVM The GVM to reset statistics for. Optional.
5416 * @param pVM The VM structure corresponding to @a pGVM.
5417 */
5418GMMR0DECL(int) GMMR0ResetStatistics(PCGMMSTATS pStats, PSUPDRVSESSION pSession, PGVM pGVM, PVM pVM)
5419{
5420 NOREF(pStats); NOREF(pSession); NOREF(pVM); NOREF(pGVM);
5421 /* Currently nothing we can reset at the moment. */
5422 return VINF_SUCCESS;
5423}
5424
5425
5426/**
5427 * VMMR0 request wrapper for GMMR0ResetStatistics.
5428 *
5429 * @returns see GMMR0ResetStatistics.
5430 * @param pGVM The global (ring-0) VM structure. Optional.
5431 * @param pVM The cross context VM structure. Optional.
5432 * @param pReq Pointer to the request packet.
5433 */
5434GMMR0DECL(int) GMMR0ResetStatisticsReq(PGVM pGVM, PVM pVM, PGMMRESETSTATISTICSSREQ pReq)
5435{
5436 /*
5437 * Validate input and pass it on.
5438 */
5439 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
5440 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
5441
5442 return GMMR0ResetStatistics(&pReq->Stats, pReq->pSession, pGVM, pVM);
5443}
5444
Note: See TracBrowser for help on using the repository browser.

© 2024 Oracle Support Privacy / Do Not Sell My Info Terms of Use Trademark Policy Automated Access Etiquette