'Assertion Failed' in PGMHandlerPhysicalDeregister

[pv32768]

3d mode in VirtualBox-7.0.4 works a little worse than in wine. I tried to figure out what was wrong and installed Virtualbox with the debug flag. The first start of the virtual machine ended with 'Assertion Failed'. The problem only appears in 3d mode.

Host: Gentoo linux

Guest: Windows 10

Backtrace:

!!Assertion Failed!!
Expression: <none>
Location  : /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/VMM/VMMAll/PGMAllHandler.cpp(570) int PGMHandlerPhysicalDeregister(PVMCC, RTGCPHYS)
Stack     :
00007f0be44cf7d0 VBoxRT.so!RTAssertMsg2V+0x3f7 (rva:0x28e7d0)

Didn't find range starting at 0000000136b26000

Thread 46 "VMSVGA FIFO" received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 0x7f0b504ff6c0 (LWP 613)]
0x00007f0b8bb68403 in PGMHandlerPhysicalDeregister (pVM=0x7f0b6bbde000, GCPhys=5212626944) at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/VMM/VMMAll/PGMAllHandler.cpp:570
570	        AssertMsgFailed(("Didn't find range starting at %RGp\n", GCPhys));
(gdb) bt
#0  0x00007f0b8bb68403 in PGMHandlerPhysicalDeregister(PVMCC, RTGCPHYS) (pVM=0x7f0b6bbde000, GCPhys=5212626944) at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/VMM/VMMAll/PGMAllHandler.cpp:570
#1  0x00007f0b8b839f3a in pdmR3DevHlp_PGMHandlerPhysicalDeregister(PPDMDEVINS, RTGCPHYS) (pDevIns=0x7f0b6b9b4000, GCPhys=5212626944)
    at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/VMM/VMMR3/PDMDevHlp.cpp:918
#2  0x00007f0b50ae8179 in PDMDevHlpPGMHandlerPhysicalDeregister(PPDMDEVINS, RTGCPHYS) (pDevIns=0x7f0b6b9b4000, GCPhys=5212626944)
    at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/include/VBox/vmm/pdmdev.h:9514
#3  0x00007f0b50aea1d9 in vmsvgaR3GboDestroy(PVMSVGAR3STATE, PVMSVGAGBO) (pSvgaR3State=0x7f0b6447f100, pGbo=0x7f0b644801f0)
    at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/Devices/Graphics/DevVGA-SVGA-cmd.cpp:629
#4  0x00007f0b50aeafb8 in vmsvgaR3OTableSetOrGrow(PVMSVGAR3STATE, SVGAOTableType, PPN64, uint32_t, uint32, SVGAMobFormat, bool)
    (pSvgaR3State=0x7f0b6447f100, type=SVGA_OTABLE_SURFACE, baseAddress=1482153, sizeInBytes=16384, validSizeInBytes=8192, ptDepth=SVGA3D_MOBFMT_PTDEPTH64_1, fGrow=true)
    at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/Devices/Graphics/DevVGA-SVGA-cmd.cpp:833
#5  0x00007f0b50af18b2 in vmsvga3dCmdGrowOTable(PVGASTATECC, SVGA3dCmdGrowOTable const*, uint32_t) (pThisCC=0x7f0b6b9b4180, pCmd=0x7f0b65773c68, cbCmd=24)
    at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/Devices/Graphics/DevVGA-SVGA-cmd.cpp:3653
#6  0x00007f0b50afb2b7 in vmsvgaR3Process3dCmd(VGAState*, VGASTATER3*, unsigned int, SVGAFifo3dCmdId, unsigned int, void const*)
    (pThis=0x7f0b6b9b4840, pThisCC=0x7f0b6b9b4180, idDXContext=4294967295, enmCmdId=SVGA_3D_CMD_GROW_OTABLE, cbCmd=24, pvCmd=0x7f0b65773c68)
    at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/Devices/Graphics/DevVGA-SVGA-cmd.cpp:5897
#7  0x00007f0b50ad8f11 in vmsvgaR3CmdBufProcessCommands(PPDMDEVINS, PVGASTATE, PVGASTATECC, uint32_t, void const*, uint32_t, uint32_t*, uint32_t*)
    (pDevIns=0x7f0b6b9b4000, pThis=0x7f0b6b9b4840, pThisCC=0x7f0b6b9b4180, idDXContext=4294967295, pvCommands=0x7f0b65773a70, cbCommands=528, poffNextCmd=0x7f0b504feb18, pu32IrqStatus=0x7f0b504feb1c)
    at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/Devices/Graphics/DevVGA-SVGA.cpp:3805
#8  0x00007f0b50ad9423 in vmsvgaR3CmdBufProcessBuffers(PPDMDEVINS, PVGASTATE, PVGASTATECC, PPDMTHREAD) (pDevIns=0x7f0b6b9b4000, pThis=0x7f0b6b9b4840, pThisCC=0x7f0b6b9b4180, pThread=0x7f0b644b0ee0)
    at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/Devices/Graphics/DevVGA-SVGA.cpp:3909
#9  0x00007f0b50adc420 in vmsvgaR3FifoLoop(PPDMDEVINS, PPDMTHREAD) (pDevIns=0x7f0b6b9b4000, pThread=0x7f0b644b0ee0)
    at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/Devices/Graphics/DevVGA-SVGA.cpp:4759
#10 0x00007f0b8b8844fd in pdmR3ThreadMain(RTTHREAD, void*) (Thread=0x7f0b644b1100, pvUser=0x7f0b644b0ee0) at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/VMM/VMMR3/PDMThread.cpp:790
#11 0x00007f0be4509d6e in rtThreadMain(PRTTHREADINT, RTNATIVETHREAD, char const*) (pThread=0x7f0b644b1100, NativeThread=139686568785600, pszThreadName=0x7f0b644b19e0 "VMSVGA FIFO")
    at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/Runtime/common/misc/thread.cpp:759
#12 0x00007f0be4693b84 in rtThreadNativeMain(void*) (pvArgs=0x7f0b644b1100) at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/Runtime/r3/posix/thread-posix.cpp:430
#13 0x00007f0be48e2d6a in start_thread () at /lib64/libc.so.6
#14 0x00007f0be49620ac in clone3 () at /lib64/libc.so.6
(gdb) c
Continuing.

!!Assertion Failed!!
Expression: RT_SUCCESS_NP(rc)
Location  : /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/Devices/Graphics/DevVGA-SVGA-cmd.cpp(630) void vmsvgaR3GboDestroy(PVMSVGAR3STATE, PVMSVGAGBO)
Stack     :
00007f0be44cf7d0 VBoxRT.so!RTAssertMsg2V+0x3f7 (rva:0x28e7d0)

VERR_PGM_HANDLER_NOT_FOUND (-1607) - Specified access handler was not found.
Qt WARNING: QCursor: Cannot create bitmap cursor; invalid bitmap(s)
Qt WARNING: QCursor: Cannot create bitmap cursor; invalid bitmap(s)

Thread 46 "VMSVGA FIFO" received signal SIGTRAP, Trace/breakpoint trap.
0x00007f0b50aea251 in vmsvgaR3GboDestroy (pSvgaR3State=0x7f0b6447f100, pGbo=0x7f0b644801f0) at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/Devices/Graphics/DevVGA-SVGA-cmd.cpp:630
630	                AssertRC(rc);
(gdb) c
Continuing.

!!Assertion Failed!!
Expression: <none>
Location  : /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/VMM/VMMAll/PGMAllHandler.cpp(570) int PGMHandlerPhysicalDeregister(PVMCC, RTGCPHYS)
Stack     :
00007f0be44cf7d0 VBoxRT.so!RTAssertMsg2V+0x3f7 (rva:0x28e7d0)

Didn't find range starting at 0000000132627000

Thread 46 "VMSVGA FIFO" received signal SIGTRAP, Trace/breakpoint trap.
0x00007f0b8bb68403 in PGMHandlerPhysicalDeregister (pVM=0x7f0b6bbde000, GCPhys=5140279296) at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/VMM/VMMAll/PGMAllHandler.cpp:570
570	        AssertMsgFailed(("Didn't find range starting at %RGp\n", GCPhys));
(gdb) c
Continuing.
Qt WARNING: QCursor: Cannot create bitmap cursor; invalid bitmap(s)

!!Assertion Failed!!
Expression: RT_SUCCESS_NP(rc)
Location  : /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/Devices/Graphics/DevVGA-SVGA-cmd.cpp(630) void vmsvgaR3GboDestroy(PVMSVGAR3STATE, PVMSVGAGBO)
Stack     :
00007f0be44cf7d0 VBoxRT.so!RTAssertMsg2V+0x3f7 (rva:0x28e7d0)

VERR_PGM_HANDLER_NOT_FOUND (-1607) - Specified access handler was not found.
Qt WARNING: QCursor: Cannot create bitmap cursor; invalid bitmap(s)

Thread 46 "VMSVGA FIFO" received signal SIGTRAP, Trace/breakpoint trap.
0x00007f0b50aea251 in vmsvgaR3GboDestroy (pSvgaR3State=0x7f0b6447f100, pGbo=0x7f0b644801f0) at /var/tmp/portage/app-emulation/virtualbox-7.0.4/work/VirtualBox-7.0.4/src/VBox/Devices/Graphics/DevVGA-SVGA-cmd.cpp:630
630	                AssertRC(rc);
(gdb) c

I found and fixed two bugs:

--- a/src/VBox/Devices/Graphics/DevVGA-SVGA-cmd.cpp	2022-11-17 22:12:54.000000000 +0300
+++ b/src/VBox/Devices/Graphics/DevVGA-SVGA-cmd.cpp	2022-11-23 13:42:19.327889043 +0300
@@ -463,6 +463,8 @@
 
     uint32_t const cPPNsPerPage = X86_PAGE_SIZE / (fGCPhys64 ? sizeof(PPN64) : sizeof(PPN));
 
+    RT_ZERO(*pGbo);
+
     pGbo->cbTotal = sizeInBytes;
     pGbo->cTotalPages = (sizeInBytes + X86_PAGE_SIZE - 1) >> X86_PAGE_SHIFT;
 
@@ -631,7 +633,7 @@
             }
         }
         RTMemFree(pGbo->paDescriptors);
-        RT_ZERO(pGbo);
+        RT_ZERO(*pGbo);
     }
 }
 

At least now the debug version starts without exceptions.

[sunlover]

Thanks for this report. This is a known issue. The fix is correct and a similar fix will be included in a future VirtualBox version.