Segfault on vm start when 3d acceleration is enabled

[Max Fedotov]

Host: Fedora 32 x86_64 VirtualBox 6.1.6 rpmfusion

VirtualBoxVM segfaults on start if 3d acceleration is enabled (Windows guest with VBoxSVGA driver or Linux guest with VMSVGA). I managed to get some backtrace: #0 0x00007f0afb2f7bb7 in ShaderInitLib () at /usr/lib64/virtualbox/VBoxSVGA3D.so #1 0x00007f0b185e8abd in vmsvga3dPowerOn(PDMDEVINSR3*, VGAState*, VGASTATER3*) () at /usr/lib64/virtualbox/VBoxDD.so #2 0x00007f0b185d5ae9 in vmsvgaR3PowerOn(PDMDEVINSR3*) () at /usr/lib64/virtualbox/VBoxDD.so #3 0x00007f0b185603d4 in vgaR3PowerOn(PDMDEVINSR3*) () at /usr/lib64/virtualbox/VBoxDD.so #4 0x00007f0b48d4a387 in PDMR3PowerOn () at /usr/lib64/virtualbox/components/VBoxVMM.so #5 0x00007f0b48d96e68 in vmR3PowerOn(VM*, VMCPU*, void*) () at /usr/lib64/virtualbox/components/VBoxVMM.so #6 0x00007f0b48da0814 in vmmR3EmtRendezvousCommon(VM*, VMCPU*, bool, unsigned int, int (*)(VM*, VMCPU*, void*), void*) () at /usr/lib64/virtualbox/components/VBoxVMM.so #7 0x00007f0b48da441e in VMMR3EmtRendezvousFF () at /usr/lib64/virtualbox/components/VBoxVMM.so #8 0x00007f0b48d9c33d in vmR3EmulationThreadWithId(RTTHREADINT*, UVMCPU*, unsigned int) () at /usr/lib64/virtualbox/components/VBoxVMM.so #9 0x00007f0b6b9d8044 in rtThreadMain () at /usr/lib64/virtualbox/VBoxRT.so #10 0x00007f0b6ba8e66e in rtThreadNativeMain(void*) () at /usr/lib64/virtualbox/VBoxRT.so #11 0x00007f0b6bd7a432 in start_thread () at /app/lib64/libpthread.so.0 #12 0x00007f0b6bca19d3 in clone () at /app/lib64/libc.so.6

VBoxTestOGL shows success

[Max Fedotov]

I tried to get better backtrace using debuginfo packages, but it's not much better:

#0  0x00007f3cc1084bb7 in ShaderInitLib () at /usr/lib64/virtualbox/VBoxSVGA3D.so
#1  0x00007f3cc23a1abd in vmsvga3dPowerOn(PDMDEVINSR3*, VGAState*, VGASTATER3*) () at /usr/lib64/virtualbox/VBoxDD.so
#2  0x00007f3cc238eae9 in vmsvgaR3PowerOn(PDMDEVINSR3*) () at /usr/lib64/virtualbox/VBoxDD.so
#3  0x00007f3cc23193d4 in vgaR3PowerOn(PDMDEVINSR3*) () at /usr/lib64/virtualbox/VBoxDD.so
#4  0x00007f3cfaae4387 in PDMR3PowerOn () at /usr/lib64/virtualbox/components/VBoxVMM.so
#5  0x00007f3cfab30e68 in vmR3PowerOn(VM*, VMCPU*, void*) () at /usr/lib64/virtualbox/components/VBoxVMM.so
#6  0x00007f3cfab3a814 in vmmR3EmtRendezvousCommon(VM*, VMCPU*, bool, unsigned int, int (*)(VM*, VMCPU*, void*), void*) () at /usr/lib64/virtualbox/components/VBoxVMM.so
#7  0x00007f3cfab3b60d in VMMR3EmtRendezvous () at /usr/lib64/virtualbox/components/VBoxVMM.so
#8  0x00007f3cfab3750b in vmR3ReqProcessOne(VMREQ*) () at /usr/lib64/virtualbox/components/VBoxVMM.so
#9  0x00007f3cfab38461 in VMR3ReqProcessU () at /usr/lib64/virtualbox/components/VBoxVMM.so
#10 0x00007f3cfab36282 in vmR3EmulationThreadWithId(RTTHREADINT*, UVMCPU*, unsigned int) () at /usr/lib64/virtualbox/components/VBoxVMM.so
#11 0x00007f3d25765044 in rtThreadMain () at /usr/lib64/virtualbox/VBoxRT.so
#12 0x00007f3d2581b66e in rtThreadNativeMain(void*) () at /usr/lib64/virtualbox/VBoxRT.so
#13 0x00007f3d25b07432 in start_thread (arg=<optimized out>) at pthread_create.c:477
        ret = <optimized out>
        pd = <optimized out>
        unwind_buf = 
              {cancel_jmp_buf = {{jmp_buf = {139899309688576, -1286539181986859283, 139899885833518, 139899885833519, 139899885833664, 139899309688576, 1179092350183225069, 1179661690941159149}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = 0
#14 0x00007f3d25a2e9d3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[Max Fedotov]

Fixed in 6.1.8 it seems, no crash.