3d acceleration with VMSVGA sometimes leads to a crash or a hang of VirtualBox

[Mikhail Kovalev]

When using a Linux VM (debian 9/10) and Windows host with VMSVGA video adapter and 3d acceleration, the VM sometimes might crash or hang. The crash/hang happens in a few minutes after we start a video conferencing app in the browser (with a webcam enabled). With 3d acceleration disabled there is no crash.

When the crash occurs, the error message is "The instruction at .... referenced memory at 0x0000000000000010..."

The steps to reproduce:

1. Start a VM
2. Forward the webcam to the VM (via VBox UI, Devices->Webcams)
3. Open Google Chrome or Chromium browser. We tried various versions, e.g. Chrome 78.0.3904.108, 64-bit and Chromium 90.0.4430.212, 64-bit. The checkbox "Use hardware acceleration when available" needs to be enabled in the browser settings (it is enabled by default).
4. Navigate to zoom.us and login (a free account is enough).
5. Host a meeting with a webcam enabled. The meeting needs to be started in the browser, not in the native linux app. There is no need for others to join the meeting.
6. If it doesn't hang or crash immediately - give it a few minutes to run. And if it doesn't crash now - close the browser (sometimes the crash only happens when closing the browser).

Attached are 2 logs: one with a crash with Chromium 90 and another one with a hang with Chromium 83. The hang has happened when the "accept cookies" window from zoom.us was loading: it turned out to be completely corrupted and the VM hang occurred. The error in the log is VMSVGA related but is different to what I see in case of a crash. In the attached logs Hyper-V is enabled. But this doesn't make a difference and we can reproduce this issues with Hyper-V disabled as well.

This problem was first reported in https://www.virtualbox.org/ticket/19154. I did more tests today trying to reproduce the problem using Nvidia GPU, but couldn't. Today I also couldn't reproduce the problem using Intel GPU and exactly the same setup as I was using previously. So the problem is not 100% reproducible, unfortunately (until today I could reproduce it all the time - ~20-30 times or so). If I see the problem again, I will do tests with NVidia and will update the ticket.