VRDP crashing VM => Fixed in SVN

[Alexandre Felipe Salgado Figueiredo]

Hi,

I´ve being experiencing problems with VBoxSVC when more than 50 VM´s are started, it locks with 13 to 15 % of system resource utilization, and if I try to stop or save one VM, it keeps locking the files preventing to change configurations/restarting the VM. The only way to unlock the files is to stop all VM´s and reboot the Host (killing VBoxSVC after all VM´s down just messes up everything ). I use this environment to test our multiuser application , and I need an environment with 70-80 VM´s running (heterogeneous , with Windows XP, 2003 and 7 as guests ), memory is not an issue, as with all VM´s running I still have 15Gb free on the host.

The host is a Core i7 4870K , with 64Gb of memory running Windows 2012R2 Enterprise. The vhd/vdi files are stored in a RAID 5 array (pretty fast ).

Also starting on Virtual Box 4.3.6 higher, VRDP doesn´t work as expected, when there are multiple people connected it crashes the VM with this log ( actually this log is from version 4.3.8 ) :

Faulting application name: VirtualBox.exe, version: 4.3.8.0, time stamp: 0x530cd30d Faulting module name: VBoxVRDP.DLL, version: 0.0.0.0, time stamp: 0x530cd502 Exception code: 0xc0000005 Fault offset: 0x000000000001e85a Faulting process id: 0x1044 Faulting application start time: 0x01cf41d9afc6f1b3 Faulting application path: C:\Program Files\Oracle\VirtualBox\VirtualBox.exe Faulting module path: C:\Program Files\Oracle\VirtualBox\ExtensionPacks\Oracle_VM_VirtualBox_Extension_Pack\win.amd64\VBoxVRDP.DLL Report Id: f4b24832-b059-11e3-8924-4c72b97cdd2d

Questions :

1- I´ve researched a lot about number of VM´s limitation on Virtual Box and all I found is that it´s limited to the host resources ( or 128 VM´s ). 2- Is this VBoxSVC problem a configuration issue or a “bug” , i´ve not found anything about this behavior on the community 3- Is this VRDP crash problem ( and also huge slowness on 4.3.8 version) a configuration issue , limitation or bug ? 4- What more information I can supply to help figuring out these issues ?

[sunlover]

Please attach a crash dump for the VRDP crash. See https://www.virtualbox.org/wiki/Core_dump for more info: "User mode crash dumps on Windows Vista and Windows 7" at the end of the page.

Also please attach VBox.log of a crashed VM.

[Alexandre Felipe Salgado Figueiredo]

Crashed VM Log ( in this log maybe there is some detail to help with the VBoxSVC issue )

[Alexandre Felipe Salgado Figueiredo]

There were no core dump when the VM Crash occured ( the only memory dump in this host is one about a faulty driver, not related with Virtual Box) , its just an entry on the Windows Application log , the most information i could get it is what is wrote above. The VBox.log of the crashed VM is attached, as i have rolled back to version 4.2.24 the VRDP crash is not heppening anymore, but the VBoxSVC issue with more than 50 VM´s starting still.

[sunlover]

The attached log is from VBox 4.2.24. I need a log of the VM crash with VBox 4.3.8.

You have to enable crash dumps as described in:

​http://msdn.microsoft.com/en-us/library/bb787181%28VS.85%29.aspx

[Alexandre Felipe Salgado Figueiredo]

Sorry .. because i had to roll back to 4.2.24 so i can use the VM´s without random crashes. I´ve attached the log before i made the roll back . I´ve also enabled the crash dumps as described in the link you sent ( Thank you ! ).

I´ll update to 4.3.8 this night and tomorrow for sure we will have the error , then i´ll retrieve the dump and attach here.

( by the way do you have also, any clues about the VBoxSVC behaviour with more than 50 vm´s running ? Or we should treat this as another topic ? )

[sunlover]

Thanks.

Technically the VBoxSVC issue with 50+ VMs is a different problem and is not related to the VRDP crash.

Please reproduce the 50+ VMs problem with VBox 4.3.8 and attach C:\Users\NAME\.VirtualBox\VBoxSVC.log

[Alexandre Felipe Salgado Figueiredo]

Ok ! This night, after i update to 4.3.8 again, i´ll start all VM´s , and let it running. Send you the logs as soon as these issues are presented.

[Alexandre Felipe Salgado Figueiredo]

Host resources utilization with 71 VM´s started

[Alexandre Felipe Salgado Figueiredo]

VBoxSVC CPU utilization with 71 VM´s running

[Alexandre Felipe Salgado Figueiredo]

VBoxSVC (4.2.24) log before update ( there were more or less 52 VM´s started ) all VM´s were saved. Host rebooted

[Alexandre Felipe Salgado Figueiredo]

VBoxSVC after upgrade to 4.3.8 , 71 VM´s started

[Alexandre Felipe Salgado Figueiredo]

Hi, I´ve just upgraded to 4.3.8 again , 71 VM´s started , the VBoxSVC is "locked" in 13-15% CPU utilization ( some screen shot´s also attached ), the VBoxSVC.log is also attached. I´ve attached also the log of the 4.2.24 version that was also with the VBoxSVC "locked" after i saved all VM´s and restarted the host , there are some error messages in the end.

Anytime now some of the VM´s should crash and i´ll also attach the dump.

Let me know if anything else is needed. Thank you very much for your help !

[sunlover]

Thanks. Please also attach a new 4.3.8 VBoxSVC.log when you will finish the test. That is when you shutdown all VMs.

[Alexandre Felipe Salgado Figueiredo]

Hi, i´ve saved all VM´s and restarted the host, there was a VirtualBox.exe fail message (occurred many, many times , i believe one per VM that was on ) , attached the screen shot. Also the VBoxSVC log is attached.

[Alexandre Felipe Salgado Figueiredo]

Message during reboot

[Alexandre Felipe Salgado Figueiredo]

4.3.8 VBoxSVC after 71 VM´s saved and host being reboot

[Alexandre Felipe Salgado Figueiredo]

One of the VM´s has just failed ( the VRDP problem ) and the dump was created, it´s attached.

[Alexandre Felipe Salgado Figueiredo]

Core dump generated by the VRDP failure in one of the running VM

[Alexandre Felipe Salgado Figueiredo]

I´ve just upgraded to 4.8.10, the VBoxSVC issue remains , when the 51th VM is started it locks on 13-15% CPU utilization.

The VRDP issue has not happened yet; once , and if , it happens, i´ll post the log.

I´ve not seen any references of these issues on the change log of 4.3.10 , but installed anyway.

[sunlover]

Thanks. I'll take a look at the dump.

[Alexandre Felipe Salgado Figueiredo]

I´ve just updated to the new package of 4.3.10 , so far , 63 VM´s running, VBoxSVC with normal behaviour, no VRDP VM´s crash .

When i started the 64th VM the VBoxSVC presented the issue of consuming 13~15% CPU, before that the service was barelly consuming 1%.

I still watching for VRDP crashe.

[Alexandre Felipe Salgado Figueiredo]

Hi, just had another VRDP Crash, dump attached.

[Alexandre Felipe Salgado Figueiredo]

New VRDP Crash Dump

[Alexandre Felipe Salgado Figueiredo]

Anothe VM has just crashed (VRDP) , Dump attached

[sunlover]

Thanks. Please also attach the VBox.log of the crash.

Do you know what the VM was doing at the time of the crash? Did it crash at the time when a RDP client was disconnecting?

I've looked at the first dump, it is not yet clear why it crashed.

[Alexandre Felipe Salgado Figueiredo]

Sure ! I´ve attached the three VBox.log of those three crash dumps, hope this helps.

On all these three crashes we were just internet browsing using Internet Explorer.

On the first crash there were three people connected, one in our local network ( 1Gbps) , another in Argentina ( ~2Mbps ) and other in US ( ~5Mbps ), i can´t tell the exact failure time, when i connected ( i would be the fourth one , connecting through local network ) it was already crashed .

On the last two crashes , there were two people connected ( or connecting , couldn´t tell the exact failure time) but also on different network speeds, one was on the local network ( 1Gbps) , the other is in another in US , connected to our network through a IPSec VPN tunnel ( ~5Mpbs ).

[Alexandre Felipe Salgado Figueiredo]

Hi ! Me again ... another crash, this time, i got all details.

There were only one user, he logged on the Host, to send the Ctrl+Alt+del to the VM ( it´s a 2003R2 SP2 ) , minimized the VM logged off the host ; logged on the VDRP on the VM, used the VM ( Internet Browsing and a Cisco VPN connection ), and when he logged off, the VM crashed .

Attached, the dump and VBox.log of the VM.

Obs.: Would you like me to keep posting the crashes dump ?

[Alexandre Felipe Salgado Figueiredo]

Complementing ... when i said logged off the VRDP, i meant disconnected ( closed the RDP )

[sunlover]

Thanks. So most likely the crash happens when a client disconnects.

I think that no more crash dumps are required.

[Alexandre Felipe Salgado Figueiredo]

Yes, it seems the crash happens when the client disconnect, because no one complaint that the VM crashed while they were using, so i only realize that the VM crashed when i log on the Host.

Now i´ll let you do your "magic" on these, good luck !

[sunlover]

I think that the crash has been fixed. Please install:

https://www.virtualbox.org/download/testcase/Oracle_VM_VirtualBox_Extension_Pack-4.3.11-93103.vbox-extpack

If it still crashes, then please upload a dump and VBox.log of the crash.

[Alexandre Felipe Salgado Figueiredo]

Excellent ! I´ve just stopped all , restarted the host ( just to be sure it´s all clear ), updated the ExtPack and started the VM´s .

They are in use now... I´m watching them closelly, let you know the results.

[Alexandre Felipe Salgado Figueiredo]

Hi, so far no VM´s crashed ( kudos to sunlover ) , we´ve being heavilly using them, so if the issue persisted, for sure one of them would have crashed. We almost can call this VRDP crash issue closed ( let it running for more a couple of days, just to be sure ).

But the VBoxSVC lock issue still ... I forgot about it, and created a new VM, after attaching the VHD, the VBox Manager frozen, i´ve killed it and opened a new one, when i tried to change the VM settings, i got this message >

Failed to open a session for the virtual machine Microsoft2.

The machine 'Microsoft2' is already locked for a session (or being unlocked).

Result Code: VBOX_E_INVALID_OBJECT_STATE (0x80BB0007) Component: Machine Interface: IMachine {480cf695-2d8d-4256-9c7c-cce4184fa048}

I´ve saved all running VM´s, but all VirtualBox.exe still running ( but no program shows on the task manager ), i´ve attached both print screen.

Would you like to keep this ticket also to manager this VBoxSVC issue, or once the VRDP crash is fully confirmed as fixed ( i believe it is) , close this ticket and open a new one ?

[Alexandre Felipe Salgado Figueiredo]

One more detail, i´ve asked the host to reboot, a Vbox Error popup showed up many times ( it seems one per "lost" running VirtualBox.exe process ) , attached the print screen.

No core dumps were generated.

[sunlover]

Thanks for feedback.

Please open a new ticket for the VBoxSVC issue.

[Alexandre Felipe Salgado Figueiredo]

Hi,

We´ve being using all VM´s intensivelly for the last 4 days, no crashes or other issues related to VRDP.

From my point of view, we can call this issue closed !

Thank you very much for your support.

[Frank Mehnert]

Thanks you for the feedback! We will close this ticket once the next VBox maintenance release is released.

[Frank Mehnert]

Fix is part of VBox 4.3.12.