VBoxManage fails due to Data Execution Prevention in rsaenh.dll

[acoulton]

I am experiencing a continuous and regular intermittent issue with VBoxManage on Vista SP2 where the vboxmanage process is terminated by Data Execution Prevention. The Windows event log shows:

Faulting application VBoxManage.exe, version 0.0.0.0, time stamp 0x50d1d0cf, 
faulting module rsaenh.dll, version 6.0.6002.18005, time stamp 0x49e037ea, 
exception code 0xc0000005, fault offset 0x00001346, 
process id 0x1040, application start time 0x01ce039be4d5df41.

The error occurs on any of the vboxmanage commands, regularly enough that it is not possible to complete the process of setting up a Vagrant virtual machine. However, the virtual machine setup stops at different points on each execution. I have now verified and can reproduce with a simple batch file that runs the vboxmanage commands in a loop - so I believe the issue is in VirtualBox and not in how it is being called.

There are some errors in the VBoxSVC.log (see attached) but from what I've been able to find these are just noise rather than critical? Note that I have uninstalled and reinstalled a number of times and chose not to install the USB drivers this time round in case that was related to the issue. I have the same result with and without the USB driver installed.

Windows refuses to switch off Data Execution Prevention for vboxmanage.

As these errors all happen independently of booting the machine (eg with the machine stopped prior to first boot) I'm not sure the VBox.log file is relevant, nonetheless I've attached it in case.

My machine is:

Windows NT  Version 6.0 Build: 6002 Service Pack 2
Product (0x3): Windows Vista (TM) Home Premium
Edition: HomePremium
BuildString: 6002.18686.x86fre.vistasp2_gdr.120824-0336
Flavor: Multiprocessor Free
Architecture: X86
LCID: 1033

Please let me know if you require any further information.

Currently Virtualbox is essentially not usable on my system - I've therefore flagged this as critical.

[acoulton]

sample VBox.log - note this issue applies to all VMs whether or not booted

[acoulton]

I have established that this behaviour still exists in 4.2.0, but not (so far) in 4.1.24