BSOD 0x3b When starting multiple VM's

[StuartMc]

Below is the debug of the memory dump. Server is Windows 2012 with 128GB Ram and quad E7-4850's.

Microsoft (R) Windows Debugger Version 6.3.9600.17336 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Users\stuadda\Desktop\MEMORY.DMP]
Kernel Bitmap Dump File: Only kernel address space is available


************* Symbol Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       SRV*C:\Windows\symbol_cache*http://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*C:\Windows\symbol_cache*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows 8 Kernel Version 9600 MP (80 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Built by: 9600.17736.amd64fre.winblue_r9.150322-1500
Machine Name:
Kernel base = 0xfffff803`6a085000 PsLoadedModuleList = 0xfffff803`6a35e850
Debug session time: Tue Jun 30 10:10:53.508 2015 (UTC - 4:00)
System Uptime: 0 days 1:23:00.009
Loading Kernel Symbols
...............................................................
................................................................
.................
Loading User Symbols
PEB is paged out (Peb.Ldr = 00007ff6`9e46f018).  Type ".hh dbgerr001" for details
Loading unloaded module list
.....
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 3B, {c000001d, fffff8013088d57c, ffffd00028f34e20, 0}

*** ERROR: Symbol file could not be found.  Defaulted to export symbols for VMMR0.r0 - 
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for VBoxDrv.sys - 
Probably caused by : VMMR0.r0 ( VMMR0!VMXR0Enter+c )

Followup: MachineOwner
---------

20: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c000001d, Exception code that caused the bugcheck
Arg2: fffff8013088d57c, Address of the instruction which caused the bugcheck
Arg3: ffffd00028f34e20, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.

Debugging Details:
------------------


EXCEPTION_CODE: (NTSTATUS) 0xc000001d - {EXCEPTION}  Illegal Instruction  An attempt was made to execute an illegal instruction.

FAULTING_IP: 
VMMR0!VMXR0Enter+c
fffff801`3088d57c 0fc7742410      vmptrld qword ptr [rsp+10h]

CONTEXT:  ffffd00028f34e20 -- (.cxr 0xffffd00028f34e20;r)
rax=00000000bee3c000 rbx=fffff8013096b380 rcx=ffffd0002b207000
rdx=ffffd0002b222000 rsi=ffffd0002b207000 rdi=ffffd0002b222000
rip=fffff8013088d57c rsp=ffffd00028f35858 rbp=ffffd00028f35c00
 r8=fffff8013096b380  r9=ffffd0002b207000 r10=ffff800000000000
r11=fffff8036a1e1470 r12=0000000000000003 r13=0000000000228307
r14=ffffe000eea1c1a0 r15=0000000000228307
iopl=0         nv up ei ng nz na pe nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00010282
VMMR0!VMXR0Enter+0xc:
fffff801`3088d57c 0fc7742410      vmptrld qword ptr [rsp+10h] ss:0018:ffffd000`28f35868=00000000bee3c000
Last set context:
rax=00000000bee3c000 rbx=fffff8013096b380 rcx=ffffd0002b207000
rdx=ffffd0002b222000 rsi=ffffd0002b207000 rdi=ffffd0002b222000
rip=fffff8013088d57c rsp=ffffd00028f35858 rbp=ffffd00028f35c00
 r8=fffff8013096b380  r9=ffffd0002b207000 r10=ffff800000000000
r11=fffff8036a1e1470 r12=0000000000000003 r13=0000000000228307
r14=ffffe000eea1c1a0 r15=0000000000228307
iopl=0         nv up ei ng nz na pe nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00010282
VMMR0!VMXR0Enter+0xc:
fffff801`3088d57c 0fc7742410      vmptrld qword ptr [rsp+10h] ss:0018:ffffd000`28f35868=00000000bee3c000
Resetting default scope

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT

BUGCHECK_STR:  0x3B

PROCESS_NAME:  VirtualBox.exe

CURRENT_IRQL:  2

ANALYSIS_VERSION: 6.3.9600.17336 (debuggers(dbg).150226-1500) amd64fre

LAST_CONTROL_TRANSFER:  from fffff80130885250 to fffff8013088d57c

STACK_TEXT:  
ffffd000`28f35858 fffff801`30885250 : ffffd000`2b222000 00000000`bee3c000 00000487`9ccc98be ffffe400`53fec830 : VMMR0!VMXR0Enter+0xc
ffffd000`28f35860 fffff801`30899ef3 : ffffd000`2b222000 00000000`00000000 00000000`00000000 00000000`00000080 : VMMR0!GVMMR0CreateVMReq+0xe00
ffffd000`28f35890 fffff801`2d0ec70f : 00000000`ffffffff 00000000`000007b4 ffffe400`53fec830 0000001e`0012019f : VMMR0!VMMR0EntryFast+0xf3
ffffd000`28f358f0 fffff801`2d0fb45b : ffffe000`ed0ec900 ffffd000`28f35cc0 00000000`00000000 ffffe000`ed0a8b38 : VBoxDrv+0x170f
ffffd000`28f35920 fffff803`6a4aa27c : ffffe000`eea1c050 00000000`00000000 00000000`00000000 fffff803`00000000 : VBoxDrv!SUPR0GetKernelFeatures+0x230b
ffffd000`28f359c0 fffff803`6a4a9d22 : 00000000`fffffb83 00007ffd`a3bd0000 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0x54c
ffffd000`28f35b60 fffff803`6a1e14b3 : 00000000`00000000 00000000`033ab000 ffffe000`ec605c00 ffffd000`28f35cc0 : nt!NtDeviceIoControlFile+0x56
ffffd000`28f35bd0 00007ffd`b818123a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`0509f568 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffd`b818123a


FOLLOWUP_IP: 
VMMR0!VMXR0Enter+c
fffff801`3088d57c 0fc7742410      vmptrld qword ptr [rsp+10h]

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  VMMR0!VMXR0Enter+c

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: VMMR0

IMAGE_NAME:  VMMR0.r0

DEBUG_FLR_IMAGE_TIMESTAMP:  55070694

STACK_COMMAND:  .cxr 0xffffd00028f34e20 ; kb

BUCKET_ID_FUNC_OFFSET:  c

FAILURE_BUCKET_ID:  0x3B_VMMR0!VMXR0Enter

BUCKET_ID:  0x3B_VMMR0!VMXR0Enter

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:0x3b_vmmr0!vmxr0enter

FAILURE_ID_HASH:  {545b206b-b31b-98cb-02a0-6bcbb5a4472b}

Followup: MachineOwner
---------

[StuartMc]

First Server Log

[StuartMc]

Second Server Log

[StuartMc]

3rd Server Log Causes Crash on Start

[Peggy]

I'm am having the same type problem, although the bugcheck offsets may be a little different. Windows Host Unresponsive when trying to start second VM with VMMR0.r0 ( VMMR0+1447c )

Guest type: Linux

Host type: Windows

Using new version VirtualBox 5.0.8

Host hardware: HP Proliant DL380 Gen9

I have 2 Virtual servers I need to spin up at the same time. They are Oracle Linux images for Peoplesoft applications. Sometimes I can get both running and other times, I get the first one running and the other guest starts to boot but then the host becomes unresponsive and I have to power reset it the Host. I have dump files. Using Windows Debugger with the mini dump file seems to points to VMMR0.r0.

Bugcheck Analysis *

• BugCheck 3B, {c000001d, fffff8009641747c, ffffd0019be86cd0, 0}

• WARNING: Unable to verify timestamp for VMMR0.r0 * ERROR: Module load completed but symbols could not be loaded for VMMR0.r0 * WARNING: Unable to verify timestamp for VBoxDrv.sys * ERROR: Module load completed but symbols could not be loaded for VBoxDrv.sys Probably caused by : VMMR0.r0 ( VMMR0+1447c )

My ticket: ​https://forums.virtualbox.org/viewtopic.php?f=6&t=74436

Has there been any discoveries on your reported bugcheck? How might I get my dump files looked at?

[Peggy]

Windows mini dump file

[Ramshankar Venkataraman]

Is this something that used to work on your HP Proliant server and if so any idea what might have changed to cause this issue?

ATM; we have no real idea what can be causing this. It's as though there is something pulling the rug from under us but we can't tell what/why. Maybe SMM/security software doing funny things with VT-x that's on these HP servers?

[Peggy]

Post by Ramshankar » 17. Nov 2015, 16:10

It might be worth checking if the firmware is up to date on the boxes which have trouble with VirtualBox.

---

Re: VMMR0 BugCheck 3B on Windows 2012 Host starting multiple

Post by danielericcio » 17. Nov 2015, 16:42

I have the exact same problem .

HP Proliant ML350 Gen9 Windows Server 2008 R2 with Service Pack 1 When I try to start the second virtual machine, the server stops with blue screen ( SYSTEM_SERVICE_EXCEPTION - VMMR0.R0 )

---

Re: VMMR0 BugCheck 3B on Windows 2012 Host starting multiple

Post by michaln » 17. Nov 2015, 17:14

Okay, that's the third HP Proliant. What are the odds that this is a generic error that magically only happens on certain Proliants? Probably not very high. Please ask HP for help.

---

Re: VMMR0 BugCheck 3B on Windows 2012 Host starting multiple

Post by Peggy » 18. Nov 2015, 00:22

@danielericcio What processor do you have in your ML350 Gen9? Mine is a Intel Xeon CPU E5-2640 v3

@Ramshankar Yes - we are fully current on all firmware

@michaln What is the possibility of the Intel processor v3 being a contributor, and not necessarily the HP server?

All, I do have a support case opened with HP.

Thanks for input! Peggy

[michaln]

The problem may be caused by VirtualBox incorrectly detecting the number of host CPUs. Anyone affected can compare the number of CPUs reported by Windows itself with the values in VBox.log.

For VirtualBox 4.3, there should be a line similar to "Logical host processors: 40 present, 40 max, 40 online, online mask: 000000ffffffffff".

For VirtualBox 5.x, there should be a line similar to "TM: GIP - u32Mode=3 (Invariant) u32UpdateHz=92 u32UpdateIntervalNS=10754700 enmUseTscDelta=2 (Pratically Zero) fGetGipCpu=0x1 cCpus=16".

If the numbers don't match (VirtualBox vs. host OS), there will be trouble. There seems to be something unusual in how Windows manages processors on these ProLiant servers, even in cases where the total number of logical processors isn't all that big. That is likely why the problem can't be reproduced on similarly configured Oracle servers.

[bird]

Just uploaded a test build which hopefully addresses this issue: https://www.virtualbox.org/download/testcase/VirtualBox-5.1.7-111271-Win.exe There will be a 5.0 testbuild later today as well (https://www.virtualbox.org/wiki/Testbuilds).

[Frank Mehnert]

Fix is part of VBox 5.0.28 and 5.1.8. Please reopen if still relevant.