VBoxUSBMon crashes

[nlopezcasad]

We are experiencing Windows crashes that point to VBoxUSBMon being the driver that causes the kernel panic.

We use ​usbipd, which in turn enables sharing usb devices with Windows Subsystem for Linux (WSL) machines. Usbipd makes use of VBox usb drivers for this.

The USB device we use are JLink JTAG probes from SEGGER.

usbipd is in auto-attach mode. The JLink software that handles these devices sometimes performs firmware updates, which trigger a device reset and re-attach.

We can't right now pinpoint a specific scenario when the crash happens.

Here's the crash analysis provided by WinDBG.

We have the full kernel memory dump available.

Loading Dump File [C:\Users\xxx\Downloads\MEMORY.DMP]

Kernel Bitmap Dump File: Kernel address space is available, User address space may not be available.

Symbol search path is: srv*

Executable search path is:

Windows 10 Kernel Version 19041 MP (8 procs) Free x64

Product: WinNt, suite: TerminalServer SingleUserTS

Edition build lab: 19041.1.amd64fre.vb_release.191206-1406

Kernel base = 0xfffff805'43a00000 PsLoadedModuleList = 0xfffff805'4462a3d0

Debug session time: Thu Sep 21 13:21:39.229 2023 (UTC + 2:00)

System Uptime: 7 days 4:42:35.116

Loading Kernel Symbols

...............................................................

................................................................

................................................................

....................................................

Loading User Symbols

PEB is paged out (Peb.Ldr = 000000e9'6f398018). Type ".hh dbgerr001" for details

Loading unloaded module list

..................................................

For analysis of this file, run !analyze -v

nt!KeBugCheckEx:

fffff805'43dfcc40 48894c2408 mov qword ptr [rsp+8],rcx ss:0018:ffffa689'6f327610=0000000000000018

3: kd> !analyze -v

*******************************************************************************

* *

* Bugcheck Analysis *

* *

*******************************************************************************

REFERENCE_BY_POINTER (18)

Arguments:

Arg1: 0000000000000000, Object type of the object whose reference count is being lowered

Arg2: ffff9689558029d0, Object whose reference count is being lowered

Arg3: 0000000000000010, Reserved

Arg4: ffff96895234f081, Reserved

The reference count of an object is illegal for the current state of the object.

Each time a driver uses a pointer to an object the driver calls a kernel routine

to increment the reference count of the object. When the driver is done with the

pointer the driver calls another kernel routine to decrement the reference count.

Drivers must match calls to the increment and decrement routines. This BugCheck

can occur because an object's reference count goes to zero while there are still

open handles to the object, in which case the fourth parameter indicates the number

of opened handles. It may also occur when the object's reference count drops below zero

whether or not there are open handles to the object, and in that case the fourth parameter

contains the actual value of the pointer references count.

Debugging Details:

------------------

KEY_VALUES_STRING: 1

Key : Analysis.CPU.mSec

Value: 3765

Key : Analysis.Elapsed.mSec

Value: 5969

Key : Analysis.IO.Other.Mb

Value: 9

Key : Analysis.IO.Read.Mb

Value: 12

Key : Analysis.IO.Write.Mb

Value: 36

Key : Analysis.Init.CPU.mSec

Value: 1843

Key : Analysis.Init.Elapsed.mSec

Value: 32639

Key : Analysis.Memory.CommitPeak.Mb

Value: 100

Key : Bugcheck.Code.KiBugCheckData

Value: 0x18

Key : Bugcheck.Code.LegacyAPI

Value: 0x18

Key : Failure.Bucket

Value: 0x18_VBoxUSBMon!ASMAtomicBitClear

Key : Failure.Hash

Value: {3ade888f-df39-202d-9e7b-2930c63fbded}

Key : Hypervisor.Enlightenments.Value

Value: 68669340

Key : Hypervisor.Enlightenments.ValueHex

Value: 417cf9c

Key : Hypervisor.Flags.AnyHypervisorPresent

Value: 1

Key : Hypervisor.Flags.ApicEnlightened

Value: 1

Key : Hypervisor.Flags.ApicVirtualizationAvailable

Value: 0

Key : Hypervisor.Flags.AsyncMemoryHint

Value: 0

Key : Hypervisor.Flags.CoreSchedulerRequested

Value: 0

Key : Hypervisor.Flags.CpuManager

Value: 1

Key : Hypervisor.Flags.DeprecateAutoEoi

Value: 0

Key : Hypervisor.Flags.DynamicCpuDisabled

Value: 1

Key : Hypervisor.Flags.Epf

Value: 0

Key : Hypervisor.Flags.ExtendedProcessorMasks

Value: 1

Key : Hypervisor.Flags.HardwareMbecAvailable

Value: 0

Key : Hypervisor.Flags.MaxBankNumber

Value: 0

Key : Hypervisor.Flags.MemoryZeroingControl

Value: 0

Key : Hypervisor.Flags.NoExtendedRangeFlush

Value: 0

Key : Hypervisor.Flags.NoNonArchCoreSharing

Value: 1

Key : Hypervisor.Flags.Phase0InitDone

Value: 1

Key : Hypervisor.Flags.PowerSchedulerQos

Value: 0

Key : Hypervisor.Flags.RootScheduler

Value: 0

Key : Hypervisor.Flags.SynicAvailable

Value: 1

Key : Hypervisor.Flags.UseQpcBias

Value: 0

Key : Hypervisor.Flags.Value

Value: 4722927

Key : Hypervisor.Flags.ValueHex

Value: 4810ef

Key : Hypervisor.Flags.VpAssistPage

Value: 1

Key : Hypervisor.Flags.VsmAvailable

Value: 1

Key : Hypervisor.RootFlags.AccessStats

Value: 1

Key : Hypervisor.RootFlags.CrashdumpEnlightened

Value: 1

Key : Hypervisor.RootFlags.CreateVirtualProcessor

Value: 1

Key : Hypervisor.RootFlags.DisableHyperthreading

Value: 0

Key : Hypervisor.RootFlags.HostTimelineSync

Value: 1

Key : Hypervisor.RootFlags.HypervisorDebuggingEnabled

Value: 0

Key : Hypervisor.RootFlags.IsHyperV

Value: 1

Key : Hypervisor.RootFlags.LivedumpEnlightened

Value: 1

Key : Hypervisor.RootFlags.MapDeviceInterrupt

Value: 1

Key : Hypervisor.RootFlags.MceEnlightened

Value: 1

Key : Hypervisor.RootFlags.Nested

Value: 0

Key : Hypervisor.RootFlags.StartLogicalProcessor

Value: 1

Key : Hypervisor.RootFlags.Value

Value: 1015

Key : Hypervisor.RootFlags.ValueHex

Value: 3f7

Key : SecureKernel.HalpHvciEnabled

Value: 0

Key : WER.OS.Branch

Value: vb_release

Key : WER.OS.Version

Value: 10.0.19041.1

BUGCHECK_CODE: 18

BUGCHECK_P1: 0

BUGCHECK_P2: ffff9689558029d0

BUGCHECK_P3: 10

BUGCHECK_P4: ffff96895234f081

FILE_IN_CAB: MEMORY.DMP

PROCESS_NAME: usbipd.exe

STACK_TEXT:

ffffa689'6f327608 fffff805'43e1bb39 : 00000000'00000018 00000000'00000000 ffff9689'558029d0 00000000'00000010 : nt!KeBugCheckEx

ffffa689'6f327610 fffff805'617f27ae : ffff9689'2e31f760 fffff805'61816360 00000000'00000000 00000000'00000000 : nt!ObfReferenceObject+0x1fb559

ffffa689'6f327650 fffff805'617f111f : ffff9689'392529f0 ffff9689'56a155b0 ffff9689'56a155b0 ffff9689'2e3fc060 : VBoxUSBMon!ASMAtomicBitClear+0x16ce

ffffa689'6f327690 fffff805'43c10665 : ffff9689'56a155b0 fffff805'43c1052d 00000000'00000000 00000000'00000000 : VBoxUSBMon!ASMAtomicBitClear+0x3f

ffffa689'6f3276f0 fffff805'43fec62f : ffffa689'6f327939 ffff9689'56a155b0 00000000'00000000 00000000'00000000 : nt!IofCallDriver+0x55

ffffa689'6f327730 fffff805'440014b0 : ffff9689'2e8f8f00 00000000'00000001 ffff9689'56a15580 ffff9689'56bc3560 : nt!IopDeleteFile+0x14f

ffffa689'6f3277b0 fffff805'43c205b7 : 00000000'00000000 00000000'00000000 ffffa689'6f327939 ffff9689'56a155b0 : nt!ObpRemoveObjectRoutine+0x80

ffffa689'6f327810 fffff805'44006d19 : ffff9689'56a15580 00000000'00000000 ffffa88a'00000000 ffff9689'56a15580 : nt!ObfDereferenceObjectWithTag+0xc7

ffffa689'6f327850 fffff805'44001a5c : 00000000'000004f4 000000e9'70dbe100 000000e9'70dbe208 ffffffff'00000000 : nt!ObCloseHandleTableEntry+0x6c9

ffffa689'6f327990 fffff805'43e104f5 : ffff9689'5e2c8000 ffff9689'00000001 ffffa689'6f327a80 ffff9689'00000000 : nt!NtClose+0xec

ffffa689'6f327a00 00007ffd'6b92d1f4 : 00000000'00000000 00000000'00000000 00000000'00000000 00000000'00000000 : nt!KiSystemServiceCopyEnd+0x25

000000e9'70dbabc8 00000000'00000000 : 00000000'00000000 00000000'00000000 00000000'00000000 00000000'00000000 : 0x00007ffd'6b92d1f4

SYMBOL_NAME: VBoxUSBMon!ASMAtomicBitClear+16ce

MODULE_NAME: VBoxUSBMon

IMAGE_NAME: VBoxUSBMon.sys

STACK_COMMAND: .cxr; .ecxr ; kb

BUCKET_ID_FUNC_OFFSET: 16ce

FAILURE_BUCKET_ID: 0x18_VBoxUSBMon!ASMAtomicBitClear

OS_VERSION: 10.0.19041.1

BUILDLAB_STR: vb_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

FAILURE_ID_HASH: {3ade888f-df39-202d-9e7b-2930c63fbded}

Followup: MachineOwner

---------

3: kd> lmvm VBoxUSBMon

Browse full module list

start end module name

fffff805'617f0000 fffff805'6182a000 VBoxUSBMon (export symbols) VBoxUSBMon.sys

Loaded symbol image file: VBoxUSBMon.sys

Image path: \SystemRoot\system32\DRIVERS\VBoxUSBMon.sys

Image name: VBoxUSBMon.sys

Browse all global symbols functions data

Timestamp: Wed Jul 12 18:34:34 2023 (64AED61A)

CheckSum: 0003BF48

ImageSize: 0003A000

Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

[Axel Dörfler]

I also have crashes with 7.0.12 when I plug USB drives in. I'll attach a Windows minidump. I guess it'll be the same issue. It's 100% reproducible.

[Axel Dörfler]

Looks like I cannot attach any files here. Well, please note if you are interested in retrieving the dump; I'll gladly send it by mail to anyone interested.