VERR_INTERNAL_ERROR_3 when loading custom device plugin

[Kaloth]

Hi folks, I use a custom virtual PCI device plugin. Unfortunately, even though I've signed it with a code-signing key from GoDaddy, VirtualBox 4.3.28 refuses to load my DLL. It gives the following error..

00:00:03.302562 VMSetError: F:\tinderbox\win-4.3\src\VBox\VMM\VMMR3\PDMLdr.cpp(307) int __cdecl pdmR3LoadR3U(struct UVM *,const char *,const char *); rc=VERR_INTERNAL_ERROR_3
00:00:03.302569 VMSetError: Unable to load R3 module C:\VMs\custom_device.dll (custom_device): : \Device\HarddiskVolume2\VMs\custom_device.dll
00:00:03.429194 ERROR [COM]: aRC=E_FAIL (0x80004005) aIID={8ab7c520-2442-4b66-8d74-4ff1e195d2b6} aComponent={Console} aText={Unable to load R3 module C:\VMs\custom_device.dll (custom_device): : \Device\HarddiskVolume2\VMs\custom_device.dll (VERR_INTERNAL_ERROR_3)}, preserve=false
00:00:03.475155 Power up failed (vrc=VERR_INTERNAL_ERROR_3, rc=E_FAIL (0X80004005))

I've had a look at the source code for VBox and unfortunately this error is returned in many different situations. To make matters worse the pdmR3LoadR3U function obliterates the original error string which makes it impossible to discover the root cause. All you get is "Unable to load R3 module", which is about as useful as a chocolate teapot.

Given that the same DLL works with older versions of VBox (prior to the hardening update) and that the DllMain is never called, I have to assume that this error is related to signature verification issues. However, every tool I've tried tells me that my DLL is signed correctly and that the certificate chain is OK.

[Klaus Espenlaub]

Yes, this sounds like your DLL is attracting the wrath of the signature verification.

Unfortunately the state of what is possible and what kind of signature is needed isn't documented, mostly due to the effort we had to sink into this project on short notice, exacerbated now by the 5.0 release which caused more overload.

We don't need the DLL binary (at least not now) if you can provide the necessary verbose signature verification information: the output of both "signtool.exe verify /v ..." and "signtool verify /kp /v ..."

[Kaloth]

Hi, thanks for the reply. "signtool verify /kp /v ..." (kernel mode driver verify) and "signtool verify /pa /v ..." (default verify) pass without any issues. However, "signtool verify /v ..." (user mode driver verify) indicates an error.

SignTool Error: A certificate chain processed, but terminated in a root
        certificate which is not trusted by the trust provider.

I've attached the output from these commands. However, it looks as though either my certificate is not installed properly or GoDaddy haven't properly generated the certificate chain.

Therefore, the outstanding action for VBox is to make the error messages due to signing errors more helpful ;)

[Kaloth]

Output from "signtool.exe verify /v <opts>"

[Kaloth]

Ok, this is now back on you guys. I've verified with GoDaddy that the signature and the signing process are correct.

Also if I verify "VBoxoglrenderspu.dll" from the VBox installation folder I get the same error from signtool.exe. This implies that something is wrong with the VBox signature verification code.

I've attached the verification output for "VBoxoglrenderspu.dll".

Thanks, Chris.

[Kaloth]

Output from "signtool.exe verify /v <opts> VBoxoglrenderspu.dll"