Startup of a VM from a saved state fails if the Windows desktop is locked

[Mikhail Kovalev]

If one tries to start a VM while the Windows screen is locked and the startup proceeds until the point where the shared clipboard service is initialized, the startup fails because the OpenClipboard() test fails.

00:00:10.477787 AssertLogRel C:\jenkins\workspace\team-bitbox\VirtualBox-x86-amd64-pipeline\VirtualBox\devel\7.0.8\src\VBox\Main\src-client\HGCM.cpp(1690) int __cdecl HGCMService::LoadState(struct SSMHANDLE *,const struct VMMR3VTABLE *,unsigned int): RT_SUCCESS_NP(vrc)
00:00:10.477807 vrc=VERR_ACCESS_DENIED, VBoxSharedClipboard
00:00:10.477831 SSM: LoadExec failed for 'HGCM' instance #0 (version 3, pass 0xffffffff): VERR_ACCESS_DENIED
00:00:10.477841 SSM: Unit at 0x19454f2f, current position: offUnit=0x4f0 offUnitUser=0x480
00:00:10.477849 VMSetError: C:\jenkins\workspace\team-bitbox\VirtualBox-x86-amd64-pipeline\VirtualBox\devel\7.0.8\src\VBox\VMM\VMMR3\SSM.cpp(8935) int __cdecl ssmR3LoadExecV2(struct VM *,struct SSMHANDLE *); rc=VERR_ACCESS_DENIED
00:00:10.477850 VMSetError: Failed to load unit 'HGCM'

The easiest way to reproduce the problem is by creating a saved state of some VM with latest GAs running (I used Debian 11), launching a simple script which does some sleep and tries to start the VM afterwards, locking Windows (Win+L) and waiting for some time until the script tries to start the VM. E.g., the following python script would do the job:

import subprocess
import time

command = [
    r'C:\Program Files\Oracle\VirtualBox\VirtualBoxVM.exe', "--startvm",  "Test VM"
]
time.sleep(30)
subprocess.Popen(command)

It all boils down to a check that VirtualBox does when initializing the shared clipboard (SharedClipboardWinGetFormats() -> SharedClipboardWinOpen () from src/VBox/GuestHost/SharedClipboard/clipboard-win.cpp).

As far as I know, it is normal for the OpenClipboard() call to fail if Windows is locked and this should not lead to a failed startup of VirtualBox. We tried to patch the function SharedClipboardWinOpen() to return VINF_SUCCESS regardless of whether OpenClipbaord() succeeds or not and in such a case VirtualBox starts up fine and the shared clipboard is working fine too. So it looks like the shared clipboard check at startup is not that important and it should be safe to ignore a failed check at this place.