Leftover signal handler in VBoxOGLcrutil.so crashes if no 3D acceleration is enabled -> fixed in 4.2 and later releases after 2 September 2014

[koe]

gnome-session on RHEL7 crashes when it receives a signal. This seems to be caused by a bug in the virtual box guest video driver.

Here are the relevant parts of a strace run:

The VBox OpenGL library is loaded and mapped into memory

open("/lib64/VBoxOGLcrutil.so", O_RDONLY|O_CLOEXEC) = 7
[...]
mmap(NULL, 1885440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 7, 0) = 0x7f6197e9e000
mprotect(0x7f6197f4f000, 1044480, PROT_NONE) = 0
mmap(0x7f619804e000, 61440, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 7, 0xb0000) = 0x7f619804e000
mmap(0x7f619805d000, 54528, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f619805d000
close(7)                                = 0
[...]

It registers several signal handlers (among them sigint at 0x7f61980b56e0)

rt_sigaction(SIGTERM, {0x7f61980b56e0, [TERM], SA_RESTORER|SA_RESTART, 0x7f61a9152a00}, {0x7f61b2377610, [], SA_RESTORER, 0x7f61b12fc130}, 8) = 0
rt_sigaction(SIGINT, {0x7f61980b56e0, [INT], SA_RESTORER|SA_RESTART, 0x7f61a9152a00}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f61a9152a00}, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f61a9152a00}, 8) = 0
[...]
rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER, 0x7f61a9152a00}, {0x7f61980b56e0, [INT], SA_RESTORER|SA_RESTART, 0x7f61a9152a00}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, 0x7f61a9152a00}, {SIG_DFL, [], 0}, 8) = 0
[...]
rt_sigaction(SIGINT, {0x7f61980b56e0, [INT], SA_RESTORER|SA_RESTART, 0x7f61a9152a00}, NULL, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7f61a9152a00}, NULL, 8) = 0
[...]

It fails to initialize since 3D acceleration is not enabled for the VM. It gets unloaded again but does *NOT* unregister the signal handlers first so they stay intact for invalid addresses.

write(2, "OpenGL Warning: Failed to connec"..., 93OpenGL Warning: Failed to connect to host. Make sure 3D acceleration is enabled for this VM.) = 93
write(2, "libGL error: ", 13libGL error: )           = 13
write(2, "failed to load driver: vboxvideo"..., 33failed to load driver: vboxvideo) = 33
write(2, "libGL error: Try again with LIBG"..., 66libGL error: Try again with LIBGL_DEBUG=verbose for more details.) = 66
munmap(0x7f619806b000, 2284776)         = 0
munmap(0x7f6197e9e000, 1885440)         = 0
[...]

The software rendering driver gets loaded. By chance it is in the same address range the VBox OpenGL driver was before. So the signal handlers now point somewhere into this library.

open("/usr/lib64/dri/swrast_dri.so", O_RDONLY|O_CLOEXEC) = 7
[...]
mmap(NULL, 9876608, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 7, 0) = 0x7f619792d000
mprotect(0x7f6197e8e000, 2093056, PROT_NONE) = 0
mmap(0x7f619808d000, 155648, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 7, 0x560000) = 0x7f619808d000
mmap(0x7f61980b3000, 1987712, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61980b3000
close(7)                                = 0
[...]

A SIGINT signal gets triggered (in this case by hand). The signal handler is called, it points into the bss section of the software rendering library which is not executable and a segfault occurs.

--- SIGINT {si_signo=SIGINT, si_code=SI_USER, si_pid=7475, si_uid=0} ---
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x7f61980b56e0} ---
+++ killed by SIGSEGV (core dumped) +++

[Michael Thayer]

Could you give this (4.3.14-compatible) Additions build a try? Thanks.

https://www.virtualbox.org/download/testcase/VBoxGuestAdditions_4.3.15-95813.iso

[koe]

In this version VBoxOGLcrutil.so stays in the process and it does not crash when the signal handler is called. Thanks for the quick fix.

[Michael Thayer]

Thanks for the feedback.

[Frank Mehnert]

Fix is part of VBox 4.3.16.