From: Red Hat Support [support@redhat.com] Sent: Thursday, February 06, 2014 6:30 PM To: Damir Cizmic Subject: (Waiting on Customer) Case #01035356 (RHEL 6.4 hosts crash when VirtualBox 3.1.8 is used) [ ref:00DA0000000HxWHMA0.500A000000HnJbyIAF:ref ] --------------------------------------- | Case Information | --------------------------------------- https://access.redhat.com/support/cases/01035356 Case Title : RHEL 6.4 hosts crash when VirtualBox 3.1.8 is used Case Number : 01035356 Case Open Date : 2014-02-05 13:29:21 Most recent comment: On 2014-02-06 13:59:37, Singh, Buland commented: "Hi Damir, The crash analysis of vmcore indicates that kernel panicked in g_abExecMemory() function of VirtualBox Linux kernel driver (vboxdrv). Here's an excerpt from the crash analysis. System Information: crash> sys KERNEL: /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.32-358.18.1.el6.x86_64/vmlinux DUMPFILE: vmcore [PARTIAL DUMP] CPUS: 24 DATE: Fri Jan 31 05:01:13 2014 UPTIME: 6 days, 04:55:22 LOAD AVERAGE: 2.63, 2.94, 2.90 TASKS: 1579 NODENAME: epgsim166.seln.ete.ericsson.se RELEASE: 2.6.32-358.18.1.el6.x86_64 VERSION: #1 SMP Fri Aug 2 17:04:38 EDT 2013 MACHINE: x86_64 (3324 Mhz) MEMORY: 96 GB PANIC: "Oops: 0002 [#1] SMP " (check log for details) Kernel Ring Buffer: crash> log [..] BUG: unable to handle kernel paging request at 00007ff90cde5880 IP: [] g_abExecMemory+0x24791/0x180000 [vboxdrv] PGD 70d928067 PUD 182bbfb067 PMD 14a17e5067 PTE 8000000cb9d96007 Oops: 0002 [#1] SMP last sysfs file: /sys/module/ipv6/initstate CPU 13 Modules linked in: bridge tun vboxnetadp(U) vboxnetflt(U) vboxdrv(U) nfs fscache mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf dell_rbu nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc target_core_iblock target_core_file target_core_pscsi target_core_mod configfs 8021q garp stp llc ipv6 ext3 jbd uinput power_meter ses enclosure sg bnx2 dcdbas microcode serio_raw iTCO_wdt iTCO_vendor_support i7core_edac edac_core ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Pid: 5157, comm: VBoxHeadless Not tainted 2.6.32-358.18.1.el6.x86_64 #1 Dell Inc. PowerEdge R610/0F0XJ6 RIP: 0010:[] [] g_abExecMemory+0x24791/0x180000 [vboxdrv] RSP: 0018:ffff881829d99a28 EFLAGS: 00010206 RAX: 0000000000000063 RBX: 00007ff90cde5880 RCX: 00007ff95a383478 RDX: 00000000000000e3 RSI: 0000000000091948 RDI: ffffc9002c3880a8 RBP: ffff881829d99aa8 R08: 00007ff95abc9008 R09: 0000000000000000 R10: 0000000000005752 R11: 0000000000000001 R12: 0000000000000003 R13: ffffc9002c3a2000 R14: ffff880051ed5000 R15: 0000000000000000 FS: 00007ff98ba0e700(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ff90cde5880 CR3: 000000010225d000 CR4: 00000000000027e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process VBoxHeadless (pid: 5157, threadinfo ffff881829d98000, task ffff88182a6fc080) Stack: ffff881829d99aa8 ffffffffa046ba63 ffffc9002c387000 00007ff98b28a000 ffffc9002c387000 010101002c387000 00007ff95a383478 00007ff98b26a478 ffff881829d99b20 00007ff90cde5880 00007ff95abc9000 00007ff90cde5880 Call Trace: [] ? g_abExecMemory+0x29443/0x180000 [vboxdrv] [] g_abExecMemory+0x2b7e4/0x180000 [vboxdrv] [] g_abExecMemory+0x29ab6/0x180000 [vboxdrv] [] g_abExecMemory+0xe6e5/0x180000 [vboxdrv] [] ? invalidate_interrupt1+0xe/0x20 [] ? g_abExecMemory+0x3de82/0x180000 [vboxdrv] [] ? g_abExecMemory+0x3d76d/0x180000 [vboxdrv] [] ? g_abExecMemory+0xc91b/0x180000 [vboxdrv] [] g_abExecMemory+0x79d2/0x180000 [vboxdrv] [] g_abExecMemory+0x3ebc7/0x180000 [vboxdrv] [] g_abExecMemory+0x110fe/0x180000 [vboxdrv] [] supdrvIOCtlFast+0x6a/0x70 [vboxdrv] [] VBoxDrvLinuxIOCtl+0x4c/0x1c0 [vboxdrv] [] ? g_abExecMemory+0x110fe/0x180000 [vboxdrv] [] ? invalidate_interrupt1+0xe/0x20 [] vfs_ioctl+0x22/0xa0 [] ? invalidate_interrupt1+0xe/0x20 [] ? bio_integrity_tag+0x10/0xb0 [] do_vfs_ioctl+0x84/0x580 [] ? unroll_tree_refs+0xed/0x120 [] ? fput+0xe/0x30 [] sys_ioctl+0x81/0xa0 [] ? __audit_syscall_exit+0x265/0x290 [] system_call_fastpath+0x16/0x1b Code: f6 c2 02 0f 85 9c 01 00 00 45 84 ff 0f 85 a7 fe ff ff 80 7d af 00 66 66 66 90 0f 85 99 fe ff ff 84 d2 0f 89 a8 01 00 00 83 c8 20 <88> 03 41 80 08 20 48 8b 55 b0 0f b6 02 89 c2 83 ca 20 80 7d ae RIP [] g_abExecMemory+0x24791/0x180000 [vboxdrv] RSP CR2: 00007ff90cde5880 [..] Backtraces: crash> bt PID: 5157 TASK: ffff88182a6fc080 CPU: 13 COMMAND: "VBoxHeadless" #0 [ffff881829d99610] machine_kexec at ffffffff81035d6b #1 [ffff881829d99670] crash_kexec at ffffffff810c0e22 #2 [ffff881829d99740] oops_end at ffffffff81511c20 #3 [ffff881829d99770] no_context at ffffffff81046c1b #4 [ffff881829d997c0] __bad_area_nosemaphore at ffffffff81046ea5 #5 [ffff881829d99810] bad_area_nosemaphore at ffffffff81046f73 #6 [ffff881829d99820] __do_page_fault at ffffffff810476d1 #7 [ffff881829d99940] do_page_fault at ffffffff81513b6e #8 [ffff881829d99970] page_fault at ffffffff81510f25 [exception RIP: g_abExecMemory+149393] <<<-----[ Exception occurred at g_abExecMemory+149393 ] RIP: ffffffffa0466db1 RSP: ffff881829d99a28 RFLAGS: 00010206 RAX: 0000000000000063 RBX: 00007ff90cde5880 RCX: 00007ff95a383478 RDX: 00000000000000e3 RSI: 0000000000091948 RDI: ffffc9002c3880a8 RBP: ffff881829d99aa8 R8: 00007ff95abc9008 R9: 0000000000000000 R10: 0000000000005752 R11: 0000000000000001 R12: 0000000000000003 R13: ffffc9002c3a2000 R14: ffff880051ed5000 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 RIP: 0000003ba94e0a47 RSP: 00007ff98ba0dca0 RFLAGS: 00000213 RAX: 0000000000000010 RBX: ffffffff8100b072 RCX: 0000003ba94e0a47 RDX: 0000000000000000 RSI: 00000000000056c1 RDI: 000000000000000f RBP: 00007ff98ba0dca0 R8: 000655cbf5d35e73 R9: 0000000001ff3739 R10: 0000000000989680 R11: 0000000000000246 R12: 00007ff98b86e780 R13: 00007ff98b86e000 R14: 00007ff98b853000 R15: 00007ff98b86e000 ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b The BUG() function was trigger while copying the value of %rbx register into lower byte (%al) of the %ax register. crash> dis -l g_abExecMemory+149393 0xffffffffa0466db1 : mov %al,(%rbx) <<< %rbx: 00007ff90cde5880 %rax: 0000|0000|0000|0063 The following is list of all unsigned(U) modules linked into the kernel. crash> mod -t NAME TAINTS vboxnetadp (U) vboxnetflt (U) vboxdrv (U) Details of VirtualBox unsigned(U) kernel modules (vboxdrv). crash> mod |grep -i vboxdrv ffffffffa05cc9e0 vboxdrv 1783385 (not loaded) [CONFIG_KALLSYMS] crash> module ffffffffa05cc9e0|grep -e version version = 0xffff881824de85a0 "3.1.8 (0x00100001)", srcversion = 0xffff88182a623ba0 "61ABD52580FB8A46A7F5C97", Please note that vboxdrv module is not shipped by Red Hat, hence I don't have a debug-info or source package of this module for further analysis. I would suggest you to contact vendor of vboxdrv module and provide this vmcore to them for analysis. If they found any issues from Operating System side then please share their finding with us for further analysis. Let me know if you have any questions. Regards, Buland Kumar Singh Red Hat, Inc." --------------------------------------- Thank you for your latest interaction with Red Hat Global Support Services. We are currently working to resolve your case. Your case has transitioned to "Waiting On Customer" status. This means that the Red Hat associate working on your case needs information or action from you to proceed. To help us resolve your case as quickly as possible, you will need to update your case online at https://access.redhat.com Once you update the case, we can continue working to resolve your issue. If you wish to contact Red Hat, visit https://access.redhat.com to find phone and web contact information relevant to your region and support contract. Thank you, Red Hat Global Support Services --------------------------------------- Have you joined the discussion in Groups yet? Visit http://red.ht/RHgroups to participate in our rapidly growing user community. Supporting success. Exceeding expectations. Red Hat Customer Portal http://red.ht/accessRH Red Hat Support Twitter http://bit.ly/rhsupport Red Hat Support Facebook http://bit.ly/RHsupportFB Red Hat Support Google+ http://bit.ly/RHsupportG If you need immediate assistance, please reference https://access.redhat.com/support/contact/technicalSupport.html --------------------------------------- [ ref:00DA0000000HxWHMA0.500A000000HnJbyIAF:ref ]