General protection fault in INTNET-RECV

[BTasker]

I've had intermittent issues with my host system hanging, so installed kdump to try and get a coredump/preserve backtraces etc.

Another hang occurred yesterday (the previous two were back in August). Unfortunately I only got a partial coredump, but it looks like the issue was Virtualbox related (I couldn't see any similar tickets).

Possibly Relevant info

• There are three guests using INTNET, one of those also has an interface bridged to a physical NIC.
• Two of the Guests are running Debian (Jessie) the other is running pfsense.
• The VBox.log files for the guests weren't updated around the time of the hang
• After the system rebooted, one of the VMs remained in an aborted state (so perhaps was involved?)

   ~# VBoxManage --version
   4.3.28r100309

Backtrace

      [2224053.164455] BUG: Bad rss-counter state mm:ffff880139ba8080 idx:0 val:139534
      [2224053.164768] BUG: Bad rss-counter state mm:ffff880139ba8080 idx:1 val:6483
      [2224053.167107] BUG: Bad rss-counter state mm:ffff880139ba8080 idx:0 val:139534
      [2224053.167412] BUG: Bad rss-counter state mm:ffff880139ba8080 idx:1 val:6483
      [2224053.173972] general protection fault: 0000 [#1] SMP 
      [2224053.174219] Modules linked in: pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) bnep nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables nls_utf8 ecb nls_cp437 uvcvideo videobuf2_vmalloc videobuf2_memops btusb cdc_ether videobuf2_core usbnet v4l2_common bluetooth vfat 6lowpan_iphc fat videodev r8152 media intel_powerclamp intel_rapl coretemp kvm_intel kvm i915 snd_hda_codec_hdmi iTCO_wdt snd_hda_codec_realtek crc32_pclmul joydev snd_hda_codec_generic ghash_clmulni_intel iTCO_vendor_support rtsx_pci_ms memstick cryptd asus_nb_wmi asus_wmi sparse_keymap rfkill snd_hda_intel snd_hda_controller efi_pstore evdev efivars pcspkr snd_hda_codec snd_hwdep psmouse serio_raw
      [2224053.177766]  drm_kms_helper drm wmi snd_pcm i2c_algo_bit i2c_i801 shpchp snd_timer i2c_core lpc_ich snd iosf_mbi snd_soc_sst_acpi soundcore video processor button ac battery autofs4 ext4 crc16 mbcache jbd2 sg sd_mod crc_t10dif sr_mod crct10dif_generic cdrom ata_generic rtsx_pci_sdmmc mmc_core crct10dif_pclmul crct10dif_common crc32c_intel ata_piix rtsx_pci mfd_core xhci_hcd thermal libata thermal_sys r8169 mii usbcore scsi_mod usb_common
      [2224053.179736] CPU: 1 PID: 15194 Comm: INTNET-RECV Tainted: G           O  3.16.0-4-amd64 #1 Debian 3.16.7-ckt11-1
      [2224053.180149] Hardware name: ASUSTeK COMPUTER INC. X551MA/X551MA, BIOS X551MA.506 04/10/2014
      [2224053.180492] task: ffff880139894ce0 ti: ffff880082fd8000 task.ti: ffff880082fd8000
      [2224053.180800] RIP: 0010:[<ffffffff8105c2d8>]  [<ffffffff8105c2d8>] pgd_free+0x68/0xb0
      [2224053.181139] RSP: 0018:ffff880082fdbb38  EFLAGS: 00010282
      [2224053.181361] RAX: ffffea0001341a08 RBX: ffff880058077000 RCX: dead000000100100
      [2224053.181653] RDX: dead000000200200 RSI: dead000000200200 RDI: ffffffff81a63868
      [2224053.181944] RBP: ffff880139ba8080 R08: ffff880082fd8000 R09: 0000000000000000
      [2224053.182234] R10: 0000000000000004 R11: 0000000000000005 R12: ffff880139ba8080
      [2224053.182527] R13: 0000000000000000 R14: 0000000000000001 R15: ffff880139894ce0
      [2224053.182824] FS:  00007f20f9b81700(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000
      [2224053.183152] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
      [2224053.183392] CR2: 00007f4e8fb61000 CR3: 000000011a60b000 CR4: 00000000001027e0
      [2224053.183681] Stack:
      [2224053.183775]  0000000000000000 ffff88013afcca20 ffffffff810642de 0000000000000000
      [2224053.184117]  ffff88013afcca20 ffff880139ba8080 ffffffff81091040 ffff8800a6273bc0
      [2224053.184455]  ffff880139894ce0 ffff88013fc92f00 ffff880139ba8080 ffffffff8150d921
      [2224053.184794] Call Trace:
      [2224053.184920]  [<ffffffff810642de>] ? __mmdrop+0x1e/0x90
      [2224053.185149]  [<ffffffff81091040>] ? finish_task_switch+0xb0/0xf0
      [2224053.185407]  [<ffffffff8150d921>] ? __schedule+0x2b1/0x710
      [2224053.185645]  [<ffffffff8150d192>] ? schedule_timeout+0x162/0x2a0
      [2224053.185904]  [<ffffffff81072aa0>] ? ftrace_raw_event_tick_stop+0xb0/0xb0
      [2224053.186190]  [<ffffffff810c7a42>] ? ktime_get_ts+0x42/0xe0
      [2224053.186453]  [<ffffffffa07bcd1a>] ? rtR0SemEventLnxWait.isra.2+0x31a/0x3a0 [vboxdrv]
      [2224053.186779]  [<ffffffff810a7a70>] ? prepare_to_wait_event+0xf0/0xf0
      [2224053.187072]  [<ffffffffa07bd6c7>] ? VBoxHost_RTSemFastMutexRelease+0x37/0x50 [vboxdrv]
      [2224053.187424]  [<ffffffffa07be4e5>] ? VBoxHost_RTSpinlockAcquire+0x25/0x30 [vboxdrv]
      [2224053.187753]  [<ffffffffa07acd06>] ? SUPR0ObjAddRefEx+0x106/0x200 [vboxdrv]
      [2224053.188059]  [<ffffffffa07b42d8>] ? supR0SemEventWaitEx+0x78/0xa0 [vboxdrv]
      [2224053.188368]  [<ffffffffa07b252f>] ? supdrvIOCtl+0xf0f/0x2c30 [vboxdrv]
      [2224053.188663]  [<ffffffffa07baa65>] ? rtR0MemAllocEx+0x175/0x240 [vboxdrv]
      [2224053.200641]  [<ffffffffa07ac521>] ? VBoxDrvLinuxIOCtl_4_3_28+0x121/0x220 [vboxdrv]
      [2224053.212701]  [<ffffffff811ba50f>] ? do_vfs_ioctl+0x2cf/0x4b0
      [2224053.224773]  [<ffffffff81233c83>] ? security_file_ioctl+0x13/0x20
      [2224053.236695]  [<ffffffff811ba771>] ? SyS_ioctl+0x81/0xa0
      [2224053.248423]  [<ffffffff815115cd>] ? system_call_fast_compare_end+0x10/0x15
      [2224053.260332] Code: 48 01 d0 48 c1 e8 0c 48 8d 14 c5 00 00 00 00 48 c1 e0 06 48 29 d0 48 ba 00 00 00 00 00 ea ff ff 48 01 d0 48 8b 48 20 48 8b 50 28 <48> 89 51 08 48 89 0a 48 b9 00 01 10 00 00 00 ad de 48 89 48 20 
      [2224053.286128] RIP  [<ffffffff8105c2d8>] pgd_free+0x68/0xb0
      [2224053.298398]  RSP <ffff880082fdbb38>

Coredump

~# crash dump.201512130857 kernel_link
      ...snip...
      This GDB was configured as "x86_64-unknown-linux-gnu"...

	    KERNEL: kernel_link                       
	  DUMPFILE: dump.201512130857  [PARTIAL DUMP]
	      CPUS: 4
	      DATE: Sun Dec 13 08:57:32 2015
	    UPTIME: 25 days, 17:17:18
      LOAD AVERAGE: 0.82, 0.81, 0.89
	    TASKS: 244
	  NODENAME: queeg
	  RELEASE: 3.16.0-4-amd64
	  VERSION: #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24)
	  MACHINE: x86_64  (2163 Mhz)
	    MEMORY: 3.9 GB
	    PANIC: ""
	      PID: 15194
	  COMMAND: "INTNET-RECV"
	      TASK: ffff880139894ce0  [THREAD_INFO: ffff880082fd8000]
	      CPU: 1
	    STATE: TASK_RUNNING (PANIC)

	crash> bt
	PID: 15194  TASK: ffff880139894ce0  CPU: 1   COMMAND: "INTNET-RECV"
	#0 [ffff880082fdb950] machine_kexec at ffffffff8104c0e2
	#1 [ffff880082fdb9a0] crash_kexec at ffffffff810df37a
	#2 [ffff880082fdba60] oops_end at ffffffff810163a8
	#3 [ffff880082fdba80] general_protection at ffffffff81513588
	    [exception RIP: pgd_free+104]
	    RIP: ffffffff8105c2d8  RSP: ffff880082fdbb38  RFLAGS: 00010282
	    RAX: ffffea0001341a08  RBX: ffff880058077000  RCX: dead000000100100
	    RDX: dead000000200200  RSI: dead000000200200  RDI: ffffffff81a63868
	    RBP: ffff880139ba8080   R8: ffff880082fd8000   R9: 0000000000000000
	    R10: 0000000000000004  R11: 0000000000000005  R12: ffff880139ba8080
	    R13: 0000000000000000  R14: 0000000000000001  R15: ffff880139894ce0
	    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
	#4 [ffff880082fdbb48] __mmdrop at ffffffff810642de
	#5 [ffff880082fdbb68] finish_task_switch at ffffffff81091040
	#6 [ffff880082fdbb90] __schedule at ffffffff8150d921
	#7 [ffff880082fdbbe8] schedule_timeout at ffffffff8150d192
	#8 [ffff880082fdbc98] rtR0SemEventLnxWait at ffffffffa07bcd1a [vboxdrv]
	#9 [ffff880082fdbd40] rtstrFormatIPv6 at ffffffffa07e3ee0 [vboxdrv]
	    RIP: 00007f2137506be7  RSP: 00007f20f9b7cab0  RFLAGS: 00000246
	    RAX: 0000000000000010  RBX: ffffffff815115cd  RCX: 0000000000000000
	    RDX: 00007f20f9b80c10  RSI: 00000000c0485687  RDI: 0000000000000007
	    RBP: 0000000000000018   R8: 00007f20f9b80c40   R9: 00007f2137556ad0
	    R10: 0000001819730211  R11: 0000000000000246  R12: 00000000fffffffa
	    R13: 00007f20f9b80d90  R14: 00007f20f9b80c00  R15: 00007f20f9b80c10
	    ORIG_RAX: 0000000000000010  CS: 0033  SS: 002b

The coredump isn't huge (3.9GB) but still too big to attach. Can make it available somewhere if it's helpful.

I don't currently have a way to repro, so if nothing else, a nudge in the right direction would be appreciated. Sorry I don't have more to go on at the moment.

[Frank Mehnert]

Hmm, normally we appreciate core dumps but in this case I hesitate. Your core dump is actually a kernel dump, right? My concern is that it will be hard to find the proper kernel + debug kernel.

[BTasker]

Hi Frank,

Sorry, didn't get a notification of your comment :)

Correct, it's a kernel dump. I can compile a debug kernel image for you, but it'll probably be missing debug symbols for the vbox* modules, which isn't likely to be much help