Guest dropping network connections after a load peak

[Garo]

I'm running Ubuntu 12.04 LTS with Docker containers (0.11.1) with kernel 3.8.0-37-generic started with Vagrant and my network from Host to Guest gets disconnected for several minutes after I do some work on the machine. Due to the nature of this issue I haven't yet been able to further pinpoint the exact requirements how to reproduce this bug, but here are the symptoms:

All network connections gets dropped due to timeouts: This includes ssh from Host to Guest and also simple netcat from Host to Guest.

During this problem I can still use the VB gui to access the terminal and the system does not report any abnormal high load but network connections (eg. ping 8.8.8.8) will report Destination Host Unreachable.

Here's what I do to reproduce:

I run 36 different Docker containers which run multiple instances of mongodb, redis and cassandra with Oracle JVM7. I execute a test suite with node.js which accesses three cassandra containers. The VM is configured to have two CPU cores with cap set to 100%, 4 GB of RAM, Chipset is setto PIIX3, I/O APIC is enabled, PAE/NX is enabled, VT-x/AMD-V is enabled, Nested Paging is enabled. No 2D/3D acceleration.

ping 8.8.8.8 shows normal ~20ms latency until the bug is triggered. This will first result that the ping jumps to around5-10 seconds for a around 10-20 seconds and then ping is stopped. There's a few ping packets which gets back but with latency of 20-120 seconds. After the bug disappears the ping gets back to normal.

vmstat 1 shows that during background operation there's around 5500 context switches per second. Once I start the test suite, which will trigger this bug, the context switches jumps to around 22000 per second and this continues around 10 seconds until the bug is triggered and then the context switches jumps back to the background noise.

Host OS (OS X in this case) shows that the VirtualBox VM uses around 135-170% or more CPU (normal background CPU usage is around 34% as reported via OS X Activity Monitor). Guest uptime shows 1.20 to 1.60 load but top does not show that any single process would be using CPU any more than just 1-3% . Guest background load is around 0.20.

Guest Memory usage: Mem: 4049112k total, 3795800k used, 253312k free, 158940k buffers Swap: 786428k total, 199172k used, 587256k free, 468212k cached

I've tested both the Intel network adapter (e1000) and the Paravirtualized Network (virtio-net). e1000 driver reports "e1000 0000:00:03.0 eth0: Reset adapter." errors in dmesg during the problem. virtio-net does not report anything on dmesg.

I can reproduce this issue every time on my environment, but I'm not sure how I can debug this further. All ideas are welcome.

[Garo]

dmesg, vmstat etc debugging info

[Garo]

VBox.log

[Garo]

It seems that the bug is specific to the NAT network type. I used vagrant with config.vm.network :private_network, ip: "192.168.33.10" option to launch the VM which creates a second adapter with "Host-only Adapter". The traffic thru this adapter is not affected from this bug.

[juan.batovi]

I had the same issue (OS X 10.8 host, Ubuntu 14.04 LTS guest) but resolved it by changing the adapter type to PCNet-FAST III keeping the NAT network type. It seems that the issue is related to the default adapter type set.

I found a similar situation at ​http://serverfault.com/questions/515765/kernel-oops-on-linux-running-in-virtualbox-breaks-some-io-related-functionality

[aeichner]

Please reopen if still relevant with a recent VirtualBox release.