Very slow IPv6 performance with bridged networking

[dremon]

On Linux 64-bit host (Ubuntu 11.04) and VirtualBox 4.1.0 the IPv6 traffic on the guest system is extremely slow, up to 10-20 kb/s. Bridged network is used on wired adapter (eth0), IPv6 gets autoconfigured from radvd daemon. This happens on Linux and Windows guests.

[dremon]

A short addition: this problem doesn't occur with virtio-net adapter.

[Jessica Allan]

I have the same problem but only in one direction. Using the virtio-net adapter reverses the direction of the problem from virtual machines receive being choked off to it's outgoing being choked off. Wireshark shows a lot of duplicate acks and retransmissions gathering in waves. Testing outside of virtual machines on the host is fine. Reducing the link speed to 100Mbps from 1Gbps helps a little.

[Kenyon Ralph]

I experienced the same problem with VirtualBox 4.0.10-dfsg-1~bpo60+1 on a Debian squeeze amd64 host and FreeBSD 9.0-BETA3 guest.

[Stephen Kou]

+1 to this bug, I see it happen on an Arch Linux host with VBox 4.1.4 installed, using bridged networking and the vboxnetflt kernel module, with a Windows 2008 R2 guest.

However, this does NOT occur on VBox 4.1.4 running on a Windows 7 host, with a Windows 7 guest. IPv6 works fine on this configuration.

Running an IPv4 socket instead of an IPv6 socket between the two VMs gets normal bandwidth, while running a IPv6 socket between the two gets 10-20KB/s. IPv6 sockets are fast on the windows 7 host/guest, but not the linux/windows one.

[Stephen Kou]

Forgot to add: I see this problem with both virtio-net and a normal intel virtualized adapter.

And I'm trying to open a socket in between the two VMs (from windows host VM to linux host VM), both using a bridged networking configuration. The Windows-host VM can handle IPv6 on both virtio-net and a virtualized adapter without a slowdown.

Replying to atanamir:

+1 to this bug, I see it happen on an Arch Linux host with VBox 4.1.4 installed, using bridged networking and the vboxnetflt kernel module, with a Windows 2008 R2 guest.

However, this does NOT occur on VBox 4.1.4 running on a Windows 7 host, with a Windows 7 guest. IPv6 works fine on this configuration.

Running an IPv4 socket instead of an IPv6 socket between the two VMs gets normal bandwidth, while running a IPv6 socket between the two gets 10-20KB/s. IPv6 sockets are fast on the windows 7 host/guest, but not the linux/windows one.

[Teemu Haapoja]

I can also confirm this bug on Ubuntu 10.04 using Virtualbox 4.1.2-dfsg-1ubuntu1~lucid1~ppa1, with two different hardware platforms and the guest using the same OS as the host. Guest has three interfaces (using Intel PRO/1000 MT Desktop virtual adapter) which are all in bridged mode, one being used for IPv6-only communication with the host and other machines on the local network. Virtual interfaces are attached to dedicated bridge interfaces on the host.

Guest traffic over IPv4 performs well, as does IPv6-over-openvpn using IPv4 transport. Using direct IPv6 (host <-> guest) speeds are stuck around 15-20kb/s. I am also getting a lot of missed segments, duplicate ACKs and out-of-order packets when transmitting more data at a time. Latency between host and guest doesn't seem to be affected.

I also see this bug was reported about 5 months ago, and didn't see any mention of it in the latest changelogs. Any idea when this could possibly get fixed?

[demonbane]

I'm experiencing the same issue. All IPv4 communication seems to be unaffected, while IPv6 is painfully slow. This appears to be true for both guest to host and guest to guest communications.

[automaciej]

I'm experiencing the same symptoms on two Ubuntu 11.10 hosts (x86_64), with VirtualBox installed from virtualbox-ose-4.1.2-dfsg-1ubuntu1. IPv6 traffic is capped to about 20-30kB/s, while IPv4 performance is unaffected.

It's funny how I left a small Subversion checkout operation running yesterday night and it's still running today's morning.

[Agent86]

I've just installed virtualbox-4.1_4.1.14-77440 on Ubuntu 12.04 with a Solaris 11/11 and Openindiana clients. IPv6 over the bridged adapter is still extremely slow. IPv4 works fine.

[imjustmatthew]

I was able to work around this bug by switching the network adapter type in the guest settings to Paravirtualized Network (virtio-net), which then allowed me to achieve near-line speeds over IPv6 again.

[furlongm]

Also seeing the same issue with VirtualBox 4.1.18 on opensuse, running windows or linux guests. Initially thought it was just slow networking, but running tcpdump now I can see it's only ipv6. Can't use virtio-net as I need to PXE boot.

[Phoenix]

I have Ubuntu Linux 12.04 LTS 64bit and pfSense 2.1 (Read: FreeBSD 8.3) 64bit.

IPv6 Performance is dead slow, no matter if I use virtio-net or e1000. IPv4 Performance is fine - both virtio-net and e1000.

Virtualbox is from the Oracle repository: 4.1.20-80170~Ubuntu~precise

I do have an IPv6 enabled network. Google is slow, Yahoo is slow, Youtube is unusable. This is not a major issue - this is considered a blocker, about to change the virtualization environment. No fix within 14 months?!

[robert82h]

Same here, Ubuntu 12.04 host and Windows 7 guest. Until I disabled IPv6 in the VM, several sites were incredibly slow (e.g., google.com). IPv4 sites worked fine.

[Alex]

Same problem for me, is there any progress on this bug?

[normanb46]

Same problem here with Gentoo host running virtualbox 4.2.6. The issue is present with both XP and gentoo guests, including virtio-nic drivers. XP is unusable with IPv6 on LAN websites. The issue is less obvious with a Gentoo guest, especially with the virtio-nic driver. However wireshark shows shows multiple packet less and retries in all cases. Both XP and Gentoo guests are usable to IPv6 internet websites.

A similar IPv6 enabled XP VMWare guest works OK to the same LAN websites, with no errors apparent in wireshark.

[Stephen Veiss]

I'm experiencing this behaviour in the guest->outside world direction. I've found that disabling TCP segmentation offloading inside the guest works around the problem for me.

This worked for both FreeBSD 9.1 and Ubuntu 12.04, using virtio-net in both cases. The host is running Windows 7 x64, and VirtualBox 4.2.6.

Example commands to test this:

freebsd # ifconfig vtnet0 -tso
ubuntu  # ethtool -K eth0 tso off

I had to restart active connections (e.g. my SSH session) to see a difference.

 # TSO enabled
[sveiss@gaim:~]$ time ssh 2001:8b0:33b:7060:a00:27ff:fe58:e3a2 cat /usr/share/dict/words > /dev/null

real	2m10.488s
user	0m0.041s
sys	0m0.040s

 # TSO disabled
[sveiss@gaim:~]$ time ssh 2001:8b0:33b:7060:a00:27ff:fe58:e3a2 cat /usr/share/dict/words > /dev/null

real	0m0.918s
user	0m0.025s
sys	0m0.016s

[Phoenix]

Virtualbox is: 4.2.6-82870~Ubuntu~precise

Throughput with IPv6 is:

• Host -> Guest: 186 Kbits/sec
• Guest -> Host: 323 Mbits/sec

Throughput with IPv4 is:

• Host -> Guest: 360 Mbits/sec
• Guest -> Host: 359 Mbits/sec

It really doesn't matter if I use e100 or virtio. With our without TSO. It is broken and not workaroundable.

When testing, do both ways!

Can somebody please Add 4.2 as the affected release? As I can't modify the ticket... or is a follow-up ticket due?

Regards Philipp

[jaredmauch]

I'm seeing a similar issue with VirtualBox-4.2-4.2.6_82870_fedora18-1.x86_64 with IPv6.

Looking for an idea how to resolve this before I start looking at another solution. IPv6 performance is around 22kb/s.

[bjacke]

a sparse comment to be cc'ed

[Sander Steffann]

Until this is resolved VirtualBox is not an option on my boxes. I can't risk having performance problems with IPv6. I'll keep an eye on this ticket.

[jaredmauch]

I set the following option on the system running virtualbox and this seemed to make it much better ...

ethtool -K eth0 gro off

The controller involved on my system is

Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

[Frank Mehnert]

We think we found and fixed this problem. No need to disable GRO anymore. Could you install and test ​this test build? Thank you!

[Frank Mehnert]

​Here is a test build for Ubuntu 12.04. Feedback is welcome!

[dremon]

I have tested it on latest Ubuntu 13.04 with Windows and Linux guests - the issue seems to be fixed.

[Stephen Veiss]

If you can supply a test build for Windows x64 hosts I'll be happy to test this with my VMs.

[Frank Mehnert]

sveiss, ​here is a Windows test build.

[nryc]

Replying to frank:

​Here is a test build for Ubuntu 12.04. Feedback is welcome!

Problem solved!

[Frank Mehnert]

Thanks for all the feedback!

[Stephen Veiss]

Replying to frank:

sveiss, ​here is a Windows test build.

This hasn't fixed the issue for me -- I still need to disable TSO to get decent performance on outgoing traffic from the VM.

Host is Windows 7 x64, and the About shows Version 4.2.7 r83916.
Guest is FreeBSD 9.1, and I'm using virtio-net.

[Aleksey Ilyushin]

Replying to sveiss:

This hasn't fixed the issue for me -- I still need to disable TSO to get decent performance on outgoing traffic from the VM.

Host is Windows 7 x64, and the About shows Version 4.2.7 r83916.
Guest is FreeBSD 9.1, and I'm using virtio-net.

This looks like a different problem to me. Please supply additional information. How exactly do you disable TSO? With 'ethtool -K ethX tso off' in FreeBSD guest? (I read your previous post I had missed earlier) How do you test performance? To the local host or remote one? Please attach the log file as well.

[Stephen Veiss]

I'm currently testing by downloading a 1M file from the VM over HTTP.

From another machine on the same network, I'm running these commands:

[sveiss@gaim:~]$ host ramona
ramona.brokenbottle.net has IPv6 address 2001:8b0:33b:7060::2


[sveiss@gaim:~]$ ssh ramona sudo ifconfig vtnet0 -tso
[sveiss@gaim:~]$ time curl http://ramona/testfile > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1024k  100 1024k    0     0  2143k      0 --:--:-- --:--:-- --:--:-- 2211k

real	0m0.489s
user	0m0.010s
sys	0m0.019s


[sveiss@gaim:~]$ ssh ramona sudo ifconfig vtnet0 tso
[sveiss@gaim:~]$ time curl http://ramona/testfile > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1024k  100 1024k    0     0   1985      0  0:08:48  0:08:48 --:--:--  2044

real	8m48.128s
user	0m0.049s
sys	0m0.113s

[Stephen Veiss]

Hrm. I've done a few more tests with this.

Running 4.2.6, I can also reproduce this both FreeBSD+virtio-net and Ubuntu+virtio-net. Disabling TSO fixes the performance issue in both Ubuntu and in FreeBSD

With the test build installed, the problem seems to be confined to FreeBSD+virtio-net. I can't reproduce it with Ubuntu.

[Frank Mehnert]

Fix is part of VBox 4.2.8. The FreeBSD+virtio problem might be a different one.

[Phoenix]

Can confirm

Host: Ubuntu Linux 12.04.2 LTS 64bit,
Guest: pfSense 2.1 (FreeBSD 8.3) 64bit,
Net Emu: Intel PRO/1000 MT

host$ iperf -c guest -V -i 10 -t 60 -r
Results are:
[  5]  0.0-60.0 sec  2.10 GBytes   301 Mbits/sec
[  4]  0.0-60.1 sec  2.12 GBytes   303 Mbits/sec

The virtio is still broken I would love to see to hit 1Gbit/sec.

I added more Cores are the Guest is at 100% System load, which actually got me a drop in performance by about 50%. The issue is described here: ​https://forums.virtualbox.org/viewtopic.php?f=7&t=47136

Thanks for the fix! At least VB gets usable again in IPv6 environments, though it has to do better to be production quality - read: wire speed performance.

[Aleksey Ilyushin]

The FreeBSD+virtio problem was addressed by changeset r44915. In fact it had nothing to do with IPv6, IPv4 was affected as well if GSO was enabled on the host.

[Frank Mehnert]

The fix is part of VBox 4.2.12.

[Max D]

Still broken in 4.2.12 I think, I am stuck at 250Mbps between 2 virtio & 500Mbps virtio<->intel.

[aeichner]

Please reopen if still relevant with a recent VirtualBox release.