Exposing multiple CPU cores makes the guest slower

[Markus Laker]

I have a 64-bit Ubuntu 9.10 host with a new quad-core Intel CPU and 8GB of memory. I start a 64-bit Ubuntu 9.10 guest. All hardware acceleration options are enabled, and guest extensions are installed. If I expose four CPU cores to the guest, it runs much more sluggishly than if I expose just one core. In particular, the VM takes a long time to respond to keyboard presses and mouse-clicks. For example, if I open a bash prompt in Konsole and hold down a key, characters appear sporadically in groups of eight or ten, rather than smoothly. If I click on an icon or press Alt+Tab, it'll take maybe half a second before anything happens. If I visit a Web site, DNS resolution will take several seconds, whereas, outside the VM, it's instantaneous. If I open a netcat-to-netcat session to another PC, it can take half a second for the remote machine to see any data when I press Enter in the VM (and it takes even longer for the VM to display anything on the screen when I type locally).

In an attempt to pinpoint the problem, I wrote the following simple benchmark, called one-sleep. It should run in under 100ms, but it often takes multiple seconds to run.

 #!/usr/bin/perl

sub doze($) {
    (my $time) = @_;
    select undef, undef, undef, $time;
}

for (1 .. 50) {
    print STDERR '.';
    doze 0.001;
}


msl@main:~/perl$ time ./one-sleep
..................................................
real    0m0.287s
user    0m0.000s
sys     0m0.000s
msl@main:~/perl$ time ./one-sleep
..................................................
real    0m6.391s
user    0m0.000s
sys     0m0.000s
msl@main:~/perl$

While this program runs, whether it runs quickly or slowly, both the guest and the host report that CPU utilisation is negligible.

So is the problem in the select() call (and its implied context switch), or is it in slow screen output? To find out, I wrote many-sleeps, which dozes ten times (rather than one) after printing each dot:

#!/usr/bin/perl

sub doze($) {
    (my $time) = @_;
    select undef, undef, undef, $time;
}

for (1 .. 50) {
    print STDERR '.';
    doze 0.001 for 1 .. 10;
}

This program *sometimes* takes over a minute to run:

msl@main:~/perl$ time ./many-sleeps
..................................................
real    0m3.721s
user    0m0.000s
sys     0m0.200s
msl@main:~/perl$ time ./many-sleeps
..................................................
real    1m4.421s
user    0m0.000s
sys     0m0.000s
msl@main:~/perl$

Despite doing the same amount of screen output as one-sleep, many-sleeps takes much longer to run. Therefore, the problem is in select() and the context switch. This tallies with the evidence that the slowdown is more likely to occur when there's at least one other program running. (In testing, I used Kontact.)

Both programs take a wildly unpredictable time to run, both in KDE 4.3 in Konsole and also in a virtual terminal (RightControl+1). When they're slow, they generally take roughly twenty times as long as when they're fast.

For comparison, here are timings taken in Konsole and KDE 4.3 when I expose only one CPU to the guest VM:

msl@main:~/perl$ time ./one-sleep
..................................................
real    0m0.148s
user    0m0.010s
sys     0m0.000s
msl@main:~/perl$ time ./one-sleep
..................................................
real    0m0.160s
user    0m0.020s
sys     0m0.000s
msl@main:~/perl$ time ./many-sleeps
..................................................
real    0m1.591s
user    0m0.010s
sys     0m0.010s
msl@main:~/perl$ time ./many-sleeps
..................................................
real    0m1.571s
user    0m0.020s
sys     0m0.020s
msl@main:~/perl$

So the single-core case is twice as fast as even the best quad-core case, and the timings don't vary by much.

The same slowdown occurs if I expose just two cores to the guest, rather than four.

A similar slowdown occurs with a 32-bit Ubuntu 9.10 guest; so it's not limited to 64-bit guests.

The same slowdown occurs with xterm on 32-bit Fluxbox (one-sleep takes either about 0.17s or about 9s); so it's not limited to KDE and Konsole.

I'll attach the config of the 64-bit VM and a VirtualBox log file, along with the contents of /proc/cpuinfo on the host.

[Markus Laker]

VirtualBox log, config of VM, and the host's /proc/cpuinfo

[rameym]

I'm experiencing a similar problem with Solaris host and Windows XP guest.

The slowdown is most notable when comparing guest boot-up times (1 core is twice as fast as 2 core and 5x faster than 4 core on average), doing video playback, and general usage (start menu is choppy, guest OS is overall less responsive). The guest cpu cores are at minimal load during this time. Each OS was installed fresh with the extra cores and they can be seen from the task manager.

Have you run your benchmarks with 2 cores? I'm wondering if the performance will be in-between 1 and 4, similar to what I am experiencing.

[Markus Laker]

For me, performance with two cores is roughly the same as with four:

msl@main:~/perl$ time ./one-sleep
..................................................
real    0m6.055s
user    0m0.000s
sys     0m0.000s
msl@main:~/perl$ time ./one-sleep
..................................................
real    0m6.171s
user    0m0.000s
sys     0m0.010s
msl@main:~/perl$ time ./many-sleeps
..................................................
real    1m0.967s
user    0m0.000s
sys     0m0.000s
msl@main:~/perl$ time ./many-sleeps
..................................................
real    1m0.848s
user    0m0.000s
sys     0m0.000s
msl@main:~/perl$ grep -c processor /proc/cpuinfo
2
msl@main:~/perl$

The only way to get the system usably responsive is to drop down to a single core.

It's interesting to hear that the problem also exists with Windows on Solaris, and is not specific to Linux on Linux.

If you want to try my benchmark on Windows on your system, download ActivePerl from <​http://www.activestate.com/activeperl/>. (You'll need to enable scripts in your browser, or the giant `download now' button won't appear. That wasted a few minutes of my time. :-))

[rameym]

The benchmarks ran equally as fast on my Windows boxes (1 to 4 core). Have you looked at this bug report: ​http://www.virtualbox.org/ticket/5501 ? It looks like you're using the same Ubuntu version as reported...and are having a similar issue.

[Markus Laker]

I hadn't seen that, but the workaround described there -- upgrading to kernel 2.6.32 on the host, with no change required on the guest -- has solved my problem. Many thanks for the pointer. I had some fun and games trying to get dkms to compile fglrx under 2.6.32, so I've updated 5501 with some Google fodder and my solution.