Huge IO-APIC/guest SMP overhead with 32 bits guests

[rafael]

Symptoms are high CPU usage while running e.g. 32 bits Windows XP/2000 guests with IO-APIC enabled.

Current status:

• fixed for AMD-V in VirtualBox 3.0.6 (requires 3.0.6 guest additions to be installed)
• fixed for VT-x, but only if the Intel CPU supports 64 bits (leaves out Core Solo/Duo); fixed in 3.0.12 (requires 3.0.12 guest additions to be installed)
• Solaris/OpenSolaris guests have similar issues, but a kernel fix is underway.

Does not apply to:

• 64 bits guests
• 32 bits Windows Vista or Windows 7

[rafael]

noticed the following, if I disable the IOAPIC and set VCPU to 1, install goes fine and fast ...

[Sander van Leeuwen]

Known problem and mentioned in the manual. 32 bits guests are extremely slow with the IO-APIC enabled on AMD cpus. Will be fixed in a future release.

[mike.mcse]

I can confirm this behavior on 3.0.2 on AMD Phenom, Mandriva 2009.1. I can also confirm that changing the HAL on the guest VM to Advanced Configuration and Power Interface (ACPI) PC, changing the VM properties to disable IO-APIC, single vCPU. The guest then runs quickly and without problems - but of course, single vCPU. Is there any roadmap for a fix?

The exact same VM runs fine with multiple vCPU and IOAPIC on Intel VT-x, what's the problem with AMD-V ?

[Sander van Leeuwen]

It's being worked on. Intel provides an optimization feature for APIC TPR access. AMD does not, so I have fix it myself.

[mike.mcse]

Similar statements also apply to VMs running Standard PC HAL compared with MPS Multiprocessor HAL. Switching to Standard speeds it up very much with the IOAPIC turned off in the VM settings.

Do we have a possible timeline for a fix, either a whole new version of VB3 incorporating AMD-friendly technology, or a patch to 3.0.2 ?

[Sander van Leeuwen]

Fixed in 3.0.6, but only after you've installed the 3.0.6 guest additions.

[mike.mcse]

Thanks for the good work on this bug. I will be testing it as soon as I can get my hands on 3.0.6.

Can I please ask you to regression-test this fix against Windows XP 64-bit Edition on AMD-V [3.0.4 crashes when Nested Paging is on, machine occasionally aborts without warning or logging an error when Nested Paging is off]. Host is 32-bit Mandriva 2009 and 2008.

[George Greer]

Windows 2000 SMP under Ubuntu 9.0.4 64-bit VT-x

[George Greer]

It's not AMD-V only; my VT-x does it too, even with 3.0.6 Guest Additions installed. (See attachment.)

Windows 2000 SMP with Ubuntu 9.0.4 64-bit VT-x.

[Sander van Leeuwen]

mike: please don't mention unrelated problems in this defect. If you still see this xp64 abort with 3.0.6, then open another defect with a proper description and VBox.log.

greerga: your Intel CPU doesn't support the APIC access optimization. I intend to enable the same fix once we get more feedback about AMD-V.

[mike.mcse]

Replying to sandervl73:Point taken about 64-bit, I will keep testing that and raise a new issue for it.

However... my testing of 3.0.6 with XP 32-bit on AMD-V has shown that it's better - and it's worse. It's better - when you choose a guest ACPI Multprocessor HAL or ACPI Uniprocessor HAL, it runs without taking half an hour to boot up and flatlining one or two CPU cores at 100%. Most of the time. However today it went back to behaving like the un-patched 3.0.4 problem. I fixed this by running sudo /etc/init.d/vboxdrv restart . Clearly something in the VirtualBox drivers can revert to the old behavior given some [unknown] circumstances.

If you build your guest with the Advanced Power Configuration Interface (ACPI) HAL [i.e. no IOAPIC, as was required on 3.0.4 to work], it runs, but not as quickly as it used to on 3.0.4. So the fix has crippled non-IOAPIC but has not completely raised the performance level of IOAPIC-based configurations.

The typical performance issue is that the SysInternals Process Explorer is showing that System and Services.exe are together taking 6% or more all the time, even with dual 2.3GHz vCPUs. This is way short of physical machine performance. So the system has a performance draining wait loop somewhere inside it, meaning that activities like Windows Installer take far longer to complete than expected unless you intervene with Process Explorer and raise the process priority to "High".

It's a shame - this so nearly worked. The fact is that 3.0.4 with IOAPIC disabled and one vCPU massively outperforms 3.0.6 with dual vCPUs, which seems counter-intuitive.

Also I have had numerous aborts when Guest Additions 3D Acceleration is enabled.

[ToddAndMargo]

Just getting myself on the Cc: list (couldn't find another way). Just ignore.

[Ralf Jung]

I have exactly the described issue with VirtualBox 3.0.10, 32bit Kubuntu 9.04 host and 32bit Windows XP guest. My processor is a Core 2 Duo E6420 which does support 64bit. However, reading the "current status", I'm not sure what exactly is in SVN only - the fix for Intel processor supporting 64bit or the fix for those which don't.

[Frank Mehnert]

VBox 3.1.0 will contain a fix for Windows guests which works if the guest additions are installed and if the host CPU supports VT-x and 64-bit. I known, it sounds a bit strange that 64-bit support is required to workaround some 32-bit issue :)

[Frank Mehnert]

Well, the current status in the ticket description should be clear.

[Ralf Jung]

Ok, thanks for clarification - I hope for version 3.1 then :)

[fuUser]

Is there any plan/solution you are working on for 32 Bit Hosts? I use a (beloved) Pentium M Notebook with 2 GB Ram which can handled non-APIC VM's with acceptable speed (for office etc) but nearly grinds to hold and suffers huge delays when using an almost exactly configured VM, which was installed (and has to run) with APIC enabled. This causes a rather huge administrative burden for me, because I have to keep two Versions of practically the same VM updated and running. One for my 64 Bit Host(s) and one for my 32 Bit Host.

[Ralf Jung]

The 3.0.12 changelog says: VMM: reduced IO-APIC overhead for 32 bits Windows NT/2000/XP/2003 guests; requires 64 bits support (VT-x only; bug #4392) However, I still experience this issue after upgrading to the latest version.

[Frank Mehnert]

ralf: Please attach a VBox.log file of such a session.

[Sander van Leeuwen]

ralf: you need to install the 3.0.12 additions and the OS type in the VM settings needs to be correct. The workaround is only enabled for 32 bits NT/2K/XP/2K3 OSes.

[Frank Mehnert]

Yes, that was the reason I was asking for the VBox.log file :)

[Ralf Jung]

The issue occurs long before I can even install the guest additions, during OS installation. I also have an old VM installed when there was no SMP, where I installed the 3.0.12 additions and after doing so copied the hal.dll and ntoskrnl.exe from a multiprocessor system. I will get you a logfile of that one ASAP. Do I have to re-install the additions after changing the HAL?

[Frank Mehnert]

The performance will not improve until you installed the guest additions and until they are active. That is, the fix/workaround will not work during boot.

I think it is not necessary to re-install the guest additions after you changed the HAL but it cannot hurt. Make sure to use 3.0.12 additions in any case.

[Ralf Jung]

Ok, I installed the new HAL and re-installed the latest guest additions. Then I started a new VirtualBox session (so the log file I'll attach soon starts here), started Windows, and indeed the "full load while doing nothing" vanished some time after the additions were loaded, so something improved. However, then I started Warcraft III (to test performance), and it is definitely slower than without IO-APIC (the sound stutters, and the animations are not fluent). I also have the impression that the task-manager process list still flickers a tiny bit, but it's nothing compared to how it behaves without the work-around. At least it's now able to play the Windows shut-down sound without stuttering (the start-up sound is of course still broken).

Any chance of a work-around which is independent of the guest doing some cooperation? In the current state, IO-APIC and SMP will stay disabled here. However, I obviously don't understand anything of the internals of this issue. Since I don't use the VM that often it's not a big deal. It'd just be nice to be able to use both cores. Anyway, keep up the good work, and thanks a lot for making such a great open-source VM :)

[J.E. Raaijmakers]

Same problem here, trying to run guest Windows XP (32-bit) on host Kubuntu 9.10 (64-bit, Intel Q9450) using more than one processor and IO-APIC is incredibly slow whilst reinstalling without IO-APIC makes it run fast again. Installing guest additions did not help in my case.

[ToddAndMargo]

Replying to JeRa:

Same problem here, trying to run guest Windows XP (32-bit) on host Kubuntu 9.10 (64-bit, Intel Q9450) using more than one processor and IO-APIC is incredibly slow whilst reinstalling without IO-APIC makes it run fast again. Installing guest additions did not help in my case.

How much RAM is your guest using? On two other bugs, boosting it to 512M solved the problem.

[J.E. Raaijmakers]

Replying to ToddAndMargo:

How much RAM is your guest using? On two other bugs, boosting it to 512M solved the problem.

I allocated 1024 MB of RAM for the guest. The host has 4096 MB of RAM available. I could try to lower the amount of RAM to 512 MB just for testing purposes (you never know), but eventually I want to run the guest using the entire gigabyte.

[Sander van Leeuwen]

JeRa: without the VBox.log from the VM session there's nothing we can do.

[J.E. Raaijmakers]

VB log of SMP WinXP (32-bit) running on Kubuntu 9.10 (64-bit) experiencing tremendous lagging and performance degradation

[J.E. Raaijmakers]

Replying to sandervl73:

JeRa: without the VBox.log from the VM session there's nothing we can do.

Well then, here you are. I'd also like to note that I had additions installed (didn't help) and switched settings on 2d and 3d acceleration (dito).

[Sander van Leeuwen]

JeRa: your problem is completely different. See #5501

[Brian J. Cohen]

I can confirm this is an issue. Running Ubuntu 9.04 host, VirtualBox 3.0.12 r54644, and Guest extensions 3.0.12. With my Windows XP guest idle., the host CPU sits at around 50% usage. Things that had no effect: disabling USB controller on the VM, setting Null Audio Driver on the VM, enabling/disabling IO-APIC on the VM. I will attach my VBox.log.

[Sander van Leeuwen]

brianjcohen: you need to enable AMD-V in your BIOS.

[Brian J. Cohen]

sandervI73: My CPU does not support AMD-V.

[Mark Cranness]

The fix in 3.0.12 does not work for me.

Host: Windows XP Pro (32bit), Intel Core 2 Duo E8400 (64bit capable), VirtualBox 3.0.12. Guest: Windows 2000 Server, GA 3.0.12 r54655 installed and running.

IO APIC on, 'ACPI Multiprocessor PC' HAL: When VM has 1 CPU: host runs at ~50% CPU usage, roughly split between each CPU. When VM has 2 CPUs: host runs at ~100% CPU usage.

hal.dll: 5.00.2195.7006 kernel32.dll: 5.00.2195.7135 ntdll.dll: 5.00.2195.7006 ntkrnlpa.exe: 5.00.2195.7319 ntoskrnl.exe: 5.00.2195.7319 win32k.sys: 5.00.2195.7322 winsrv.dll: 5.00.2195.7135

IO APIC on, 'ACPI Uniprocessor PC' HAL or IO APIC xx, 'Advanced Configuration and Power Interface (ACPI) PC' HAL, host CPU is as expected.

[Mark Cranness]

SMP Windows 2000 Server

[Marc Steffens]

Can anyone let me know if there are there plans for a fix for Core Duo processors? I have an Intel T2500 Core Duo and have the problem when trying to use both cores in a VM.

[Sander van Leeuwen]

MarkCranness: IO-APIC overhead has been solved for your CPU since 3.0 (iirc) as it supports the hardware workaround. There must be some other reason why the VM is using so much cpu time.

[Marc Steffens]

Ok I must've misunderstood the notes in the description (leaves out Core Solo/Duo)? It'd be greatly appreciated if you have time to look at the log I just attached to see if there's anything obvious in there. When I run resource intensive apps in the VM on a single core it's quite slow so it'd be great to get both cores going. thanks

[Michael Kromer]

I have the same Problem with an Windows 2003 R2 Guest on Various HP ML 385 Machines ... One host with 8 CPUs, and 5 CPU's dedicated for the Guest. Only one other VBox is running, however with only 1 CPU, and it doesn't take any load at all. The System (5 CPUs) is a terminal server, as soon as >2 people are doing things on the machine (like opening such applications as office) performance is lousy. Ticking Kernel Time in Task Manager shows up Extremely high Kernel Times ... Also top/htop reveal extremely high "System" load, consumed by VBox. The same machine without virtualization and same load not even takes 10% of what I needs under VirtualBox 3.1.2 running on 2.6.31.7 x86_64 kernel. Log follows in a second

Any suggestions?

[Michael Kromer]

VBox.log with bad 2003 r2 32-bit performance

[hansb]

Just checked with Virtualbox 3.1.4. r57640, Ubuntu Karmic kernel 2.6.32-02063205-generic.

W2k guest with SMP enabled is unusable, because it uses 100% of the host's cpu as soon as IO APIC is enabled. The guest itself seems to run fine, but he host is borked. Current guest additions installed. All non-essential peripherals disabled. Log file follows.

[Sander van Leeuwen]

hansb: karmic problems should be discussed in ticket #5501

[Frank Mehnert]

Marc Steffens, please have a look at #6250, this problem applies to you as well.

[Richard]

I can confirm this issue exists in VirtualBox 3.2.0. I am running Windows XP Professional SP3 as a guest OS with Gentoo Linux as a host. I am using app-emulation/virtualbox-bin-3.2.0 from Gentoo Linux's portage tree.

Here are some details regarding my host system:

# uname -a Linux Laptop 2.6.34 #1 SMP PREEMPT Tue May 18 09:57:44 EST 2010 i686 Genuine Intel(R) CPU T2400 @ 1.83GHz GenuineIntel GNU/Linux

[Vincent Bernat]

I am using VirtualBox 3.2.4 from Debian and still have the issue. VT-X is enabled in the BIOS. The processor is Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz. I have set perf_counter to 2. The kernel is stock 2.6.32 from Debian Sid.

I have installed 3.2.4 additions without any change.

[Vincent Bernat]

VBOX.log with 100% CPU usage, VT-X enabled, on a QuadCore

[Frank Mehnert]

Please reopen if still relevant with VBox 4.3.2.