vbox claims SVME, even when in use by KVM -> fixed in SVN

[Bernhard M.]

vbox claims SVME, even when it is already in use by KVM

How to reproduce: have a CPU/system AMD VT-x (SVM) start a VM using KVM start VirtualBox ; start a VM that uses hardware virtualisation (SVM)

Actual Results: VBox VM started, but at the same time KVM VM stopped execution and could only be terminated with killall -9 qemu-kvm with some bugs visible in dmesg. A reboot was needed to get it back to a working state.

Expected Results: when a VM using HVM is actually started, availability of SVME should be checked and an error be reported to the user like "hardware virtualisation is already in use - either stop kvm or disable HVM on this VM"

maybe related to the fix of bug 3058 ​http://www.virtualbox.org/ticket/3058

[Sander van Leeuwen]

VirtualBox checks if somebody else is using AMD-V. It all depends on what KVM does to initialize AMD-V. Never *ever* start two hypervisors at the same time.

[Bernhard M.]

VirtualBox checks if somebody else is using AMD-V

I found this to not be true for version 3.0.8 and 3.0.10.

example test code I used:

# Makefile KERNELVER=uname -r KERNELDIR="/lib/modules/${KERNELVER}/build"

ifndef KERNELRELEASE all:

make -C ${KERNELDIR} M=pwd

clean:

make -C ${KERNELDIR} M=pwd clean #rm -f *.o *.ko .*.cmd rm -f modules.order Module.markers

else obj-m += svmealloc.o endif

svmealloc.c only works right with maxcpus=1 #include <linux/kernel.h> #include <linux/module.h>

#define EFER_SVME (1<<12)

static int init init_svme(void) {

uint64_t efer; int me = raw_smp_processor_id(); printk("loading svmealloc on CPU %i\n", me); rdmsrl(MSR_EFER, efer); printk("EFER=%08llX\n", efer); if (efer & EFER_SVME) {

printk("SVME already allocated by someone else\n"); return -EBUSY;

} wrmsrl(MSR_EFER, efer | EFER_SVME); rdmsrl(MSR_EFER, efer); printk("EFER=%08llX\n", efer); if(!(efer & EFER_SVME)) {

printk("allocating EFER failed\n");

} return 0;

}

static void exit cleanup_svme(void) {

uint64_t efer; int me = raw_smp_processor_id(); printk("unloading svmealloc on CPU %i\n", me); rdmsrl(MSR_EFER, efer); printk("EFER=%08llX\n", efer); if (!(efer & EFER_SVME)) {

printk("error: someone freed SVME while we used it\n");

} wrmsrl(MSR_EFER, efer & ~EFER_SVME); rdmsrl(MSR_EFER, efer); printk("EFER=%08llX\n", efer);

}

module_init(init_svme); module_exit(cleanup_svme);

# example call (with a VirtualBox VM started and stopped during the sleep): insmod ./svmealloc.ko && sleep 20 && rmmod svmealloc ; dmesg |tail # produced: loading svmealloc on CPU 0 EFER=00000D01 EFER=00001D01 unloading svmealloc on CPU 0 EFER=00000D01 error: someone freed SVME while we used it EFER=00000D01

# for comparison, dmesg where openSUSE's kvm-84 properly checks: loading svmealloc on CPU 0 EFER=00000D01 EFER=00001D01 kvm: enabling virtualization on CPU0 failed unloading svmealloc on CPU 0 EFER=00001D01 EFER=00000D01

Never *ever* start two hypervisors at the same time.

It will eventually happen by mistake (as it did to me) and you do not want VirtualBox to crash machines in those cases. Just fix the check.

[Bernhard M.]

should have used preview...

#Makefile
KERNELVER=`uname -r`
KERNELDIR="/lib/modules/${KERNELVER}/build"

ifndef KERNELRELEASE
all:
	make -C ${KERNELDIR} M=`pwd`
clean:
	make -C ${KERNELDIR} M=`pwd` clean
	#rm -f *.o *.ko .*.cmd
	rm -f modules.order Module.markers
else
obj-m += svmealloc.o
endif

// svmealloc.c
// only works right with maxcpus=1
#include <linux/kernel.h>
#include <linux/module.h>

#define EFER_SVME (1<<12)

static int __init init_svme(void)
{
	uint64_t efer;
	int me = raw_smp_processor_id();
	printk("loading svmealloc on CPU %i\n", me);
	rdmsrl(MSR_EFER, efer);
	printk("EFER=%08llX\n", efer);
	if (efer & EFER_SVME) {
		printk("SVME already allocated by someone else\n");
		return -EBUSY;
	}
	wrmsrl(MSR_EFER, efer | EFER_SVME);
	rdmsrl(MSR_EFER, efer);
	printk("EFER=%08llX\n", efer);
	if(!(efer & EFER_SVME)) {
		printk("allocating EFER failed\n");
	}
	return 0;
}

static void __exit cleanup_svme(void)
{
	uint64_t efer;
	int me = raw_smp_processor_id();
	printk("unloading svmealloc on CPU %i\n", me);
	rdmsrl(MSR_EFER, efer);
	printk("EFER=%08llX\n", efer);
	if (!(efer & EFER_SVME)) {
		printk("error: someone freed SVME while we used it\n");
	} 
	wrmsrl(MSR_EFER, efer & ~EFER_SVME);
	rdmsrl(MSR_EFER, efer);
	printk("EFER=%08llX\n", efer);
}

module_init(init_svme);
module_exit(cleanup_svme);

[Sander van Leeuwen]

Check again with 3.0.10. Should fail there.

[Bernhard M.]

retested this today and it is still what I wrote three days ago

VirtualBox checks if somebody else is using AMD-V

I found this to not be true for version 3.0.8 and 3.0.10.

so to summarize:

• start VBox, insmod svmealloc.ko fails nicely (-EBUSY)

• start VBox, start kvm fails nicely

• start kvm, insmod svmealloc.ko fails nicely

• insmod svmealloc.ko, start kvm fails nicely

• insmod svmealloc.ko, start VBox, stop VBox shows "error: someone freed SVME while we used it" on rmmod svmealloc

• insmod svmealloc.ko, start VBox, rmmod svmealloc causes VBox to hang because hardware virtualisation is suddenly unavailable

(* start kvm, start VBox causes kvm to hang because hardware virtualisation is suddenly unavailable)

conclusion: VBox allocates/uses SVME, even when in use by someone else before, thus failing badly.

[Sander van Leeuwen]

The code checks for it and the failure paths look fine. Could you provide a VBox.log file for the failed experiment above? I can't repeat your experiment as I don't run Linux.

[Bernhard M.]

​http://www.zq1.de/~bernhard/temp/Logs/VBox.log.2 this is from just start+stop

​http://www.zq1.de/~bernhard/temp/Logs/VBox.log.1 is from case 5 above and virtually identical to the above

​http://www.zq1.de/~bernhard/temp/Logs/VBox.log is from case 6

​http://www.zq1.de/~bernhard/temp/Logs/dmesg including my .ko output and vboxdrv failure

btw: the case 7 (kvm) fail message also starts with "invalid opcode"

another sidenote: I tried to locate the SVM-check but could not find where/how the ASMRdMsr/ASMWrMsr function of /usr/share/virtualbox/src/vboxdrv are used. Documentation pointers welcome.

[Sander van Leeuwen]

Ok, found the problem. Fixed in SVN now.

You were looking at the wrong source code. The AMD-V code is not in vboxdrv, but in src/vbox/vmm.

[Bernhard M.]

I can see the fix now in r24368. And I saw, that this was introduced with r15987:15989 dated 2009-01-16 11:50:47 - exactly the time of bug #3058 that I linked in the top post.

great.

[Frank Mehnert]

The fix is in VBox 3.1.0, please reopen if this problem persists with this version.