Severe time drift in guest OS

[geegee]

Installed ubuntu 14 lts guest on centos7 virtualbox Version 5.0.14 r105127

ubuntu guest has latest version: 5.0.14r105127

everything is standard and updated.

Problem is that time drift is so extreme that i run into trouble.

the guest loses about 10 seconds every 15 minutes!

i already tried setting a ntp daemon.. but it does nothing.. also adding a crontab with ntpdate does nothing.

also adding --timesync-set-start --timesync-set-on-restore 1 --timesync-set-threshold 5000 to /etc/init.d/vboxadd-service does not help.

[geegee]

log

[Frank Mehnert]

The time should be properly synced by the VirtualBox Guest Additions. Please do not install ntp in the guest as it would probably conflict with the Guest Additions service. Is the host which you run your VM on is very busy, ie do you execute VMs in parallel to the Ubuntu VM? Also, any improvement if you reduce the number of VCPUs to 3, 2 or 1?

[geegee]

i uninstalled ntp again and rebooted.

also checking load averages in both host and guest and loads avg dont exceed 0.2.

also not running other guests on this host.

how can i verify that the guest additions are actually running?

[geegee]

i left it running idle for an hour and the guest is now about 30 seconds behind.

[Frank Mehnert]

The best way for verifying the time synchronization mechanism is to start the VBoxService process with verbose output:

1. Stop the running VBoxService process with

   /sbin/rcvboxadd-service stop
   

2. Start the VBoxService process with verbose output on a terminal:

   /usr/sbin/VBoxService -fvvv
   

The process will show verbose output on the current console including information about time synchronization.

[geegee]

ok, well something is amiss then:

00:01:50.005537 timesync vgsvcTimeSyncWorker: Host:    2016-02-29T19:51:36.953000000Z    (MinAdjust: 100 ms)
00:01:50.005600 timesync vgsvcTimeSyncWorker: Guest: - 2016-02-29T19:50:38.431210000Z => 58 521 790 000 ns drift
00:01:50.005626 timesync vgsvcTimeSyncAdjust: adjtime by 58 521 790 000 ns
00:01:50.265625 vminfo   Guest Property: /VirtualBox/HostInfo/VRDP/ActiveClient not found
00:01:50.265681 vminfo   VRDP: Handling location awareness done
00:01:55.266569 vminfo   Guest Property: /VirtualBox/HostInfo/VRDP/ActiveClient not found
00:01:55.266643 vminfo   VRDP: Handling location awareness done
00:02:00.005755 timesync vgsvcTimeSyncWorker: Host:    2016-02-29T19:51:47.022000000Z    (MinAdjust: 100 ms)
00:02:00.005817 timesync vgsvcTimeSyncWorker: Guest: - 2016-02-29T19:50:48.431428000Z => 58 590 572 000 ns drift
00:02:00.005842 timesync vgsvcTimeSyncAdjust: adjtime by 58 590 572 000 ns
00:02:00.267636 vminfo   Guest Property: /VirtualBox/HostInfo/VRDP/ActiveClient not found
00:02:00.267705 vminfo   VRDP: Handling location awareness done
00:02:05.268323 vminfo   Guest Property: /VirtualBox/HostInfo/VRDP/ActiveClient not found

as you can see it detects the difference but fails to actually adjust the difference.

[Frank Mehnert]

Looks to me like the timer adaption code is working but the time difference is just too big for some reason. The service uses adjtime() which slowly adapts the time. If the time difference between guest and host is bigger then the adaption is too slow. The real challenge is to find out why the guest time diverts that quickly from the host time.

Does it make any difference if you disable the KVM paravirt provider? VM settings / System / Extended / Paravirtualization => None.

Could you also attach the output of dmesg executed in the guest?

[geegee]

do you mean setting the: "system->acceleration->paravirualization interface" dropdown to "none"?

i did that and now it seems to stay in sync... not sure if that was actually did the trick or that some other factor came into play..

[geegee]

if i set it to KVM it immediately starts loosing time.

[geegee]

guest dmesg

[Frank Mehnert]

Sorry, I now saw that 'KVM' was already active (this is the default if you create a new Linux VM). Could you repeat the test after you disabled paravirtualization (set it to 'None')? Does that make any difference? The KVM provider is preferred because it makes the guest work better together with the VMM but I want to rule out any bugs in that provider.

[geegee]

that is what i was trying to say in my previous posts:

quote: "if i set it to KVM it immediately starts loosing time."

so, as long as i dont use KVM option all is fine.

[Klaus Espenlaub]

I would suspect that the root cause is the apparently rather big measurement error for the CPU frequency. VirtualBox measured the clock to be 2.412904761GHz for a CPU rated at 2.40GHz. This is a probable error (don't know the exact clock frequency, it's unlikely to be exactly 2.4GHz) of 0.5%, which will slow down the time perceived by the guest.

That would explain a deviation of approx. 20 seconds per hour. Such big differences can't be usually handled by the time adjustment mechanism.

[FranzS]

I have nearly the same behavior.

I was running an Ubuntu 15.10 VM without Paravirualization => No drift Now I've created a new Ubuntu 16.04 VM with Paravirualization => extreme dift.

Some guy's from the Forum already did some analysis: ​https://forums.virtualbox.org/viewtopic.php?f=3&t=77680

The very strange thing is the same VM has no time drift on one of my Notebooks. But a high drift on the other Notebook.

socratis:      6 * 10-9
Perry :      934 * 10-9
Mine :   2392220 * 10-9 =  2,3 * 10-3  (Not Working)
Working: 5998384 * 10-9 (Working)

[FranzS]

So,

as promised a short review after trying something new. After setting Paravirtualization to "none" the Time drift is "gone" (like a poem).

But nevertheless I've found some other strange things:

This error occurs if i run the VM in the "problematic VM Host"

May 17 19:10:26 fortuna kernel: [    0.000000] NR_IRQS:16640 nr_irqs:256 16
May 17 19:10:26 fortuna kernel: [    0.000000] Console: colour VGA+ 80x25
May 17 19:10:26 fortuna kernel: [    0.000000] console [tty0] enabled
May 17 19:10:26 fortuna kernel: [    0.000000] tsc: Fast TSC calibration failed
May 17 19:10:26 fortuna kernel: [    0.000000] tsc: Unable to calibrate against PIT
May 17 19:10:26 fortuna kernel: [    0.000000] tsc: using PMTIMER reference calibration

It looks like this in the Working VM:

May  9 09:50:30 fortuna kernel: [    0.000000] console [tty0] enabled
May  9 09:50:30 fortuna kernel: [    0.000000] tsc: Detected 2594.000 MHz processor
May  9 09:50:30 fortuna kernel: [    0.062760] Calibrating delay loop (skipped) preset value.. 5188.00 BogoMIPS (lpj=10376000)
May  9 09:50:30 fortuna kernel: [    0.063445] pid_max: default: 32768 minimum: 301

Kernel: 4.4.0-22-generic (Ubuntu 16.04)

[meixner]

Ticket #15359 lists the same issue when running on a Windows host.

[Frank Mehnert]

Please never forget to attach a VBox.log file for such a VM session. A guest timing drift is the result of some timing problems. There are many settings which can be used to trigger guest timing problems and the log file would show them.

[Timothe Litt]

Another thing to consider is that Linux has a number of competing time sync mechanisms. It's possible to have more that one enabled - and if you do, things go wild. Besides time variance, I've seen very high load averages due to logging of the competing daemons.

Besides vboxadd-service's timesync, there are at least:

ntpd (classic) (which may depend on a separate ntpdate to start, unless the ntp init script does an ntpd -q before starting the daemon)

ntpd (bsd)

xntpd

systemd-timesyncd (a sntp client that's particularly hard to keep disabled)

chrony

sntp

Some of these run as threads and won't show up on the usual ps / top commands.

So, while VirtualBox may be at fault - it may also be the result of competition in the guest.

My first reaction isn't to look for things to add to the guest; rather to look for things to disable...

[Klaus Espenlaub]

Disabling paravirt (as in comment:13) will of course make timekeeping significantly more expensive. It can easily hide a bug in VirtualBox.

A while ago we discovered a bug in the 'omni timer' support on some systems when they're running Windows. It looks like VirtualBox is doing something wrong, and that can cause the calibration determine the wrong values. This should only happen for reasonably old CPUs which don't report their TSC as invariant and the code which checks for the TSC delta between CPUs also gets a value which is close enough to 0. We didn't find time to investigate when this happens, but it appears specific to certain manufacturers (probably they use a different way to declare the CPUs in the ACPI table, ending up with different identifiers for CPUs).