Paused VM causes swap storm after 5-10 minutes

[David Harris]

This might be a little difficult to describe - if I'm not clear I apologise in advance and I'd be more than willing to follow-up with more information as requested.

Host: Linux 2.6.30, 2.6.31, 2.6.32 CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 5200+ Memory: 2GB RAM, 8GB swap Guest: Windows XP, fully up-to-date, Guest Additions installed

When I pause (Host+p) the VM, after about 5-10 minutes, the host starts swapping things out slowly but surely. After about 5 minutes, the machine is unusable and requires a hard reset. This happens consistently every time. Leaving the VM running doesn't cause this problem. Suspending the VM (closing it and saving state) doesn't have this problem. There's nothing unexpected in the logs.

Bluntly I expect it should be pretty easy to reproduce on similar hardware (it happened with VirtualBox 3.0.12 as well), but if any more information is required, please let me know.

Thanks,

Dave

[David Harris]

And by "things", I mean everything on the host. The guest is configured to use 384MB of memory, and here's the output of 'free' after the VM has just booted:

[ dbharris@willow: ~/ ]$ free  -m
             total       used       free     shared    buffers     cached
Mem:          2010       1978         31          0          4        796
-/+ buffers/cache:       1178        831
Swap:         8191         15       8176

And just after the VM has paused:

[ dbharris@willow: ~/ ]$ free  -m
             total       used       free     shared    buffers     cached
Mem:          2010       1986         23          0          4        791
-/+ buffers/cache:       1191        819
Swap:         8191         15       8176

And, shortly after the swapping starts:

[ dbharris@willow: ~/ ]$ free  -m
             total       used       free     shared    buffers     cached
Mem:          2010       1614        396          0          3         56
-/+ buffers/cache:       1553        456
Swap:         8191        181       8010

Also, here's what the last of ~/.VirtualBox/Machines/winxppro/Logs/VBox.log has; note that no additional lines are appended after the VM is paused.

[ dbharris@willow: ~/ ]$ tail -F .VirtualBox/Machines/winxppro/Logs/VBox.log
00:00:58.111 Guest Log: VBoxService.exe: Started. Verbose level = 0
00:00:59.796 PCNet#0: Init: ss32=1 GCRDRA=0x0288a420[64] GCTDRA=0x0288a020[64]
00:01:18.416 PIT: mode=2 count=0x4ad (1197) - 996.81 Hz (ch=0)
00:01:49.730 Guest Log: VBoxTray: Started.
00:01:50.297 Starting host clipboard service
00:01:50.297 Initializing X11 clipboard backend
00:01:50.300 Shared clipboard: starting shared clipboard thread
00:01:50.333 Guest Additions capability report: (0x5) seamless: yes, hostWindowMapping: no, graphics: yes
00:01:56.291 PCNet#0: Init: ss32=1 GCRDRA=0x0288a420[64] GCTDRA=0x0288a020[64]
00:02:09.154 Changing the VM state from 'RUNNING' to 'SUSPENDED'.

Close examination of iostat and iotop (iostat excerpt included below) indicates that the swap-outs start happening within a minute or two of the VM being paused, and continues until I unpause the VM. Trying to use swapped-out applications while the VM is paused works, but the swapping-out is pretty aggressive so doesn't work for long.

vmstat excerpt for when idle (but VM is running):

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 3  0 170340 626952  12380 261288    0    0    64   100 1110 3200 28 18 55  0
 2  0 170340 626860  12380 261420    0    0   148    16 1218 3743 24 20 55  1
 0  0 170340 624892  12380 261536    0    0    96     0 1049 3312 39 17 42  3
 2  0 170340 623412  12388 261652    0    0    80    28 1320 3971 27 20 51  1

vmstat excerpt for about 15 minutes after the VM is paused, when swap-out is happening, but machine is still responsive:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0  25348 297440  11532 357048    0  172   776   172  778 2495 15  6 79  0
 0  0  25348 297060  11532 357208    0    0   192     0  900 2778 13  7 80  0
 0  0  25428 296992  11528 356752    0   80   256    80  773 2564 14  5 81  0
 0  0  25604 297212  11520 356128    0  176   244   176  895 2841 13  6 80  0
 0  0  25864 298364  11516 354696    0  260    16   260  762 2411 15  7 77  0
 0  0  25864 298028  11516 354668    0    0    16     0  845 2698 15  6 78  0
 0  0  25864 298680  11516 353828    0    0    24     0  797 2643 16  6 76  1
 0  0  25864 298328  11516 354228    0    0   352   104  928 2755 14  9 77  0
 1  0  26268 298900  11516 353072    0  404    20   452 1061 2655 11  9 78  2
 0  0  26268 298580  11516 353388    0    0   384     0 1179 3128 14  7 78  0
 0  0  26268 299024  11520 352660    0    0    24     0  767 2470 17  6 75  1

vmstat excerpt for when machine is becoming quite unresponsive, about 25 minutes after VM was paused:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  1 186556 397304   3672  56332  808 1900  6680  2040 1796 64378 22 17 27 34
 2  0 186028 407556   3684  57256  896    0  1848    24 1387 61833 36 14 28 23
 2  1 185808 404856   3952  59120  740  116  3448   132 1623 57250 33 12 24 32
 1  0 185600 391756   3936  58836  616  188   868   188 1724 53971 45  8 31 17
 1  1 185560 390216   3936  59600   76    0   908     0 1537 88822 17  9 68  6
 1  0 186624 386324   3892  63260 1016 2168  6284  2168 1973 74277 24 12 21 44
 0  0 184980 395532   3908  62712  680   76  1096   228 1504 71426 39  9 32 21
 0  0 184900 394920   3908  63360  116    0   856    32 1201 81639 20 11 61  9

Thanks again,

Dave

P.S.: Very consistent, I've tested this a dozen times (across different kernel versions, builds, and VirtualBox builds and versions).

[Technologov]

Added myself as CC on this issue.

-Technologov

[David Harris]

Hey, the ethtool trick in #5260 fixed this for me.

Thanks,

Dave

[Frank Mehnert]

Aha, interesting observation!

[Frank Mehnert]

Please reopen if still relevant with VBox 4.0.4.