Corrupted data on shared folders => Fixed in SVN

[Fran]

On a Windows XP host, I use a linux guest as local web server, where the /var/www directory is a shared folder (mounted with gid/uid to be accessible by apache). I handle version control with Mercurial.

All has been working fine, until I stumbled upon a bug in version 2.1.4 and 2.2.0 - data on the shared directory is corrupted, while read or written. I can reproduce this consistently, and I have found a minimal* text file that can trigger the bug. ( * if I split the file in two, both the parts will be read correctly )

    fran@inside:/var/www/test$ rm -f .hg
    fran@inside:/var/www/test$ hg init
    fran@inside:/var/www/test$ hg add minfail
    fran@inside:/var/www/test$ hg commit -m "Boom"
    fran@inside:/var/www/test$ hg verify
    checking changesets
    checking manifests
    crosschecking files in changesets and manifests
    checking files
    minfail@0: broken revlog! (index data/minfail.i unknown format 25920)
    warning: orphan revlog 'data/minfail.i'
    1 files, 1 changesets, 0 total revisions
    1 warnings encountered!
    1 integrity errors encountered!
    (first damaged changeset appears to be 0)
    fran@inside:/var/www/test$

The problem is not present if I run mercurial on the host machine, or I copy the files on the virtual machine hard drive. This made me think that Mercurial works fine, but VirtualBox has trouble writing on the shared folder.

The minfail file is just text, more or less 8kB. You can find here attached.

[Fran]

Text file that can be used to reproduce the bug

[vboxHHuser]

Hi, with Version 2.2.2 r46594 I have created under Win XP shared folders for Ubuntu as guest. When e.g. Firefox in Ubuntu saves a download into a shared folder then the real filesize seems to be mixed and unpredictable. Also data is corrupted.

When I used midnight commander (mc) in xterm mc complains about problems to change directory.

[Dann Martens]

I'm having exactly the same problem, using Windows XP as host and Debian Lenny as guest. File operations inside the mounted virtual folder fail unexpectedly; midnight commander complains consistently that it cannot read the directory.

[ltorrent]

Using Windows XP as a host and Fedora (kernel 2.6.29.4-167.fc11.i686.PAE) or Mandriva as a guest I've got similar problem: shared folder is mounted and naulilus or ls read directory entry well, but midnight commander (mc) produce cannot read the directory entry error. Wonderfully after pressing Ctrl-O twice one can see directory contents in mc, but it should be pressed twice after each directory changing.

I've only tried read-only folders, and read operations works probably well. But shared folders are useless for me if they can't be accessed in midnight commander properly. And Virtual Machine is useless without shared folders...

[Dann Martens]

The problem is still there with VirtualBox 3.0.0; I've installed the guest additions that come with this version, but the problem remains.

[Fran]

I can also confirm the original problem described in this bug, using VirtualBox 3.0.0 (r49315).

[Kevin Montagne]

I have the same problem using VirtualBox 3.0.0, Windows Vista as a host and Solaris 10 as the guest OS. If I am in the Solaris 10 guest and I copy a file to the shared folder it works fine. However, if I make any write to that file (vi, etc.) the file is corrupted and is reported by both Windows and Solaris as being 4GB in size.

[rengeade22]

I have also this Problem on VirtualBox 3.0.4 and WinXP host. This feature is pretty annoying while cross developing. It seams always to happen when i try to synchronise my local repo up to the server, mounted as a shared folder. Cloning the repo up to the server does no harm so far.

[am]

Same problem here.

I have a windows folder (e.g. 'foo') on Windows Vista 64 bit (host) that I mount on Ubuntu 9.04 (guest) using VirtualBox 3.08 r5318.

If I du -sh on foo on my Linux guest I get 8.6 MB, but foo has an actual size of 68 MB on my Windows host. If I go down the folder tree on foo, the file size differences reported by the host and the guest decreases. VBox always reports smaller file sizes than Windows does.

I tried older versions of VBox with no success.Cygwin can accurately report 68 MB though at the top of the folder tree.

By the way, I also have a mercurial repository on this folder and it always crashes if I try to commit anything from the Linux guest.

Thanks,

Amvr

[Frank Mehnert]

amvr, your observation is correct, the disk usage of the file is displayed wrongly. There are two bugs, the host side and the guest side. This Linux guest side and the Linux host side are both fixed in SVN but there is still no fix for the Windows host side available.

But this disk usage problem is minor has nothing to do with the original problem (corrupted data). As the origin of this problem is currently unknown I would like if you could check if you still can encounter corrupted data with the latest release (VBox 3.0.8). Please don't forget to update the guest additions as well. amvr, you reported that you are already using VBox 3.0.8 and observe mercurial crashes, did you upgrade your guest additions?

dreadnaut, I cannot reproduce your example with a Ubuntu Jaunty guest on a Vista 32 host.

[am]

Hi,

Yes, I upgraded my guest additions as well.

Cloning repositories on the Linux guest works fine. But if I commit a change, and then I hg status, I get the following message from Mercurial:

abort: index 00changelog.i is corrupted!

I found the following thread on Mercurial's archived mailing list from last July: ​http://selenic.com/pipermail/mercurial/2009-July/026882.html

So the disk usage problem and the data corruption problem are not related?

Thanks Amvr

[Frank Mehnert]

A simple testcase is still appreciated, for instance like the above one from the original reporter. But please make sure that this test case shows this error with VBox 3.0.8 and the VBox 3.0.8 additions installed (the above testcase works fine here).

[am]

Ok,

Here is a compule of simple test cases:

Example 1: Mount a Windows directory on the Linux guest, cd to the directory on Linux and type the following on the terminal:

$ hg clone ​http://www.selenic.com/repo/hello my-hello $ cd my-hello $ echo 'some text' > newfile.txt $ hg commit -Am 'Comitting changes' $ hg st abort: index 00changelog.i is corrupted

The following example also fails:

Example 2: Mount a Windows directory on the Linux guest, cd to the directory on Linux and type the following on the terminal:

$ hg clone ​http://www.selenic.com/repo/hello my-hello $ rm -R my-hello/ rm: cannot remove `my-hello/.hg/store/data': No such file or directory

Both examples work with no problem on Cygwin. Thanks AMVR

[am]

Sorry for the formatting problems above. See below for a better formatted version of the previous post:

Here are a couple of simple test cases:

Example 1: Mount a Windows directory on the Linux guest, cd to the directory on Linux and type the following on the terminal:

$ hg clone ​http://www.selenic.com/repo/hello my-hello
$ cd my-hello
$ echo 'some text' > newfile.txt
$ hg commit -Am 'Comitting changes'
$ hg st abort: index 00changelog.i is corrupted

The following example also fails:

Example 2: Mount a Windows directory on the Linux guest, cd to the directory on Linux and type the following on the terminal:

$ hg clone ​http://www.selenic.com/repo/hello my-hello
$ rm -R my-hello/
rm: cannot remove `my-hello/.hg/store/data': No such file or directory

Both examples work with no problem on Cygwin.
Thanks
AMVR

[am]

Last line of Example 1 should say:

$ hg st
abort: index 00changelog.i is corrupted

AMVR

[Dann Martens]

I can't confirm the problem with wrong file sizes. I have checked the contents of a Shared Folder, and both Windows XP and the Debian Lenny Linux guest (with 3.0.8. Guest Additions Installed) report the same file sizes.

However, midnight commander still complains consistently that it cannot read the directory.

[Fran]

I have updated to 3.0.10 today (both vm and guest addition) and I can confirm the behaviour I initially reported. However, the original test file is not enough to trigger the bug anymore, I need to add two files to the repository, but any file works.

$ touch a
$ touch b
$ hg init
$ hg add a
$ hg commit -m "a"
$ hg verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
1 files, 1 changesets, 1 total revisions
$ hg add b
$ hg commit -m "b"
$ hg verify
** unknown exception encountered, details follow
[...]
ValueError: corrupt index file

There is no difference between adding the files in one commit operation, or two subsequent ones. Of course everything works fine on the guest virtual drive, or on the host.

If it can be of any help, the guest is running Slackware 12.2 with the default 2.6.27.7-smp kernel.

[Frank Mehnert]

Finally reproduced with a Jaunty guest, not reproducible with a Debian/Lenny guest.

[Frank Mehnert]

Replacement for /usr/src/vboxvfs-3.0.10/regops.c which should fix this issue

[Frank Mehnert]

I've attached the modified file regops.c. Please copy this file to /usr/src/vboxvfs-3.0.10/regops.c, rebuild the guest additions module with /etc/init.d/vboxadd setup and reboot your guest. After that, the problem should be fixed.

Feedback appreciated.

[Fran]

Compiled and running, it seems to work fine now. For reference, my vboxadd script was in /etc/rc.d/vboxadd.rc being on Slackware.

Thank you!

[Frank Mehnert]

The relevant changeset in the public repository is r24293.

[Spyros]

Microsoft Access databases are corrupted. Index and/or PrimaryKey in database tables may deleted with no reason. Compact and Repair does not work and produce in shared folders a new mdb file with name db1.mdb next time to try Compact and Repair from menu db2.mdb is made. The shared folder is mapped in drive letter D. This bug is allways reproduced. Tested in Windows XP SP3 vm client/guest in VBoxHeadless with host as Windows 2003 SP2 in Workgroup environment, (not Domain Controller). The bug does not happens if I don't use VirtualBox shared folders but instead use the common Microsoft Network sharing to map a drive letter. Tested in VirtualBox 3.0.10 r54097 with latest Vbox guest additions. This bug has also a high possibility to corrupt the data in other shared files (not only inside Access mdb files). Seems there is a lock issue. I don't test different permissions with the shared folders.

[moppel]

I had problems with corrupted data on shared folders under VirtualBox 2.2.4.
host: Debian (64-bit)
guests: Debian 64-bit, Ubuntu 32-bit.

With VirtualBox 3.0.10 and new regops.c it works fine now. Thanks!

[Frank Mehnert]

The fix made it into 3.0.12. suez-zeus, could you check if your problem is fixed as well by chance?

[zsero]

I don't know about the corruption, but the whole mounting disappears many times randomly on any Ubuntu guest newer than 8.04! (My host is 64-bit Win 7 with 3.1.2 but it was like this with XP 64 also or when I was using older versions of VBox.)

The best check is to enter into midnight commander (mc) and hit enter continuosly on a directory. It will go in, go out, go in, go out, after the 2x or 3x turn it will surely fail.

It's not only mc, actually the whole mount disappears many times inside programs randomly too. It doesn't work with ls, ls is 100% reliable.

[Spyros]

Replying to frank:

The fix made it into 3.0.12. suez-zeus, could you check if your problem is fixed as well by chance?

The problem still exists. Sorry for the late anwser. I set a new ticket (6053) because the ticket system doesn't remember my posts (tickets my me=0). There is logs and a screenshot there. I install the latest version 3.1.2 to the server and I install the new VirtualBox tools to the client (reboot etc.) with the same problems. I also have many windows services disable (as usual) like tcp/ip netbios helper with no issues. Is this bug possible to happen from a feature like prefetch in network commands (SMB) which windows server 2003 and xp had? Windows 2008 had SMB2 and many users say that the new protocol had more increase in transfer speed when both ends using the new SMB2. If you need a server and a client admin remote access via rdp or vnc to see things by yourself please tell me. You can also install and uninstall anything. Also I can't understand why the ticket say that corruption is fixed in SVN. I believe it and change it from ms network mapped drive to VirtualBox share folders only to corrupt the database again:( I mean we don't have to hurry if we are not sure only to close some more tickets with fixed label.

[Dann Martens]

Since VirtualBox 3.1.8 the problem seems to have been resolved.

Midnight commander doesn't complain anymore about not being able to read the directory, as described above.

I also checked on the reported file sizes and they appear to be consistent.

(Yay!)

[Frank Mehnert]

Please reopen if still relevant.

[sp]

I still have this bug on the VirtualBox 4.1.20 r80170.

I'm running VirtualBox in Windows 7 x64 environment.

Guest is Windows XP.

If I download anything on Firefox into guest's HDD - everything is fine.

If I do download on the shared folder - sometimes file is corrupted (about 20% chance).

NOTE: shared folder is located on PGP encrypted drive, but I don't think it's the case though.

I will try latest 4.2.4 and also check if it happens on regular shared drive and report back.

[Frank Mehnert]

As I didn't see a report for 4.2.4 I assume the problem does not happen there. Closing.