Re: [Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and later

[Stefan Hajnoczi]

On Wed, Aug 20, 2014 at 12:20 AM, Andrew Martin <address@hidden> wrote:
> ----- Original Message -----
>> From: "Stefan Hajnoczi" <address@hidden>
>> To: "Andrew Martin" <address@hidden>
>> Cc: address@hidden
>> Sent: Tuesday, August 19, 2014 9:59:25 AM
>> Subject: Re: [Qemu-devel] Using cache=writeback safely on qemu 1.4.0 and 
>> later
>>
>> If you strace -f the QEMU process on the host, you will see fdatasync(2)
>> system calls when the guest flushes the disk.
>>
>> You can find the file descriptor number by checking ls -l
>> /proc/$PID_OF_QEMU/fd and looking for the disk image file.
>
> When the disk is set to cache=writethrough on one of the same VMs, I see 
> frequent
> fdatasync(2) calls (every few seconds). However, when I change the disk over 
> to
> cache=writeback, since boot I have not yet seen a single fdatasync(2) call, 
> even
> after writing data 2x the amount of RAM:

There is a misconception here.  Writing data to disk does not make it
persistent across power failure.  The behavior you observed is
actually expected.

Flushing the disk cache is a deliberate operation that applications
must perform to ensure data is safe on disk.

You can use sync(1) to manually flush file system buffers for testing.
On the host you should see an fdatasync(2).

>> > I recently experienced UPS failure on several hosts which caused a hard
>> > shutdown. After restarting, 3 of the guests had corruption on their disks
>> > and
>> > required a fairly long fsck to fix. Afterwards, data that had been written
>> > to
>> > the disks several hours before the crash was corrupted, which makes me
>> > think
>> > that it was never fsync()-ed to the non-volatile storage.
>>
>> What exactly was the "corruption" you encountered?  Which application,
>> error message, etc.
>
> Two of the servers are web servers with apache2. In one case, a python daemon
> copies JPGs onto the server - the last 100 copied onto the server were 
> corrupted.
> In another case, some files had been uploaded several days prior to the 
> www-root,
> but after the hard reset said files were no longer present in the filesystem.

Did the Python daemon fsync the files and directories it modified/created?

Did you sync(1) after copying files to www-root?

Also, you didn't explain what "corrupted" means.  Where the jpg files
missing, were they zero bytes in size, were they filled with junk,
etc?

>> > Is it safe in this setup to use cache=writeback? Or, should I use
>> > cache=writethrough instead?
>>
>> Ubuntu 12.04 is recent and sends write cache flushes.
>>
>> Are you sure the file system and/or application workload are flushing
>> the disk cache?  Please check the mount options and application-specific
>> configuration.
>
> The mount options for the ext4 filesystem in the VM in both cases are:
> rw,relatime,errors=remount-ro,data=ordered
>
> Similarly, the host's ext4 filesystem holding the images is mounted with:
> rw,relatime,data=ordered
>
> I did not see any errors in the kernel log in the guest, probably because the
> root filesystem was read-only until the fsck had completed.

Good, that is a sane setup.  I guess the behavior you saw would also
happen on a physical machine, it is necessary to fsync the file or
sync the file system to guarantee that data is safely on disk.

Stefan