[Qemu-devel] [Bug 1368815] Re: qemu-img convert intermittently corrupts output images

[Serge Hallyn]

** Description changed:

+ ==========================================================
+ Impact: occasional qcow2 corruption
+ Test case: see the qemu-img command below
+ Regression potential: this cherrypicks a patch from upstream to a 
not-insignificantly older qemu source tree.  While the cherrypick seems sane, 
it's possible that there are subtle interactions with the other delta.  I'd 
really like for a full qa-regression-test qemu testcase to be run against this 
package.
+ ==========================================================
+ 
  -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on
  Ubuntu 14.04 using Ext4 filesystems.
  
  The command
  
    qemu-img convert -O raw inputimage.qcow2 outputimage.raw
  
  intermittently creates corrupted output images, when the input image is
  not yet fully synchronized to disk. While the issue has actually been
  discovered in operation of of OpenStack nova, it can be reproduced
  "easily" on command line using
  
    cat $SRC_PATH > $TMP_PATH && $QEMU_IMG_PATH convert -O raw $TMP_PATH
  $DST_PATH && cksum $DST_PATH
  
  on filesystems exposing this behavior. (The difficult part of this
  exercise is to prepare a filesystem to reliably trigger this race. On my
  test machine some filesystems are affected while other aren't, and
  unfortunately I haven't found the relevant difference between them, yet.
  Possible it's timing issues completely out of userspace control ...)
  
  The root cause, however, is the same as in
  
    http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html
  
  and it can be solved the same way as suggested in
  
    http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html
  
  In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change
  
      f.fm.fm_flags = 0;
  
  to
  
      f.fm.fm_flags = FIEMAP_FLAG_SYNC;
  
  As discussed in the thread mentioned above, retrieving a page cache
  coherent map of file extents is possible only after fsync on that file.
  
  See also
  
    https://bugs.launchpad.net/nova/+bug/1350766
  
  In that bug report filed against nova, fsync had been suggested to be
  performed by the framework invoking qemu-img. However, as the choice of
  fiemap -- implying this otherwise unneeded fsync of a temporary file  --
  is not made by the caller but by qemu-img, I agree with the nova bug
  reviewer's objection to put it into nova. The fsync should instead be
  triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC, specifically
  intended for that purpose.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1368815

Title:
  qemu-img convert intermittently corrupts output images

Status in OpenStack Compute (Nova):
  In Progress
Status in QEMU:
  In Progress
Status in “qemu” package in Ubuntu:
  Fix Released
Status in “qemu” source package in Trusty:
  Triaged
Status in “qemu” source package in Utopic:
  Triaged
Status in “qemu” source package in Vivid:
  Fix Released

Bug description:
  ==========================================================
  Impact: occasional qcow2 corruption
  Test case: see the qemu-img command below
  Regression potential: this cherrypicks a patch from upstream to a 
not-insignificantly older qemu source tree.  While the cherrypick seems sane, 
it's possible that there are subtle interactions with the other delta.  I'd 
really like for a full qa-regression-test qemu testcase to be run against this 
package.
  ==========================================================

  -- Found in releases qemu-2.0.0, qemu-2.0.2, qemu-2.1.0. Tested on
  Ubuntu 14.04 using Ext4 filesystems.

  The command

    qemu-img convert -O raw inputimage.qcow2 outputimage.raw

  intermittently creates corrupted output images, when the input image
  is not yet fully synchronized to disk. While the issue has actually
  been discovered in operation of of OpenStack nova, it can be
  reproduced "easily" on command line using

    cat $SRC_PATH > $TMP_PATH && $QEMU_IMG_PATH convert -O raw $TMP_PATH
  $DST_PATH && cksum $DST_PATH

  on filesystems exposing this behavior. (The difficult part of this
  exercise is to prepare a filesystem to reliably trigger this race. On
  my test machine some filesystems are affected while other aren't, and
  unfortunately I haven't found the relevant difference between them,
  yet. Possible it's timing issues completely out of userspace control
  ...)

  The root cause, however, is the same as in

    http://lists.gnu.org/archive/html/coreutils/2011-04/msg00069.html

  and it can be solved the same way as suggested in

    http://lists.gnu.org/archive/html/coreutils/2011-04/msg00102.html

  In qemu, file block/raw-posix.c use the FIEMAP_FLAG_SYNC, i.e change

      f.fm.fm_flags = 0;

  to

      f.fm.fm_flags = FIEMAP_FLAG_SYNC;

  As discussed in the thread mentioned above, retrieving a page cache
  coherent map of file extents is possible only after fsync on that
  file.

  See also

    https://bugs.launchpad.net/nova/+bug/1350766

  In that bug report filed against nova, fsync had been suggested to be
  performed by the framework invoking qemu-img. However, as the choice
  of fiemap -- implying this otherwise unneeded fsync of a temporary
  file  -- is not made by the caller but by qemu-img, I agree with the
  nova bug reviewer's objection to put it into nova. The fsync should
  instead be triggered by qemu-img utilizing the FIEMAP_FLAG_SYNC,
  specifically intended for that purpose.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1368815/+subscriptions