Re: Race condition in overlayed qcow2?

[dovgaluk]

Vladimir Sementsov-Ogievskiy писал 2020-02-21 13:09:
> 21.02.2020 12:49, dovgaluk wrote:
> > Vladimir Sementsov-Ogievskiy писал 2020-02-20 12:36:
> > > > > 1 or 2 are ok, and 4 or 8 lead to the failures.
> > > > >
> > > > >
> > > > > That is strange. I could think, that it was caused by the bugs in
> > > > > deterministic CPU execution, but the first difference in logs
> > > > > occur in READ operation (I dump read/write buffers in 
> > > > > blk_aio_complete).
> > > >
> > > > Aha, yes, looks strange.
> > > >
> > > > Then next steps:
> > > >
> > > > 1. Does problem hit into the same offset every time?
> > > > 2. Do we write to this region before this strange read?
> > > >
> > > > 2.1. If yes, we need to check that we read what we write.. You say 
> > > > you dump buffers
> > > > in blk_aio_complete... I think it would be more reliable to dump at 
> > > > start of
> > > > bdrv_co_pwritev and at end of bdrv_co_preadv. Also, guest may modify 
> > > > its buffers
> > > > during operation which would be strange but possible.
> > > >
> > > > 2.2 If not, hmm...
> > >
> > > Another idea to check: use blkverify
> >
> > I added logging of file descriptor and discovered that different 
> > results are obtained
> > when reading from the backing file.
> > And even more - replay runs of the same recording produce different 
> > results.
> > Logs show that there is a preadv race, but I can't figure out the 
> > source of the failure.
> >
> > Log1:
> > preadv c 30467e00
> > preadv c 30960000
> > --- sum = a2e1e
> > bdrv_co_preadv_part complete offset: 30467e00 qiov_offset: 0 len: 8200
> > --- sum = 10cdee
> > bdrv_co_preadv_part complete offset: 30960000 qiov_offset: 8200 len: 
> > ee00
> >
> > Log2:
> > preadv c 30467e00
> > --- sum = a2e1e
> > bdrv_co_preadv_part complete offset: 30467e00 qiov_offset: 0 len: 8200
> > preadv c 30960000
> > --- sum = f094f
> > bdrv_co_preadv_part complete offset: 30960000 qiov_offset: 8200 len: 
> > ee00
> >
> >
> > Checksum calculation was added to preadv in file-posix.c
>
> So, preadv in file-posix.c returns different results for the same
> offset, for file which is always opened in RO mode? Sounds impossible
> :)

True.
Maybe my logging is wrong?

static ssize_t
qemu_preadv(int fd, const struct iovec *iov, int nr_iov, off_t offset)
{
    ssize_t res = preadv(fd, iov, nr_iov, offset);
    qemu_log("preadv %x %"PRIx64"\n", fd, (uint64_t)offset);
    int i;
    uint32_t sum = 0;
    int cnt = 0;
    for (i = 0 ; i < nr_iov ; ++i) {
        int j;
        for (j = 0 ; j < (int)iov[i].iov_len ; ++j)
        {
            sum += ((uint8_t*)iov[i].iov_base)[j];
            ++cnt;
        }
    }
    qemu_log("size: %x sum: %x\n", cnt, sum);
    assert(cnt == res);
    return res;
}

This code prints preadv checksum.
But when I calculate the same with the standalone program, then it gives 
me another values of the checksums for the same offsets:

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/uio.h>

unsigned char buf[0x100000];

int main(int argc, char **argv)
{
  if (argc < 4) return 1;
  int f = open(argv[1], O_RDONLY);
  unsigned int cnt;
  unsigned int offs;
  sscanf(argv[2], "%x", &offs);
  sscanf(argv[3], "%x", &cnt);
  printf("file: %s offset: %x size: %x\n", argv[1], offs, cnt);
  struct iovec iov = {buf, (size_t)cnt};
  size_t sz = preadv(f, &iov, 1, offs);
  printf("read %x\n", (int)sz);
  int i;
  unsigned int sum = 0;
  for (i = 0 ; i < cnt ; ++i)
    sum += buf[i];
  printf("sum = %x\n", sum);
}



Pavel Dovgalyuk