Vladimir Sementsov-Ogievskiy писал 2020-02-21 13:09:
21.02.2020 12:49, dovgaluk wrote:
Vladimir Sementsov-Ogievskiy писал 2020-02-20 12:36:
1 or 2 are ok, and 4 or 8 lead to the failures.
That is strange. I could think, that it was caused by the bugs in
deterministic CPU execution, but the first difference in logs
occur in READ operation (I dump read/write buffers in blk_aio_complete).
Aha, yes, looks strange.
Then next steps:
1. Does problem hit into the same offset every time?
2. Do we write to this region before this strange read?
2.1. If yes, we need to check that we read what we write.. You say you dump
buffers
in blk_aio_complete... I think it would be more reliable to dump at start of
bdrv_co_pwritev and at end of bdrv_co_preadv. Also, guest may modify its buffers
during operation which would be strange but possible.
2.2 If not, hmm...
Another idea to check: use blkverify
I added logging of file descriptor and discovered that different results are
obtained
when reading from the backing file.
And even more - replay runs of the same recording produce different results.
Logs show that there is a preadv race, but I can't figure out the source of the
failure.
Log1:
preadv c 30467e00
preadv c 30960000
--- sum = a2e1e
bdrv_co_preadv_part complete offset: 30467e00 qiov_offset: 0 len: 8200
--- sum = 10cdee
bdrv_co_preadv_part complete offset: 30960000 qiov_offset: 8200 len: ee00
Log2:
preadv c 30467e00
--- sum = a2e1e
bdrv_co_preadv_part complete offset: 30467e00 qiov_offset: 0 len: 8200
preadv c 30960000
--- sum = f094f
bdrv_co_preadv_part complete offset: 30960000 qiov_offset: 8200 len: ee00
Checksum calculation was added to preadv in file-posix.c
So, preadv in file-posix.c returns different results for the same
offset, for file which is always opened in RO mode? Sounds impossible
:)
True.
Maybe my logging is wrong?
static ssize_t
qemu_preadv(int fd, const struct iovec *iov, int nr_iov, off_t offset)
{
ssize_t res = preadv(fd, iov, nr_iov, offset);
qemu_log("preadv %x %"PRIx64"\n", fd, (uint64_t)offset);
int i;
uint32_t sum = 0;
int cnt = 0;
for (i = 0 ; i < nr_iov ; ++i) {
int j;
for (j = 0 ; j < (int)iov[i].iov_len ; ++j)
{
sum += ((uint8_t*)iov[i].iov_base)[j];
++cnt;
}
}
qemu_log("size: %x sum: %x\n", cnt, sum);
assert(cnt == res);
return res;
}
This code prints preadv checksum.
But when I calculate the same with the standalone program, then it gives me
another values of the checksums for the same offsets:
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/uio.h>
unsigned char buf[0x100000];
int main(int argc, char **argv)
{
if (argc < 4) return 1;
int f = open(argv[1], O_RDONLY);
unsigned int cnt;
unsigned int offs;
sscanf(argv[2], "%x", &offs);
sscanf(argv[3], "%x", &cnt);
printf("file: %s offset: %x size: %x\n", argv[1], offs, cnt);
struct iovec iov = {buf, (size_t)cnt};
size_t sz = preadv(f, &iov, 1, offs);
printf("read %x\n", (int)sz);
int i;
unsigned int sum = 0;
for (i = 0 ; i < cnt ; ++i)
sum += buf[i];
printf("sum = %x\n", sum);
}