Re: [Qemu-devel] [RFC] Propose the Fast Virtual Disk (FVD) image format that outperforms QCOW2 by 249%

[Chunqiang Tang]

> The community block I/O test suite is qemu-iotests:
> http://git.kernel.org/?p=linux/kernel/git/hch/qemu-iotests.git;a=summary
> If you have tests that you'd like to contribute, please put them into
> that framework so other developers can run them as part of their
> regular testing.

Hi Stefan,

What I described is not a qemu-io test case. I also use qemu-io, which is 
very helpful, but I observed that qemu-io has several limitations in 
discovering elusive bugs:

B1) qemu-io cannot trigger many race condition bugs, because it does not 
fully control the timing of events. For example, qemu-io cannot test this 
scenario: three concurrent writes a, b, and c are processed by 
bdrv_aio_writev() in the order of Pa, Pb, and Pc; their writes are 
actually persisted on disk in another order of Wc, Wa, and Wb; and finally 
their callbacks are invoked in yet another order of Vb, Vc, and Va. Some 
race condition bugs may exist in the code (e.g., inappropriate locking), 
because it does not anticipate these orders of events are possible. This 
is just one example. In theory, there can be 100 concurrent reads or 
writes, and their events can happen in an arbitrary permutation order. It 
is nearly impossible to manually generating test cases for all of them.

B2) Even if a race condition bug is triggered by chance, its behavior 
depends on subtle event timing that is hard to repeat and hence hard to 
debug. 

B3) With qemu-io, it is hard to test code paths that handle I/O failures. 
For example, a disk write may fail due to disk media error. Because these 
errors are rare, the failure handling code paths may never be tested, 
which for example may contain a null pointer bug that can crash the entire 
VM or gradually leaks resources (e.g., memory) due to incomplete cleanup.

B4) qemu-io requires manually creating test cases, which is not only time 
consuming but also leads a low coverage in testing. This is because many 
bugs happen in scenarios that the developers do not anticipate, and hence 
do not know how to create test cases in the first place. 

The FVD patch includes a new testing framework that addresses the above 
issues. This testing framework is orthogonal to FVD and can be used to 
test other block device drivers as well. This testing framework includes 
two components that can be used both separately and in a combination

T1) To address the problems of B1- B3, I implemented an emulated disk in 
block/sim.c, which allows a full control of event timings, either manually 
or automatically. Given the three concurrent writes example above, their 9 
events (Pa, Pb, Pc, Wa, Wb, Wc, Va, Vb, and Vc) can be precisely 
controlled to be executed in any given order. Moreover, the emulated disk 
can inject disk I/O errors in a controlled manner. For example, it can 
fail a specific read or write to test how the code handles that, or it can 
even fail as many as 90% of the reads/writes to test if the code has 
resource leaks. qemu-io is extended with a module qemu-io-sim.c to work 
with the emulated disk block/sim.c, so that the tester can use the qemu-io 
console to manually control the order of events or fail disk reads or 
writes.

T2) The solution in T1 still does not address the problem of B3), i.e., 
manually generating test cases is time consuming and has a low coverage. 
This problem is solved by a new testing tool called qemu-test. qemu-test 
can 1) automatically generate an unlimited number of randomized test cases 
that, e.g., execute 1,000 concurrent disk reads or writes on overlapping 
disk regions; 2) automatically generate the corresponding anticipated 
correct results, automatically run the tests, and automatically compare 
the actual test results with the anticipated correct results. Once it 
discovers a difference, which indicates a bug, it halts testing and waits 
for the developer to debug.  The randomized test cases created by 
qemu-test are controlled by a pseudo random number generator, and hence 
the behavior is completely repeatable. Therefore, once a bug is triggered, 
it can be precisely repeated for an unlimited number of times to 
facilitate debugging, even if this bug happens extremely rare in real runs 
of a VM. qemu-test is fully automated. Once started, it can continuously 
run, e.g., for months to test an enormous number of test cases.

The implementation of qemu-test is actually not that complicated. It opens 
two virtual disks, the so-called truth image and test image, respectively. 
The truth image is served by a trivial synchronous block device driver so 
that its behavior is guaranteed to be correct. The test image is served a 
real block device driver (e.g., FVD or QCOW2) that we want to test. 
qemu-test submits the same sequence of disk I/O requests (which is 
randomly generated) to the truth image and the test image, and expect that 
the two images’ contents never diverge. Otherwise, it indicates a bug in 
the test image’s block device driver. qemu-test works with the emulated 
disk block/sim.c so that it can randomize event timings in a controlled 
manner and can inject disk I/O errors randomly. 

I found qemu-test extremely powerful in discovering elusive bugs that I 
never anticipated, and using qemu-test is effortless. Whenever I completed 
some major code upgrade, I simply started qemu-test in the evening and 
came back in the morning to collect bugs, if any. Debugging them is also 
easy because the bugs are precisely repeatable even if they are hard to 
trigger.

As for the QCOW2 bug I mentioned previously, it can be triggered by 
test-qcow2.sh. A faster way to trigger it is to bypass those correct test 
runs by executing the commands below:

dd if=/dev/zero of=/var/ramdisk/truth.raw count=0 bs=1 seek=1155683840
dd if=/dev/zero of=/var/ramdisk/zero-500M.raw count=0 bs=1 seek=609064448
./qemu-img create -f qcow2 -b /var/ramdisk/zero-500M.raw 
/var/ramdisk/test.qcow2 1155683840
./qemu-test --seed=116579177 --truth=/var/ramdisk/truth.raw 
--test=/var/ramdisk/test.qcow2 --verify_write=true --compare_before=false 
--compare_after=true --round=100000 --parallel=100 --io_size=10485760 
--fail_prob=0 --cancel_prob=0 --instant_qemubh=true

As for the FVD patch that includes the new testing framework, I tried to 
post it on the mailing list twice but it always got bounced back, either 
because the message is too big or because of a Notes client configuration 
issue. Until I figure it out, please down the FVD patch from 
https://researcher.ibm.com/researcher/files/us-ctang/FVD-01-14-2011.patch 
.

Best regards,
ChunQiang (CQ) Tang, Ph.D.
Homepage: http://www.research.ibm.com/people/c/ctang