[Qemu-devel] [PATCH V2 1/6] docs: Specification for the image fuzzer

[Maria Kustova]

'Overall fuzzer requirements' chapter contains the current product vision and
features done and to be done. This chapter is still in progress.

v1 -> v2:
 * added description for fuzzer configuration files
 * added the example fuzzer.conf file
 * various fixes based on the Eric's review

Signed-off-by: Maria Kustova <address@hidden>
---
 tests/image-fuzzer/docs/fuzzer.conf      |  37 ++++++
 tests/image-fuzzer/docs/image-fuzzer.txt | 216 +++++++++++++++++++++++++++++++
 2 files changed, 253 insertions(+)
 create mode 100644 tests/image-fuzzer/docs/fuzzer.conf
 create mode 100644 tests/image-fuzzer/docs/image-fuzzer.txt

diff --git a/tests/image-fuzzer/docs/fuzzer.conf 
b/tests/image-fuzzer/docs/fuzzer.conf
new file mode 100644
index 0000000..856cec7
--- /dev/null
+++ b/tests/image-fuzzer/docs/fuzzer.conf
@@ -0,0 +1,37 @@
+# Python version < 2.7 doesn't support empty options
+# Option values will not be interpreted
+[header]
+magic=True
+version=True
+backing_file_offset=True
+backing_file_size=True
+cluster_bits=True
+size=True
+crypt_method=True
+l1_size=True
+l1_table_offset=True
+refcount_table_offset=True
+refcount_table_clusters=True
+nb_snapshots=True
+snapshots_offset=True
+incompatible_features=True
+compatible_features=True
+autoclear_features=True
+refcount_order=True
+header_length=True
+
+[backing_file_format_ext]
+ext_magic=True
+ext_length=True
+bf_data=True
+
+[feature_name_table_ext]
+ext_magic=True
+ext_length=True
+feat_type=True
+feat_bit_number=True
+feat_name=True
+
+[end_ext]
+ext_magic=True
+ext_length=True
\ No newline at end of file
diff --git a/tests/image-fuzzer/docs/image-fuzzer.txt 
b/tests/image-fuzzer/docs/image-fuzzer.txt
new file mode 100644
index 0000000..0ebd0f7
--- /dev/null
+++ b/tests/image-fuzzer/docs/image-fuzzer.txt
@@ -0,0 +1,216 @@
+Image fuzzer
+============
+
+Description
+-----------
+
+The goal of the image fuzzer is to catch crashes of qemu-io/qemu-img
+by providing to them randomly corrupted images.
+Test images are generated from scratch and have valid inner structure with some
+elements, e.g. L1/L2 tables, having random invalid values.
+
+
+Test runner
+-----------
+
+The test runner generates test images, executes tests utilizing generated
+images, indicates their results and collects all test related artifacts (logs,
+core dumps, test images).
+The test means one start of a system under test (SUT), e.g. qemu-io, with
+specified arguments and one test image.
+By default, the test runner generates new tests and executes them until
+keyboard interruption. But if a test seed is specified via the '-s' runner
+parameter, then only one test with this seed will be executed, after its finish
+the runner will exit.
+
+The runner uses an external image fuzzer to generate test images. An image
+generator should be specified as a mandatory parameter of the test runner.
+Details about interactions between the runner and fuzzers see "Module
+interfaces".
+
+The runner activates generation of core dumps during test executions, but it
+assumes that core dumps will be generated in the current working directory.
+For comprehensive test results, please, set up your test environment
+properly.
+
+Path to a SUT can be specified via the '--binary' parameter of the test runner.
+If no SUT is provided, then the runner tries to get it from environment
+variables (now only qemu_img).If the environment check fails the runner will
+use qemu-img installed in system paths.
+For details about environment variables see qemu-iotests/check.
+
+The runner accepts files with lists of fields expected to be fuzzed. The file
+has the same structure as regular *.conf or *.ini file. Complex image elements
+having multiple fields should be specified as sections, individual fields as
+options for the sections.
+Example,
+        [header]
+        l1_table_offset=True
+        nb_snapshots=True
+
+        [feature_name_table_ext]
+
+If all fields in a section are expected to be fuzzed, leave the section empty.
+Option values play no role and added just for compatibility.
+The full list of supported fields are available in
+image-fuzzer/docs/fuzzer.conf.
+
+
+Qcow2 image generator
+---------------------
+
+The 'qcow2' generator is a Python package providing 'create_image' method as
+a single public API. See details in 'Test runner/image fuzzer' in 'Module
+interfaces'.
+
+Qcow2 contains two submodules: fuzz.py and layout.py.
+
+'fuzz.py' contains all fuzzing functions, one per image field. It's assumed
+that
+after code analysis every field will have own constraints for its value. For
+now only universal potentially dangerous values are used, e.g. type limits for
+integers or unsafe symbols as '%s' for strings. For bitmasks random amount of
+bits are set to ones. All fuzzed values are checked on non-equality to the
+current valid value of the field. In case of equality the value will be
+regenerated.
+
+'layout.py' creates a random valid image, fuzzes a random subset of the image
+fields by 'fuzz.py' module and writes a fuzzed image to the file specified.
+If a fuzzer configuration is specified, then it has the next interpretation:
+
+    1. If a section has no options, then some random portion of fields of
+    the image element specified as the section name will be fuzzed every test.
+    The same behavior is applied for the entire image if no configuration is
+    used. This case is useful for the test specialization.
+
+    2. When a section has options, then all fields specified in these options
+    are always fuzzed for every test. This case is useful for regression
+    testing.
+
+For now only header fields and header extensions are generated, the remaining
+file is filled with zeros.
+
+
+Module interfaces
+-----------------
+
+* Test runner/image fuzzer
+
+The runner calls an image generator specifying the path to a test image file.
+An image generator is expected to provide a
+   'create_image(test_img_path, fuzz_config=None)'
+method that creates a test image and writes it to the specified file. The file
+should be created if it doesn't exist or overwritten otherwise.
+'fuzz_config' has a form of a list of lists. Every sublist can have one
+or two elements: the first element is a name of a parent image element,
+the second one if exists is a name of a field in this element.
+Example,
+        [['header', 'l1_table_offset'],
+         ['header', 'nb_snapshots'],
+         ['feature_name_table_ext']]
+
+The full list of available names can be found in docs/fuzzer.conf.
+
+Random seed is set by the runner at every test execution for the regression
+purpose, so an image generator is not recommended to modify it internally.
+
+* Test runner/SUT
+
+A full test command is composed from the SUT, its arguments specified
+via '-c' runner parameter and the name of generated image. To indicate where
+a test image name should be placed TEST_IMG placeholder can be used.
+For example, for the next test command
+
+ qemu-img convert -O vmdk fuzzed_image test.vmdk
+
+a call of the image fuzzer will be following:
+
+ ./runner.py -b qemu-img -c 'convert -O vmdk TEST_IMG test.vmdk' work_dir qcow2
+
+
+Overall fuzzer requirements
+===========================
+
+Input data:
+----------
+
+ - image template (generator)
+ - work directory
+ - test run duration (optional)
+ - action vector (optional)
+ - seed (optional)
+ - SUT and its arguments (optional)
+
+
+Fuzzer requirements:
+-------------------
+
+1.  Should be able to inject random data
+2.  Should be able to select a random value from the manually pregenerated
+    vector (boundary values, e.g. max/min cluster size)
+3.  Image template should describe a general structure invariant for all
+    test images (image format description)
+4.  Image template should be autonomous and other fuzzer parts should not
+    rely on it
+5.  Image template should contain reference rules (not only block+size
+    description)
+6.  Should generate the test image with the correct structure based on an image
+    template
+7.  Should accept a seed as an argument (for regression purpose)
+8.  Should generate a seed if it is not specified as an input parameter.
+9.  The same seed should generate the same image for the same action vector,
+    specified or generated.
+10. Should accept a vector of actions as an argument (for test reproducing and
+    for test case specification, e.g. group of tests for header structure,
+    group of test for snapshots, etc)
+11. Action vector should be randomly generated from the pool of available
+    actions, if it is not specified as an input parameter
+12. Pool of actions should be defined automatically based on an image template
+13. Should accept a SUT and its call parameters as an argument or select them
+    randomly otherwise. As far as it's expected to be rarely changed, the list
+    of all possible test commands can be available in the test runner
+    internally.
+14. Should accept a test run duration as an argument. Tests should be executed
+    during a minimum period from a test run duration and time while fuzzer can
+    generate different test images
+15. Should support an external cancellation of a test run
+16. Seed and action vector should be logged (for regression purpose)
+17. All files related to test result should be collected: a test image,
+    SUT logs, fuzzer logs and crash dumps
+18. Should be compatible with python version 2.4-2.7
+19. Usage of external libraries should be limited as much as possible.
+
+
+Image formats:
+-------------
+
+Main target image format is qcow2, but support of image templates should
+provide an ability to add any other image format.
+
+
+Effectiveness:
+-------------
+
+Fuzzer can be controlled via template, seed and action vector;
+it makes the fuzzer itself invariant to an image format and test logic.
+It should be able to perform rather complex and precise tests, that can be
+specified via action vector. Otherwise, knowledge about an image structure
+allows the fuzzer to generate the pool of all available areas can be fuzzed
+and randomly select some of them and so compose its own action vector.
+Also complexity of a template defines complexity of the fuzzer, so its
+functionality can be varied from simple model-independent fuzzing to smart
+model-based one.
+
+
+Glossary:
+--------
+
+Action vector is a sequence of structure elements retrieved from an image
+format, each of them will be fuzzed for the test image. It's a subset of
+elements of the action pool. Example: header, refcount table, etc.
+Action pool is all available elements of an image structure that generated
+automatically from an image template.
+Image template is a formal description of an image structure and relations
+between image blocks.
+Test image is an output image of the fuzzer defined by the current seed and
+action vector.
-- 
1.9.3