[Qemu-devel] [RFC] Static instrumentation (aka guest code tracing)

[Lluís]

Ok, sorry for the delay.

Here's a "report" on the current status. Please comment if you feel that any
decision has been taken through the wrong path. Also, if you send me patches
I'll happily push them into the repository.


Quick status summary
--------------------

 * minimal set of instrumentation points in place for testing (FETCH, VMEM)
 * examples at ./backdoor/examples and ./instrument/examples
 * code available at:
   https://projects.gso.ac.upc.edu/projects/qemu-instrument
   git clone https://code.gso.ac.upc.edu/git/qemu-instrument


How instrumentation currently works
-----------------------------------

Instrumentation points have the form of preprocessor macro calls. The user
defines each of these on a separate file (selected at configure time).

Each CPU has an "instrumentation state" variable, that can be dynamically
changed (e.g., by defining a backdoor that calls the instrumentation control
API) by the host. The number of states is defined by the user.

The macros are called at code generation time, where the user can check if a
specific instrumentation state is active on the current CPU (assuming
'cpu_single_env' points to the cpu object that originated the request for
disassembling the current instruction).

Changing the instrumentation state triggers TB flushes to allow for new
disassembly calls to take into account the new instrumentation state.

As of now, all CPUs must have the same state (see below).


What is lacking
---------------

1) immediate end of TB on backdoor instruction

I use backdoor instructions to control the instrumentation state from the guest,
triggering a call to a host-side code helper associated to the backdoor
instruction.

For this to work when controlling instrumentation state, the disassembly of an
instrumentation backdoor must immediately end the current TB.

The problem is that calling 'end_eob' (i386) produces code that infinitely
reexecutes that backdoor instruction.


2) instrumenting i386 is extremely time-consuming (for the developer)

As my work is not tied to a specific target architecture, I was thinking of
shifting into PPC, as the ISA is pretty regular and that would certainly make
the process easier by just patching a small set of places in the code.


3) per-CPU instrumentation state

The goal is to achieve minimal performance impact when executing TBs: no
instrumentation state checks when executing TBs (perform checks at TB generation
time), and negligible performance impact when executing non-instrumented TBs
(see if the modifications described below have no performance impact).

The original idea was to expand the arrays holding TBs ('tbs' and
'tb_phys_hash') into 2-dimensional arrays, where the first dimension would
contain one entry for each possible instrumentation state.

When a CPU looks up a TB, it is searched/added on the array for the current
state.

If the CPU-specific state changes, 'tb_jmp_cache' is flushed and lookups will
continue wherever they must according to the current state.

It is still unclear if PageDesc should also contain an array of 'first_tb' or if
'l1_map' should be 2-dimensional; I still have to look into that code in more
detail to see the feasibility and performance costs of each one.


4) KVM

I've performed tests only on i386-linux-user, but backdoor instructions and
calls to the instrumentation control API should switch from KVM to softmmu (and
disabling all instrumentation should jump back to KVM).

I dont' really know what would happen right now.


What needs to be decided
------------------------

1) instrumentation points

Which static instrumentation points must be present, and which arguments should
they have in order to have a target-agnostinc interface.

The current example points are:

FETCH(vaddress, size, used_registers, defined_registers)
VMEM(vaddress, size, read_or_write)


2) instrumentation from code helpers

It might be unavoidable the need to add a second set of calls to user-provided
macros to instrument from code helpers, as these must not generate code, but
call the user-provided instrumentation code helpers.

Another option would be to re-define INSTR_GEN_* into a plain function call to
the user macro.

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth