Re: [Qemu-devel] [RFC PATCH 00/21] Trace updates and plugin RFC

[Alex Bennée]

Emilio G. Cota <address@hidden> writes:

> On Fri, Oct 05, 2018 at 16:48:49 +0100, Alex Bennée wrote:
> (snip)
>> ==Known Limitations==
>>
>> Currently there is only one hook allowed per trace event. We could
>> make this more flexible or simply just error out if two plugins try
>> and hook to the same point. What are the expectations of running
>> multiple plugins hooking into the same point in QEMU?
>
> It's very common. All popular instrumentation tools (e.g. PANDA,
> DynamoRIO, Pin) support multiple plugins.

Fair enough.

>
>> ==TCG Hooks==
>>
>> Thanks to Lluís' work the trace API already splits up TCG events into
>> translation time and exec time events and provides the machinery for
>> hooking a trace helper into the translation stream. Currently that
>> helper is unconditionally added although perhaps we could expand the
>> call convention a little for TCG events to allow the translation time
>> event to filter out planting the execution time helper?
>
> A TCG helper is suboptimal for these kind of events, e.g. instruction/TB/
> mem callbacks, because (1) these events happen *very* often, and
> (2) a helper then has to iterate over a list of callbacks (assuming
> we support multiple plugins). That is, one TCG helper call,
> plus cache misses for the callback pointers, plus function calls
> to call the callbacks. That adds up to 2x average slowdown
> for SPEC06int, instead of 1.5x slowdown when embedding the
> callbacks directly into the generated code. Yes, you have to
> flush the code when unsubscribing from the event, but that cost
> is amortized by the savings you get when the callbacks occur,
> which are way more frequent.

What would you want instead of a TCG helper? But certainly being able be
selective about which instances of each trace point are instrumented
will be valuable.

> Besides performance, to provide a pleasant plugin experience we need
> something better than the current tracing callbacks.

What I hope to avoid in re-using trace points is having a whole bunch of
a additional hook points just for plugins. However nothing stops us
adding more tracepoints at more useful places for instrumentation. We
could also do it on a whitelist basis similar to the way the tcg events
are marked.

>
>> ===Instruction Tracing===
>>
>> Pavel's series had a specific hook for instrumenting individual
>> instructions. I have not yet added it to this series but I think it be
>> done in a slightly cleaner way now we have the ability to insert TCG
>> ops into the instruction stream.
>
> I thought Peter explicitly disallowed TCG generation from plugins.
> Also, IIRC others also mentioned that exposing QEMU internals
> (e.g. "struct TranslationBlock", or "struct CPUState") to plugins
> was not on the table.

We definitely want to avoid plugin controlled code generation but the
tcg tracepoint mechanism is transparent to the plugin itself. I think
the pointers should really be treated as anonymous handles rather than
windows into QEMU's internals. Arguably some of the tracepoints should
be exporting more useful numbers (I used cpu->cpu_index in the TLB trace
points) but I don't know if we can change existing trace point
definitions to clean that up.

Again if we whitelist tracepoints for plugins we can be more careful
about the data exported.

>
>> If we add a tracepoint for post
>> instruction generation which passes a buffer with the instruction
>> translated and method to insert a helper before or after the
>> instruction. This would avoid exposing the cpu_ldst macros to the
>> plugins.
>
> Again, for performance you'd avoid the tracepoint (i.e. calling
> a helper to call another function) and embed directly the
> callback from TCG. Same thing applies to TB's.

OK I see what you mean. I think that is doable although it might take a
bit more tcg plumbing.

>
>> So what do people think? Could this be a viable way to extend QEMU
>> with plugins?
>
> For frequent events such as the ones mentioned above, I am
> not sure plugins can be efficiently implemented under
> tracing.

I assume some form of helper-per-instrumented-event/insn is still going
to be needed though? We are not considering some sort of EBF craziness?

> For others (e.g. cpu_init events), sure, they could.
> But still, differently from tracers, plugins can come and go
> anytime, so I am not convinced that merging the two features
> is a good idea.

I don't think we have to mirror tracepoints and plugin points but I'm in
favour of sharing the general mechanism and tooling rather than having a
whole separate set of hooks. We certainly don't want anything like:

          trace_exec_tb(tb, pc);
          plugin_exec_tb(tb, pc);

scattered throughout the code where the two do align.

>
> Thanks,
>
>               Emilio


--
Alex Bennée