Re: [Qemu-devel] [PATCH 00/13] instrument: Add basic event instrumentation

[Lluís Vilanova]

Daniel P Berrange writes:

> On Fri, Jul 28, 2017 at 02:34:30PM +0100, Stefan Hajnoczi wrote:
>> On Thu, Jul 27, 2017 at 04:45:35PM +0100, Daniel P. Berrange wrote:
>> > On Thu, Jul 27, 2017 at 04:33:01PM +0100, Peter Maydell wrote:
>> > > On 27 July 2017 at 16:21, Daniel P. Berrange <address@hidden> wrote:
>> > > > On Thu, Jul 27, 2017 at 11:54:29AM +0100, Peter Maydell wrote:
>> > > >> That said, yes, I was going to ask if we could do this via
>> > > >> leveraging the tracepoint infrastructure and whatever scripting
>> > > >> facilities it provides. Are there any good worked examples of
>> > > >> this sort of thing? Can you do it as an ordinary non-root user?
>> > > >
>> > > > Do you have a particular thing you'd like to see an example of ?
>> > > >
>> > > > To dynamically probe a function which doesn't have a tracepoint
>> > > > defined you can do:
>> > > >
>> > > > probe process("/usr/bin/qemu-x86_64").function("helper_syscall") {
>> > > >   printf("syscall stasrt\n")
>> > > > }
>> > > >
>> > > > but getting access to the function args is not as easy as with
>> > > > pre-defined tracepoints.
>> > > 
>> > > How do I go about actually running that script? What I
>> > > have in mind by "worked example" is something like a blog
>> > > post that says "ok, here's a problem, we want to find out
>> > > what QEMU is doing in situation X, here's how you do this
>> > > with $TRACING_THINGY" and generally steps you through how
>> > > it works assuming you know nothing at all about whatever
>> > > the tracing facility you're using is.
>> > 
>> > Ok, so something like this example that I wrote for libvirt a
>> > while back then
>> > 
>> >   
>> > https://www.berrange.com/posts/2011/11/30/watching-the-libvirt-rpc-protocol-using-systemtap/
>> > 
>> > 
>> > > > You can't typically run this as root,
>> > > 
>> > > Do you mean "non-root" ?
>> > 
>> > Sigh, yes, of course.
>> > 
>> > > > however, I don't think that's a
>> > > > huge issue, because most QEMU deployments are not running as your own
>> > > > user account anyway, so you can't directly interact with them no
>> > > > matter what.
>> > > 
>> > > It is important, because almost all uses of TCG QEMU are
>> > > running it from the command line as non-root normal users,
>> > > especially if they're trying to debug what's going on with a
>> > > guest binary. So any tracing solution for this kind of usecase
>> > > must work without requiring root access, I think.
>> > 
>> > None of the Linux integrated tracing tools allow direct non-root access
>> > afaik. systemtap has ability to launch probes as non-root, via a privileged
>> > daemon, but it is restricted to probe scripts that the administrator has
>> > pre-defined.
>> 
>> One exception is gdb's static userspace probes support.  If you can run
>> gdb on QEMU then you can trace the same events as SystemTap.  I have
>> never tried this GDB feature:
>> 
>> https://sourceware.org/gdb/onlinedocs/gdb/Static-Probe-Points.html
>> 
>> It should work out of the box if your distro builds QEMU with the
>> 'dtrace' backend enabled.

> Wow, that's great to learn about. It does indeed work !

> If you knew alot about ptrace() you could probably build something
> that use ptrace() and these probe points to call your dynamic
> instrumentation code with reasonable low overheads.

I don't think so. Ptrace traps into the kernel and stops the process while a
separate process decides what to do. That's between 3 and 4 orders of magnitude
slower than calling an instrumentor function.


Cheers,
  Lluis