Re: [Qemu-devel] [PATCH 00/13] instrument: Add basic event instrumentation

[Daniel P. Berrange]

On Fri, Jul 28, 2017 at 02:34:30PM +0100, Stefan Hajnoczi wrote:
> On Thu, Jul 27, 2017 at 04:45:35PM +0100, Daniel P. Berrange wrote:
> > On Thu, Jul 27, 2017 at 04:33:01PM +0100, Peter Maydell wrote:
> > > On 27 July 2017 at 16:21, Daniel P. Berrange <address@hidden> wrote:
> > > > On Thu, Jul 27, 2017 at 11:54:29AM +0100, Peter Maydell wrote:
> > > >> That said, yes, I was going to ask if we could do this via
> > > >> leveraging the tracepoint infrastructure and whatever scripting
> > > >> facilities it provides. Are there any good worked examples of
> > > >> this sort of thing? Can you do it as an ordinary non-root user?
> > > >
> > > > Do you have a particular thing you'd like to see an example of ?
> > > >
> > > > To dynamically probe a function which doesn't have a tracepoint
> > > > defined you can do:
> > > >
> > > > probe process("/usr/bin/qemu-x86_64").function("helper_syscall") {
> > > >   printf("syscall stasrt\n")
> > > > }
> > > >
> > > > but getting access to the function args is not as easy as with
> > > > pre-defined tracepoints.
> > > 
> > > How do I go about actually running that script? What I
> > > have in mind by "worked example" is something like a blog
> > > post that says "ok, here's a problem, we want to find out
> > > what QEMU is doing in situation X, here's how you do this
> > > with $TRACING_THINGY" and generally steps you through how
> > > it works assuming you know nothing at all about whatever
> > > the tracing facility you're using is.
> > 
> > Ok, so something like this example that I wrote for libvirt a
> > while back then
> > 
> >   
> > https://www.berrange.com/posts/2011/11/30/watching-the-libvirt-rpc-protocol-using-systemtap/
> > 
> > 
> > > > You can't typically run this as root,
> > > 
> > > Do you mean "non-root" ?
> > 
> > Sigh, yes, of course.
> > 
> > > > however, I don't think that's a
> > > > huge issue, because most QEMU deployments are not running as your own
> > > > user account anyway, so you can't directly interact with them no
> > > > matter what.
> > > 
> > > It is important, because almost all uses of TCG QEMU are
> > > running it from the command line as non-root normal users,
> > > especially if they're trying to debug what's going on with a
> > > guest binary. So any tracing solution for this kind of usecase
> > > must work without requiring root access, I think.
> > 
> > None of the Linux integrated tracing tools allow direct non-root access
> > afaik. systemtap has ability to launch probes as non-root, via a privileged
> > daemon, but it is restricted to probe scripts that the administrator has
> > pre-defined.
> 
> One exception is gdb's static userspace probes support.  If you can run
> gdb on QEMU then you can trace the same events as SystemTap.  I have
> never tried this GDB feature:
> 
>   https://sourceware.org/gdb/onlinedocs/gdb/Static-Probe-Points.html
> 
> It should work out of the box if your distro builds QEMU with the
> 'dtrace' backend enabled.

Wow, that's great to learn about. It does indeed work !

If you knew alot about ptrace() you could probably build something
that use ptrace() and these probe points to call your dynamic
instrumentation code with reasonable low overheads.

> > That pretty much leaves re-building QEMU, LD_PRELOADS, or something
> > ptrace(), or qemu's built-in simpletrace feature, as the remaining
> > options.  We have a scripts/simpletrace.py that lets you load a
> > trace file into python and process it, but as written that's aimed
> > as post-processing a tracefile you've previously collected.
> > 
> > It would be desirable to write a more advanced simpletrace python
> > module that could collect & process the trace data live, and also
> > interact with the qemu monitor to change what events are enabled
> > dynamically.  Basically we'd need a way for the simpletrace backend
> > to output its data to a fifo, instead of creating an file on disk,
> > then you could dynanically consume it.
> 
> That would be interesting, I know Alex Bennee has wrangled with large
> (~10 GB?) simpletrace files and it's not a pleasant experience :).
> 
> Lluís/Peter: What are the requirements for instrumentation code
> interacting with the running QEMU instance?  simpletrace is
> asynchronous, meaning it does not wait for anyone handle the trace event
> before continuing execution, and is therefore not suitable for
> SystemTap-style scripts that can interact with the program while
> handling a trace event.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|