Re: [Qemu-devel] [PATCH 1/2] trace: Add simple tracing support

[Stefan Hajnoczi]

On Fri, May 21, 2010 at 5:52 PM, Jan Kiszka <address@hidden> wrote:
> I would just like to avoid that too much efforts are spent on
> re-inventing smart trace buffers, trace daemons, or trace visualization
> tools. Then better pick up some semi-perfect approach (e.g. [1], it
> unfortunately still seems to lack kernel integration) and drive it
> according to our needs.

I agree we have to consider existing solutions.  The killer is the
usability: what dependencies are required to build with tracing?  Is a
patched kernel or module required?  How easy is it to add static trace
events during debugging?

If there are too many dependencies, especially to unpackaged software,
many people will stop right there and not bother.  A patched kernel or
module isn't acceptable since the hassle of reconfiguring a system for
tracing becomes too great (or in some cases changing the kernel is not
possible/allowed).

Adding new static trace events should be easy, too.  Ideally it
doesn't require adding information about the trace event in multiple
places (header files, C files, etc).  It also shouldn't require
learning about the tracing system, adding a trace event should be
self-explanatory so anyone can easily add one for debugging.

A lot of opinions there, but what I'm saying is that friction must be
low.  If the tracing system is a pain to use, then no-one will use it.

http://lttng.org/files/ust/manual/ust.html

LTTng Userspace Tracer looks interesting - no kernel support required
AFAICT.  Toggling trace events in a running process supported.
Similar to kernel tracepoint.h and existing report/visualization tool.

x86 (32- and 64-bit) only.  Like you say, no correlation with kernel trace data.

I'll try to give LTTng UST a spin by converting my trace events to use
UST.  This seems closest to an existing tracing system we can drop in.

http://sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps

Requires kernel support - not sure if enough of utrace is in mainline
for this to work out-of-the-box across distros.

Unclear how exactly SystemTap userspace probing would work out.  Does
anyone have experience or want to try this?

Stefan