Re: [Qemu-devel] Tracing: outstanding tasks

[Stefan Hajnoczi]

On Wed, Jun 30, 2010 at 11:20 AM, Prerna Saxena
<address@hidden> wrote:
> On 06/26/2010 01:36 PM, Stefan Hajnoczi wrote:
>>
>> Here are the outstanding tasks for QEMU tracing, which Prerna and I have
>> been working on.  Tracing aids debugging, profiling, and observing
>> execution via lightweight logging at key points in the code path.
>>
>> The current prototype is available from the 'tracing' branch at:
>>
>> http://repo.or.cz/w/qemu/stefanha.git/shortlog/refs/heads/tracing
>>
>> This email is both to help Prerna and me focus our efforts, as well as a
>> roadmap for the QEMU community to discuss and comment on.
>>
>> Here are the outstanding tasks for a tracing patchset that can be
>> proposed for merge:
>>
>> 1. Integration with QMP
>>
>> Owner: Prerna
>>
>> Currently the trace commands are available from the monitor but a QMP
>> interface
>> is needed.
>>
>
> Agree. I'd suggest we let this be a TODO till the trace infrastructure makes
> its way upstream.
>
>> 2. More tracepoints need to be added for instrumenting other qemu
>> components
>> such as virtio drivers, etc.
>>
>> Owner: ?
>>
>> QEMU must come with a useful set of trace events that allows people to get
>> up
>> and running quickly.  Focus areas include:
>>
>>  * guest device emulation
>>  * host devices
>>  * lifecycle and runloop
>>  * memory management
>>  * live migration
>>
>> 3. Documentation
>>
>> Owner: Stefan
>>
>> User documentation that explains how to collect traces and add new trace
>> events.  I have committed documentation to the tracing branch here:
>>
>>
>> http://repo.or.cz/w/qemu/stefanha.git/blob_plain/91fde34dc6bfe01af6d5e9265f6a81535d6add15:/docs/tracing.txt
>>
>> 4. Fix i386-linux-user build
>>
>> Owner: Prerna
>>
>>   LINK  i386-linux-user/qemu-i386
>> ../simpletrace.o: In function `do_info_all_trace_events':
>> /home/stefanha/qemu/simpletrace.c:88: undefined reference to
>> `monitor_printf'
>> ../simpletrace.o: In function `do_info_trace':
>> /home/stefanha/qemu/simpletrace.c:77: undefined reference to
>> `monitor_printf'
>>
>
> I'm looking into this.
>
>> 5. Out-of-line trace file write-out
>>
>> Owner: Stefan
>>
>> Trace buffers are written out to file synchronously.  The vcpu thread
>> should
>> not be blocked so an async write-out mechanism is needed.
>>
>> 6. Trace file command
>>
>> Owner: ?
>>
>> Traces are written out to hardcoded /tmp/trace.log.  This must be
>> configurable.
>> Tracing at startup time should still be possible so configuration needs to
>> happen early.
>
> Agree, it is a good-to-have feature.
>
>>
>> 7. Binary trace format finalization
>>
>> Owner: Stefan
>>
>> We should leave room for extension.  I suggest partitioning the Event ID
>> namespace into normal events and special events.  The __trace_begin
>> special
>> event is defined to contain the file format version and/or trace record
>> size in
>> bytes as the first trace record in the file.
>>
>> This way, post-processing tools can check the format of the binary trace
>> file.
>>
>> 8. QMP/monitor command review
>>
>> Owner: Prerna, Stefan
>>
>>
>
> 9. In its present format, the tracing infrastructure causes at least two
> function calls even for trace events that are disabled. Ideally, there
> should be minimal performance overhead for disabled trace events.
> Investigating scope for further optimization.
>
> Owner : Prerna

Have you looked at the immediate value abstraction in LTTng?  I didn't
look very far, but I believe it abstracts how you enable/disable
breakpoints.  A simple immediate value implementation stores the state
in a variable like we do for the simple trace backend
(TraceEvent.state).  A "smarter" implementation is an unconditional
jump over the machine instructions that call into trace().  When the
state of a trace event is toggled, the jump instruction is patched.

That said, I think we should keep the "simple" trace backend simple as
long as possible.  Hopefully platform tracing mechanism like LTTng or
SystemTap will mature and become more widely supported so that we
don't need to re-invent all the hard parts of tracing.

Stefan