Re: [Qemu-devel] [PATCH v2 0/5] backdoor: lightweight guest-to-QEMU backdoor channel

[Lluís Vilanova]

Anthony Liguori writes:
>> Well, both backdoor and trace instrumentation are implemented using the same
>> approach (a static library selected at compile-time). The user sets which 
>> events
>> to instrument in the "trace-events" file. This has the effect that the 
>> tracetool
>> script will not generate the tracing functions for those events, and instead 
>> the
>> user must provide the implementation for these events on a static library
>> provided at compile time.

> I don't think this is the right approach to tracing.  What not just use
> something like SystemTap to implement logic around tracing?

> http://blog.vmsplice.net/2011/03/how-to-write-trace-analysis-scripts-for.html

This would only allow me to post-process all the traces, with no possible
interaction with qemy from the analysis library.

Compiling the analysis library into qemu itself lets me control when TCG code to
actually trace an instruction is generated (with guest code, trace events are
instrumented at both code translation and code execution time):

translate:
    gen_helper_trace_mem(...)
      -- trace instrumentation -->  ... if (...) 
gen_helper_trace_mem_backend(...); ...
      -- otherwise             -->  gen_helper_trace_mem_backend(...);

execute:
    helper_trace_mem(...)
      trace_mem(...)
        -- trace instrumentation --> ... whatever ...
        -- otherwise             --> trace_fetch_backend(...);

One could argue that gen_helper_trace_mem can just check some in-qemu central
structure to see if it must generate the call to helper_trace_mem, but then the
best you can do is an all-or-nothing on a per-event basis.

You must have in mind that we're talking about an extreme tracing bandwidth
here, so the analyzer is willing to filter-out as many events as soon as
possible (e.g., just tracing a specific set of IPs can be optimized at
translation time, and tracing instructions using specific registers can be
optimized by generating extra checks with TCG). This requires some sort of
in-line communication between qemu and the analyzer.

Still, I tried to build this into perfectly separate and independent features:

  * Support for tracing guest code through TCG.
  * Support for different sets of enabled guest tracing events on a per-vCPU
    basis.
  * Support for instrumenting tracing events (including those for guest code).


>> The analyses one might want to perform are ad-hoc in nature. For example, 
>> some
>> recent work sent here did coverage analyses through ad-hoc modifications on 
>> the
>> core of qemu, I want to generate detailed traces of the guest code that is
>> executed (although only when certain conditions in the guest are met). There 
>> is
>> no silver bullet here, but the development of analyzers can be simplified by
>> letting users perform static instrumentation of the tracing events, which 
>> act as
>> regular tracing events otherwise.
>> 
>> In this environment, the backdoor is just a handy channel to let the guest
>> communicate with code on that analyzer library, like signalling semanticful
>> events of interest (e.g., I have a trivial linux kernel module that installs 
>> a
>> tracepoint callback in the process switch routine and passes that information
>> down to the analyzer library through the backdoor channel).
>> 
>> As these analyzes are ad-hoc in nature, I don't see any way other than 
>> providing
>> this extension functionality.

> But the effect is that the infrastructure is useless without out-of-tree code
> which means it's impossible to test.  Furthermore, when the ad-hoc analyzer
> library breaks because of an API change in QEMU, it puts us in a very 
> difficult
> situation.

The infrastructure itself is pretty simple (the least trivial part is code in
tracetool), and can be tested by having a testing library that can be shipped
with qemu, as well as can be used to test qemu itself.

I also think that the point of breaking third-party analyzers is less important
if the access to TCG code generation and the tracing events are enough for most
of them (the TCG API is pretty stable). Some analyzers might want to access
register values or something else, but they would still do that even without
instrumentation through ad-hoc changes in a rapidly aging version of qemu. My
only intent is to ease the effort and the performance costs of developing such
analyzers.


>> Does it make sense as a whole? I thought from previous mails that being able 
>> to
>> trace and analyze guest code was something perceived as a useful feature to 
>> have
>> in qemu.

> I think there's wide agreement that extending tracing to guests is very useful
> and something we should do.

> I don't think there's agreement that we should have a mechanism to pull in
> third-party code to implement the logic for these trace hooks within QEMU.

> The right way to do analysis is via something like SystemTap.

Well, as I tried to explain above, this comes at the expense of performance, as
the analyzer is not inlined into qemu.


Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth