[PATCH for 4.2 v5 00/55] Support for TCG plugins

[Alex Bennée]

Hi,

This is the latest iteration of the TCG plugins series. From the documentation:

  QEMU TCG plugins provide a way for users to run experiments taking
  advantage of the total system control emulation can have over a guest.
  It provides a mechanism for plugins to subscribe to events during
  translation and execution and optionally callback into the plugin
  during these events. TCG plugins are unable to change the system state
  only monitor it passively. However they can do this down to an
  individual instruction granularity including potentially subscribing
  to all load and store operations.

I think I've addressed most of the comments from the v4 posting. I've
made a couple of changes to API which I intend to keep as separate
patches in the series to aid review. They are:

  qemu_plugin_install - new info block
  qemu_plugin_outs - output a string via QEMU's log system

The first allows plugins to know about the guest architecture - in a
fairly simple way by exposing TARGET_NAME. Its enough that a plugin
could decide not to install anything if it is not what it is expecting
but not so much that it has detailed information about the target. For
that sort of subtly you will have to pass details down via the plugin
options. I did briefly look at exposing ElfMachine information but
that got too gnarly. The only user in the examples is the howvec
plugin which will now run on all architectures although it can only
classify aarch64 and sparc64 instructions into groups.

The ability of the plugins to be fully "multiarch" aware meant I could
enable check-tcg to run most tests with each available plugin. This
proved very effective at exposing some bugs in the code that didn't
show up on my main test architecture (ARM). I can still trigger
crashes in sparc64-linux-user but rth informs me it is currently quite
broken so I skip it for now in the CI config.

The qemu_plugin_outs patch is mostly mechanical but it does neaten up
the output of plugins and prevent them just spamming stdout. Currently
the output goes via the logging system but I can envisage future
tweaks for system emulation which could push the output via a chardev
and allow nicer integration with test harnesses.

One requested feature that didn't make it in was a way to introspect
the state of the registers. It's not a deal breaker as the plugin can
always decode the instruction itself and has access to all
loads/stores so could track the data itself. I didn't want to hold up
the merging of what we have given softfreeze is so close so I think
this is something to think about for the next cycle. I'm open to ideas
of the best way to do this. One idle thought I had was giving the
plugin the ability to trigger a debug exception which would then punt
the problem to the gdbstub. I'm not sure if triggering a non-guest
visible exception violates the passive monitoring contract of TCG
plugins.

I've added some more words to the documentation to hopefully ally
fears about TCG plugins being used to end-run around the GPL.

The only other change of note is fixing a bunch of races by using
__thread for per-CPU data structures. There are more details bellow
the --- comments in each patch.

I'm hoping to get it merged this cycle so please review:

  02 - trace add mmu_index to mem_info
  17 - plugins implement helpers for resolving hwaddr
  40 - tests tcg Makefile.target fix path to config host
  41 - tests tcg set QEMU_OPTS for all cris runs
  42 - tests tcg move virtual tests to EXTRA_TESTS
  43 - tests tcg drop test i386 fprem from TESTS when no
  44 - tests tcg enable plugin testing
  45 - tests plugin add a hotblocks plugin
  47 - tests plugin add instruction execution breakdown
  48 - tests plugin add hotpages plugin to breakdown mem
  51 - plugins expand the plugin_init function to includ
  52 - plugins make howvec plugin more generic
  53 - plugins add sparc64 instruction classification ta
  54 - plugins add qemu_plugin_outs and use it
  55 - .travis.yml add enable plugins tests

Alex Bennée (21):
  trace: add mmu_index to mem_info
  docs/devel: add plugins.rst design document
  configure: add --enable-plugins (MOVE TO END)
  plugin: add implementation of the api
  plugins: implement helpers for resolving hwaddr
  tests/tcg/Makefile.target: fix path to config-host.mak
  tests/tcg: set QEMU_OPTS for all cris runs
  tests/tcg: move "virtual" tests to EXTRA_TESTS
  tests/tcg: drop test-i386-fprem from TESTS when not SLOW
  tests/tcg: enable plugin testing
  tests/plugin: add a hotblocks plugin
  plugin: add qemu_plugin_insn_disas helper
  tests/plugin: add instruction execution breakdown
  tests/plugin: add hotpages plugin to breakdown memory access patterns
  accel/stubs: reduce headers from tcg-stub
  include/exec: wrap cpu_ldst.h in CONFIG_TCG
  plugins: expand the plugin_init function to include an info block
  plugins: make howvec plugin more generic
  plugins: add sparc64 instruction classification table
  plugins: add qemu_plugin_outs and use it
  .travis.yml: add --enable-plugins tests

Emilio G. Cota (32):
  trace: expand mem_info:size_shift to 4 bits
  cpu: introduce cpu_in_exclusive_context()
  translate-all: use cpu_in_exclusive_work_context() in tb_flush
  plugin: add user-facing API
  plugin: add core code
  queue: add QTAILQ_REMOVE_SEVERAL
  cputlb: document get_page_addr_code
  cputlb: introduce get_page_addr_code_hostp
  tcg: add tcg_gen_st_ptr
  plugin-gen: add module for TCG-related code
  atomic_template: add inline trace/plugin helpers
  tcg: let plugins instrument virtual memory accesses
  translate-all: notify plugin code of tb_flush
  *-user: notify plugin of exit
  *-user: plugin syscalls
  cpu: hook plugin vcpu events
  plugin-gen: add plugin_insn_append
  translator: add translator_ld{ub,sw,uw,l,q}
  target/arm: fetch code with translator_ld
  target/ppc: fetch code with translator_ld
  target/sh4: fetch code with translator_ld
  target/i386: fetch code with translator_ld
  target/hppa: fetch code with translator_ld
  target/m68k: fetch code with translator_ld
  target/alpha: fetch code with translator_ld
  target/riscv: fetch code with translator_ld
  target/sparc: fetch code with translator_ld
  target/xtensa: fetch code with translator_ld
  target/openrisc: fetch code with translator_ld
  translator: inject instrumentation from plugins
  plugin: add API symbols to qemu-plugins.symbols
  tests/plugin: add sample plugins

Lluís Vilanova (2):
  vl: support -plugin option
  linux-user: support -plugin option

 .travis.yml                               |  15 +
 Makefile                                  |  16 +-
 Makefile.target                           |   2 +
 accel/stubs/tcg-stub.c                    |   1 -
 accel/tcg/Makefile.objs                   |   1 +
 accel/tcg/atomic_common.inc.c             |  54 ++
 accel/tcg/atomic_template.h               |  94 ++-
 accel/tcg/cpu-exec.c                      |   8 +-
 accel/tcg/cputlb.c                        |  53 +-
 accel/tcg/plugin-gen.c                    | 932 ++++++++++++++++++++++
 accel/tcg/plugin-helpers.h                |   5 +
 accel/tcg/translate-all.c                 |  15 +-
 accel/tcg/translator.c                    |  20 +
 accel/tcg/user-exec.c                     |   3 +
 bsd-user/syscall.c                        |  24 +-
 configure                                 |  86 ++
 cpus-common.c                             |   4 +
 cpus.c                                    |  10 +
 disas.c                                   | 110 +++
 docs/devel/index.rst                      |   1 +
 docs/devel/plugins.rst                    | 112 +++
 exec.c                                    |   2 +
 hw/core/cpu.c                             |   2 +
 include/disas/disas.h                     |   2 +
 include/exec/cpu-defs.h                   |   1 +
 include/exec/cpu_ldst.h                   |  11 +
 include/exec/cpu_ldst_template.h          |  37 +-
 include/exec/cpu_ldst_useronly_template.h |  29 +-
 include/exec/exec-all.h                   |  84 +-
 include/exec/helper-gen.h                 |   1 +
 include/exec/helper-proto.h               |   1 +
 include/exec/helper-tcg.h                 |   1 +
 include/exec/plugin-gen.h                 |  71 ++
 include/exec/translator.h                 |  58 +-
 include/hw/core/cpu.h                     |  19 +
 include/qemu/bswap.h                      |   5 +
 include/qemu/log.h                        |   1 +
 include/qemu/plugin.h                     | 261 ++++++
 include/qemu/qemu-plugin.h                | 388 +++++++++
 include/qemu/queue.h                      |  10 +
 include/user/syscall-trace.h              |  40 +
 linux-user/exit.c                         |   1 +
 linux-user/main.c                         |  18 +
 linux-user/syscall.c                      |   7 +-
 plugins/.gitignore                        |   2 +
 plugins/Makefile.objs                     |  21 +
 plugins/api.c                             | 334 ++++++++
 plugins/core.c                            | 504 ++++++++++++
 plugins/loader.c                          | 384 +++++++++
 plugins/plugin.h                          |  95 +++
 plugins/qemu-plugins.symbols              |  40 +
 qemu-options.hx                           |  17 +
 scripts/tracetool/transform.py            |   1 +
 target/alpha/translate.c                  |   2 +-
 target/arm/arm_ldst.h                     |  15 +-
 target/hppa/translate.c                   |   2 +-
 target/i386/translate.c                   |  10 +-
 target/m68k/translate.c                   |   2 +-
 target/openrisc/translate.c               |   2 +-
 target/ppc/translate.c                    |   8 +-
 target/riscv/translate.c                  |   2 +-
 target/sh4/translate.c                    |   4 +-
 target/sparc/translate.c                  |   2 +-
 target/xtensa/translate.c                 |   4 +-
 tcg/tcg-op.c                              |  40 +-
 tcg/tcg-op.h                              |  16 +
 tcg/tcg-opc.h                             |   3 +
 tcg/tcg.c                                 |  22 +
 tcg/tcg.h                                 |  23 +
 tests/Makefile.include                    |  11 +-
 tests/plugin/Makefile                     |  31 +
 tests/plugin/bb.c                         |  64 ++
 tests/plugin/empty.c                      |  30 +
 tests/plugin/hotblocks.c                  | 143 ++++
 tests/plugin/hotpages.c                   | 174 ++++
 tests/plugin/howvec.c                     | 351 ++++++++
 tests/plugin/insn.c                       |  61 ++
 tests/plugin/mem.c                        |  97 +++
 tests/tcg/Makefile.target                 |  43 +-
 tests/tcg/aarch64/Makefile.softmmu-target |   2 +-
 tests/tcg/aarch64/Makefile.target         |   6 +
 tests/tcg/arm/Makefile.softmmu-target     |   1 +
 tests/tcg/arm/Makefile.target             |   6 +
 tests/tcg/cris/Makefile.target            |   2 +-
 tests/tcg/i386/Makefile.target            |  11 +-
 trace-events                              |   8 +-
 trace/mem-internal.h                      |  39 +-
 trace/mem.h                               |   7 +-
 util/log.c                                |   3 +
 vl.c                                      |  11 +
 90 files changed, 5111 insertions(+), 161 deletions(-)
 create mode 100644 accel/tcg/atomic_common.inc.c
 create mode 100644 accel/tcg/plugin-gen.c
 create mode 100644 accel/tcg/plugin-helpers.h
 create mode 100644 docs/devel/plugins.rst
 create mode 100644 include/exec/plugin-gen.h
 create mode 100644 include/qemu/plugin.h
 create mode 100644 include/qemu/qemu-plugin.h
 create mode 100644 include/user/syscall-trace.h
 create mode 100644 plugins/.gitignore
 create mode 100644 plugins/Makefile.objs
 create mode 100644 plugins/api.c
 create mode 100644 plugins/core.c
 create mode 100644 plugins/loader.c
 create mode 100644 plugins/plugin.h
 create mode 100644 plugins/qemu-plugins.symbols
 create mode 100644 tests/plugin/Makefile
 create mode 100644 tests/plugin/bb.c
 create mode 100644 tests/plugin/empty.c
 create mode 100644 tests/plugin/hotblocks.c
 create mode 100644 tests/plugin/hotpages.c
 create mode 100644 tests/plugin/howvec.c
 create mode 100644 tests/plugin/insn.c
 create mode 100644 tests/plugin/mem.c

-- 
2.20.1