Re: [PATCH v4 00/20] Add virtual device fuzzing support

[Darren Kenny]

Hi Alexander,

I've been trying out these patches, and I'm seeing a high volume of
crashes - where for v3, there were none in a run of over 3 weeks -
so it was a bit of a surprise :)

The question is what may have changed that is causing that level of
crashes - are you seeing this for the virtio-net-fork-fuzz tests?

But also, I've been trying to debug some of these crashes - and the
expectation is that you pass the crash-XXXX file as an argument to
the qemu-fuzz-* binary - and when I do, I see the crash - but when I
try to debug it, it ends up running through and exiting.

My assumption is that because of the fork in the test, the crash is
in one of the children.

(ASIDE: I think it might be worth adding a debugging/analysing
section to the documentation you've added to help people debug such
crashes)

Setting follow-fork-mode to child does get me there, and each crash
seems, at least in the samples that I've taken, to be in iov_copy:

 #0  0x00007ffff4cff377 in raise () from /lib64/libc.so.6
 #1  0x00007ffff4d00a68 in abort () from /lib64/libc.so.6
 #2  0x00007ffff4cf8196 in __assert_fail_base () from
 /lib64/libc.so.6
 #3  0x00007ffff4cf8242 in __assert_fail () from /lib64/libc.so.6
 #4  0x00005555574d4026 in iov_copy ()
 #5  0x000055555640dbd8 in virtio_net_flush_tx ()
 #6  0x000055555640c8ef in virtio_net_tx_bh ()
 #7  0x00005555574a05bb in aio_bh_call ()
 #8  0x00005555574a0a34 in aio_bh_poll ()
 #9  0x00005555574b1687 in aio_dispatch ()
 #10 0x00005555574a35f9 in aio_ctx_dispatch ()
 #11 0x00007ffff5e5d099 in g_main_context_dispatch () from
 /lib64/libglib-2.0.so.0
 #12 0x00005555574ae9fd in glib_pollfds_poll ()
 #13 0x00005555574ad972 in os_host_main_loop_wait ()
 #14 0x00005555574ad62c in main_loop_wait ()
 #15 0x000055555736c653 in flush_events ()
 #16 0x00005555573710a4 in virtio_net_fork_fuzz ()
 #17 0x000055555736cb85 in LLVMFuzzerTestOneInput ()
 ...

Have you seen these kind of crashes, or is this just me?

Just wondering if I should dig into it as a real issue, or some
mis-merge I've done (not all the patches were cleanly applied for
me when I cloned from master).

Thanks,

Darren.

On Wed, Oct 30, 2019 at 02:49:47PM +0000, Oleinik, Alexander wrote:
> This series adds a framework for coverage-guided fuzzing of
> virtual-devices. Fuzzing targets are based on qtest and can make use of
> the libqos abstractions.
>
> V4:
> * add/transfer license headers to new files
> * restructure the added QTestClientTransportOps struct
> * restructure the FuzzTarget struct and fuzzer skeleton
> * fork-based fuzzer now directly mmaps shm over the coverage bitmaps
> * fixes to i440 and virtio-net fuzz targets
> * undo the changes to qtest_memwrite
> * possible to build /fuzz and /all in the same build-dir
> * misc fixes to address V3 comments
>
> V3:
> * rebased onto v4.1.0+
> * add the fuzzer as a new build-target type in the build-system
> * add indirection to qtest client/server communication functions
> * remove ramfile and snapshot-based fuzzing support
> * add i440fx fuzz-target as a reference for developers.
> * add linker-script to assist with fork-based fuzzer
>
> V2:
> * split off changes to qos virtio-net and qtest server to other patches
> * move vl:main initialization into new func: qemu_init
> * moved useful functions from qos-test.c to a separate object
> * use struct of function pointers for add_fuzz_target(), instead of
>   arguments
> * move ramfile to migration/qemu-file
> * rewrite fork-based fuzzer pending patch to libfuzzer
> * pass check-patch
>
> Alexander Oleinik (20):
>  softmmu: split off vl.c:main() into main.c
>  libqos: Rename i2c_send and i2c_recv
>  fuzz: Add FUZZ_TARGET module type
>  qtest: add qtest_server_send abstraction
>  libqtest: Add a layer of abstraciton to send/recv
>  module: check module wasn't already initialized
>  qtest: add in-process incoming command handler
>  tests: provide test variables to other targets
>  libqos: split qos-test and libqos makefile vars
>  libqos: move useful qos-test funcs to qos_external
>  libqtest: make qtest_bufwrite send "atomic"
>  libqtest: add in-process qtest.c tx/rx handlers
>  fuzz: add configure flag --enable-fuzzing
>  fuzz: Add target/fuzz makefile rules
>  fuzz: add fuzzer skeleton
>  fuzz: add support for fork-based fuzzing.
>  fuzz: add support for qos-assisted fuzz targets
>  fuzz: add i440fx fuzz targets
>  fuzz: add virtio-net fuzz target
>  fuzz: add documentation to docs/devel/
>
> Makefile                     |  16 ++-
> Makefile.objs                |   4 +
> Makefile.target              |  18 ++-
> configure                    |  39 ++++++
> docs/devel/fuzzing.txt       | 119 ++++++++++++++++++
> exec.c                       |  12 +-
> include/qemu/module.h        |   4 +-
> include/sysemu/qtest.h       |   4 +
> include/sysemu/sysemu.h      |   4 +
> main.c                       |  52 ++++++++
> qtest.c                      |  30 ++++-
> tests/Makefile.include       |  75 +++++------
> tests/fuzz/Makefile.include  |  11 ++
> tests/fuzz/fork_fuzz.c       |  51 ++++++++
> tests/fuzz/fork_fuzz.h       |  23 ++++
> tests/fuzz/fork_fuzz.ld      |  37 ++++++
> tests/fuzz/fuzz.c            | 177 ++++++++++++++++++++++++++
> tests/fuzz/fuzz.h            |  66 ++++++++++
> tests/fuzz/i440fx_fuzz.c     | 176 ++++++++++++++++++++++++++
> tests/fuzz/qos_fuzz.c        | 232 +++++++++++++++++++++++++++++++++++
> tests/fuzz/qos_fuzz.h        |  33 +++++
> tests/fuzz/virtio_net_fuzz.c | 123 +++++++++++++++++++
> tests/libqos/i2c-imx.c       |   8 +-
> tests/libqos/i2c-omap.c      |   8 +-
> tests/libqos/i2c.c           |  10 +-
> tests/libqos/i2c.h           |   4 +-
> tests/libqos/qos_external.c  | 168 +++++++++++++++++++++++++
> tests/libqos/qos_external.h  |  28 +++++
> tests/libqtest.c             | 109 ++++++++++++++--
> tests/libqtest.h             |   4 +
> tests/pca9552-test.c         |  10 +-
> tests/qos-test.c             | 140 +--------------------
> util/module.c                |   7 ++
> vl.c                         |  36 ++----
> 34 files changed, 1601 insertions(+), 237 deletions(-)
> create mode 100644 docs/devel/fuzzing.txt
> create mode 100644 main.c
> create mode 100644 tests/fuzz/Makefile.include
> create mode 100644 tests/fuzz/fork_fuzz.c
> create mode 100644 tests/fuzz/fork_fuzz.h
> create mode 100644 tests/fuzz/fork_fuzz.ld
> create mode 100644 tests/fuzz/fuzz.c
> create mode 100644 tests/fuzz/fuzz.h
> create mode 100644 tests/fuzz/i440fx_fuzz.c
> create mode 100644 tests/fuzz/qos_fuzz.c
> create mode 100644 tests/fuzz/qos_fuzz.h
> create mode 100644 tests/fuzz/virtio_net_fuzz.c
> create mode 100644 tests/libqos/qos_external.c
> create mode 100644 tests/libqos/qos_external.h
>
> --
> 2.23.0