Re: [Qemu-devel] [RFC PATCH 00/17] reverse debugging

[Ciro Santilli]

On Thu, Apr 26, 2018 at 1:34 PM, Pavel Dovgalyuk <address@hidden> wrote:

> > From: Ciro Santilli [mailto:address@hidden
> > On Wed, Apr 25, 2018 at 1:45 PM, Pavel Dovgalyuk
> > <address@hidden> wrote:
> > > GDB remote protocol supports reverse debugging of the targets.
> > > It includes 'reverse step' and 'reverse continue' operations.
> > > The first one finds the previous step of the execution,
> > > and the second one is intended to stop at the last breakpoint that
> > > would happen when the program is executed normally.
> > >
> > > Reverse debugging is possible in the replay mode, when at least
> > > one snapshot was created at the record or replay phase.
> > > QEMU can use these snapshots for travelling back in time with GDB.
> > >
> >
> > Hi Pavel,
> >
> > 1)
> >
> > Can you provide more details on how to run the reverse debugging? In
> > particular how to take the checkpoint?
>
> There is some information in docs/replay.txt, but I guess, that I can give
> some more.
>
> >
> > My test setup is described in detail at:
> > https://github.com/cirosantilli/qemu-test/tree/8127452e5685e
> d233dc7357a1fe34b7a2d173480
> > command "x86_64/reverse-debug".
> >
> > Here are the actual commands:
> >
> > #!/usr/bin/env bash
> > set -eu
> > dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/.."
> > cmd="\
> > time \
> > ./x86_64-softmmu/qemu-system-x86_64 \
> > -M pc \
> > -append 'root=/dev/sda console=ttyS0 nokaslr printk.time=y -
> > lkmc_eval=\"/rand_check.out;/sbin/ifup -a;wget -S
> > google.com;/poweroff.out;\"' \
> > -kernel '${dir}/out/x86_64/buildroot/images/bzImage' \
> > -nographic \
> > -serial mon:stdio \
> > -monitor telnet::45454,server,nowait \
> > \
> > -drive file='${dir}/out/x86_64/buildroot/images/rootfs.ext2.qcow2,
> if=none,id=img-
> > direct,format=qcow2,snapshot'
>
> The main thing for reverse debugging is snapshotting.
> Therefore you should have an image that does not use temporary overlay
> file (snapshot option).
> I'm using the following command line for record:
>
> rm ./images/xp.ovl
> # create overlay to avoid modifying the original image
> ./bin/qemu-img create -f qcow2 -b xp.qcow2 ./images/xp.ovl
> ./bin/qemu-system-i386 \
> # This is workaround for XP. I wonder is it needed for the current version
> or not.
>  -global apic-common.vapic=off \
> # using newly created overlay instead of the original image
> # rrsnapshot creates the snapshot at the start
>  -icount shift=7,rr=record,rrfile=xp.replay,rrsnapshot=init -drive
> file=./images/xp.ovl,if=none,id=img-direct \
>  -drive driver=blkreplay,if=none,image=img-direct,id=img-replay -device
> ide-hd,drive=img-replay -net none -m 256M -monitor stdio
>
> While recording I can create some snapshots with savevm.
> Command line for replaying differs only in "rr" option. rrsnapshot there
> loads the initial snapshot.
> Any of the previously created snapshots may be specified.
> You can also create new snapshots while replaying.
>
>
> > \
> > -drive driver=blkreplay,if=none,image=img-direct,id=img-blkreplay \
> > -device ide-hd,drive=img-blkreplay \
> > \
> > -netdev user,id=net1 \
> > -device rtl8139,netdev=net1 \
> > -object filter-replay,id=replay,netdev=net1 \
> > "
> > cmd="${cmd} $@"
> > echo "$cmd"
> > eval "$cmd -icount 'shift=7,rr=record,rrfile=replay.bin'"
> > eval "$cmd -icount 'shift=7,rr=replay,rrfile=replay.bin' -S -s"
> >
> > Then I take a snapshot right at the beginning of the execution:
> >
> > telnet 45454
> > savevm a
> >
> > And on another shell:
> >
> > /data/git/linux-kernel-module-cheat/out/x86_64/buildroot/hos
> t/usr/bin/x86_64-linux-gdb
> > \
> > -q \
> > -ex 'file vmlinux' \
> > -ex 'target remote localhost:1234' \
> > -ex 'break start_kernel' \
> > -ex 'continue' \
> >
> > But now if I try on GDB:
> >
> > next
> > next
> > next
> > reverse-continue
> >
> > hoping to go back to start_kernel, but nothing happens.
>
> Yes, because you are missing your snapshot, that was actually created in
> the temporary overlay.
>
> > Same behavior if I take the snapshot after reaching start_kernel instead.
> >
> > 2)
> >
> > I wonder if it would be possible to expose checkpoint taking through
> > GDB example via:
> > https://sourceware.org/gdb/onlinedocs/gdb/Checkpoint_002fRestart.html
>
> We'll check this out.
>

Actually, this is not needed, I have learnt now that you can send QEMU
monitor commands with `monitor <qemu-monitor-command>`, so e.g. `monitor
savevm a`.


>
> > Or some other more convenient checkpoint generation method, e.g.
> > automatically take checkpoints every N instructions.
>
> We implemented 'taking snapshots every N seconds', but I'll prefer to
> submit
> it later, after approving the main idea.
>
> > > Running the execution in replay mode allows using GDB reverse debugging
> > > commands:
> > >  - reverse-stepi (or rsi): Steps one instruction to the past.
> > >    QEMU loads on of the prior snapshots and proceeds to the desired
> > >    instruction forward. When that step is reaches, execution stops.
> > >  - reverse-continue (or rc): Runs execution "backwards".
> > >    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
> > >    and replaying the execution. Then QEMU loads snapshots again and
> > >    replays to the latest breakpoint. When there are no breakpoints in
> > >    the examined section of the execution, QEMU finds one more snapshot
> > >    and tries again. After the first snapshot is processed, execution
> > >    stops at this snapshot.
> > >
> > > The set of patches include the following modifications:
> > >  - gdbstub update for reverse debugging support
> > >  - functions that automatically perform reverse step and reverse
> > >    continue operations
> > >  - hmp/qmp commands for manipulating the replay process
> > >  - improvement of the snapshotting for saving the execution step
> > >    in the snapshot parameters
> > >  - other record/replay fixes
> > >
> > > The patches are available in the repository:
> > > https://github.com/ispras/qemu/tree/rr-180207
>
> Pavel Dovgalyuk
>
>