Re: [Qemu-devel] [PATCH v3] qqq: module for synchronizing with a simulation

[Nutaro, James J.]

I realizes my reply wasn't sent to the forum. Thanks Greg for the feedback. 
I'll keep this in mind for the next revision.

Jim
________________________________________
From: Greg Kurz <address@hidden>
Sent: Wednesday, September 07, 2016 2:33 PM
To: Nutaro, James J.
Cc: address@hidden
Subject: Re: [Qemu-devel] [PATCH v3] qqq: module for synchronizing with a 
simulation

On Wed,  7 Sep 2016 13:53:18 -0400
"James J. Nutaro" <address@hidden> wrote:

> This patch adds an interface for pacing the execution of
> QEMU to match its rtc with an external simulation clock.
> Its aim is to permit QEMU to be used as a module within a
> larger simulation system. This revision (v3) corrects
> formatting errors detected by Patchew.
>

The changelog should only mention what the patch actually does. If you have
to re-post newer versions before the patch is finally accepted, changes should
be mentioned...

> Signed-off-by: James J. Nutaro <address@hidden>
> ---

... here, below the ---, so they don't appear in the commit changelog.

Cheers.

--
Greg

>  Makefile.target          |   3 ++
>  docs/simulation-sync.txt |  60 ++++++++++++++++++++++++++
>  qemu-options.hx          |  16 +++++++
>  qqq.c                    | 109 
> +++++++++++++++++++++++++++++++++++++++++++++++
>  qqq.h                    |  35 +++++++++++++++
>  vl.c                     |  32 ++++++++++++++
>  6 files changed, 255 insertions(+)
>  create mode 100644 docs/simulation-sync.txt
>  create mode 100644 qqq.c
>  create mode 100644 qqq.h
>
> diff --git a/Makefile.target b/Makefile.target
> index 8c7a072..b7f6c4c 100644
> --- a/Makefile.target
> +++ b/Makefile.target
> @@ -103,6 +103,9 @@ obj-$(CONFIG_LIBDECNUMBER) += libdecnumber/dpd/decimal32.o
>  obj-$(CONFIG_LIBDECNUMBER) += libdecnumber/dpd/decimal64.o
>  obj-$(CONFIG_LIBDECNUMBER) += libdecnumber/dpd/decimal128.o
>
> +# qqq
> +obj-y += qqq.o
> +
>  #########################################################
>  # Linux user emulator target
>
> diff --git a/docs/simulation-sync.txt b/docs/simulation-sync.txt
> new file mode 100644
> index 0000000..2133093
> --- /dev/null
> +++ b/docs/simulation-sync.txt
> @@ -0,0 +1,60 @@
> += Synchronizing the virtual clock with an external source =
> +
> +QEMU has a protocol for synchronizing its virtual clock
> +with the clock of a simulator in which QEMU is embedded
> +as a component. This options is enabled with the -qqq
> +argument, and it should generally be accompanied by the
> +following additional command line arguments:
> +
> +-icount 1,sleep=off -rtc clock=vm
> +
> +The -qqq argument is used to supply file descriptors
> +for two Unix pipes. The read pipe is used by QEMU to
> +receive synchronization data from the external simulator.
> +The write pipe is used by QEMU to supply synchronization
> +data to the external emulator. The typical procedure for
> +launching QEMU in is synchronization mode has three steps:
> +
> +(1) Create two pairs of pipes with the Linux pipe function.
> +    The code segment that does this might look like
> +
> +       int pipefd1[2];
> +       int pipefd2[2];
> +       pipe(pipefd1);
> +       pipe(pipefd2);
> +
> +(2) Fork QEMU with the appropriate command line arguments.
> +    The -qqq part of the argument will look something like
> +
> +       -qqq write=pipefd1[1],read=pipefd2[0]
> +
> +(3) After forking QEMU, close pipefd1[1] and pipefd2[0].
> +    Retain the other pair of pipes for communicating with QEMU.
> +
> +The synchronization protocol is very simple. To start, the
> +external simulator writes an integer to its write pipe with
> +the amount of time in microseconds that QEMU is allowed to
> +advance. The code segment that does this might look like:
> +
> +    int ta = 1000; // Advance by 1 millisecond
> +    write(pipefd2[1],&ta,sizeof(int));
> +
> +The external simulator can then advance its clock by this
> +same amount. During this time, QEMU and the external simulator
> +will be executing in parallel. When the external simulator
> +completes its time advance, it waits for QEMU by reading from
> +its read pipe. The value read will be the actual number of
> +virtual microseconds by which QEMU has advanced its virtual clock.
> +This will be greater than or equal to the requested advance.
> +The code that does this might look like:
> +
> +   read(pipefd1[0],&ta,sizeof(int));
> +
> +These steps are repeated until either (1) the external simulator
> +closes its pipes thereby causing QEMU to terminate or (2) QEMU
> +stops executing (e.g., if the emulated computer is shutdown) and
> +causes SIGPIPE to be generated by the closing of its pipes.
> +
> +You can find an example of a simulator using this protocol in
> +the adevs simulation package at http://sourceforge.net/projects/adevs/
> +
> diff --git a/qemu-options.hx b/qemu-options.hx
> index a71aaf8..d52cc9c 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -3359,6 +3359,22 @@ many timer interrupts were not processed by the 
> Windows guest and will
>  re-inject them.
>  ETEXI
>
> +DEF("qqq", HAS_ARG, QEMU_OPTION_qqq, \
> +    "-qqq read=fd,write=fd\n" \
> +    "                enable synchronization of the virtual clock \n" \
> +    "                with an external simulation clock\n", QEMU_ARCH_ALL)
> +STEXI
> address@hidden -qqq address@hidden,address@hidden
> address@hidden -qqq
> +Qemu will use the supplied pipes to synchronize its virtual clock with
> +an external simulation clock. Qemu will wait until a time slice size in
> +microseconds is supplied on the read pipe. Then it will execute for at
> +least that number of virtual microseconds before writing the actual
> +virtual time that has elapsed in microseconds to the write pipe. This
> +cycle will repeat until a zero is elaspsed time is requested, which
> +will cause qemu to exit.
> +ETEXI
> +
>  DEF("icount", HAS_ARG, QEMU_OPTION_icount, \
>      "-icount 
> [shift=N|auto][,align=on|off][,sleep=on|off,rr=record|replay,rrfile=<filename>]\n"
>  \
>      "                enable virtual instruction counter with 2^N clock ticks 
> per\n" \
> diff --git a/qqq.c b/qqq.c
> new file mode 100644
> index 0000000..f94711f
> --- /dev/null
> +++ b/qqq.c
> @@ -0,0 +1,109 @@
> +
> +#include "qemu/osdep.h"
> +#include "qemu/timer.h"
> +#include "qemu/main-loop.h"
> +#include "sysemu/cpus.h"
> +#include "qqq.h"
> +#include <time.h>
> +
> +/* This is a Linux only feature */
> +
> +#ifndef _WIN32
> +
> +#include <unistd.h>
> +#include <assert.h>
> +
> +static int elapsed;
> +static int time_advance = -1;
> +static int read_fd = -1, write_fd = -1;
> +static int64_t t;
> +static QEMUTimer *sync_timer;
> +
> +static void cleanup_and_exit(void)
> +{
> +    close(read_fd);
> +    close(write_fd);
> +    exit(0);
> +}
> +
> +static void write_mem_value(int val)
> +{
> +    if (write(write_fd, &val, sizeof(int)) != sizeof(int)) {
> +        /* If the pipe is no good, then assume this is an
> +         * indication that we should exit.
> +         */
> +        cleanup_and_exit();
> +    }
> +}
> +
> +static int read_mem_value(void)
> +{
> +    int tmp;
> +    if (read(read_fd, &tmp, sizeof(int)) != sizeof(int)) {
> +        /* If the pipe is no good, then assume this is an
> +         * indication that we should exit.
> +         */
> +        cleanup_and_exit();
> +    }
> +    return tmp;
> +}
> +
> +static void schedule_next_event(void)
> +{
> +    /* If we got the time advance in fd_read, then don't do it
> +     * again here. */
> +    if (time_advance < 0) {
> +        /* Otherwise read the value from the pipe */
> +        time_advance = read_mem_value();
> +    }
> +    /* Schedule the next synchronization point */
> +    timer_mod(sync_timer, t + time_advance);
> +    /* Note that we need to read the time advance again on the next pass */
> +    time_advance = -1;
> +}
> +
> +static void sync_func(void *data)
> +{
> +    /* Report the actual elapsed time. */
> +    int64_t tnow = qemu_clock_get_us(QEMU_CLOCK_VIRTUAL);
> +    elapsed = tnow - t;
> +    write_mem_value(elapsed);
> +    /* Update our time of last event */
> +    t = tnow;
> +    /* Schedule the next event */
> +    schedule_next_event();
> +}
> +
> +static void fd_read(void *opaque)
> +{
> +    /* Read the time advance if its becomes available
> +     * before our timer expires */
> +    time_advance = read_mem_value();
> +}
> +
> +void setup_qqq(QemuOpts *opts)
> +{
> +    /* Initialize the simulation clock */
> +    t = 0;
> +    /* Get the communication pipes */
> +    read_fd = qemu_opt_get_number(opts, "read", 0);
> +    write_fd = qemu_opt_get_number(opts, "write", 0);
> +    /* Start the timer to ensure time warps advance the clock */
> +    sync_timer = timer_new_us(QEMU_CLOCK_VIRTUAL, sync_func, NULL);
> +    /* Get the time advance that is requested by the simulation */
> +    schedule_next_event();
> +    /* Register the file descriptor with qemu. This should ensure
> +     * the emulator doesn't pause for lack of I/O and thereby
> +     * cause the attached simulator to pause with it. */
> +    qemu_set_fd_handler(read_fd, fd_read, NULL, NULL);
> +}
> +
> +#else
> +
> +void setup_qqq(QemuOpts *opts)
> +{
> +    fprintf(stderr, "-qqq is not supported on Windows, exiting\n");
> +    exit(0);
> +}
> +
> +#endif
> diff --git a/qqq.h b/qqq.h
> new file mode 100644
> index 0000000..40a24e3
> --- /dev/null
> +++ b/qqq.h
> @@ -0,0 +1,35 @@
> +/*
> + * This work is licensed under the terms of the GNU GPL
> + * version 2. Seethe COPYING file in the top-level directory.
> + *
> + * A module for pacing the rate of advance of the computer
> + * clock in reference to an external simulation clock. The
> + * basic approach used here is adapted from QBox from Green
> + * Socs. The mode of operation is as follows:
> + *
> + * The simulator uses pipes to exchange time advance data.
> + * The external simulator starts the exchange by forking a
> + * QEMU process and passing is descriptors for a read and
> + * write pipe. Then the external simulator writes an integer
> + * (native endian) to the pipe to indicate the number of
> + * microseconds that QEMU should advance. QEMU advances its
> + * virtual clock by this amount and writes to its write pipe
> + * the actual number of microseconds that have advanced.
> + * This process continues until a pipe on either side is
> + * closed, which will either cause QEMU to exit (if the
> + * external simulator closes a pipe) or raise SIGPIPE in the
> + * external simulator (if QEMU closes a pipe).
> + *
> + * Authors:
> + *   James Nutaro <address@hidden>
> + *
> + */
> +#ifndef QQQ_H
> +#define QQQ_H
> +
> +#include "qemu/osdep.h"
> +#include "qemu-options.h"
> +
> +void setup_qqq(QemuOpts *opts);
> +
> +#endif
> diff --git a/vl.c b/vl.c
> index ee557a1..fe72990 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -122,6 +122,8 @@ int main(int argc, char **argv)
>  #include "sysemu/replay.h"
>  #include "qapi/qmp/qerror.h"
>
> +#include "qqq.h"
> +
>  #define MAX_VIRTIO_CONSOLES 1
>  #define MAX_SCLP_CONSOLES 1
>
> @@ -231,6 +233,23 @@ static struct {
>      { .driver = "virtio-vga",           .flag = &default_vga       },
>  };
>
> +static QemuOptsList qemu_qqq_opts = {
> +    .name = "qqq",
> +    .implied_opt_name = "",
> +    .merge_lists = true,
> +    .head = QTAILQ_HEAD_INITIALIZER(qemu_qqq_opts.head),
> +    .desc = {
> +        {
> +            .name = "read",
> +            .type = QEMU_OPT_NUMBER,
> +        }, {
> +            .name = "write",
> +            .type = QEMU_OPT_NUMBER,
> +        },
> +        { /* end of list */ }
> +    },
> +};
> +
>  static QemuOptsList qemu_rtc_opts = {
>      .name = "rtc",
>      .head = QTAILQ_HEAD_INITIALIZER(qemu_rtc_opts.head),
> @@ -2952,6 +2971,7 @@ int main(int argc, char **argv, char **envp)
>      DisplayState *ds;
>      int cyls, heads, secs, translation;
>      QemuOpts *hda_opts = NULL, *opts, *machine_opts, *icount_opts = NULL;
> +    QemuOpts *qqq_opts = NULL;
>      QemuOptsList *olist;
>      int optind;
>      const char *optarg;
> @@ -2987,6 +3007,7 @@ int main(int argc, char **argv, char **envp)
>
>      module_call_init(MODULE_INIT_QOM);
>
> +    qemu_add_opts(&qemu_qqq_opts);
>      qemu_add_opts(&qemu_drive_opts);
>      qemu_add_drive_opts(&qemu_legacy_drive_opts);
>      qemu_add_drive_opts(&qemu_common_drive_opts);
> @@ -3847,6 +3868,13 @@ int main(int argc, char **argv, char **envp)
>                      exit(1);
>                  }
>                  break;
> +            case QEMU_OPTION_qqq:
> +                qqq_opts = qemu_opts_parse_noisily(qemu_find_opts("qqq"),
> +                                                      optarg, true);
> +                if (!qqq_opts) {
> +                    exit(1);
> +                }
> +                break;
>              case QEMU_OPTION_incoming:
>                  if (!incoming) {
>                      runstate_set(RUN_STATE_INMIGRATE);
> @@ -4350,6 +4378,10 @@ int main(int argc, char **argv, char **envp)
>      /* spice needs the timers to be initialized by this point */
>      qemu_spice_init();
>
> +    if (qqq_opts) {
> +        setup_qqq(qqq_opts);
> +    }
> +
>      cpu_ticks_init();
>      if (icount_opts) {
>          if (kvm_enabled() || xen_enabled()) {