Re: [Qemu-devel] [PATCH v5 23/45] migrate_start_postcopy: Command to trigger transition to postcopy

[David Gibson]

On Thu, Mar 19, 2015 at 09:33:31AM +0000, Dr. David Alan Gilbert wrote:
> * David Gibson (address@hidden) wrote:
> > On Wed, Mar 18, 2015 at 05:59:51PM +0000, Dr. David Alan Gilbert wrote:
> > > * David Gibson (address@hidden) wrote:
> > > > On Fri, Mar 13, 2015 at 11:19:06AM +0000, Dr. David Alan Gilbert wrote:
> > > > > * David Gibson (address@hidden) wrote:
> > > > > > On Wed, Feb 25, 2015 at 04:51:46PM +0000, Dr. David Alan Gilbert 
> > > > > > (git) wrote:
> > > > > > > From: "Dr. David Alan Gilbert" <address@hidden>
> > > > > > > 
> > > > > > > Once postcopy is enabled (with migrate_set_capability), the 
> > > > > > > migration
> > > > > > > will still start on precopy mode.  To cause a transition into 
> > > > > > > postcopy
> > > > > > > the:
> > > > > > > 
> > > > > > >   migrate_start_postcopy
> > > > > > > 
> > > > > > > command must be issued.  Postcopy will start sometime after this
> > > > > > > (when it's next checked in the migration loop).
> > > > > > > 
> > > > > > > Issuing the command before migration has started will error,
> > > > > > > and issuing after it has finished is ignored.
> > > > > > > 
> > > > > > > Signed-off-by: Dr. David Alan Gilbert <address@hidden>
> > > > > > > Reviewed-by: Eric Blake <address@hidden>
> > > > > > > ---
> > > > > > >  hmp-commands.hx               | 15 +++++++++++++++
> > > > > > >  hmp.c                         |  7 +++++++
> > > > > > >  hmp.h                         |  1 +
> > > > > > >  include/migration/migration.h |  3 +++
> > > > > > >  migration/migration.c         | 22 ++++++++++++++++++++++
> > > > > > >  qapi-schema.json              |  8 ++++++++
> > > > > > >  qmp-commands.hx               | 19 +++++++++++++++++++
> > > > > > >  7 files changed, 75 insertions(+)
> > > > > > > 
> > > > > > > diff --git a/hmp-commands.hx b/hmp-commands.hx
> > > > > > > index e37bc8b..03b8b78 100644
> > > > > > > --- a/hmp-commands.hx
> > > > > > > +++ b/hmp-commands.hx
> > > > > > > @@ -985,6 +985,21 @@ Enable/Disable the usage of a capability 
> > > > > > > @var{capability} for migration.
> > > > > > >  ETEXI
> > > > > > >  
> > > > > > >      {
> > > > > > > +        .name       = "migrate_start_postcopy",
> > > > > > > +        .args_type  = "",
> > > > > > > +        .params     = "",
> > > > > > > +        .help       = "Switch migration to postcopy mode",
> > > > > > > +        .mhandler.cmd = hmp_migrate_start_postcopy,
> > > > > > > +    },
> > > > > > > +
> > > > > > > +STEXI
> > > > > > > address@hidden migrate_start_postcopy
> > > > > > > address@hidden migrate_start_postcopy
> > > > > > > +Switch in-progress migration to postcopy mode. Ignored after the 
> > > > > > > end of
> > > > > > > +migration (or once already in postcopy).
> > > > > > > +ETEXI
> > > > > > > +
> > > > > > > +    {
> > > > > > >          .name       = "client_migrate_info",
> > > > > > >          .args_type  = 
> > > > > > > "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
> > > > > > >          .params     = "protocol hostname port tls-port 
> > > > > > > cert-subject",
> > > > > > > diff --git a/hmp.c b/hmp.c
> > > > > > > index b47f331..df9736c 100644
> > > > > > > --- a/hmp.c
> > > > > > > +++ b/hmp.c
> > > > > > > @@ -1140,6 +1140,13 @@ void hmp_migrate_set_capability(Monitor 
> > > > > > > *mon, const QDict *qdict)
> > > > > > >      }
> > > > > > >  }
> > > > > > >  
> > > > > > > +void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict)
> > > > > > > +{
> > > > > > > +    Error *err = NULL;
> > > > > > > +    qmp_migrate_start_postcopy(&err);
> > > > > > > +    hmp_handle_error(mon, &err);
> > > > > > > +}
> > > > > > > +
> > > > > > >  void hmp_set_password(Monitor *mon, const QDict *qdict)
> > > > > > >  {
> > > > > > >      const char *protocol  = qdict_get_str(qdict, "protocol");
> > > > > > > diff --git a/hmp.h b/hmp.h
> > > > > > > index 4bb5dca..da1334f 100644
> > > > > > > --- a/hmp.h
> > > > > > > +++ b/hmp.h
> > > > > > > @@ -64,6 +64,7 @@ void hmp_migrate_set_downtime(Monitor *mon, 
> > > > > > > const QDict *qdict);
> > > > > > >  void hmp_migrate_set_speed(Monitor *mon, const QDict *qdict);
> > > > > > >  void hmp_migrate_set_capability(Monitor *mon, const QDict 
> > > > > > > *qdict);
> > > > > > >  void hmp_migrate_set_cache_size(Monitor *mon, const QDict 
> > > > > > > *qdict);
> > > > > > > +void hmp_migrate_start_postcopy(Monitor *mon, const QDict 
> > > > > > > *qdict);
> > > > > > >  void hmp_set_password(Monitor *mon, const QDict *qdict);
> > > > > > >  void hmp_expire_password(Monitor *mon, const QDict *qdict);
> > > > > > >  void hmp_eject(Monitor *mon, const QDict *qdict);
> > > > > > > diff --git a/include/migration/migration.h 
> > > > > > > b/include/migration/migration.h
> > > > > > > index e6a814a..293c83e 100644
> > > > > > > --- a/include/migration/migration.h
> > > > > > > +++ b/include/migration/migration.h
> > > > > > > @@ -104,6 +104,9 @@ struct MigrationState
> > > > > > >      int64_t xbzrle_cache_size;
> > > > > > >      int64_t setup_time;
> > > > > > >      int64_t dirty_sync_count;
> > > > > > > +
> > > > > > > +    /* Flag set once the migration has been asked to enter 
> > > > > > > postcopy */
> > > > > > > +    bool start_postcopy;
> > > > > > >  };
> > > > > > >  
> > > > > > >  void process_incoming_migration(QEMUFile *f);
> > > > > > > diff --git a/migration/migration.c b/migration/migration.c
> > > > > > > index a4fc7d7..43ca656 100644
> > > > > > > --- a/migration/migration.c
> > > > > > > +++ b/migration/migration.c
> > > > > > > @@ -372,6 +372,28 @@ void 
> > > > > > > qmp_migrate_set_capabilities(MigrationCapabilityStatusList 
> > > > > > > *params,
> > > > > > >      }
> > > > > > >  }
> > > > > > >  
> > > > > > > +void qmp_migrate_start_postcopy(Error **errp)
> > > > > > > +{
> > > > > > > +    MigrationState *s = migrate_get_current();
> > > > > > > +
> > > > > > > +    if (!migrate_postcopy_ram()) {
> > > > > > > +        error_setg(errp, "Enable postcopy with 
> > > > > > > migration_set_capability before"
> > > > > > > +                         " the start of migration");
> > > > > > > +        return;
> > > > > > > +    }
> > > > > > > +
> > > > > > > +    if (s->state == MIG_STATE_NONE) {
> > > > > > > +        error_setg(errp, "Postcopy must be started after 
> > > > > > > migration has been"
> > > > > > > +                         " started");
> > > > > > > +        return;
> > > > > > > +    }
> > > > > > > +    /*
> > > > > > > +     * we don't error if migration has finished since that would 
> > > > > > > be racy
> > > > > > > +     * with issuing this command.
> > > > > > > +     */
> > > > > > > +    atomic_set(&s->start_postcopy, true);
> > > > > > 
> > > > > > Why atomic_set?
> > > > > 
> > > > > It's being read by the migration thread, this is happening in the 
> > > > > main thread.
> > > > > 
> > > > > There's no strict ordering requirement or anything.
> > > > 
> > > > I don't think you need an atomic then.  AFAIK an atomic_set() in
> > > > isolation without some sort of atomic on the other side is pretty much
> > > > meaningless.
> > > 
> > > The other side has an atomic_read:
> > > 
> > >                 if (migrate_postcopy_ram() &&
> > >                     s->state != MIGRATION_STATUS_POSTCOPY_ACTIVE &&
> > >                     pend_nonpost <= max_size &&
> > >                     atomic_read(&s->start_postcopy)) {
> > > 
> > >                     if (!postcopy_start(s, &old_vm_running)) {
> > >                         current_active_type = 
> > > MIGRATION_STATUS_POSTCOPY_ACTIVE;
> > >                         entered_postcopy = true;
> > >                     }
> > > 
> > > so it is at least symmetric.
> > 
> > But still pointless.  Atomicity isn't magic pixie dust; it only makes
> > sense if you're making atomic specific operations that need to be.
> > Simple integer loads and stores are already atomic.  Unless at least
> > some of the atomic operations are something more complex, there's
> > really no point to atomically marked operations.
> 
> OK, I'll kill it off.
> 
> It'll work in practice, but I still believe that what you're saying isn't
> safe C:
>    1) There's no barrier after the write, so there's no guarantee the other
>       thread will eventually see it (in practice we've got other pthread ops
>       we take so we will get a barrier somewhere, and most CPUs eventually
>       do propagate the store).

Sorry, I should have been clearer.  If you need a memory barrier, by
all means include a memory barrier.  But that should be explicit:
atomic set/read operations often include barriers, but it's not
obvious which side will include what barrier.

>    2) The read side could legally be optimised out of the loop by the 
> compiler.
>       (but in practice wont be because compilers won't optimise that far).

That one's a trickier question.  Compilers are absolutely capable of
optimizing that far, *but* the C rules about when it's allowed to
assume in-memory values remain unchanged are pretty conservative.  I
think any function call in the loop will require it to reload the
value, for example.  That said, a (compiler only) memory barrier might
be appropriate to ensure that reload.

-- 
David Gibson                    | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
                                | _way_ _around_!
http://www.ozlabs.org/~dgibson

pgp6VJAojxkzn.pgp
Description: PGP signature