Re: [PATCH v6 1/3] multifd: Create property multifd-flush-after-each-section

[Peter Xu]

On Wed, Feb 15, 2023 at 07:02:29PM +0100, Juan Quintela wrote:
> We used to flush all channels at the end of each RAM section
> sent.  That is not needed, so preparing to only flush after a full
> iteration through all the RAM.
> 
> Default value of the property is false.  But we return "true" in
> migrate_multifd_flush_after_each_section() until we implement the code
> in following patches.
> 
> Signed-off-by: Juan Quintela <quintela@redhat.com>

This line can be dropped, after (I assume) git commit helped to add the
other one below. :)

Normally that's also why I put R-bs before my SoB because I should have two
SoB but then I merge them into the last; git is happy with that too.

> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> Signed-off-by: Juan Quintela <quintela@redhat.com>

Acked-by: Peter Xu <peterx@redhat.com>

But some nitpicks below (I'll leave those to you to decide whether to
rework or keep them as is..).

>
> ---
> 
> Rename each-iteration to after-each-section
> Rename multifd-sync-after-each-section to
>        multifd-flush-after-each-section
> ---
>  qapi/migration.json   | 21 ++++++++++++++++++++-
>  migration/migration.h |  1 +
>  hw/core/machine.c     |  1 +
>  migration/migration.c | 17 +++++++++++++++--
>  4 files changed, 37 insertions(+), 3 deletions(-)
> 
> diff --git a/qapi/migration.json b/qapi/migration.json
> index c84fa10e86..3afd81174d 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -478,6 +478,24 @@
>  #                    should not affect the correctness of postcopy migration.
>  #                    (since 7.1)
>  #
> +# @multifd-flush-after-each-section: flush every channel after each
> +#                                    section sent.  This assures that
> +#                                    we can't mix pages from one
> +#                                    iteration through ram pages with
> +#                                    pages for the following
> +#                                    iteration.  We really only need
> +#                                    to do this flush after we have go
> +#                                    through all the dirty pages.
> +#                                    For historical reasons, we do
> +#                                    that after each section.  This is
> +#                                    suboptimal (we flush too many
> +#                                    times).
> +#                                    Default value is false.
> +#                                    Setting this capability has no
> +#                                    effect until the patch that
> +#                                    removes this comment.
> +#                                    (since 8.0)

IMHO the core of this new "cap" is the new RAM_SAVE_FLAG_MULTIFD_FLUSH bit
in the stream protocol, but it's not referenced here.  I would suggest
simplify the content but highlight the core change:

 @multifd-lazy-flush:  When enabled, multifd will only do sync flush after
                       each whole round of bitmap scan.  Otherwise it'll be
                       done per RAM save iteration (which happens with a much
                       higher frequency).

                       Please consider enable this as long as possible, and
                       keep this off only if any of the src/dst QEMU binary
                       doesn't support it.

                       This capability is bound to the new RAM save flag
                       RAM_SAVE_FLAG_MULTIFD_FLUSH, the new flag will only
                       be used and recognized when this feature bit set.

I know you dislike multifd-lazy-flush, but that's still the best I can come
up with when writting this (yeah I still like it :-p), please bare with me
and take whatever you think the best.

> +#
>  # Features:
>  # @unstable: Members @x-colo and @x-ignore-shared are experimental.
>  #
> @@ -492,7 +510,8 @@
>             'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate',
>             { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] },
>             'validate-uuid', 'background-snapshot',
> -           'zero-copy-send', 'postcopy-preempt'] }
> +           'zero-copy-send', 'postcopy-preempt',
> +           'multifd-flush-after-each-section'] }
>  
>  ##
>  # @MigrationCapabilityStatus:
> diff --git a/migration/migration.h b/migration/migration.h
> index 2da2f8a164..7f0f4260ba 100644
> --- a/migration/migration.h
> +++ b/migration/migration.h
> @@ -424,6 +424,7 @@ int migrate_multifd_channels(void);
>  MultiFDCompression migrate_multifd_compression(void);
>  int migrate_multifd_zlib_level(void);
>  int migrate_multifd_zstd_level(void);
> +bool migrate_multifd_flush_after_each_section(void);
>  
>  #ifdef CONFIG_LINUX
>  bool migrate_use_zero_copy_send(void);
> diff --git a/hw/core/machine.c b/hw/core/machine.c
> index f73fc4c45c..602e775f34 100644
> --- a/hw/core/machine.c
> +++ b/hw/core/machine.c
> @@ -54,6 +54,7 @@ const size_t hw_compat_7_1_len = 
> G_N_ELEMENTS(hw_compat_7_1);
>  GlobalProperty hw_compat_7_0[] = {
>      { "arm-gicv3-common", "force-8-bit-prio", "on" },
>      { "nvme-ns", "eui64-default", "on"},
> +    { "migration", "multifd-flush-after-each-section", "on"},

[same note to IRC: need to revert if rename, but otherwise don't bother]

Thanks,

>  };
>  const size_t hw_compat_7_0_len = G_N_ELEMENTS(hw_compat_7_0);
>  
> diff --git a/migration/migration.c b/migration/migration.c
> index 90fca70cb7..cfba0da005 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -167,7 +167,8 @@ 
> INITIALIZE_MIGRATE_CAPS_SET(check_caps_background_snapshot,
>      MIGRATION_CAPABILITY_XBZRLE,
>      MIGRATION_CAPABILITY_X_COLO,
>      MIGRATION_CAPABILITY_VALIDATE_UUID,
> -    MIGRATION_CAPABILITY_ZERO_COPY_SEND);
> +    MIGRATION_CAPABILITY_ZERO_COPY_SEND,
> +    MIGRATION_CAPABILITY_MULTIFD_FLUSH_AFTER_EACH_SECTION);
>  
>  /* When we add fault tolerance, we could have several
>     migrations at once.  For now we don't need to add
> @@ -2701,6 +2702,17 @@ bool migrate_use_multifd(void)
>      return s->enabled_capabilities[MIGRATION_CAPABILITY_MULTIFD];
>  }
>  
> +bool migrate_multifd_flush_after_each_section(void)
> +{
> +    MigrationState *s = migrate_get_current();
> +
> +    /*
> +     * Until the patch that remove this comment, we always return that
> +     * the capability is enabled.
> +     */
> +    return true || 
> s->enabled_capabilities[MIGRATION_CAPABILITY_MULTIFD_FLUSH_AFTER_EACH_SECTION];

(I'd rather not care about what happens if someone applies only this patch
 not the latter two by dropping "true ||" directly here, but again, no a
 huge deal)

> +}
> +
>  bool migrate_pause_before_switchover(void)
>  {
>      MigrationState *s;
> @@ -4535,7 +4547,8 @@ static Property migration_properties[] = {
>      DEFINE_PROP_MIG_CAP("x-zero-copy-send",
>              MIGRATION_CAPABILITY_ZERO_COPY_SEND),
>  #endif
> -
> +    DEFINE_PROP_MIG_CAP("multifd-flush-after-each-section",
> +                        
> MIGRATION_CAPABILITY_MULTIFD_FLUSH_AFTER_EACH_SECTION),
>      DEFINE_PROP_END_OF_LIST(),
>  };
>  
> -- 
> 2.39.1
> 
> 

-- 
Peter Xu