Re: [PATCH v4] i386: Add ratelimit for bus locks acquired in guest

[Chenyi Qiang]

On 7/27/2021 4:28 PM, Dr. David Alan Gilbert wrote:
> * Chenyi Qiang (chenyi.qiang@intel.com) wrote:
> > A bus lock is acquired through either split locked access to writeback
> > (WB) memory or any locked access to non-WB memory. It is typically >1000
> > cycles slower than an atomic operation within a cache and can also
> > disrupts performance on other cores.
> >
> > Virtual Machines can exploit bus locks to degrade the performance of
> > system. To address this kind of performance DOS attack coming from the
> > VMs, bus lock VM exit is introduced in KVM and it can report the bus
> > locks detected in guest. If enabled in KVM, it would exit to the
> > userspace to let the user enforce throttling policies once bus locks
> > acquired in VMs.
> >
> > The availability of bus lock VM exit can be detected through the
> > KVM_CAP_X86_BUS_LOCK_EXIT. The returned bitmap contains the potential
> > policies supported by KVM. The field KVM_BUS_LOCK_DETECTION_EXIT in
> > bitmap is the only supported strategy at present. It indicates that KVM
> > will exit to userspace to handle the bus locks.
> >
> > This patch adds a ratelimit on the bus locks acquired in guest as a
> > mitigation policy.
> >
> > Introduce a new field "bus_lock_ratelimit" to record the limited speed
> > of bus locks in the target VM. The user can specify it through the
> > "bus-lock-ratelimit" as a machine property. In current implementation,
> > the default value of the speed is 0 per second, which means no
> > restrictions on the bus locks.
> >
> > As for ratelimit on detected bus locks, simply set the ratelimit
> > interval to 1s and restrict the quota of bus lock occurence to the value
> > of "bus_lock_ratelimit". A potential alternative is to introduce the
> > time slice as a property which can help the user achieve more precise
> > control.
> >
> > The detail of bus lock VM exit can be found in spec:
> > https://software.intel.com/content/www/us/en/develop/download/intel-architecture-instruction-set-extensions-programming-reference.html
> >
> > Signed-off-by: Chenyi Qiang <chenyi.qiang@intel.com>
>
> Hi Chenyi,
>
>    I noticed in this patch:
>
>
> > +static void kvm_rate_limit_on_bus_lock(void)
> > +{
> > +    uint64_t delay_ns = ratelimit_calculate_delay(&bus_lock_ratelimit_ctrl, 1);
> > +
> > +    if (delay_ns) {
> > +        g_usleep(delay_ns / SCALE_US);
> > +    }
> > +}
>
> and wondered if this would block cpu kicks, and what would happen if
> delay_ns got quite big - Eduardo thinks it might get upto 1s.

I did a rough test, force the delay_ns to 1s and see how long it will 
take to sleep 20s in guest. Actually, for 1-vcpu VM, the output of 
elapsed time is 20.4~20.6s, so I assume the applications in guest may 
lose some precision. Changing to a more refined time slice control is an 
solution. (But concerning that such ratelimit only happen in a malicious 
guest, maybe it is acceptable to lose some accuracy.)

> Also, it feels similar to what migration does during 'auto converge';
> see softmuu/cpu-throttle.c - instead of doing your own g_usleep
> you could call cpu_throttle_set with a given throttle rate.

Yes, looked at the cpu-throttle code, cpu_throttle_set works similarly, 
but need some refactor. Migration uses the static throttle_percentage to 
control the global throttling, so if bus lock throttling calls 
cpu_throttle_set, it needs to distinguish with migration.

> Dave
>
> > +
> >   MemTxAttrs kvm_arch_post_run(CPUState *cpu, struct kvm_run *run)
> >   {
> >       X86CPU *x86_cpu = X86_CPU(cpu);
> > @@ -4237,6 +4271,9 @@ MemTxAttrs kvm_arch_post_run(CPUState *cpu, struct 
> > kvm_run *run)
> >       } else {
> >           env->eflags &= ~IF_MASK;
> >       }
> > +    if (run->flags & KVM_RUN_X86_BUS_LOCK) {
> > +        kvm_rate_limit_on_bus_lock();
> > +    }
> >   
> >       /* We need to protect the apic state against concurrent accesses from
> >        * different threads in case the userspace irqchip is used. */
> > @@ -4595,6 +4632,10 @@ int kvm_arch_handle_exit(CPUState *cs, struct kvm_run 
> > *run)
> >           ioapic_eoi_broadcast(run->eoi.vector);
> >           ret = 0;
> >           break;
> > +    case KVM_EXIT_X86_BUS_LOCK:
> > +        /* already handled in kvm_arch_post_run */
> > +        ret = 0;
> > +        break;
> >       default:
> >           fprintf(stderr, "KVM: unknown exit reason %d\n", run->exit_reason);
> >           ret = -1;
> > --
> > 2.17.1