Re: Error "cannot bind memory to host NUMA nodes: Operation not permitted" running inside docker

[Daniel P . Berrangé]

On Thu, Apr 30, 2020 at 01:45:58PM +0200, Igor Mammedov wrote:
> On Thu, 30 Apr 2020 09:52:15 +0100
> Daniel P. Berrangé <address@hidden> wrote:
> 
> > On Wed, Apr 29, 2020 at 11:40:32PM +0300, Manuel Hohmann wrote:
> > > Hi,
> > > 
> > > I encountered the following error message on the QEMU 5.0.0 release, 
> > > compiled and run inside a docker image:
> > > 
> > > "cannot bind memory to host NUMA nodes: Operation not permitted"  
> > 
> > The error is reporting that mbind() failed.
> > 
> > mbind() man page says it gives EPERM when
> > 
> >   "The  flags argument included the MPOL_MF_MOVE_ALL flag and
> >    the caller does not have the CAP_SYS_NICE privilege."
> > 
> > QEMU always uses the MPOL_MF_MOVE flag though.
> > 
> > Looking at the kernel source,  mbind can also return EPERM if the
> > process is not permitted to access the requested nodes which seems
> > more plausible as a cause.
> > 
> > I guess the container the bound to some sub-set of nodes and QEMU is
> > trying to place the VM on different nodes that the container isn't
> > allowed to accesss.
> 
> 
> mbind call in this case should be nop since it's 'reapplying' the same
> default policy the RAM was allocated with (modulo flags which are not
> issue in this default case).
> 
> It looks like there is configuration issue with container (blacklisted mbind) 
> [2]
> Is it possible to try run container with '--security-opt seccomp=unconfined'
> to see if it's the issue.

Oh, yes, I forgot about seccomp - that is almost certainly the problem,
given that this is a public CI system with locked down container config.

> 
> From QEMU side we may skip mbind if hostnodes bitmap is empty to workaround
> the issue.
> But I'm not sure if it should be done instead of whitelisting mbind in 
> container,
> since usecases that are using host-nodes will still be broken due to 
> blacklisted mbind.
> (looks like mysql has the same [1] problem (but it just warning for them, so 
> it's not so severe issue),
> and they were inclined towards fixing container config)

Telling users to reconfigure the container to allow mbind() is not a viable
approach. In a public cloud scenario the users will not have any direct
control over the container, and it is entirely unsurprising for mbind to
be blocked.


> > > The QEMU command line to reproduce this behavior (it happens also on 
> > > -x86_64, -arm, -aarch64 with similar command line):
> > > 
> > > qemu-system-i386 -m 64 -M pc -smp 1 -display none -monitor stdio -drive 
> > > file=mp-acpi/NOS.iso,media=cdrom,id=d -boot order=d -d cpu_reset  
> > 
> > There is no reference to host mem backend or NUMA binding, so I'm
> > puzzled why QEMU would be doing an mbind() at all. That seems bad.
> 
> since 5.0 all guest RAM allocation was consolidated around hostmem.

Ok, so QEMU shouldn't be calling mbind() at all unless there's some config
on the CLI that requests us to change the default binding, which there
is not in this case.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|