[Qemu-discuss] VirtFS/9P Passthru ... inconsistent/duplicate inodes? (ldconfig -> `Aborted`, localedef -> `Invalid argument`)

[C Anthony Risinger]

hello,

not sure qemu is the root problem, but i've been stumped for weeks,
and have already exhausted all (known) options including hours of
researching ... hoping someone may have the solution (or *absolutely
anything* to bring me closer).

will be as concise as possible -- lmk if i missed anything, or need
further detail.  information is primarily relevant to `ldconfig`
problem -- `localedef` mentioned twice, but possibly unrelated.  until
a recent glibc upgrade, everything *appeared* to be running perfectly
(at least 30 days).  mapped(?) mode 9P passthru as rootfs for 3
similar VMs; all have same issue.  x86_64 host, x86_64 VMs (Archlinux)
... identical package builds/versions.  normal/physical machines do
not exhibit issues.  cross-VM, host-based `--bind` mounts are present
in VM rootfs directories.  VM kernels are direct booted from their
respective rootfs directories (one of main motivations ... avoids
mucking with a bootloader entirely).  qemu runs as `nobody:kvm`.

(skip to DIAGNOSTICS if short on time or completely brilliant)

==== PROBLEM (output) ====

# ldconfig; echo return: $?
Aborted
return: 134

# ldconfig -v; echo return: $?
ldconfig: Can't stat /usr/lib64: No such file or directory
/usr/lib/libfakeroot:
        libfakeroot-0.so -> libfakeroot.so
/usr/lib/perl5/core_perl/CORE:
        libperl.so -> libperl.so
/lib:
Aborted
return: 134

# ldconfig -N; echo return: $?
return: 0
^^^^^^^^^^^^^^^ works fine

# localedef -i en_US -c -f ISO-8859-1 -A
/usr/share/locale/locale.alias en_US; echo return: $?
cannot map archive header: Invalid argument
return: 1

==== PROBLEM (summary) ====

`ldconfig` fails if a cache file is written; if the cache file is
skipped the links are updated properly, but anything outside the
default path (eg. perl) cannot be found.  `ld.so.cache` and
`aux-cache` removed prior to execution.  strace appears uninteresting.
 gdb output will follow (and provides the title of this thread)

`localedef` fails ONLY IF the output directory (/usr/lib/locale/)
resides on a 9P filesystem (mounting/binding a tmpfs, then writing out
`locale-archive` works fine).  per `strace -ff -etrace=file,mmap`,
it's failing at mmap() syscall:

[ ... ]
open("/usr/lib/locale/locale-archive", O_RDWR) = 3
stat("/usr/lib/locale/locale-archive", {st_mode=S_IFREG|0644,
st_size=1852464, ...}) = 0
mmap(NULL, 536870912, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f6aa600d000
mmap(0x7f6aa600d000, 1852464, PROT_READ|PROT_WRITE,
MAP_SHARED|MAP_FIXED, 3, 0) = -1 EINVAL (Invalid argument)
cannot map archive header: Invalid argument

==== ENVIRONMENT (host) ====

using `qemu-kvm` as i was under the impression they are virtually --
if not completely -- identical at this point?  per ...
https://wiki.archlinux.org/index.php/QEMU#Difference_between_qemu_and_qemu-kvm
... apologies if this is not true.  happy to try with `qemu` if need
be.

DISTRO
    Archlinux
ARCH
    x86_64
KERNEL
    3.1.5-1-ARCH
QEMU
    QEMU emulator version 1.0 (qemu-kvm-1.0)
LIBVIRT
    libvirtd (libvirt) 0.9.8
KSM?
    yes
HOST FS
    ext4
9P PASSTHRU TYPE
    mapped <------------------ hunch ... still supported/recommended?
COMPLETE CMD (1-of-3, but same prob on each VM)

# xargs -0 -n1 echo < /proc/$(pgrep -n qemu-kvm)/cmdline | pr -w80 -o4 -T | fold
    /usr/bin/qemu-kvm
    -S
    -M
    pc-0.14
    -cpu
    phenom,+osvw,+3dnowprefetch,+misalignsse,+sse4a,+abm,+cr8legacy,+extapic,+cm
p_legacy,+lahf_lm,+rdtscp,+pdpe1gb,+popcnt,+cx16,+ht,+vme
    -enable-kvm
    -m
    256
    -smp
    1,maxcpus=4,sockets=4,cores=1,threads=1
    -name
    beb6d03a495d89158422a27e07e61054
    -uuid
    beb6d03a-495d-8915-8422-a27e07e61054
    -nodefconfig
    -nodefaults
    -chardev
    socket,id=charmonitor,path=/var/lib/libvirt/qemu/beb6d03a495d89158422a27e07e
61054.monitor,server,nowait
    -mon
    chardev=charmonitor,id=monitor,mode=control
    -rtc
    base=utc
    -no-shutdown
    -kernel
    /var/lib/libvirt/assets/beb6d03a495d89158422a27e07e61054.pool/vfs/boot/vmlin
uz-linux
    -initrd
    /var/lib/libvirt/assets/beb6d03a495d89158422a27e07e61054.pool/vfs/boot/initr
amfs-linux.img
    -append
    root=/dev/root rootfstype=9p rootflags=trans=virtio,version=9p2000.L logleve
l=4 init=/bin/systemd elevator=deadline console=ttyS0,115200 console=tty0 ro
    -drive
    file=/var/lib/libvirt/assets/beb6d03a495d89158422a27e07e61054.pool/swap,if=n
one,id=drive-virtio-disk2,format=raw,cache=none,aio=native
    -device
    virtio-blk-pci,bus=pci.0,addr=0x8,drive=drive-virtio-disk2,id=virtio-disk2
    -fsdev
    local,security_model=mapped,id=fsdev-fs0,path=/var/lib/libvirt/assets/beb6d0
3a495d89158422a27e07e61054.pool/vfs
    -device
    virtio-9p-pci,id=fs0,fsdev=fsdev-fs0,mount_tag=/dev/root,bus=pci.0,addr=0x7
    -netdev
    tap,fd=19,id=hostnet0,vhost=on,vhostfd=22
    -device
    virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:e2:d2:03,bus=pci.0,addr=
0x3
    -chardev
    pty,id=charserial0
    -device
    isa-serial,chardev=charserial0,id=serial0
    -usb
    -device
    usb-tablet,id=input0
    -vnc
    127.0.0.1:2
    -vga
    cirrus
    -device
    virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

==== ENVIRONMENT (guest) ====

this VM doesn't run much more than basic services and a light httpd
process (darkhttpd) ... it's job is to rsync an upstream mirror, which
is then bind mounted into other guests.  not sure what to provide here
...

DISTRO
    Archlinux
ARCH
    x86_64
KERNEL
    3.1.5-1-ARCH
MODULES IN USE

# lsmod |grep virt
virtio_blk              5690  1
virtio_net             12290  0
virtio_balloon          3764  0
9pnet_virtio            5190  1
9pnet                  42052  2 9p,9pnet_virtio
virtio_pci              6447  0
virtio_ring             4261  5
virtio_blk,virtio_net,virtio_balloon,9pnet_virtio,virtio_pci
virtio                  3682  5
virtio_blk,virtio_net,virtio_balloon,9pnet_virtio,virtio_pci

==== DIAGNOSTICS (gdb) ====

slightly more detail here ...
http://mailman.archlinux.org/pipermail/arch-general/2011-December/023668.html
... and accompanying thread.

quick: after evaluating the source and gdb output, `ldconfig`
explicitly last-minute abort()s on attempt to add a duplicate entry to
the `aux-cache` file (something that should not happen by this point).
 not only is there somehow a duplicate, but the failing library also
initially shows up with a different inode number.  the original inode
matches the HOST inode ... the duplicates match the GUEST inode ... in
all tests, the guest inode is EXACTLY "off-by-two" greater than the
host(?) ... not sure if that's intended.  failure is consistent, but
the actual library doesn't matter -- if removed it just fails on
another in the same way.

`gdb --args ldconfig -v` (breakpoint, output, backtrace, and
source context provided below):

----------------------------------------------------------------------------
Reading symbols from /sbin/ldconfig...done.
(gdb) break cache.c:620 if (soname!=0x0 && strcmp(soname, "libgcrypt.so.11")==0)
Breakpoint 1 at 0x402e14: file cache.c, line 620.
(gdb) commands
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
>silent
>printf "\n---- soname: %s\n---- inode:  %i\n---- hash:   %i\n\n", soname, 
>id->ino, hash
>continue
>end
(gdb) run
Starting program: /sbin/ldconfig -v
/sbin/ldconfig: Can't stat /usr/lib64: No such file or directory

---- soname: libgcrypt.so.11
---- inode:  15344348
---- hash:   722

/usr/lib/libfakeroot:
        libfakeroot-0.so -> libfakeroot.so
/usr/lib/perl5/core_perl/CORE:
        libperl.so -> libperl.so
/lib:

---- soname: libgcrypt.so.11
---- inode:  15344350
---- hash:   834


---- soname: libgcrypt.so.11
---- inode:  15344350
---- hash:   834


Program received signal SIGABRT, Aborted.
0x000000000044f4fc in raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
64        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0  0x000000000044f4fc in raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x000000000040c20e in abort () at abort.c:93
#2  0x0000000000402e57 in insert_to_aux_cache (id=0x7fffffffd1b0,
flags=771, osversion=0, soname=0x6db360 "libgcrypt.so.11", used=1) at
cache.c:625
#3  0x0000000000403dea in add_to_aux_cache (stat_buf=<optimized out>,
flags=<optimized out>, osversion=<optimized out>, soname=<optimized
out>) at cache.c:650
#4  0x00000000004023cd in search_dir (entry=0x6d09d0) at ldconfig.c:880
#5  0x0000000000402d09 in search_dirs () at ldconfig.c:1023
#6  main (argc=2, argv=<optimized out>) at ldconfig.c:1372
(gdb) list cache.c:620,625
620       for (entry = aux_hash[hash]; entry; entry = entry->next)
621         if (id->ino == entry->id.ino
622             && id->ctime == entry->id.ctime
623             && id->size == entry->id.size
624             && id->dev == entry->id.dev)
625           abort ();
----------------------------------------------------------------------------

... per the above, `ldconfig` tries to add inode `15344348` as
`libgcrypt.so.11.7.0`, but that is totally WRONG from guest
perspective:

(guest) # stat --format="%i %n" /lib/l* | grep 15344348
15344348 /lib/libext2fs.so.2.4

... but that exact inode matches the host:

(host) # stat --format="%i %n" ./lib/libgcrypt.so.11.7.0
15344348 ./lib/libgcrypt.so.11.7.0

(guest) # stat --format="%i %n" /lib/libgcrypt.so.11.7.0
15344350 /lib/libgcrypt.so.11.7.0

==== CONCLUSION (confused) ====

i have no idea how `ldconfig` is getting the inodes crossed.  i
haven't yet tried none/passthru mode because i want qemu to run
unprivileged, and i thought `mapped` was the preferred way from
various papers/sources i've read.   comments?  i will manually run
whatever tests, as needed/requested, to track this down.

i realize i spewed a fair amount of output -- hopefully most is useful
-- but i've been stuck on this problem for far too long.  i'm only
willing to potentially waste other peoples time after burning a
mountain of my own, and while i learned a spectacular amount RE:gdb
and friends in the process ... i would be eternally grateful for *any*
assistance one could give, no matter how seemingly obvious.  i tried
hard -- i promise!!! :-)

thanks for your time thus far, very much appreciated.

-- 

C Anthony