Re: qemu-ppc64-linux-user: signal handler endianness in elfv2

[David Gibson]

On Thu, Feb 27, 2020 at 01:40:53PM -0600, Vincent Fazio wrote:
> David,
> 
> On 2/18/20 9:14 PM, David Gibson wrote:
> > On Sat, Feb 08, 2020 at 12:57:40PM -0600, Vincent Fazio wrote:
> > > I ran into a crash scenario in ppc64-linux-user when trying to run openssl
> > > compiled with gcc & musl which emits an ELFv2 binary. The target CPU was a
> > > big endian model (e6500)
> > > 
> > > Example:
> > > 
> > > 
> > > vfazio@vfazio1 ~/development/buildroot/output/target :( $ gdb
> > > ~/development/qemu/ppc64-linux-user/qemu-ppc64
> > > (gdb) run -d guest_errors -cpu e6500 -E 
> > > LD_LIBRARY_PATH="/home/vfazio/development/buildroot/output/target/lib/:/home/vfazio/development/buildroot/output/target/usr/lib/"
> > > lib/libc.so bin/bash
> > > Starting program: 
> > > /home/vfazio/development/qemu/ppc64-linux-user/qemu-ppc64
> > > -d guest_errors -cpu e6500 -E 
> > > LD_LIBRARY_PATH="/home/vfazio/development/buildroot/output/target/lib/:/home/vfazio/development/buildroot/output/target/usr/lib/"
> > > lib/libc.so bin/bash
> > > [Thread debugging using libthread_db enabled]
> > > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > > [New Thread 0x7ffff7ff9700 (LWP 27186)]
> > > invalid/unsupported opcode: 1f - 0c - 1a - 00 (7c005698) 0000000010028e58 > > > 0
> > > 
> > > Thread 1 "qemu-ppc64" received signal SIGSEGV, Segmentation fault.
> > > 0x00000000600a9257 in ldl_he_p (ptr=0x4c2c061000000000) at
> > > /home/vfazio/development/qemu/include/qemu/bswap.h:351
> > > 351        __builtin_memcpy(&r, ptr, sizeof(r));
> > > 
> > > (gdb) !readelf -h bin/bash | grep abi
> > >    Flags:                             0x2, abiv2
> > > (gdb) !readelf -h lib/libc.so | grep abi
> > >    Flags:                             0x2, abiv2
> > > <<<<<
> > > 
> > > openssl does "probing" for PPC features (Altivec, Crypto, etc) by trying 
> > > to
> > > execute possibly illegal instructions and handling them... hence that
> > > SIGILL. The problem is when it comes to jumping to the signal handler
> > > 
> > > 
> > > #0  0x00000000600a9257 in ldl_he_p (ptr=0x4c2c061000000000) at
> > > /home/vfazio/development/qemu/include/qemu/bswap.h:351
> > > #1  0x00000000600a92fe in ldl_be_p (ptr=0x4c2c061000000000) at
> > > /home/vfazio/development/qemu/include/qemu/bswap.h:449
> > > #2  0x00000000600c0790 in translator_ldl_swap (env=0x62923150,
> > > pc=5488768711647035392, do_swap=false) at
> > > /home/vfazio/development/qemu/include/exec/translator.h:201
> > > #3  0x000000006011c1ab in ppc_tr_translate_insn (dcbase=0x7fffffffd250,
> > > cs=0x6291ae80) at 
> > > /home/vfazio/development/qemu/target/ppc/translate.c:7856
> > > #4  0x000000006005ae70 in translator_loop (ops=0x60805fc0 <ppc_tr_ops>,
> > > db=0x7fffffffd250, cpu=0x6291ae80, tb=0x60a5f900
> > > <static_code_gen_buffer+1681600>, max_insns=512)
> > >      at /home/vfazio/development/qemu/accel/tcg/translator.c:102
> > > <<<<<
> > > 
> > > The handler has what looks like an LE address but then swaps it since the
> > > host is LE and the target is BE
> > > 
> > > 
> > > setup_rt_frame (sig=4, ka=0x628c5f80 <sigact_table+96>, info=0x62931658,
> > > set=0x7fffffffd2f8, env=0x62923150) at
> > > /home/vfazio/development/qemu/linux-user/ppc/signal.c:575
> > > 575            qemu_log_mask(LOG_GUEST_ERROR, "sa_handler NIP to "
> > > TARGET_FMT_lx "\n", ka->_sa_handler);
> > > (gdb) p/x ka->_sa_handler
> > > $11 = 0x10062c4c
> > > 579            env->nip = tswapl((target_ulong) ka->_sa_handler);
> > > (gdb) p/x env->nip
> > > $12 = 0x4c2c061000000000
> > > <<<<<
> > > 
> > > the memcpy later faults because that's obviously not a valid address from
> > > which to grab the PPC instructions that need to be translated.
> > > 
> > > I'm not familiar with the ELFv2 ABI or PPC assembly in general, so I'm not
> > > sure what to expect here. Typically ELFv2 was reserved for ppc64le, but 
> > > musl
> > > uses ELFv2 for all ppc64 targets. This likely wouldn't be an issue in
> > > ppc64LE since x86_64 is LE as well and no swap would take place.
> > > 
> > > Is the signal handler address tied to the endianess of the host? I noticed
> > > there was no swapping in elfload.c so wasn't sure...removing the swap gets
> > > me a little further in program execution but i do eventually crash (but 
> > > that
> > > may not be related to this specific issue).
> > I've had a look at this code, and as far as I can reason the tswap
> > here is correct.  I think the error must be somewhere else.
> > 
> > AFAICT, the ka structure is a verbatim copy of the sigaction structure
> > that the target supplied earlier with sigaction().  So, it should be
> > in target endianness.  env->nip needs to be in host endianness, so a
> > tswap() looks right.
> > 
> > I think we need to look earlier to see how an apparently LE value is
> > getting into the sigaction structure with a BE binary.
> 
> I spent a little time over the weekend looking at this.

Sorry, I should have replied to this earlier.

> In the ELFv2 ABI, function pointers are relative to the TOC stored in GPR2.
> There's no hop into .opd to pull an address that needs to be endian swapped
> relative to the host as in ELFv1.

Right.

> As best as I can tell, GPR2 is in QEMU's host endianness as it's calculated

Sort of.  In hardware the register has no endianness, since it's not
byte-addressable.  In qemu's modelling, env->gpr[2], like all the
registers, is stored in host endianness.  That's because the fact
those registers have an in memory representation is a qemu/host side
detail.

> from GPR12 or the ELF load address + entry point so there should be no need
> to swap the value of the signal handler function pointer in ELFv2.

Erm... if the address was coming from a register, yes.  But AFAICT
it's not coming from a register, it's coming from the guest's struct
sigaction, which I'd expect to be in target/guest endianness.


> 
> >>>
> vfazio@vfazio1 ~/development/buildroot $
> output/host/powerpc64-buildroot-linux-musl/bin/objdump -x
> output/build/libopenssl-1.1.1d/libcrypto.so | egrep -e
> ".TOC|\<_init|ill_handler"
> 0000000000186014 l     F .text    0000000000000034 0x60 ill_handler
> 00000000002b7f00 l     O .got    0000000000000000 .TOC.
> 000000000007de00 l     F .init    0000000000000010 0x60 _init
> 
> 000000000007de00 <_init>:
>    7de00:       3c 4c 00 24     addis   r2,r12,36
>    7de04:       38 42 a1 00     addi    r2,r2,-24320
> 
> (gdb) p/x 0x7de00 + (36 << 16) - 24320
> $69 = 0x2b7f00
> 
>    7f180:       3f a2 ff ed     addis   r29,r2,-19
>    7f184:       3b bd e1 14     addi    r29,r29,-7916
> 
> (gdb) p/x 0x2b7f00 - (19 << 16) - 7916
> $75 = 0x186014
> 
> (gdb) where
> #0  setup_rt_frame (sig=4, ka=0x628c5f80 <sigact_table+96>, info=0x629314d8,
> set=0x7fffffffd368, env=0x629240c0) at
> /home/vfazio/development/qemu/linux-user/ppc/signal.c:579
> #1  0x000000006008d6b3 in handle_pending_signal (cpu_env=0x629240c0, sig=4,
> k=0x629314d0) at /home/vfazio/development/qemu/linux-user/signal.c:922
> #2  0x000000006008d84b in process_pending_signals (cpu_env=0x629240c0) at
> /home/vfazio/development/qemu/linux-user/signal.c:960
> #3  0x0000000060097ec7 in cpu_loop (env=0x629240c0) at
> /home/vfazio/development/qemu/linux-user/ppc/cpu_loop.c:469
> #4  0x000000006006444e in main (argc=9, argv=0x7fffffffde48,
> envp=0x7fffffffde98) at /home/vfazio/development/qemu/linux-user/main.c:865
> 
> (gdb) p/x env->gpr[2]
> $83 = 0x4000c21f00
> (gdb) p/x env->gpr[2] - (19 << 16) - 7916
> $84 = 0x4000af0014
> (gdb) p/x ka->_sa_handler
> $85 = 0x4000af0014
> (gdb) p/x *ka->_sa_handler
> $86 = 0x13004c3c
> 
> 0000000000186014 <ill_handler>:
>   186014:       3c 4c 00 13     addis   r2,r12,19
>   186018:       38 42 1e ec     addi    r2,r2,7916
>   18601c:       7c 08 02 a6     mflr    r0
> 
> <<<
> 
> If this seems reasonable, I can submit a patch.
> 

-- 
David Gibson                    | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
                                | _way_ _around_!
http://www.ozlabs.org/~dgibson

signature.asc
Description: PGP signature