Re: qemu-ppc64-linux-user: signal handler endianness in elfv2

[Vincent Fazio]

David,

On 2/18/20 9:14 PM, David Gibson wrote:
> On Sat, Feb 08, 2020 at 12:57:40PM -0600, Vincent Fazio wrote:
> > I ran into a crash scenario in ppc64-linux-user when trying to run openssl
> > compiled with gcc & musl which emits an ELFv2 binary. The target CPU was a
> > big endian model (e6500)
> >
> > Example:
> >
> >
> > vfazio@vfazio1 ~/development/buildroot/output/target :( $ gdb
> > ~/development/qemu/ppc64-linux-user/qemu-ppc64
> > (gdb) run -d guest_errors -cpu e6500 -E 
> > LD_LIBRARY_PATH="/home/vfazio/development/buildroot/output/target/lib/:/home/vfazio/development/buildroot/output/target/usr/lib/"
> > lib/libc.so bin/bash
> > Starting program: /home/vfazio/development/qemu/ppc64-linux-user/qemu-ppc64
> > -d guest_errors -cpu e6500 -E 
> > LD_LIBRARY_PATH="/home/vfazio/development/buildroot/output/target/lib/:/home/vfazio/development/buildroot/output/target/usr/lib/"
> > lib/libc.so bin/bash
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > [New Thread 0x7ffff7ff9700 (LWP 27186)]
> > invalid/unsupported opcode: 1f - 0c - 1a - 00 (7c005698) 0000000010028e58 0
> >
> > Thread 1 "qemu-ppc64" received signal SIGSEGV, Segmentation fault.
> > 0x00000000600a9257 in ldl_he_p (ptr=0x4c2c061000000000) at
> > /home/vfazio/development/qemu/include/qemu/bswap.h:351
> > 351        __builtin_memcpy(&r, ptr, sizeof(r));
> >
> > (gdb) !readelf -h bin/bash | grep abi
> >    Flags:                             0x2, abiv2
> > (gdb) !readelf -h lib/libc.so | grep abi
> >    Flags:                             0x2, abiv2
> > <<<<<
> >
> > openssl does "probing" for PPC features (Altivec, Crypto, etc) by trying to
> > execute possibly illegal instructions and handling them... hence that
> > SIGILL. The problem is when it comes to jumping to the signal handler
> >
> >
> > #0  0x00000000600a9257 in ldl_he_p (ptr=0x4c2c061000000000) at
> > /home/vfazio/development/qemu/include/qemu/bswap.h:351
> > #1  0x00000000600a92fe in ldl_be_p (ptr=0x4c2c061000000000) at
> > /home/vfazio/development/qemu/include/qemu/bswap.h:449
> > #2  0x00000000600c0790 in translator_ldl_swap (env=0x62923150,
> > pc=5488768711647035392, do_swap=false) at
> > /home/vfazio/development/qemu/include/exec/translator.h:201
> > #3  0x000000006011c1ab in ppc_tr_translate_insn (dcbase=0x7fffffffd250,
> > cs=0x6291ae80) at /home/vfazio/development/qemu/target/ppc/translate.c:7856
> > #4  0x000000006005ae70 in translator_loop (ops=0x60805fc0 <ppc_tr_ops>,
> > db=0x7fffffffd250, cpu=0x6291ae80, tb=0x60a5f900
> > <static_code_gen_buffer+1681600>, max_insns=512)
> >      at /home/vfazio/development/qemu/accel/tcg/translator.c:102
> > <<<<<
> >
> > The handler has what looks like an LE address but then swaps it since the
> > host is LE and the target is BE
> >
> >
> > setup_rt_frame (sig=4, ka=0x628c5f80 <sigact_table+96>, info=0x62931658,
> > set=0x7fffffffd2f8, env=0x62923150) at
> > /home/vfazio/development/qemu/linux-user/ppc/signal.c:575
> > 575            qemu_log_mask(LOG_GUEST_ERROR, "sa_handler NIP to "
> > TARGET_FMT_lx "\n", ka->_sa_handler);
> > (gdb) p/x ka->_sa_handler
> > $11 = 0x10062c4c
> > 579            env->nip = tswapl((target_ulong) ka->_sa_handler);
> > (gdb) p/x env->nip
> > $12 = 0x4c2c061000000000
> > <<<<<
> >
> > the memcpy later faults because that's obviously not a valid address from
> > which to grab the PPC instructions that need to be translated.
> >
> > I'm not familiar with the ELFv2 ABI or PPC assembly in general, so I'm not
> > sure what to expect here. Typically ELFv2 was reserved for ppc64le, but musl
> > uses ELFv2 for all ppc64 targets. This likely wouldn't be an issue in
> > ppc64LE since x86_64 is LE as well and no swap would take place.
> >
> > Is the signal handler address tied to the endianess of the host? I noticed
> > there was no swapping in elfload.c so wasn't sure...removing the swap gets
> > me a little further in program execution but i do eventually crash (but that
> > may not be related to this specific issue).
> I've had a look at this code, and as far as I can reason the tswap
> here is correct.  I think the error must be somewhere else.
>
> AFAICT, the ka structure is a verbatim copy of the sigaction structure
> that the target supplied earlier with sigaction().  So, it should be
> in target endianness.  env->nip needs to be in host endianness, so a
> tswap() looks right.
>
> I think we need to look earlier to see how an apparently LE value is
> getting into the sigaction structure with a BE binary.

I spent a little time over the weekend looking at this.

In the ELFv2 ABI, function pointers are relative to the TOC stored in 
GPR2. There's no hop into .opd to pull an address that needs to be 
endian swapped relative to the host as in ELFv1.

As best as I can tell, GPR2 is in QEMU's host endianness as it's 
calculated from GPR12 or the ELF load address + entry point so there 
should be no need to swap the value of the signal handler function 
pointer in ELFv2.

>>>
vfazio@vfazio1 ~/development/buildroot $ 
output/host/powerpc64-buildroot-linux-musl/bin/objdump -x 
output/build/libopenssl-1.1.1d/libcrypto.so | egrep -e 
".TOC|\<_init|ill_handler"
0000000000186014 l     F .text    0000000000000034 0x60 ill_handler
00000000002b7f00 l     O .got    0000000000000000 .TOC.
000000000007de00 l     F .init    0000000000000010 0x60 _init

000000000007de00 <_init>:
   7de00:       3c 4c 00 24     addis   r2,r12,36
   7de04:       38 42 a1 00     addi    r2,r2,-24320

(gdb) p/x 0x7de00 + (36 << 16) - 24320
$69 = 0x2b7f00

   7f180:       3f a2 ff ed     addis   r29,r2,-19
   7f184:       3b bd e1 14     addi    r29,r29,-7916

(gdb) p/x 0x2b7f00 - (19 << 16) - 7916
$75 = 0x186014

(gdb) where
#0  setup_rt_frame (sig=4, ka=0x628c5f80 <sigact_table+96>, 
info=0x629314d8, set=0x7fffffffd368, env=0x629240c0) at 
/home/vfazio/development/qemu/linux-user/ppc/signal.c:579
#1  0x000000006008d6b3 in handle_pending_signal (cpu_env=0x629240c0, 
sig=4, k=0x629314d0) at 
/home/vfazio/development/qemu/linux-user/signal.c:922
#2  0x000000006008d84b in process_pending_signals (cpu_env=0x629240c0) 
at /home/vfazio/development/qemu/linux-user/signal.c:960
#3  0x0000000060097ec7 in cpu_loop (env=0x629240c0) at 
/home/vfazio/development/qemu/linux-user/ppc/cpu_loop.c:469
#4  0x000000006006444e in main (argc=9, argv=0x7fffffffde48, 
envp=0x7fffffffde98) at /home/vfazio/development/qemu/linux-user/main.c:865

(gdb) p/x env->gpr[2]
$83 = 0x4000c21f00
(gdb) p/x env->gpr[2] - (19 << 16) - 7916
$84 = 0x4000af0014
(gdb) p/x ka->_sa_handler
$85 = 0x4000af0014
(gdb) p/x *ka->_sa_handler
$86 = 0x13004c3c

0000000000186014 <ill_handler>:
  186014:       3c 4c 00 13     addis   r2,r12,19
  186018:       38 42 1e ec     addi    r2,r2,7916
  18601c:       7c 08 02 a6     mflr    r0

<<<

If this seems reasonable, I can submit a patch.

--
Vincent Fazio
Embedded Software Engineer - Linux
Extreme Engineering Solutions, Inc
http://www.xes-inc.com