Re: [Qemu-devel] [PATCH v2 13/22] target/openrisc: Fix cpu_mmu_index

[Stafford Horne]

Hello,

I think I found out something.

in: target/openrisc/sys_helper.c:92

When we write to `env->tlb.dtlb[idx].tr`  in helper_mtspr():
  93          case TO_SPR(1, 640) ... TO_SPR(1, 640 + TLB_SIZE - 1):
/* DTLBW0TR 0-127 */
  94              idx = spr - TO_SPR(1, 640);
  95              env->tlb.dtlb[idx].tr = rb;


Somehow we are overlapping with `cpu->tb_jmp_cache`,  these are both
pointing to the same spot in memory.

(gdb) p &cs->tb_jmp_cache[3014]
$9 = (struct TranslationBlock **) 0x55555608b300
(gdb) p &env->tlb.dtlb[idx].tr
$10 = (uint32_t *) 0x55555608b304


I can't see why yet, but it should be something simple.  Still looking.

-Stafford
On Sun, Jun 24, 2018 at 12:44 PM Stafford Horne <address@hidden> wrote:
>
> On Tue, Jun 19, 2018 at 3:41 AM Richard Henderson
> <address@hidden> wrote:
> >
> > The code in cpu_mmu_index does not properly honor SR_DME.
> > This bug has workarounds elsewhere in that we flush the
> > tlb more often than necessary, on the state changes that
> > should be reflected in a change of mmu_index.
> >
> > Fixing this means that we can respect the mmu_index that
> > is given to tlb_flush.
> >
> > Signed-off-by: Richard Henderson <address@hidden>
> > ---
> >  target/openrisc/cpu.h              | 23 +++++++++++++--------
> >  target/openrisc/interrupt.c        |  4 ----
> >  target/openrisc/interrupt_helper.c | 15 +++-----------
> >  target/openrisc/mmu.c              | 33 +++++++++++++++++++++++++++---
> >  target/openrisc/sys_helper.c       |  4 ----
> >  target/openrisc/translate.c        |  2 +-
> >  6 files changed, 49 insertions(+), 32 deletions(-)
>
>
> Hello,
>
> I am trying to test these patches running a linux kernel.
>
> For some reason this is causing a strange failure with SMP but not
> single core, I see an OpenRISC target pointer is making its way into
> the tb_jmp_cache.  I don't think this is right and I am trying to
> figure out why this happens and why this patch triggers it.
>
> When bisecting to this commit I get:
>
> [New Thread 0x7fffe9f11700 (LWP 4210)]
>
> [    0.000000] Compiled-in FDT at (ptrval)
> [    0.000000] Linux version
> 4.18.0-rc1-simple-smp-00006-gd5d0782e3db9-dirty
> (address@hidden) (gcc version 9.0.0 20180426
> (experimental) (GCC)) #1013 SMP Sat Jun 23 17:11:42 JST 2018
> [    0.000000] CPU: OpenRISC-0 (revision 0) @20 MHz
> [    0.000000] -- dcache disabled
> [    0.000000] -- icache disabled
> [    0.000000] -- dmmu:   64 entries, 1 way(s)
> [    0.000000] -- immu:   64 entries, 1 way(s)
> [    0.000000] -- additional features:
> [    0.000000] -- power management
> [    0.000000] -- PIC
> [    0.000000] -- timer
> [    0.000000] setup_memory: Memory: 0x0-0x2000000
> [    0.000000] Setting up paging and PTEs.
> [    0.000000] map_ram: Memory: 0x0-0x2000000
> [    0.000000] itlb_miss_handler (ptrval)
> [    0.000000] dtlb_miss_handler (ptrval)
> [    0.000000] OpenRISC Linux -- http://openrisc.io
> [    0.000000] percpu: Embedded 6 pages/cpu @(ptrval) s18880 r8192 d22080 
> u49152
> [    0.000000] Built 1 zonelists, mobility grouping off.  Total pages: 4080
> [    0.000000] Kernel command line: earlycon
> [    0.000000] earlycon: ns16550a0 at MMIO 0x90000000 (options '115200')
> [    0.000000] bootconsole [ns16550a0] enabled
> [    0.000000] Dentry cache hash table entries: 4096 (order: 1, 16384 bytes)
> [    0.000000] Inode-cache hash table entries: 2048 (order: 0, 8192 bytes)
> [    0.000000] Sorting __ex_table...
> [    0.000000] Memory: 22336K/32768K available (3309K kernel code, 96K
> rwdata, 736K rodata, 5898K init, 91K bss, 10432K reserved, 0K
> cma-reserved)
> [    0.000000] mem_init_done ...........................................
> [    0.000000] Hierarchical RCU implementation.
> [    0.000000] NR_IRQS: 32, nr_irqs: 32, preallocated irqs: 0
> [    0.000000] clocksource: openrisc_timer: mask: 0xffffffff
> max_cycles: 0xffffffff, max_idle_ns: 95563022313 ns
> [    0.000000] 40.00 BogoMIPS (lpj=200000)
> [    0.000000] pid_max: default: 32768 minimum: 301
> [    0.000000] Mount-cache hash table entries: 2048 (order: 0, 8192 bytes)
> [    0.000000] Mountpoint-cache hash table entries: 2048 (order: 0, 8192 
> bytes)
>
>
> (gdb) bt
> #0  0x00005555556d3e59 in tb_lookup__cpu_state (cf_mask=0,
> flags=<synthetic pointer>, cs_base=<synthetic pointer>, pc=<synthetic
> pointer>, cpu=0x555555f81300)
>     at /home/shorne/work/openrisc/qemu/include/exec/tb-lookup.h:31
> #1  0x00005555556d3e59 in tb_find (cf_mask=0, tb_exit=0,
> last_tb=0x7fffe223ff00 <code_gen_buffer+2358995>, cpu=0x555555f81300)
> at /home/shorne/work/openrisc/qemu/accel/tcg/cpu-exec.c:390
> #2  0x00005555556d3e59 in cpu_exec (address@hidden) at
> /home/shorne/work/openrisc/qemu/accel/tcg/cpu-exec.c:735
> #3  0x00005555556a0d2b in tcg_cpu_exec (address@hidden)
> at /home/shorne/work/openrisc/qemu/cpus.c:1362
> #4  0x00005555556a238e in qemu_tcg_rr_cpu_thread_fn (arg=<optimized
> out>) at /home/shorne/work/openrisc/qemu/cpus.c:1461
> #5  0x0000555555886005 in qemu_thread_start (args=0x555555f93ef0) at
> /home/shorne/work/openrisc/qemu/util/qemu-thread-posix.c:507
> #6  0x00007ffff2a18564 in start_thread () at /lib64/libpthread.so.0
> #7  0x00007ffff274c31f in clone () at /lib64/libc.so.6
> (gdb) l
> 26          uint32_t hash;
> 27
> 28          cpu_get_tb_cpu_state(env, pc, cs_base, flags);
> 29          hash = tb_jmp_cache_hash_func(*pc);
> 30          tb = atomic_rcu_read(&cpu->tb_jmp_cache[hash]);
> 31          if (likely(tb &&
> 32                     tb->pc == *pc &&
> 33                     tb->cs_base == *cs_base &&
> 34                     tb->flags == *flags &&
> 35                     tb->trace_vcpu_dstate == *cpu->trace_dstate &&
> (gdb) p tb
> $1 = (TranslationBlock *) 0xc03c90a8
>
>
> To reproduce I am running qemu with:
>   qemu-system-or1k -cpu or1200 -M or1k-sim -kernel
> or1k-linux-4.18-rc1-smp -serial stdio -nographic -monitor none -smp
> cpus=2 -m 128
>
> Kernel (need to gunzip):
>   SMP - http://shorne.noip.me/downloads/or1k-linux-4.18-rc1-smp.gz
>   Single - http://shorne.noip.me/downloads/or1k-linux-4.18-rc1.gz
>
> I will continue to investigate, I just figured out SMP triggers it so
> maybe that will uncover something more.
>
> Sorry, if this mail gets clobbered I am using the gmail web interface.
>
> -Stafford