Re: mac99 SMP

[Andrew Randrianasulu]

On Wed, Mar 5, 2025 at 5:21 PM Andrew Randrianasulu
<randrianasulu@gmail.com> wrote:
>
>
>
> ср, 5 мар. 2025 г., 16:33 BALATON Zoltan <balaton@eik.bme.hu>:
>>
>> On Wed, 5 Mar 2025, Andrew Randrianasulu wrote:
>> > On Wed, Mar 5, 2025 at 5:46 AM Andrew Randrianasulu
>> > <randrianasulu@gmail.com> wrote:
>> >>
>> >> On Wed, Mar 5, 2025 at 5:13 AM Andrew Randrianasulu
>> >> <randrianasulu@gmail.com> wrote:
>> >>>
>> >>>
>> >>>
>> >>> ср, 5 мар. 2025 г., 04:10 BALATON Zoltan <balaton@eik.bme.hu>:
>> >>>>
>> >>>> On Wed, 5 Mar 2025, Andrew Randrianasulu wrote:
>> >>>>>> I tried  "set trace-commands on" before setting logging on.
>> >>>>>>
>> >>>>>> It looks better now?
>> >>>>
>> >>>> Yes, this is more readable but still don't know where it is stuck when 
>> >>>> you
>> >>>> just run with continue as this shows when you're stepping through that
>> >>>> works. We would need a backtrace when you start it with continue then 
>> >>>> when
>> >>>> it's waiting for the second CPU after it started it but before it 
>> >>>> reports
>> >>>> it's stuck press ctrl-c then "thread apply all backtrace full" to get
>> >>>> where the CPUs are at that point.
>> >>
>> >> +target remote 10.0.2.2:1234
>> >> Remote debugging using 10.0.2.2:1234
>> >> 0xfff00100 in ?? ()
>> >> +c
>> >> Continuing.
>> >>
>> >> Thread 1 hit Breakpoint 1.1, 0xc0067d4c in smp_core99_kick_cpu () at
>> >> ./arch/powerpc/include/asm/io.h:167
>> >> 167    DEF_MMIO_OUT_D(out_8,   8, stb);
>> >> +c
>> >> Continuing.
>> >>
>> >> Thread 1 hit Breakpoint 1.2, smp_core99_kick_cpu (nr=1) at
>> >> arch/powerpc/platforms/powermac/smp.c:802
>> >> 802        if (nr < 0 || nr > 3)
>> >> +c
>> >> Continuing.
>> >>
>> >> Thread 2 received signal SIGINT, Interrupt.
>> >> [Switching to Thread 1.2]
>> >> 0xc000439c in InstructionAccess_virt () at
>> >> arch/powerpc/kernel/head_book3s_32.S:308
>> >> 308        b    interrupt_return
>> >> +thread apply all backtrace full
>> >>
>> >> Thread 2 (Thread 1.2 (CPU#1 [running])):
>> >> +backtrace full
>> >> #0  0xc000439c in InstructionAccess_virt () at
>> >> arch/powerpc/kernel/head_book3s_32.S:308
>> >> No locals.
>> >> #1  0x00000000 in ?? ()
>> >> No symbol table info available.
>> >> Backtrace stopped: Cannot access memory at address 0xefff4124
>> >>
>> >> Thread 1 (Thread 1.1 (CPU#0 [running])):
>> >> +backtrace full
>> >> #0  0xc0023044 in __cpu_up (cpu=1, tidle=<optimized out>) at
>> >> arch/powerpc/kernel/smp.c:1329
>> >>         boot_spin_ms = 5000
>> >>         booting = <optimized out>
>> >>         hp_spin_ms = 1
>> >>         deadline = <optimized out>
>> >>         rc = 0
>> >>         spin_wait_ms = 5000
>> >>         __dummy = <optimized out>
>> >>         __dummy2 = <optimized out>
>> >>         __dummy = <optimized out>
>> >>         __dummy2 = <optimized out>
>> >> #1  0xc008a454 in bringup_cpu (cpu=1) at kernel/cpu.c:877
>> >>         st = 0xeeddcb24
>> >>         idle = 0xc19956a0
>> >>         ret = <optimized out>
>> >>         __vpp_verify = <optimized out>
>> >> #2  0xc008ad6c in cpuhp_invoke_callback (cpu=cpu@entry=1,
>> >> state=CPUHP_BRINGUP_CPU, bringup=bringup@entry=true,
>> >> node=node@entry=0x0, lastp=lastp@entry=0x0) at kernel/cpu.c:194
>> >>         st = 0xeeddcb24
>> >>         step = 0xc1591990 <cpuhp_hp_states+1740>
>> >>         cbm = <optimized out>
>> >>         cb = 0xc008a3f4 <bringup_cpu>
>> >>         ret = <optimized out>
>> >>         cnt = <optimized out>
>> >>         __vpp_verify = <optimized out>
>> >> #3  0xc008bde4 in __cpuhp_invoke_callback_range
>> >> (bringup=bringup@entry=true, cpu=cpu@entry=1, st=st@entry=0xeeddcb24,
>> >> target=target@entry=CPUHP_BRINGUP_CPU, nofail=nofail@entry=false) at
>> >> kernel/cpu.c:965
>> >>         err = <optimized out>
>> >>         state = <optimized out>
>> >>         ret = <optimized out>
>> >> #4  0xc008c11c in cpuhp_invoke_callback_range (bringup=true, cpu=1,
>> >> st=0xeeddcb24, target=CPUHP_BRINGUP_CPU) at kernel/cpu.c:989
>> >> No locals.
>> >> #5  cpuhp_up_callbacks (target=CPUHP_BRINGUP_CPU, cpu=1,
>> >> st=0xeeddcb24) at kernel/cpu.c:1020
>> >>         prev_state = <optimized out>
>> >>         ret = 0
>> >>         prev_state = <optimized out>
>> >>         ret = <optimized out>
>> >>         __func__ = "cpuhp_up_callbacks"
>> >>         __UNIQUE_ID_ddebug747 = {modname = 0xc0fb2a54 "cpu", function
>> >> = 0xc0dd4bec <__func__.0> "cpuhp_up_callbacks", filename = 0xc0fae87c
>> >> "kernel/cpu.c", format = 0xc0fae96c "CPU UP failed (%d) CPU %u state
>> >> %s (%d)\n", lineno = 1022, class_id = 63, flags = 0, key =
>> >> {dd_key_true = {key = {enabled = {counter = 0}, {type = 3238560028,
>> >> entries = 0xc108811c, next = 0xc108811c}}}, dd_key_false = {key =
>> >> {enabled = {counter = 0}, {type = 3238560028, entries = 0xc108811c,
>> >> next = 0xc108811c}}}}}
>> >>         branch = <optimized out>
>> >>         __ret_warn_on = <optimized out>
>> >> #6  _cpu_up (tasks_frozen=0, target=CPUHP_BRINGUP_CPU, cpu=1) at
>> >> kernel/cpu.c:1690
>> >>         st = 0xeeddcb24
>> >>         idle = <optimized out>
>> >>         ret = <optimized out>
>> >>         st = <optimized out>
>> >>         idle = <optimized out>
>> >>         ret = <optimized out>
>> >>         out = <optimized out>
>> >>         __vpp_verify = <optimized out>
>> >>         __ptr = <optimized out>
>> >>         __UNIQUE_ID_x_756 = <optimized out>
>> >>         __UNIQUE_ID_y_757 = <optimized out>
>> >> #7  cpu_up (cpu=cpu@entry=1, target=CPUHP_ONLINE) at kernel/cpu.c:1722
>> >>         err = 0
>> >> #8  0xc14188a8 in cpuhp_bringup_mask (mask=<optimized out>,
>> >> target=<optimized out>, ncpus=3) at kernel/cpu.c:1788
>> >>         st = <optimized out>
>> >>         __vpp_verify = <optimized out>
>> >>         cpu = 1
>> >> #9  bringup_nonboot_cpus (max_cpus=<optimized out>) at kernel/cpu.c:1896
>> >> No locals.
>> >> #10 0xc1420480 in smp_init () at kernel/smp.c:1009
>> >>         num_nodes = 1
>> >>         num_cpus = <optimized out>
>> >> #11 0xc1403ee8 in kernel_init_freeable () at init/main.c:1572
>> >> No locals.
>> >> #12 0xc0008c74 in kernel_init (unused=<optimized out>) at init/main.c:1469
>> >>         ret = <optimized out>
>> >> #13 0xc00212ec in ret_from_kernel_user_thread () at
>> >> arch/powerpc/kernel/entry_32.S:193
>> >> No locals.
>> >>
>> >> =====
>> >>
>> >> Ugh ... "InstructionAccess_virt () at 
>> >> arch/powerpc/kernel/head_book3s_32.S:308"
>> >>
>> >> https://elixir.bootlin.com/linux/v6.12.17/source/arch/powerpc/kernel/head_book3s_32.S#L308
>> >>
>> >> Data Access Expection? But why .....
>> >
>> > Si tried this maintenance packet trick.
>> >
>> > it still works with 0x5 (IRQ enabled, timers not) and fail if I
>> > disable it (0x0) completely
>> >
>> >
>> >
>> > Remote connection closed
>> > +target remote 10.0.2.2:1234
>> > Remote debugging using 10.0.2.2:1234
>> >
>> > Thread 1 received signal SIGTRAP, Trace/breakpoint trap.
>> > 0xfff00100 in ?? ()
>> > +list breakpoints
>> > Function "breakpoints" not defined.
>> > +maintenance packet Qqemu.sstep=0x5
>> > sending: Qqemu.sstep=0x5
>> > received: "OK"
>> > +c
>> > Continuing.
>> >
>> > Thread 1 hit Breakpoint 1.1, 0xc0067d4c in smp_core99_kick_cpu () at
>> > ./arch/powerpc/include/asm/io.h:167
>> > 167    DEF_MMIO_OUT_D(out_8,   8, stb);
>> > +c
>> > Continuing.
>> >
>> > Thread 1 hit Breakpoint 1.2, smp_core99_kick_cpu (nr=1) at
>> > arch/powerpc/platforms/powermac/smp.c:802
>> > 802        if (nr < 0 || nr > 3)
>> > +s 100
>> > pte_update (mm=<error reading variable: Cannot access memory at
>> > address 0x0>, clr=4294967293, huge=0, addr=<optimized out>,
>> > p=0xc182b000, set=12535189) at
>> > ./arch/powerpc/include/asm/book3s/32/pgtable.h:267
>> > 267            asm volatile(
>> > +maintenance packet Qqemu.sstep=0x0
>> > sending: Qqemu.sstep=0x0
>> > Remote connection closed
>> > +target remote 10.0.2.2:1234
>> > Remote debugging using 10.0.2.2:1234
>> > 0xfff00100 in ?? ()
>> > +maintenance packet Qqemu.sstep=0x0
>> > sending: Qqemu.sstep=0x0
>> > received: "OK"
>> > +c
>> > Continuing.
>> >
>> > Thread 1 hit Breakpoint 1.1, 0xc0067d4c in smp_core99_kick_cpu () at
>> > ./arch/powerpc/include/asm/io.h:167
>> > 167    DEF_MMIO_OUT_D(out_8,   8, stb);
>> > +c
>> > Continuing.
>> >
>> > Thread 1 hit Breakpoint 1.2, smp_core99_kick_cpu (nr=1) at
>> > arch/powerpc/platforms/powermac/smp.c:802
>> > 802        if (nr < 0 || nr > 3)
>> > +s 100
>> >
>> > Thread 1 received signal SIGINT, Interrupt.
>> > 0xc0023034 in __cpu_up (cpu=1, tidle=<optimized out>) at
>> > arch/powerpc/kernel/smp.c:1329
>> > 1329        spin_until_cond(cpu_callin_map[cpu] ||
>> > time_is_before_jiffies(deadline));
>> > +thread apply all bt full
>> >
>> > Thread 2 (Thread 1.2 (CPU#1 [running])):
>> > +bt full
>> > #0  0x00000100 in ?? ()
>> > No symbol table info available.
>> >
>> > Thread 1 (Thread 1.1 (CPU#0 [running])):
>> > +bt full
>> > #0  0xc0023034 in __cpu_up (cpu=1, tidle=<optimized out>) at
>> > arch/powerpc/kernel/smp.c:1329
>> >        __dummy = <optimized out>
>> >        __dummy2 = 0
>> >        boot_spin_ms = 5000
>> >        booting = <optimized out>
>> >        hp_spin_ms = 1
>> >        deadline = <optimized out>
>> >        rc = 0
>> >        spin_wait_ms = 5000
>> >        __dummy = <optimized out>
>> >        __dummy2 = <optimized out>
>> >        __dummy = <optimized out>
>> >        __dummy2 = <optimized out>
>> > #1  0xc008a454 in bringup_cpu (cpu=1) at kernel/cpu.c:877
>> >        st = 0xeeddcb24
>> >        idle = 0xc1990000
>> >        ret = <optimized out>
>> >        __vpp_verify = <optimized out>
>> > #2  0xc008ad6c in cpuhp_invoke_callback (cpu=cpu@entry=1,
>> > state=CPUHP_BRINGUP_CPU, bringup=bringup@entry=true,
>> > node=node@entry=0x0, lastp=lastp@entry=0x0) at kernel/cpu.c:194
>> >        st = 0xeeddcb24
>> >        step = 0xc1591990 <cpuhp_hp_states+1740>
>> >        cbm = <optimized out>
>> >        cb = 0xc008a3f4 <bringup_cpu>
>> >        ret = <optimized out>
>> >        cnt = <optimized out>
>> >        __vpp_verify = <optimized out>
>> > #3  0xc008bde4 in __cpuhp_invoke_callback_range
>> > (bringup=bringup@entry=true, cpu=cpu@entry=1, st=st@entry=0xeeddcb24,
>> > target=target@entry=CPUHP_BRINGUP_CPU, nofail=nofail@entry=false) at
>> > kernel/cpu.c:965
>> >        err = <optimized out>
>> >        state = <optimized out>
>> >        ret = <optimized out>
>> > #4  0xc008c11c in cpuhp_invoke_callback_range (bringup=true, cpu=1,
>> > st=0xeeddcb24, target=CPUHP_BRINGUP_CPU) at kernel/cpu.c:989
>> > No locals.
>> > #5  cpuhp_up_callbacks (target=CPUHP_BRINGUP_CPU, cpu=1,
>> > st=0xeeddcb24) at kernel/cpu.c:1020
>> >        prev_state = <optimized out>
>> >        ret = 0
>> >        prev_state = <optimized out>
>> >        ret = <optimized out>
>> >        __func__ = "cpuhp_up_callbacks"
>> >        __UNIQUE_ID_ddebug747 = {modname = 0xc0fb2a54 "cpu", function
>> > = 0xc0dd4bec <__func__.0> "cpuhp_up_callbacks", filename = 0xc0fae87c
>> > "kernel/cpu.c", format = 0xc0fae96c "CPU UP failed (%d) CPU %u state
>> > %s (%d)\n", lineno = 1022, class_id = 63, flags = 0, key =
>> > {dd_key_true = {key = {enabled = {counter = 0}, {type = 3238560028,
>> > entries = 0xc108811c, next = 0xc108811c}}}, dd_key_false = {key =
>> > {enabled = {counter = 0}, {type = 3238560028, entries = 0xc108811c,
>> > next = 0xc108811c}}}}}
>> >        branch = <optimized out>
>> >        __ret_warn_on = <optimized out>
>> > #6  _cpu_up (tasks_frozen=0, target=CPUHP_BRINGUP_CPU, cpu=1) at
>> > kernel/cpu.c:1690
>> >        st = 0xeeddcb24
>> >        idle = <optimized out>
>> >        ret = <optimized out>
>> >        st = <optimized out>
>> >        idle = <optimized out>
>> >        ret = <optimized out>
>> >        out = <optimized out>
>> >        __vpp_verify = <optimized out>
>> >        __ptr = <optimized out>
>> >        __UNIQUE_ID_x_756 = <optimized out>
>> >        __UNIQUE_ID_y_757 = <optimized out>
>> > #7  cpu_up (cpu=cpu@entry=1, target=CPUHP_ONLINE) at kernel/cpu.c:1722
>> >        err = 0
>> > #8  0xc14188a8 in cpuhp_bringup_mask (mask=<optimized out>,
>> > target=<optimized out>, ncpus=3) at kernel/cpu.c:1788
>> >        st = <optimized out>
>> >        __vpp_verify = <optimized out>
>> >        cpu = 1
>> > #9  bringup_nonboot_cpus (max_cpus=<optimized out>) at kernel/cpu.c:1896
>> > No locals.
>> > #10 0xc1420480 in smp_init () at kernel/smp.c:1009
>> >        num_nodes = 1
>> >        num_cpus = <optimized out>
>> > #11 0xc1403ee8 in kernel_init_freeable () at init/main.c:1572
>> > No locals.
>> > #12 0xc0008c74 in kernel_init (unused=<optimized out>) at init/main.c:1469
>> >        ret = <optimized out>
>> > #13 0xc00212ec in ret_from_kernel_user_thread () at
>> > arch/powerpc/kernel/entry_32.S:193
>> > No locals.
>> >
>> >
>> > So, timers ?
>>
>> I don't know. I hope this info in above logs gives somebody with more
>> understanding an idea and could help further. I still don't see where it's
>> stuck when CPU does not start or how interrupts to other CPU could prevent
>> CPU1 to run or be detected to run. I don't know how CPU bringup works so
>> to know if it's correct or not reading Linux sources and understanding
>> what it's supposed to do would be needed. Then check that what happens on
>> QEMU corresponds to what's expected and find where it's not. That's too
>> much effort for me for something I'm not interested in using so I only do
>> what I can to help but won't get into it deeper. At least these traces
>> show places in Linux kernel that should be read to find out what it does.
>>
>> I also don't know what qemu.sstep value controls but the defines says bit
>> 1 is emulated HW single step so maybe you need that enabled at least. What
>> is with other values such as 1 and 3? Disabling only timers is 3 not 0.
>
>
> yeah, with 0x0 I can't single step. Text from documentation webpage says
>
> ====
>
> maintenance packet qqemu.sstepbits
>
> This will display the MASK bits used to control the single stepping IE:
>
> (gdb) maintenance packet qqemu.sstepbits
> sending: "qqemu.sstepbits"
> received: "ENABLE=1,NOIRQ=2,NOTIMER=4"
>
> =====


Trying to resend second time

I tried 0x3 and was able single step to second cpu bringup

Simple 0x1 does not work (it single steps until irq_save (?) and then
just go forward into hang on secondary cpu)


>
>>
>> Regards,
>> BALATON Zoltan

gdb-trace-hang3.txt
Description: Text document