Re: [Qemu-devel] State of ARM FIQ in Qemu

[Greg Bellows]

On 14 November 2014 09:34, Tim Sander <address@hidden> wrote:

> > > 0xbfffe000? You where talking about the fact that the security
> > > extensions
> > > where not implemented. I was not aware that the different vbar's where
> > > already part of the security stuff?
> >
> > MVBAR is part of the Security extensions. HVBAR is part of the
> > Virtualization extensions. In mainline QEMU we implement neither
> > of those extensions, and so don't implement the associated
> > registers. (Strictly speaking, VBAR is also only in the
> > Security extensions, but we provide it as a workaround for
> > guests that assume our CPUs should implement it.)
>
> Peter beat me to it.  None of the VBAR registers should matter in your case
> which coincides with the use of hivecs.
While writing this mail i found out that the integrated debugger is causing
harm in combination with the fiq. So everything below the braces seems to
be related to the this problem. But i still wanted to keep the data points for
reference:

{
Ok, so qemu only implements the SCTLR.V bit to control the memory address of
the interrupt vector. So its either 0 or 0xffff0000. That is fine with me.
Currently i have the problem that a call to set_fiq_handler does not place the
binary stuff loaded at the address where qemu is jumping to which is presumably
0xffff1240. I have checked that SCTLR.V =1 under linux which is fine.

The background info to set_fiq_handler from my understanding is that it copies
the given stuff directly at the address where the FIQ vector is located. This
works as the FIQ is the last entry and thus there is some memory space for a
short interrupt handler. I checked the memory when entering the FIQ with the
integrated gdb:
(gdb) info reg
r0             0x0      0
r1             0x0      0
r2             0x1      1
r3             0x76eb34c8       1995125960
r4             0x76eb34c8       1995125960
r5             0x76f633b8       1995846584
r6             0x2a     42
r7             0x76f4c28c       1995752076
r8             0xf8200100       -132120320
r9             0xe0040000       -536608768
r10            0x60004059       1610629209
r11            0x0      0
r12            0x0      0
sp             0x908be000       0x908be000
lr             0x76dfc108       1994375432
pc             0xffff1240       0xffff1240 <firq_fiq_handler>
cpsr           0x600f01d1       1611596241
(gdb) x 0xffff1240
0xe599b00c

But my firq_fiq_handler starts with 0xee12af10? I know that this works on real
hardware so i suspect that this an error within qemu? Or at least that there
is something amiss in the way the memory is initialized or handled.

Is there a way to instrument the memory below the vector table to get debug
logs if the memory is modified?
}

> It may be worthwhile to put a kernel breakpoint in handle_fiq_as_nmi() just
> to see where it goes.  If CONFIG_ARM_GIC is enabled it should take you to
> your handler I suspect.  Plus, if you get there then we have likely proven
> that QEMU is getting the kernel to the right place.  I set a BP in this
> routine on my A9 run and appear to be hitting it correctly.
So you are talking about the linux kernel, right? CONFIG_ARM_GIC=y check but
i can't find handle_fiq_as_nmi? Even a fuzzier "rgrep nmi * |grep fiq" does not
find anything.

Maybe we are working off different versions of the kernel sources.  I'm using a kernel variant of v3.18-rc1.  I took a look at my 3.15 kernel and it does not have the routine, so perhaps yours is an earlier version as well.  

I don't spend much time working in or tracking the Linux kernel, so I am not sure when the difference was introduced.  I just found it to be a convenient function to set a BP for early FIQ debugging, you may have something different.

 

Interestingly, as I researched the Linux FIQ support I found this mail thread...

http://www.spinics.net/lists/arm-kernel/msg14960.html

As I don't have access to your code, I could not verify that the SVC SPSR was being preserved, but it may be worth you looking into it as I can see this potentially resulting in all kinds of random behavior.  More interestingly, this comment and code appears to have been changed in later versions of the FIQ code, so perhaps this has been fixed or improved (My 3.18 kernel does not have the comment).

Concerning the fact that qemu is jumping to the right address:
To i have put a breakpoint to 0xffff001c which is the fiq base vector address.
There is an instruction 0xea000480 which seems to be a pc relative branch to
0x1224 which then lands at 0xffff1240.

But the internal debugger gives me some concerns. If i do at the gdb command
line:
hb *0xffff001c
hb *0xffff1240
The debugger only stops at the first breakpoint. If i leave the first breakpoint
away the debugger stops at 0xffff1240. As i know that at 0xffff01c it should jump
right to 0xffff1240 i would expect that both breakpoints are triggered.

Then if i reach the breakpoint at 0xffff1240 i know i am at the fiq code. But
(gdb) x 0xffff1240 gives the wrong value. Nevertheless i see now (after
correcting the static map of the GIC) the following debug output of my test
device when single-stepping from PC=0xffff1240:
Taking exception 6 [FIQ]
pml: pml_write: update control flags: 1
pml: pml_update: stop timer
pml: pml_update: lower irq
pml: pml_read: read magic
pml: pml_write: update control flags: 3
pml: pml_update: stop timer

This means that there has been some code executed, most probably my FIQ
handler, but the debugger showed me:
Breakpoint 1, firq_fiq_handler () at fiq.S:26
26              mrc p15, 0, r10, c2, c0, 0         @ read TTBR0   < ok
(gdb) s        <- oh my why is it single stepping into the kernel from FIQ?
test_ti_thread_flag (flag=1, ti=0x8f84e000) at include/asm-generic/preempt.h:71
71              return !--*preempt_count_ptr() && tif_need_resched();
(gdb) s      <- next step does not look any better...
test_bit (addr=0x8f84e000, nr=1) at include/asm-generic/bitops/non-
atomic.h:105
105             return 1UL & (addr[BIT_WORD(nr)] >> (nr & (BITS_PER_LONG-1)));

The second run is even stranger:
Breakpoint 1, firq_fiq_handler () at fiq.S:26
26              mrc p15, 0, r10, c2, c0, 0         @ read TTBR0
(gdb) s
Cannot access memory at address 0x4
(gdb) c
Continuing.
Cannot access memory at address 0x4
...
qemu seems completly unusable from here on...

I am pretty sure now that my FIQ handler is executed.
I see multiple accesses to my virtual pml test hardware:
arm_gic: Raised pending FIQ 49 (cpu 0)
pml: pml_write: update control flags: 1
pml: pml_update: start timer
pml: pml_update: lower irq
pml: pml_read: read magic
pml: pml_write: update control flags: 3
pml: pml_update: start timer
arm_gic: Enabled IRQ 37
[  OK  ] Found device /dev/ttyAMA0.
pml: pml_timer_tick: raise_irq
arm_gic: Raised pending FIQ 49 (cpu 0)
pml: pml_write: update control flags: 1
pml: pml_update: start timer
pml: pml_update: lower irq
pml: pml_read: read magic
pml: pml_write: update control flags: 3
pml: pml_update: start timer
pml: pml_timer_tick: raise_irq

Which seems like normal operation. Especially the log
message shows that other stuff gets executed.

But after a while the interrupts stop and nothing happens
The system is not reacting to keypresses anymore. Not even
Ctrl-A-X. But this seems as if the debug output in the GIC and/or
my pml test driver locked the qemu up?

Hmmm... almost sounds like we lost an interrupt or ack which could be in QEMU. Does execution cease if run as A15?  

 

Also if i connect to the gdb port while the fiq is running the
qemu stops the execution.

But besides the problems with the debugger which set me of course
the qemu seems to happy emulate FIQs, which is really nice :-)

I'm happy to hear that we found a working scenario, but hangs and such should not happen.  I need to determine a way to  look into this more myself to see if it is related to grouping or FIQ support. 

 

Best regards
Tim