Re: [Qemu-devel] State of ARM FIQ in Qemu

[Tim Sander]

Hi Greg

Am Freitag, 14. November 2014, 10:50:40 schrieb Greg Bellows:
> On 14 November 2014 09:34, Tim Sander <address@hidden> wrote:
> > > > > 0xbfffe000? You where talking about the fact that the security
> > > > > extensions
> > > > > where not implemented. I was not aware that the different vbar's
> > 
> > where
> > 
> > > > > already part of the security stuff?
> > > > 
> > > > MVBAR is part of the Security extensions. HVBAR is part of the
> > > > Virtualization extensions. In mainline QEMU we implement neither
> > > > of those extensions, and so don't implement the associated
> > > > registers. (Strictly speaking, VBAR is also only in the
> > > > Security extensions, but we provide it as a workaround for
> > > > guests that assume our CPUs should implement it.)
> > > 
> > > Peter beat me to it.  None of the VBAR registers should matter in your
> > 
> > case
> > 
> > > which coincides with the use of hivecs.
> > 
> > While writing this mail i found out that the integrated debugger is
> > causing
> > harm in combination with the fiq. So everything below the braces seems to
> > be related to the this problem. But i still wanted to keep the data points
> > for
> > reference:
> > 
> > {
> > Ok, so qemu only implements the SCTLR.V bit to control the memory address
> > of
> > the interrupt vector. So its either 0 or 0xffff0000. That is fine with me.
> > Currently i have the problem that a call to set_fiq_handler does not place
> > the
> > binary stuff loaded at the address where qemu is jumping to which is
> > presumably
> > 0xffff1240. I have checked that SCTLR.V =1 under linux which is fine.
> > 
> > The background info to set_fiq_handler from my understanding is that it
> > copies
> > the given stuff directly at the address where the FIQ vector is located.
> > This
> > works as the FIQ is the last entry and thus there is some memory space for
> > a
> > short interrupt handler. I checked the memory when entering the FIQ with
> > the
> > integrated gdb:
> > (gdb) info reg
> > r0             0x0      0
> > r1             0x0      0
> > r2             0x1      1
> > r3             0x76eb34c8       1995125960
> > r4             0x76eb34c8       1995125960
> > r5             0x76f633b8       1995846584
> > r6             0x2a     42
> > r7             0x76f4c28c       1995752076
> > r8             0xf8200100       -132120320
> > r9             0xe0040000       -536608768
> > r10            0x60004059       1610629209
> > r11            0x0      0
> > r12            0x0      0
> > sp             0x908be000       0x908be000
> > lr             0x76dfc108       1994375432
> > pc             0xffff1240       0xffff1240 <firq_fiq_handler>
> > cpsr           0x600f01d1       1611596241
> > (gdb) x 0xffff1240
> > 0xe599b00c
> > 
> > But my firq_fiq_handler starts with 0xee12af10? I know that this works on
> > real
> > hardware so i suspect that this an error within qemu? Or at least that
> > there
> > is something amiss in the way the memory is initialized or handled.
> > 
> > Is there a way to instrument the memory below the vector table to get
> > debug
> > logs if the memory is modified?
> > }
> > 
> > > It may be worthwhile to put a kernel breakpoint in handle_fiq_as_nmi()
> > 
> > just
> > 
> > > to see where it goes.  If CONFIG_ARM_GIC is enabled it should take you
> > > to
> > > your handler I suspect.  Plus, if you get there then we have likely
> > 
> > proven
> > 
> > > that QEMU is getting the kernel to the right place.  I set a BP in this
> > > routine on my A9 run and appear to be hitting it correctly.
> > 
> > So you are talking about the linux kernel, right? CONFIG_ARM_GIC=y check
> > but
> > i can't find handle_fiq_as_nmi? Even a fuzzier "rgrep nmi * |grep fiq"
> > does not
> > find anything.
> 
> Maybe we are working off different versions of the kernel sources.  I'm
> using a kernel variant of v3.18-rc1.  I took a look at my 3.15 kernel and
> it does not have the routine, so perhaps yours is an earlier version as
> well.
I am on 3.14 as i am working with rt-preempt kernels right now.

> I don't spend much time working in or tracking the Linux kernel, so I am
> not sure when the difference was introduced.  I just found it to be a
> convenient function to set a BP for early FIQ debugging, you may have
> something different.
> 
> Interestingly, as I researched the Linux FIQ support I found this mail
> thread...
> 
> http://www.spinics.net/lists/arm-kernel/msg14960.html
> 
> As I don't have access to your code, I could not verify that the SVC SPSR
> was being preserved, but it may be worth you looking into it as I can see
> this potentially resulting in all kinds of random behavior.  More
> interestingly, this comment and code appears to have been changed in later
> versions of the FIQ code, so perhaps this has been fixed or improved (My
> 3.18 kernel does not have the comment).
I have not following the 3.18 kernel concering the FIQ but i will take a look.
But regarding the above link i think preserving SPSR is only needed if mode 
switching is beeing done from fiq. But as i just return from the handler i am 
assuming that the problem above is not mine. The only problem i have (besides 
the qemu debugger) is that i am missing some static mappings so i get a bad 
mode error when hitting a pagefault in FIQ mode. 

> > Concerning the fact that qemu is jumping to the right address:
> > To i have put a breakpoint to 0xffff001c which is the fiq base vector
> > address.
> > There is an instruction 0xea000480 which seems to be a pc relative branch
> > to
> > 0x1224 which then lands at 0xffff1240.
> > 
> > But the internal debugger gives me some concerns. If i do at the gdb
> > command
> > line:
> > hb *0xffff001c
> > hb *0xffff1240
> > The debugger only stops at the first breakpoint. If i leave the first
> > breakpoint
> > away the debugger stops at 0xffff1240. As i know that at 0xffff01c it
> > should jump
> > right to 0xffff1240 i would expect that both breakpoints are triggered.
> > 
> > Then if i reach the breakpoint at 0xffff1240 i know i am at the fiq code.
> > But
> > (gdb) x 0xffff1240 gives the wrong value. Nevertheless i see now (after
> > correcting the static map of the GIC) the following debug output of my
> > test
> > device when single-stepping from PC=0xffff1240:
> > Taking exception 6 [FIQ]
> > pml: pml_write: update control flags: 1
> > pml: pml_update: stop timer
> > pml: pml_update: lower irq
> > pml: pml_read: read magic
> > pml: pml_write: update control flags: 3
> > pml: pml_update: stop timer
> > 
> > This means that there has been some code executed, most probably my FIQ
> > handler, but the debugger showed me:
> > Breakpoint 1, firq_fiq_handler () at fiq.S:26
> > 26              mrc p15, 0, r10, c2, c0, 0         @ read TTBR0   < ok
> > (gdb) s        <- oh my why is it single stepping into the kernel from
> > FIQ?
> > test_ti_thread_flag (flag=1, ti=0x8f84e000) at
> > include/asm-generic/preempt.h:71
> > 71              return !--*preempt_count_ptr() && tif_need_resched();
> > (gdb) s      <- next step does not look any better...
> > test_bit (addr=0x8f84e000, nr=1) at include/asm-generic/bitops/non-
> > atomic.h:105
> > 105             return 1UL & (addr[BIT_WORD(nr)] >> (nr &
> > (BITS_PER_LONG-1)));
> > 
> > The second run is even stranger:
> > Breakpoint 1, firq_fiq_handler () at fiq.S:26
> > 26              mrc p15, 0, r10, c2, c0, 0         @ read TTBR0
> > (gdb) s
> > Cannot access memory at address 0x4
> > (gdb) c
> > Continuing.
> > Cannot access memory at address 0x4
> > ...
> > qemu seems completly unusable from here on...
> > 
> > I am pretty sure now that my FIQ handler is executed.
> > I see multiple accesses to my virtual pml test hardware:
> > arm_gic: Raised pending FIQ 49 (cpu 0)
> > pml: pml_write: update control flags: 1
> > pml: pml_update: start timer
> > pml: pml_update: lower irq
> > pml: pml_read: read magic
> > pml: pml_write: update control flags: 3
> > pml: pml_update: start timer
> > arm_gic: Enabled IRQ 37
> > [  OK  ] Found device /dev/ttyAMA0.
> > pml: pml_timer_tick: raise_irq
> > arm_gic: Raised pending FIQ 49 (cpu 0)
> > pml: pml_write: update control flags: 1
> > pml: pml_update: start timer
> > pml: pml_update: lower irq
> > pml: pml_read: read magic
> > pml: pml_write: update control flags: 3
> > pml: pml_update: start timer
> > pml: pml_timer_tick: raise_irq
> > 
> > Which seems like normal operation. Especially the log
> > message shows that other stuff gets executed.
> > 
> > But after a while the interrupts stop and nothing happens
> > The system is not reacting to keypresses anymore. Not even
> > Ctrl-A-X. But this seems as if the debug output in the GIC and/or
> > my pml test driver locked the qemu up?
> 
> Hmmm... almost sounds like we lost an interrupt or ack which could be in
> QEMU. Does execution cease if run as A15?
I think by now that its not related to the CPU core but the gdb debug port.
As soon as the debugger is open and a fiq is hit, problems start. This was 
a bit unfortunate for my tests as i was using the integrated debugger to debug 
the fiq. But results are completly bogus and definetly do not represent the
qemu execution as this is running fine as i can see from my debug output from 
my virtual hardware driver.

> > Also if i connect to the gdb port while the fiq is running the
> > qemu stops the execution.
> > 
> > But besides the problems with the debugger which set me of course
> > the qemu seems to happy emulate FIQs, which is really nice :-)
> 
> I'm happy to hear that we found a working scenario, but hangs and such
> should not happen.  I need to determine a way to  look into this more
> myself to see if it is related to grouping or FIQ support.
I can prepare you a ptxdist.org based environment and my patches for my 
testdriver if you need a target. This should give you a linux environment in 
less than 30minutes of work and about 30min of compile time (depending on 
cpu). 

Best regards
Tim