Re: [Qemu-ppc] [OpenBIOS] Running client with MMU off

[BALATON Zoltan]

On Sat, 21 Jun 2014, BALATON Zoltan wrote:
> On Fri, 20 Jun 2014, Mark Cave-Ayland wrote:
> > As for the code that generates the ISIs, is this in MorphOS as opposed to 
> > OpenBIOS? I guess something must have previously accessed an entry on the 
> > same page before the registers were updated, or maybe there is some kind of 
> > hardware readahead?
>
> The code is in the MorphOS boot loader and what it does is trying to take 
> over memory management. Unfortunately it seems there is a period when it 
> already replaced the vectors but have not set up the TLB hash table yet so it 
> cannot actually handle exceptions. I could prevent DSIs but running the code 
> during this period generates ISI-s. If the code is run in the same order on 
> real hardware then it's not likely that the page is accessed there and not on 
> QEMU. A readahead could explain it but I don't know if that happens. I have 
> no better idea now than manually generating faults for all pages where the 
> client code is loaded before calling it. I'll try to implement that unless 
> someone can suggest a better solution.

Experimenting with it some more I could not make it work that way probably 
because by the time the ISI happens the vectors are already replaced and 
the sr0 register is overwritten so even if I manage to put translations in 
our hash table for that code that won't be correct any more by the time 
the ISI that causes a crash is handled.

So the only things that might work are using IBATs or disabling the MMU 
bits when MorphOS overwrites the vectors. I think this latter option is a 
bit cleaner but how can I get an interrupt when a memory address is 
written to? (Apart from setting up a watch point for it.) In the OpenBIOS 
code there are comments stating that page 0 is not mapped to catch NULL 
pointer dereferences but this does not seem to work as MorphOS can write 
over page 0 and only get an exception when reaching the next page. (Also 
I've found documented in one version of the PPC programming environments 
document that 0x00-0xff is reserved for operating system use so they can 
legally write there.) I'm sure I'm missing something here again.

It could possibly work on real hardware because of caches and that's where 
the read ahead might happen during the critical part. This is what seems 
to happen during MorphOS's MMU take over as far as I understand:

1. memcpy to 0, len=0x2000
2. fixup jumps and write base address to 0x80 (this is what's zeroed at
   the earlier write at the beginning)
3. Set MSR_BE and tweak HID0 (this may cause code to be preloaded in
   cache?)
4. Set sr0-15 from stack variables
5. Values for IBAT and DBAT registers are loaded from the stack and SDR1
   is set to 0 dropping the hash table
6. MSR_DR and MSR_IR are cleared
7. BAT registers are set up as shown in this message:
   http://www.openfirmware.info/pipermail/openbios/2014-June/008419.html
   which means that the stack should be within the first 256MB
8. TLB entries are invalidated then MMU bits are re-enabled (but
   hashed page tables are not there yet so it relies on BATs at this point)
9. After doing some other stuff eventually a function is called that sets
   up the hash table
10. Vectors are replaced again later during boot after the microkernel has
    started its servers

Currently we get a DSI at 0x1000 in step 1. and crash due to our handler 
already replaced, probably we don't get a DSI earlier due to a prevoius 
zeroing of 0x80 at the very beginning while our handler still works so we 
have a translation for that. If we managed to get through this we get a 
DSI in step 5. while accessing the stack variable and crash there or get 
an ISI running code that replaces sr0 and crash due to that when managed 
to avoid DSIs. I have patches to move the stack to after the executable as 
was shown in my oftest results and clearing it plus writing to 0x1000 
before calling the client code can avoid DSI-s but I still get the ISI. 
Alternatively adding code to ofmem_arch_map_pages which is called from 
ofmem_map to add translations to the hash table like this:

void ofmem_arch_map_pages(phys_addr_t phys, ucell virt, ucell size, ucell mode)
{
    unsigned int i;

    if (virt < OF_CODE_START) {
        for (i = 0; i * PAGE_SIZE < size; i++) {
            hash_page(virt + i * PAGE_SIZE, phys + i * PAGE_SIZE, mode);
        }
    }
}

also seems to help with DSIs but cannot avoid the ISI (maybe because sr0 
is replaced that could possible be used to look up the hash table).

Can anyone see a way through this or has an idea what would be an 
acceptable hack to allow this code run? I'm out of ideas now.

Regards,
BALATON Zoltan