[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH v4 4/7] hw/riscv: Use pre-built bios image of generic platfor
|
From: |
Alistair Francis |
|
Subject: |
Re: [PATCH v4 4/7] hw/riscv: Use pre-built bios image of generic platform for virt & sifive_u |
|
Date: |
Tue, 11 Aug 2020 20:25:10 -0700 |
On Sun, Aug 2, 2020 at 11:51 PM Bin Meng <bmeng.cn@gmail.com> wrote:
>
> Hi Alistair,
>
> On Wed, Jul 29, 2020 at 1:10 PM Bin Meng <bmeng.cn@gmail.com> wrote:
> >
> > Hi Alistair,
> >
> > On Wed, Jul 29, 2020 at 1:05 PM Alistair Francis <alistair23@gmail.com>
> > wrote:
> > >
> > > On Tue, Jul 28, 2020 at 9:51 PM Bin Meng <bmeng.cn@gmail.com> wrote:
> > > >
> > > > Hi Alistair,
> > > >
> > > > On Wed, Jul 29, 2020 at 2:26 AM Alistair Francis <alistair23@gmail.com>
> > > > wrote:
> > > > >
> > > > > On Tue, Jul 28, 2020 at 8:46 AM Bin Meng <bmeng.cn@gmail.com> wrote:
> > > > > >
> > > > > > Hi Alistair,
> > > > > >
> > > > > > On Tue, Jul 28, 2020 at 11:39 PM Alistair Francis
> > > > > > <alistair23@gmail.com> wrote:
> > > > > > >
> > > > > > > On Wed, Jul 15, 2020 at 9:55 PM Bin Meng <bmeng.cn@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Alistair,
> > > > > > > >
> > > > > > > > On Mon, Jul 13, 2020 at 9:53 AM Bin Meng <bmeng.cn@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > On Sun, Jul 12, 2020 at 1:34 AM Alistair Francis
> > > > > > > > > <alistair23@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 9, 2020 at 10:07 PM Bin Meng
> > > > > > > > > > <bmeng.cn@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > From: Bin Meng <bin.meng@windriver.com>
> > > > > > > > > > >
> > > > > > > > > > > Update virt and sifive_u machines to use the opensbi
> > > > > > > > > > > fw_dynamic bios
> > > > > > > > > > > image built for the generic FDT platform.
> > > > > > > > > > >
> > > > > > > > > > > Remove the out-of-date no longer used bios images.
> > > > > > > > > > >
> > > > > > > > > > > Signed-off-by: Bin Meng <bin.meng@windriver.com>
> > > > > > > > > > > Reviewed-by: Anup Patel <anup@brainfault.org>
> > > > > > > > > > > Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
> > > > > > > > > >
> > > > > > > > > > This patch seems to break 32-bit Linux boots on the
> > > > > > > > > > sifive_u and virt machines.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > It looks only Linux boot on sifive_u is broken. On our side,
> > > > > > > > > we have
> > > > > > > > > been using VxWorks to test 32-bit OpenSBI on sifive_u so this
> > > > > > > > > issue
> > > > > > > > > gets unnoticed. I will take a look.
> > > > > > > >
> > > > > > > > I've figured out the issue of 32-bit Linux booting failure on
> > > > > > > > sifive_u. A patch has been sent to Linux upstream:
> > > > > > > > http://lists.infradead.org/pipermail/linux-riscv/2020-July/001213.html
> > > > > > >
> > > > > > > Thanks for that. What change in QEMU causes this failure though?
> > > > > > >
> > > > > >
> > > > > > There is nothing wrong in QEMU.
> > > > >
> > > > > There is. This patch causes a regression for 32-bit Linux boot on the
> > > > > sifive_u. Your v5 has not addressed this.
> > > >
> > > > The 32-bit Linux boot failure was fixed by:
> > > > http://lists.infradead.org/pipermail/linux-riscv/2020-July/001213.html
> > > >
> > > > What additional issue did you see?
> > > >
> > > > >
> > > > > With this patch, the Linux boot stops here:
> > > > >
> > > > > OpenSBI v0.8
> > > > > ____ _____ ____ _____
> > > > > / __ \ / ____| _ \_ _|
> > > > > | | | |_ __ ___ _ __ | (___ | |_) || |
> > > > > | | | | '_ \ / _ \ '_ \ \___ \| _ < | |
> > > > > | |__| | |_) | __/ | | |____) | |_) || |_
> > > > > \____/| .__/ \___|_| |_|_____/|____/_____|
> > > > > | |
> > > > > |_|
> > > > >
> > > > > Platform Name : SiFive HiFive Unleashed A00
> > > > > Platform Features : timer,mfdeleg
> > > > > Platform HART Count : 4
> > > > > Boot HART ID : 3
> > > > > Boot HART ISA : rv64imafdcsu
> > > >
> > > > This is a 64-bit hardware.
> > >
> > > You are right. It's not 32-bit, that was my mistake. I'm used to my
> > > first test being 32-bit, but in this case it's not.
> > >
> > > It looks like this commit instead breaks the sifive_u for 64-bit with
> > > the 5.3 kernel.
> > >
> > > >
> > > > > BOOT HART Features : pmp,scounteren,mcounteren
> > > > > BOOT HART PMP Count : 16
> > > > > Firmware Base : 0x80000000
> > > > > Firmware Size : 116 KB
> > > > > Runtime SBI Version : 0.2
> > > > >
> > > > > MIDELEG : 0x0000000000000222
> > > > > MEDELEG : 0x000000000000b109
> > > > > PMP0 : 0x0000000080000000-0x000000008001ffff (A)
> > > > > PMP1 : 0x0000000000000000-0xffffffffffffffff (A,R,W,X)
> > > > > [ 0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
> > > > > [ 0.000000] Linux version 5.3.0 (oe-user@oe-host) (gcc version
> > > >
> > > > It seems that you are using quite an old kernel. Can you please try
> > > > the latest version?
> > >
> > > It is an old kernel, but old kernels should still keep working (or we
> > > should at least know why they don't)
> > >
> > > >
> > > > > 9.2.0 (GCC)) #1 SMP Thu Sep 19 18:34:52 UTC 2019
> > > > > [ 0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
> > > > > [ 0.000000] printk: bootconsole [sbi0] enabled
> > > > > [ 0.000000] initrd not found or empty - disabling initrd
> > > > > [ 0.000000] Zone ranges:
> > > > > [ 0.000000] DMA32 [mem 0x0000000080200000-0x00000000bfffffff]
> > > > > [ 0.000000] Normal empty
> > > > > [ 0.000000] Movable zone start for each node
> > > > > [ 0.000000] Early memory node ranges
> > > > > [ 0.000000] node 0: [mem 0x0000000080200000-0x00000000bfffffff]
> > > > > [ 0.000000] Initmem setup node 0 [mem
> > > > > 0x0000000080200000-0x00000000bfffffff]
> > > > > [ 0.000000] OF: fdt: Invalid device tree blob header
> > > > > [ 0.000000] software IO TLB: mapped [mem 0xbb1fe000-0xbf1fe000]
> > > > > (64MB)
> > > > >
> > > > > Without this patch I can boot all the way to looking for a rootFS.
> > > > >
> > > > > Please don't send new versions of patches without addresses
> > > > > regressions.
> > > >
> > > > The patches were sent after addressing all regressions you reported
> > > > (well the 32-bit Linux booting issue is actually not a QEMU
> > > > regression, but one that exists in the Linux kernel side for a long
> > > > time).
> > >
> > > Yep, that is my mistake. Sorry about the confusion.
> > >
> > > >
> > > > I just tested 64-bit Linux boot on both virt and sifive_u, and they
> > > > both can boot all the way to looking for a root fs.
> > >
> > > Can you test with older kernels?
> > >
> >
> > OK I will investigate.
>
> I've located the commit in the newer Linux kernel that fixed the boot
> failure of the 5.3 kernel. See below:
>
> commit 922b0375fc93fb1a20c5617e37c389c26bbccb70
> Author: Albert Ou <aou@eecs.berkeley.edu>
> Date: Fri Sep 27 16:14:18 2019 -0700
>
> riscv: Fix memblock reservation for device tree blob
>
> This fixes an error with how the FDT blob is reserved in memblock.
> An incorrect physical address calculation exposed the FDT header to
> unintended corruption, which typically manifested with of_fdt_raw_init()
> faulting during late boot after fdt_totalsize() returned a wrong value.
> Systems with smaller physical memory sizes more frequently trigger this
> issue, as the kernel is more likely to allocate from the DMA32 zone
> where bbl places the DTB after the kernel image.
>
> Commit 671f9a3e2e24 ("RISC-V: Setup initial page tables in two stages")
> changed the mapping of the DTB to reside in the fixmap area.
> Consequently, early_init_fdt_reserve_self() cannot be used anymore in
> setup_bootmem() since it relies on __pa() to derive a physical address,
> which does not work with dtb_early_va that is no longer a valid kernel
> logical address.
>
> The reserved[0x1] region shows the effect of the pointer underflow
> resulting from the __pa(initial_boot_params) offset subtraction:
>
> [ 0.000000] MEMBLOCK configuration:
> [ 0.000000] memory size = 0x000000001fe00000 reserved size =
> 0x0000000000a2e514
> [ 0.000000] memory.cnt = 0x1
> [ 0.000000] memory[0x0]
> [0x0000000080200000-0x000000009fffffff], 0x000000001fe00000 bytes
> flags: 0x0
> [ 0.000000] reserved.cnt = 0x2
> [ 0.000000] reserved[0x0]
> [0x0000000080200000-0x0000000080c2dfeb], 0x0000000000a2dfec bytes
> flags: 0x0
> [ 0.000000] reserved[0x1]
> [0xfffffff080100000-0xfffffff080100527], 0x0000000000000528 bytes
> flags: 0x0
>
> With the fix applied:
>
> [ 0.000000] MEMBLOCK configuration:
> [ 0.000000] memory size = 0x000000001fe00000 reserved size =
> 0x0000000000a2e514
> [ 0.000000] memory.cnt = 0x1
> [ 0.000000] memory[0x0]
> [0x0000000080200000-0x000000009fffffff], 0x000000001fe00000 bytes
> flags: 0x0
> [ 0.000000] reserved.cnt = 0x2
> [ 0.000000] reserved[0x0]
> [0x0000000080200000-0x0000000080c2dfeb], 0x0000000000a2dfec bytes
> flags: 0x0
> [ 0.000000] reserved[0x1]
> [0x0000000080e00000-0x0000000080e00527], 0x0000000000000528 bytes
> flags: 0x0
>
> Fixes: 671f9a3e2e24 ("RISC-V: Setup initial page tables in two stages")
> Signed-off-by: Albert Ou <aou@eecs.berkeley.edu>
> Tested-by: Bin Meng <bmeng.cn@gmail.com>
> Reviewed-by: Anup Patel <anup@brainfault.org>
> Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
>
> As the commit message says, this commit is a fix to commit
> 671f9a3e2e24 ("RISC-V: Setup initial page tables in two stages") which
> is in the 5.3 kernel:
>
> commit 671f9a3e2e24cdeb2d2856abee7422f093e23e29
> Author: Anup Patel <Anup.Patel@wdc.com>
> Date: Fri Jun 28 13:36:21 2019 -0700
>
> RISC-V: Setup initial page tables in two stages
>
> Currently, the setup_vm() does initial page table setup in one-shot
> very early before enabling MMU. Due to this, the setup_vm() has to map
> all possible kernel virtual addresses since it does not know size and
> location of RAM. This means we have kernel mappings for non-existent
> RAM and any buggy driver (or kernel) code doing out-of-bound access
> to RAM will not fault and cause underterministic behaviour.
>
> Further, the setup_vm() creates PMD mappings (i.e. 2M mappings) for
> RV64 systems. This means for PAGE_OFFSET=0xffffffe000000000 (i.e.
> MAXPHYSMEM_128GB=y), the setup_vm() will require 129 pages (i.e.
> 516 KB) of memory for initial page tables which is never freed. The
> memory required for initial page tables will further increase if
> we chose a lower value of PAGE_OFFSET (e.g. 0xffffff0000000000)
>
> This patch implements two-staged initial page table setup, as follows:
> 1. Early (i.e. setup_vm()): This stage maps kernel image and DTB in
> a early page table (i.e. early_pg_dir). The early_pg_dir will be used
> only by boot HART so it can be freed as-part of init memory free-up.
> 2. Final (i.e. setup_vm_final()): This stage maps all possible RAM
> banks in the final page table (i.e. swapper_pg_dir). The boot HART
> will start using swapper_pg_dir at the end of setup_vm_final(). All
> non-boot HARTs directly use the swapper_pg_dir created by boot HART.
>
> We have following advantages with this new approach:
> 1. Kernel mappings for non-existent RAM don't exists anymore.
> 2. Memory consumed by initial page tables is now indpendent of the
> chosen PAGE_OFFSET.
> 3. Memory consumed by initial page tables on RV64 system is 2 pages
> (i.e. 8 KB) which has significantly reduced and these pages will be
> freed as-part of the init memory free-up.
>
> The patch also provides a foundation for implementing strict kernel
> mappings where we protect kernel text and rodata using PTE permissions.
>
> Suggested-by: Mike Rapoport <rppt@linux.ibm.com>
> Signed-off-by: Anup Patel <anup.patel@wdc.com>
> [paul.walmsley@sifive.com: updated to apply; fixed a checkpatch warning]
> Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
>
> I cherry-picked 922b0375fc93fb1a20c5617e37c389c26bbccb70 on top of a
> 5.3 kernel, and with that the kernel boots again with this patch set.
Thanks for looking into this. It's unfortunate that OpenSBI won't work
with older kernels, but people can always use their own OpenSBI build
in that case. 5.4 is an LTS and it works so that seems fine to me.
Alistair
>
> >
> > > If we can't support older kernels with the default bios option we at
> > > least need to know why and list that in the release notes.
> > >
>
> I will include the above findings in my commit message of this series,
> mentioning that to test an older kernel like 5.3 version Linux,
> 922b0375fc93fb1a20c5617e37c389c26bbccb70 should be cherry-picked.
>
> Regards,
> Bin