Re:

[Anup Patel]

On 12/11/21, 3:51 PM, "Alex Bennée" <alex.bennee@linaro.org> wrote:

    Subject: Re: riscv64 system emulation maximum core count with virt machine
    Date: Fri, 12 Nov 2021 10:16:34 +0000
    References: 
<CADJL7eegP=UajgcOxcb=cs1QFim+W3wt0ieq_On6siznEU-byw@mail.gmail.com> 
<D5150EE4-3E9E-43F3-828D-630CA3673A2F@getmailspring.com> 
<CADJL7ecO089N_1GC6GPY5SJmRbVh81i7hE=8nsPZaEDuXHbR2w@mail.gmail.com>
    User-agent: mu4e 1.7.4; emacs 28.0.60
    In-reply-to: 
<CADJL7ecO089N_1GC6GPY5SJmRbVh81i7hE=8nsPZaEDuXHbR2w@mail.gmail.com>


    Mark Wyse <wysem@cs.washington.edu> writes:

    > Thanks Rahul,
    >
    > After some digging I came to the same conclusions. I'll dig deeper to see 
if there are any impacts in raising max CPU count for RISC-V Virt
    > machine, but I definitely agree that it should be set higher.

    If you look at the commit that added the #define:

    18df0b4695 (hw/riscv: virt: Allow creating multiple NUMA sockets)

    You'll see amongst other things the number of vCPUs affected the size of
    the one of the regions (IRQ controller?):

         [VIRT_RTC] =         {   0x101000,        0x1000 },
         [VIRT_CLINT] =       {  0x2000000,       0x10000 },
         [VIRT_PCIE_PIO] =    {  0x3000000,       0x10000 },
    -    [VIRT_PLIC] =        {  0xc000000,     0x4000000 },
    +    [VIRT_PLIC] =        {  0xc000000, VIRT_PLIC_SIZE(VIRT_CPUS_MAX * 2) },
         [VIRT_UART0] =       { 0x10000000,         0x100 },
         [VIRT_VIRTIO] =      { 0x10001000,        0x1000 },

    which I suspects limits the total number you can handle without messing
    around with the MMIO memory map.

Various devices such as SiFive CLINT, ACLINT, PLIC, AIA APLIC, and
AIA IMSIC have different number of max CPUs support and proportionate
memory map.

1) SiFive CLINT and ACLINT
Max CPUs 4095
Max memory 8K

2) PLIC
Max CPUs 7936
Max memory 16MB

3) AIA APLIC (work-in-progress)
Max CPUs 16384
Max memory 528KB

4) AIA IMSIC (work-in-progress)
This is a per-CPU devices so no limit on max CPUs
Per-CPU memory 4KB to 256KB memory based on number of VS-files

What is a good value for max CPUs ?

We can organize virt machine memory layout based on this.

Regards,
Anup

    > FWIW, I was able to run a 32-core kernel after adjusting code and kernel 
appropriately and don't recall any warning or error messages
    > popping up.

    I should also point out from a emulation performance point of view you
    will run into diminishing returns as you increases the number of CPUs as
    the emulator has to spend more time ensuring cross-vCPU consistency when
    things like the TLBs are flushed.

    >
    > Best,
    > Mark
    >
    > On Wed, Sep 29, 2021 at 10:16 AM Rahul Pathak <rpathakmailbox@gmail.com> 
wrote:
    >
    >  If you want to boot with more than 8 harts then you also need to 
increase the CONFIG_NR_CPUS in the kernel. Along with change in
    >  QEMU - VIRT_CPUS_MAX in  "include/hw/riscv/virt.h" 
    >
    >  But, the limit of 8 is low and should be increased. Don't know the 
reason why its 8.
    >
    >  On Sep 22 2021, at 10:19 pm, Mark Wyse <wysem@cs.washington.edu> wrote:
    >
    >  Hello all,
    >
    >  Apologies if I missed the answer in the docs, but I am wondering why the 
virt machine is limited to only 8 cores when running
    >  RISC-V 64 system emulation? I am interested in booting a 16, 24, or even 
32 core SMP Linux kernel with OpenSBI as the
    >  bootloader on the generic virt machine before adding a custom machine 
definition. I was able to successfully boot Linux kernel
    >  5.9.0 with OpenSBI v0.9 and 8 cores/harts with an ext4 filesystem 
provided by Buildroot.
    >
    >  I poked around the docs and source code a bit, and found mention of the 
core count limit in the docs
    >  (https://qemu.readthedocs.io/en/latest/system/riscv/virt.html), but no 
reason as to why this is the limit. It also appears that
    >  providing a custom device tree blob and setting QEMU's -smp option to 
match might overcome this limitation, but it would be
    >  better if more than 8 cores was supported directly.
    >
    >  Best,
    >  Mark
    >
    >  Mark Wyse
    >  pronouns: he/him/his
    >  PhD Student
    >  Paul G. Allen School of Computer Science & Engineering
    >  University of Washington


    -- 
    Alex Bennée