[RFC PATCH 0/8] riscv: implement Ssqosid extension and CBQRI controllers

[Drew Fustini]

This RFC series implements the Ssqosid extension and the sqoscfg CSR as
defined in the RISC-V Capacity and Bandwidth Controller QoS Register
Interface (CBQRI) specification [1]. Quality of Service (QoS) in this
context is concerned with shared resources on an SoC such as cache
capacity and memory bandwidth.

These patches are available as a public git branch [2].

sqoscfg CSR
-----------
The sqoscfg CSR provides a mechanism by which a software workload (e.g.
a process or a set of processes) can be associated with a resource
control ID (RCID) and a monitoring counter ID (MCID) that accompanies
each request made by the hart to shared resources like cache.

CBQRI defines operations to configure resource usage limits, in the form
of capacity or bandwidth, for an RCID. CBQRI also defines operations to
configure counters to track the resource utilization per MCID.

CBQRI controllers
-----------------
This series also implements an CBQRI capacity controller and an CBQRI
bandwidth controller as well as an example SoC that instantiates the
controllers with a specific configuration:

  - L2 cache controllers
    - Resource type: Capacity
    - Number of capacity blocks (NCBLKS): 12
        - In the context of a set-associative cache, the number of
          capacity blocks can be thought of as the number of ways
    - Number of access types: 2 (code and data)
    - Usage monitoring not supported
    - Capacity allocation operations: CONFIG_LIMIT, READ_LIMIT

  - Last-level cache (LLC) controller
    - Resource type: Capacity
    - Number of capacity blocks (NCBLKS): 16
    - Number of access types: 2 (code and data)
    - Usage monitoring operations: CONFIG_EVENT, READ_COUNTER
    - Event IDs supported: None, Occupancy
    - Capacity allocation ops: CONFIG_LIMIT, READ_LIMIT, FLUSH_RCID

  - Memory controllers
    - Resource type: Bandwidth
    - Number of bandwidth blocks (NBWBLKS): 1024
       - Bandwidth blocks do not have a unit but instead represent a
         portion of the total bandwidth resource. For NWBLKS of 1024,
         each block represents about 0.1% of the bandwidth resource.
    - Maximum reserved bandwidth blocks (MRBWB): 819 [80% of NBWBLKS]
    - Number of access types: 1 (no code/data differentiation)
    - Usage monitoring operations: CONFIG_EVENT, READ_COUNTER
    - Event IDs supported: None, Total read/write byte count, Total
                           read byte count, Total write byte count
    - Bandwidth allocation operations: CONFIG_LIMIT, READ_LIMIT

The memory map for this example SoC:

  Base addr  Size
  0x4820000  4KB  Cluster 0 L2 cache controller
  0x4821000  4KB  Cluster 1 L2 cache controller
  0x4828000  4KB  Memory controller 0
  0x4829000  4KB  Memory controller 1
  0x482A000  4KB  Memory controller 2
  0x482B000  4KB  Shared LLC cache controller

This configuration is meant to provide a "concrete" example for software
(like Linux) to test against. It represents just one of many possible
ways for hardware to implement the CBQRI spec.

In addition, please note that this RFC series only implements the
register interface that CBQRI specifies. It does not attempt to emulate
the performance impact of configuring limits on shared resources like
cache and memory bandwidth. Similarly, the code does not attempt to
emulate cache and memory bandwidth utilization, like what would be
observed on a real hardware system implementing CBQRI.

Status of CBQRI
---------------
The CBQRI spec is still in a draft state and is undergoing review [3].
It is possible there will be changes to the Ssqosid extension and the
CBQRI spec. For example, the sqoscfg CSR address is not yet finalized.

The goal of this Qemu patch series, along with complimentary Linux patch
series, is to satisfy the software proof of concept requirement for
CBQRI to be frozen.

Note: CBQRI was previously known as CMQRI and the github repo with the
spec has not yet been renamed [4].

Future work
-----------
Currently the configuration of the CBQRI controller parameters is done
directly in cbqri_example_soc.c but this requires a user to edit the
code for a different configuration. In the future, the configuration of
CBQRI controller parameters will be done as command line arguments.

In addition, the device tree generation will be expanded to include
CBQRI-related properties. Currently, these are added to the dumped dtb
that Linux consumes for the purposes of testing this RFC.

For those interested in the software using this Qemu implementation,
I will follow-up with a link to the Linux patches once posted.

[1] https://github.com/riscv-non-isa/riscv-cmqri/blob/main/riscv-cbqri.pdf
[2] https://gitlab.baylibre.com/baylibre/qemu/-/tree/riscv-cbqri-rfc
[3] https://lists.riscv.org/g/tech-cmqri/message/38
[4] https://lists.riscv.org/g/tech-cmqri/message/41

Kornel Dulęba (1):
  riscv: implement Ssqosid extension and sqoscfg CSR

Nicolas Pitre (7):
  hw/riscv: define capabilities of CBQRI controllers
  hw/riscv: implement CBQRI capacity controller
  hw/riscv: implement CBQRI memory controller
  hw/riscv: instantiate CBQRI controllers for an example SoC
  hw/riscv: Kconfig: add CBQRI options
  hw/riscv: meson: add CBQRI options to the build
  hw/riscv: virt: initialize the CBQRI example SoC

 disas/riscv.c                |   1 +
 hw/riscv/Kconfig             |   8 +
 hw/riscv/cbqri_bandwidth.c   | 511 +++++++++++++++++++++++++++++++++
 hw/riscv/cbqri_capacity.c    | 532 +++++++++++++++++++++++++++++++++++
 hw/riscv/cbqri_example_soc.c | 124 ++++++++
 hw/riscv/meson.build         |   4 +
 hw/riscv/virt.c              |   3 +
 include/hw/riscv/cbqri.h     |  81 ++++++
 target/riscv/cpu.c           |   2 +
 target/riscv/cpu.h           |   3 +
 target/riscv/cpu_bits.h      |   5 +
 target/riscv/csr.c           |  34 +++
 12 files changed, 1308 insertions(+)
 create mode 100644 hw/riscv/cbqri_bandwidth.c
 create mode 100644 hw/riscv/cbqri_capacity.c
 create mode 100644 hw/riscv/cbqri_example_soc.c
 create mode 100644 include/hw/riscv/cbqri.h

-- 
2.34.1