Re: [Qemu-devel] Supporting multiple CPU AddressSpaces and memory transaction attributes

[Edgar E. Iglesias]

On Mon, Sep 08, 2014 at 12:53:57PM +0100, Peter Maydell wrote:
> On 7 September 2014 02:47, Edgar E. Iglesias <address@hidden> wrote:
> > On Thu, Sep 04, 2014 at 06:47:58PM +0100, Peter Maydell wrote:
> >> We introduce the concept of memory transaction attributes,
> >> which are a guest-CPU specific bunch of bits (say, a
> >> uint32_t). We also allow the CPU to have more than one
> >> AddressSpace, and have the guest CPU code provide
> >> a function returning the AddressSpace to use for a
> >> given set of transaction attributes. For ARM we'd
> >> put all of (S/NS, mmu_idx, CPU number) into the attributes,
> >> use an AddressSpace each for S and NS, and use the
> >> S/NS bit in the attributes to select the AddressSpace
> >> The default is that we have one AddressSpace and always
> >> use that, ie the transaction attributes are ignored.
> >> (Maybe the function should return an integer index
> >> into a cpu->as[] array?)
> >
> > The new read_with_attrs (or another name) sounds good.
> > It allows us to incrementally change the codebase. It would be nice if the
> > arguments to that function be passed in a structure so we can
> > make future changes easier.
> >
> > Maybe it could even end up going this far:
> > void access(BusPayload *pay);
> >
> > With *pay having fields for attributes, data, slave error signaling etc.
> 
> Yes, picking an API that gives us expansion room
> for things like "let device signal an error response"
> would be a good idea.
> 
> > Im a little concerned about the guest CPU defining the attribute
> > format. Maybe we could start by making the attributes generic
> > (look the same for everyone).
> 
> I don't think this works at all -- our very first
> use case is ARM S/NS signalling, which is a very
> ARM specific concept. We could make it be partly
> generic and partly guest-defined, but really the
> attributes seem to me to be pretty much bus
> specific, and guest-CPU-specific is about the
> best proxy we have for that. I'd rather just have
> the core code pass these things around as an
> opaque pile of bits.

I'll try to clarify what worries me.

If we let the guest decide the bit format and ordering we'll end up with
something like following:

struct {
    union {
        struct {
            unsigned int secure : 1;
            unsigned int featureA : 1;
            unsigned int featureB : 1;
        } arm;
        struct {
            unsigned int featureC : 1;
            unsigned int featureD : 1;
        } mips;
        struct {
            unsigned int featureX : 1;
            unsigned int featureY : 1;
        } cris;
    };
};

Once we start adding masters and slaves all writing and reading these bits
in different formats we will have trouble mixing and matching because the
interpretations will clash. If we add a type to the
attr object, we could disallow incompatible setups or maybe add
conversion routines along the path (this is what I meant with attribute
specialization and conversion below) but this becomes quite complex
to enforce.

What happens on real SoCs is that allthough various CPUs typically have their
native bus protocols, they can speak to all kinds of devices through
bus bridges. Typically bus specific features get translated or dropped.

If we instead describe the attributes like the following
(no union, i.e the attrs have globally allocated bits and look and mean
the same to everyone):

struct {
    struct {
        unsigned int master_id;
        unsigned int featureA;
    } generic;

    struct {
        unsigned int secure : 1;
        unsigned int featureB : 1;
    } axi;

    struct {
        unsigned int featureC : 1;
    } ocp;

    struct {
        unsigned int featureX : 1;
        unsigned int featureY : 1;
    } smif;
};

As soon as a feature bit needs to be in multiple bus specific sub structs,
it becomes generic. As there will never be dups, the sub structs may not
make much sense, i.e everything is really generic. It's the highlevel
feature we describe, not the fact that it's on a specific bus protocol.

Anyway, masters that set axi.secure can talk to non-TZ OCP slaves without
enabling ocp.featureC accidentally. Features will naturally get dropped
along the buspath if they don't match between master and slave. E.g,
a slave that does not know the notion of a secure bit will never try to
read it. IMO this is a friendlier default behaviour to work with.

For cases where we need specific attr translations, this can be done through
modeling bridges. The IOMMU framework with attributes support would work for 
that
as a bridge would be a subset of an IOMMU (does attribute translation but no
address translation). I think this would become the exception rather than the
rule.

Cheers,
Edgar