Re: [PATCH v2 2/5] apic: add support for x2APIC mode

[Bui Quang Minh]

On 4/3/23 23:38, Bui Quang Minh wrote:
> On 4/3/23 17:27, David Woodhouse wrote:
> > On Wed, 2023-03-29 at 22:30 +0700, Bui Quang Minh wrote:
> > > > I do some more testing on my hardware, your point is correct when dest
> > > > == 0xffffffff, the interrupt is delivered to all APICs regardless of
> > > > their mode.
> > >
> > > To be precisely, it only broadcasts to CPUs in xAPIC mode if the IPI
> > > destination mode is physical. In case the destination mode is logical,
> > > flat model/cluster model rule applies to determine if the xAPIC CPUs
> > > accept the IPI. Wow, this is so complicated :)
> >
> > So even if you send to *all* of the first 8 CPUs in a cluster (e.g.
> > cluster 0x0001 giving a destination 0x000100FF, a CPU in xAPIC mode
> > doesn't see that as a broadcast because it's logical mode?
>
> I mean if the destination is 0xffffffff, the xAPIC CPU will see 
> destination as 0xff. 0xff is broadcast in physical destination mode 
> only, in logical destination, it may not be a broadcast. It may depend 
> on the whether it is flat model or cluster model in logical destination 
> mode.
>
> In flat model, 8 bits are used as mask, so in theory, this model can 
> only support 8 CPUs, each CPU reserves its own bit by setting the upper 
> 8 bits of APIC LDR register. In Intel SDM, it is said that 0xff can be 
> interpreted as a broadcast, this is true in normal case, but I think if 
> the CPU its APIC LDR to 0, the IPI should not deliver to this CPU. This 
> also matches with the current flat model of logical destination mode 
> implementation of userspace APIC in Qemu before my series. However, I 
> see this seems like a corner case, I didn't test it on real hardware.
>
> With cluster model, when writing the above paragraphs, I think that 0xff 
> will be delivered to all APICs (mask = 0xf) of cluster 15 (0xf). 
> However, reading the SDM more carefully, I see that the there are only 
> 15 clusters with address from 0 to 14 so there is no cluster with 
> address 15. 0xff is interpreted as broadcast to all APICs in all 
> clusters too.
>
> In conclusion, IPI with destination 0xffffffff can be a broadcast to all 
> xAPIC CPUs too if we just ignore the corner case in flat model of 
> logical destination mode (we may need to test more)

I do some tests on my machine with custom Linux kernel module (I can't 
use kvm-unit-tests because the display on my laptop is on embedded 
display port not serial port so I don't know how to get the output). I 
find out that
- In flat model, if the CPU intentionally set its 8 bit address to 0 in 
APIC_LDR, the 0xff logical IPI does not deliver to this CPU.
- In cluster model, the 4 higher bits used as cluster address, if these 
4 bits is equal to 0xf, the IPI is broadcasted to all cluster. The 4 
lower bits are used to address APICs in the cluster. If the CPU 
intentionally set these 4 lower bits to 0 in APIC_LDR, same as the flat 
model, this CPU does not receive the logical IPI. For example, the CPUs 
set its address to 0x20 (cluster 2, address 0 in cluster), the logical 
IPI with destination == 0xff does deliver to cluster 2 but does not 
deliver to this CPU.

So I choose to stick with the current implementation, 0xffffffff in IPI 
is seen as 0xff IPI in xAPIC CPUs. This IPI if in physical mode is a 
broadcast but not in logical mode.