[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH v2 2/5] apic: add support for x2APIC mode
|
From: |
Bui Quang Minh |
|
Subject: |
Re: [PATCH v2 2/5] apic: add support for x2APIC mode |
|
Date: |
Mon, 3 Apr 2023 23:01:58 +0700 |
|
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 |
On 3/30/23 15:28, Igor Mammedov wrote:
On Wed, 29 Mar 2023 22:30:44 +0700
Bui Quang Minh <minhquangbui99@gmail.com> wrote:
On 3/29/23 21:53, Bui Quang Minh wrote:
On 3/28/23 22:58, Bui Quang Minh wrote:
On 3/27/23 23:49, David Woodhouse wrote:
On Mon, 2023-03-27 at 23:35 +0700, Bui Quang Minh wrote:
On 3/27/23 23:22, David Woodhouse wrote:
On Mon, 2023-03-27 at 22:45 +0700, Bui Quang Minh wrote:
Maybe I'm misreading the patch, but to me it looks that
if (dest == 0xff) apic_get_broadcast_bitmask() bit applies even in
x2apic mode? So delivering to the APIC with physical ID 255 will be
misinterpreted as a broadcast?
In case dest == 0xff the second argument to
apic_get_broadcast_bitmask
is set to false which means this is xAPIC broadcast
Yeah, but it *isn't* xAPIC broadcast. It's X2APIC unicast to APIC#255.
I think you want (although you don't have 'dev') something like this:
static void apic_get_delivery_bitmask(uint32_t *deliver_bitmask,
uint32_t dest, uint8_t
dest_mode)
{
APICCommonState *apic_iter;
int i;
memset(deliver_bitmask, 0x00, max_apic_words *
sizeof(uint32_t));
/* x2APIC broadcast id for both physical and logical
(cluster) mode */
if (dest == 0xffffffff) {
apic_get_broadcast_bitmask(deliver_bitmask, true);
return;
}
if (dest_mode == 0) {
apic_find_dest(deliver_bitmask, dest);
/* Broadcast to xAPIC mode apics */
- if (dest == 0xff) {
+ if (dest == 0xff && is_x2apic_mode(dev)) {
apic_get_broadcast_bitmask(deliver_bitmask, false);
}
} else {
Hmm, the unicast case is handled in apic_find_dest function, the logic
inside the if (dest == 0xff) is for handling the broadcast case only.
This is because when dest == 0xff, it can be both a x2APIC unicast and
xAPIC broadcast in case we have some CPUs that are in xAPIC and others
are in x2APIC.
Ah! Yes, I see it now.
Shouldn't apic_get_broadcast_bitmask(… true) add *all* APICs to the
mask, regardless of their mode? An APIC which is still in xAPIC mode
will only look at the low 8 bits and see 0xFF which it also interprets
as broadcast? Or is that not how real hardware behaves?
This is interesting. Your point looks reasonable to me but I don't
know how to verify it, I'm trying to write kernel module to test it
but there are just too many things running on Linux that uses
interrupt so the system hangs.
This raises another question: when dest == 0x102 in IPI, does the
xAPIC mode CPU with APIC ID 0x2 accept the IPI? I can't see this
stated clearly in the Intel SDM.
I do some more testing on my hardware, your point is correct when dest
== 0xffffffff, the interrupt is delivered to all APICs regardless of
their mode.
To be precisely, it only broadcasts to CPUs in xAPIC mode if the IPI
destination mode is physical. In case the destination mode is logical,
flat model/cluster model rule applies to determine if the xAPIC CPUs
accept the IPI. Wow, this is so complicated :)
It would be nice if you could update apic kvm unit test with your
findings if it doesn't test those variants yet.
And when dest == 0x102 in IPI, xAPIC mode CPU with APIC ID
0x2 also accepts that IPI.
KVM does not do the same way as the real hardware in these cases, if the
destination of IPI is 0xffffffff, IPI is broadcasted to x2APIC CPUs but
not xAPIC CPUs. The same with IPI has destination 0x102 does not deliver
to xAPIC CPU with APIC ID 0x2. This is the intended behavior as I see
some comments mentioning it.
diff --git a/x86/apic.c b/x86/apic.c
index 20c3a1a..8c91d27 100644
--- a/x86/apic.c
+++ b/x86/apic.c
@@ -679,7 +679,7 @@ static void set_ldr(void *__ldr)
apic_write(APIC_LDR, ldr << 24);
}
-static int test_fixed_ipi(u32 dest_mode, u8 dest, u8 vector,
+static int test_fixed_ipi(u32 dest_mode, u32 dest, u8 vector,
int nr_ipis_expected, const char *mode_name)
{
u64 start = rdtsc();
@@ -913,6 +913,38 @@ static void test_aliased_xapic_physical_ipi(void)
report(!f, "IPI to aliased xAPIC physical IDs");
}
+static void reset_apic_cpu(void *arg)
+{
+ u8 *id = (u8 *)arg;
+ reset_apic();
+ *id = apic_id();
+}
+
+static void test_physical_ipi_with_x2apic_id(void)
+{
+ u8 vector = 0xf1;
+ int f = 0;
+ u8 apic_id_cpu1;
+
+ if (cpu_count() < 2)
+ return;
+
+ if (!is_x2apic_enabled())
+ return;
+
+ on_cpu(1, reset_apic_cpu, &apic_id_cpu1);
+ handle_irq(vector, handle_ipi);
+
+ /*
+ * CPU1 is in xAPIC so it accepts the IPI because the (destination &
0xff)
+ * matches its APIC ID.
+ */
+ f += test_fixed_ipi(APIC_DEST_PHYSICAL, apic_id_cpu1 | 0x100, vector,
1, "physical");
+ f += test_fixed_ipi(APIC_DEST_PHYSICAL, 0xffffffff, vector,
cpu_count(), "physical");
+
+ report(!f, "IPI with x2apic id to xapic CPU");
+}
+
typedef void (*apic_test_fn)(void);
int main(void)
@@ -950,6 +982,7 @@ int main(void)
test_apic_id,
test_apicbase,
test_aliased_xapic_physical_ipi,
+ test_physical_ipi_with_x2apic_id,
};
assert_msg(is_apic_hw_enabled() && is_apic_sw_enabled(),
With this patch in kvm-unit-test, the version 3 of this series, which I
will post soon, passes the test but not KVM. So I am not sure if I
should post this test to kvm-unit-test.