qemu-devel
[Top][All Lists]
Advanced
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:1

From: Gavin Shan
Subject: Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159
Date: Thu, 6 Jun 2024 15:05:02 +1000
User-agent: Mozilla Thunderbird 

On 6/6/24 01:56, Jean-Philippe Brucker wrote:

On Wed, Jun 05, 2024 at 11:28:47AM +1000, Gavin Shan wrote:

    WriteSections64(): 
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
 AARCH64 small code model requires identical ELF and PE/COFF section offsets 
modulo 4 KB.


Ah I've seen this once but it disappeared as I tried to investigate and
I've since changed the implementation, so I don't have many notes about
it.

Maybe you could try to bisect from "ArmVirtPkg: ArmCcaIoMmu: Provide an
implementation for SetAttribute", but it may give false positives if the
error depends on some random linker placement. Could be
"ArmVirtPkg/ArmPlatformLibQemu: Setup early UART mapping in a Realm" which
adds a 4k page to the data section for the ealy RSI config call, though
that has explicit 4kB alignment.

In my notes I also wrote that changing "-z common-page-size=0x20" to 4k in
the link flags may have made the error disappear, but I doubt it's the
right fix.

I'll try GCC 11 to see if I can reproduce.



Ok. I run a git-bisect and the first problematic commit is 1153ae939c
("ArmVirtPkg/ArmPlatformLibQemu: Add a third-level page table for the UART 
idmap")


Ah thanks, I'm able to reproduce the problem now, it was my local config
that masked it.



I'm not familiar with edk2. The error is raised by 
BaseTools/Source/C/GenFw/Elf64Convert.c::WriteSections64()
where the relocatable address isn't properly aligned to 4KB. So I modified the 
code
as below, but I have to run two consecutive builds. In the first attempt build, 
I
still hit the same error.


This seems to be because GenFw generates a file even on error, so it
doesn't retry the second time.

This commit moves the page tables from .rodata to .data. When linking
IdMap.obj into ArmPlatformPrePeiCore.dll, the alignment of the .text
section changes from 0x1000 to 0x800. This change comes from the linker
script putting .rodata into .text. I don't know why the included .rodata
alignment affects the .text alignment, but I don't think it matters here.

In GenFw, ScanSections64() calculates a mCoffAlignment as the max
.text/.data/.hii section alignement. Since with this commit, .data
alignement (0x1000) becomes larger than .text (0x800), it picks 0x1000 as
the output text offset, and then WriteSections64() complains that this
offset isn't equal to the input .text alignment modulo 0x1000.

The linker script says:

   /*
    * The alignment of the .data section should be less than or equal to the
    * alignment of the .text section. This ensures that the relative offset
    * between these sections is the same in the ELF and the PE/COFF versions of
    * this binary.
    */

but that's not what we're getting. I don't have a fix yet, other than
forcing the .text and .data alignment to 4k.



Jean, thanks for your explanation. Right, the issue is caused by mismatched
alignments for ELF and PE/COFF sections. I ever dumped the variables at the
failing point, showing the mismatched alignments (0x800 vs 0x1000). Apart from
that, the virtual address of 'text' section is aligned to 0x800 instead of
0x1000 after ArmPlatformPrePeiCore.dll is dumped by 'readelf'.

SecHdr->sh_addr:                    0x800              <<< Mismatched alignment 
between ELF and PE/COFF
SecOffset:                          0x1000
SymShdr->sh_addr:                   0x800
mCoffSectionsOffset[Sym->st_shndx]: 0x1000
GenFw: ERROR 3000: Invalid
  WriteSections64(): 
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
 AARCH64 small code model requires identical ELF and PE/COFF section offsets 
modulo 4 KB.

# readelf -S 
Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .text             PROGBITS         0000000000000800  00000800       <<< 
Aligned to 0x800
       00000000000051b8  0000000000000000  AX       0     0     2048

With the following changes, I'm able to build the firmware successfully. I don't
see how COMMONPAGESIZE is sorted out because I don't find its definition in the
source code.

diff --git a/BaseTools/Scripts/GccBase.lds b/BaseTools/Scripts/GccBase.lds
index 9f27e83bb0..5463df47a9 100644
--- a/BaseTools/Scripts/GccBase.lds
+++ b/BaseTools/Scripts/GccBase.lds
@@ -20,7 +20,8 @@ SECTIONS {
    */
   . = PECOFF_HEADER_SIZE;
- .text : ALIGN(CONSTANT(COMMONPAGESIZE)) { 
+  /* .text : ALIGN(CONSTANT(COMMONPAGESIZE)) { */^M
+  .text : ALIGN(4096) {^M

# <rebuild>
# readelf -S 
Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
There are 24 section headers, starting at offset 0x63050:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .text             PROGBITS         0000000000001000  00001000     <<< 
Aligned to 0x1000 now
       00000000000051b8  0000000000000000  AX       0     0     4096

Even the edk2 for the guest can be built successfully, but I'm not able to try 
it
because I'm unable to bring up the host now. I tried to rebuild the environment
from scratch, the host runs into crash inside EDK2 unfortunately...

  TF-RMM:   https://git.codelinaro.org/linaro/dcap/rmm.git                      
 (branch: cca/v2)
  EDK2:     git@github.com:tianocore/edk2.git                                   
 (tag:    edk2-stable202402)
  TF-A:     https://git.codelinaro.org/linaro/dcap/tf-a/trusted-firmware-a.git  
 (branch: cca/v2)
  QEMU:     https://git.qemu.org/git/qemu.git                                   
 (branch: master)
  KERNEL:   https://git.gitlab.arm.com/linux-arm/linux-cca.git                  
 (branch: cca-full/v2)
  BuildRoot: <doesn't matter at present>

arm64-server# home/gavin/sandbox/qemu.main/build/qemu-system-aarch64      \
              -M virt,virtualization=on,secure=on,gic-version=3,acpi=off  \
              -cpu max,x-rme=on -m 8G -smp 8                              \
              -monitor none -serial mon:stdio -nographic -nodefaults      \
              -bios /home/gavin/sandbox/CCA/tf-a/flash.bin                \
              -kernel /home/gavin/sandbox/CCA/linux/arch/arm64/boot/Image \
              -append console=ttyAMA0 root=/dev/vda                       \
              -drive 
format=raw,if=none,file=/home/gavin/sandbox/CCA/buildroot/output/images/rootfs.ext4,id=hd0
 \
              -device virtio-blk-pci,drive=hd0                            \
              -netdev 
tap,id=tap0,vhost=false,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
              -device virtio-net-pci,netdev=tap0,mac=52:54:00:f1:26:b0          
                \
              -fsdev 
local,security_model=none,path=/home/gavin/sandbox/CCA,id=shr0             \
              -device virtio-9p-device,fsdev=shr0,mount_tag=shr0
                 :
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.10.0(debug):99e0b97aa-dirty
NOTICE:  BL1: Built : 23:14:56, Jun  5 2024
INFO:    BL1: RAM 0xe0ee000 - 0xe0f7000
INFO:    BL1: Loading BL2
INFO:    Loading image id=1 at address 0xe06b000
INFO:    Image id=1 loaded: 0xe06b000 - 0xe0742d1
NOTICE:  BL1: Booting BL2
INFO:    Entry point address = 0xe06b000
INFO:    SPSR = 0x3cd
INFO:    [GPT] Boot Configuration
INFO:      PPS/T:     0x2/40
INFO:      PGS/P:     0x0/12
INFO:      L0GPTSZ/S: 0x0/30
INFO:      PAS count: 0x6
INFO:      L0 base:   0xedfe000
INFO:    [GPT] PAS[0]: base 0xe001000, size 0xff000, GPI 0xa, type 0x1
INFO:    [GPT] PAS[1]: base 0xe100000, size 0xcfe000, GPI 0x8, type 0x1
INFO:    [GPT] PAS[2]: base 0xedfe000, size 0x202000, GPI 0xa, type 0x1
INFO:    [GPT] PAS[3]: base 0x40000000, size 0x100000, GPI 0x9, type 0x1
INFO:    [GPT] PAS[4]: base 0x40100000, size 0x2800000, GPI 0xb, type 0x1
INFO:    [GPT] PAS[5]: base 0x42900000, size 0x1fd700000, GPI 0x9, type 0x1
INFO:    Enabling Granule Protection Checks
NOTICE:  BL2: v2.10.0(debug):99e0b97aa-dirty
NOTICE:  BL2: Built : 23:14:56, Jun  5 2024
INFO:    BL2: Doing platform setup
INFO:    Reserved RMM memory [0x40100000, 0x428fffff] in Device tree
INFO:    BL2: Loading image id 3
INFO:    Loading image id=3 at address 0xe0a0000
INFO:    Image id=3 loaded: 0xe0a0000 - 0xe0b10c4
INFO:    BL2: Loading image id 35
INFO:    Loading image id=35 at address 0x40100000
INFO:    Image id=35 loaded: 0x40100000 - 0x403033b0
INFO:    BL2: Loading image id 5
INFO:    Loading image id=5 at address 0x60000000
INFO:    Image id=5 loaded: 0x60000000 - 0x60200000
NOTICE:  BL2: Booting BL31
INFO:    Entry point address = 0xe0a0000
INFO:    SPSR = 0x3cd
NOTICE:  BL31: v2.10.0(debug):99e0b97aa-dirty
NOTICE:  BL31: Built : 23:14:56, Jun  5 2024
INFO:    GICv3 without legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    Maximum SPI INTID supported: 287
INFO:    BL31: Initializing runtime services
INFO:    RMM setup done.
INFO:    BL31: Initializing RMM
INFO:    RMM init start.
Booting RMM v.0.4.0(debug) 17924bc Built with GCC 11.4.1
RMM-EL3 Interface v.0.2
Boot Manifest Interface v.0.3
RMI/RSI ABI v.1.0/1.0 built: Jun  5 2024 23:03:00
INFO:    RMM init end.
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x60000000
INFO:    SPSR = 0x3c9
Loading driver at 0x00060009160 EntryPoint=0x00000000000
ArmVirtGetMemoryMap: Dumping System DRAM Memory Map:
        PhysicalBase: 0x40000000
        VirtualBase: 0x40000000
        Length: 0x200000000
UEFI firmware (version  built at 23:28:51 on Jun  5 2024)
PlatformPeim: PL011 UART (console) @ 0x9000000
PlatformPeim: PL011 UART (debug) @ 0x9000000
  :
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable
SetMemoryAttributes: BaseAddress == 0x22DC00000, Length == 0x1CE0000, 
Attributes == 0x20000
SetMemoryAttributes: BaseAddress == 0x22F8E0000, Length == 0xE50000, Attributes 
== 0x4000
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
EFI stub: update_fdt() ... done
EFI stub: efi_exit_boot_services: enter
EFI stub: efi_exit_boot_services: efi_pci_disable_bridge_busmaster
EFI stub: efi_exit_boot_services: efi_get_memory_map
EFI stub: efi_exit_boot_services: priv_func
=====> CoreExitBootServices
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x000000023BE60000 - 0x0000000000040000 
(0x0000000000000008)
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x0000000238AF0000 - 0x0000000000040000 
(0x0000000000000008)
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x0000000238AA0000 - 0x0000000000040000 
(0x0000000000000008)
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x0000000238A50000 - 0x0000000000040000 
(0x0000000000000008)
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x0000000238960000 - 0x0000000000040000 
(0x0000000000000008)
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x000000023BE20000 - 0x0000000000030000 
(0x0000000000000008)
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x0000000238860000 - 0x0000000000030000 
(0x0000000000000008)
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x0000000238820000 - 0x0000000000030000 
(0x0000000000000008)
CoreExitBootServices: MemoryProtectionExitBootServicesCallback
CoreExitBootServices: SaveAndSetDebugTimer
CoreExitBootServices: gCpu->DisableInterrupt
CoreExitBootServices: CalculateEfiHdrCrc
CoreExitBootServices: Return with status=0x0


Synchronous Exception at 0x000000023248E9B4
PC 0x00023248E9B4
PC 0x00023248EA70
PC 0x00023248EBA8
PC 0x000232490FC8
PC 0x00023248A004
PC 0x00023248973C
PC 0x0002324894DC
PC 0x00023F2C7FA8 (0x00023F2C1000+0x00006FA8) [ 1] DxeCore.dll
PC 0x00023BCC6604 (0x00023BCBE000+0x00008604) [ 2] BdsDxe.dll
PC 0x00023F2CBC68 (0x00023F2C1000+0x0000AC68) [ 3] DxeCore.dll
[ 1] 
/home/gavin/sandbox/CCA/edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
[ 2] 
/home/gavin/sandbox/CCA/edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/BdsDxe/BdsDxe/DEBUG/BdsDxe.dll
[ 3] 
/home/gavin/sandbox/CCA/edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll

  X0 0x0000000000000000   X1 0x000000023F2C0740   X2 0x000000000000000A   X3 
0x00000002325D3E26
  X4 0x0000000000000020   X5 0xFFFFFFFFFFFFFFFE   X6 0x0000000000000000   X7 
0x0000000000000000
  X8 0x0000000000000000   X9 0x0000000238820000  X10 0x000000000000002E  X11 
0x00000000000023D0
 X12 0x00000000000023C4  X13 0x0000000000000001  X14 0x0000000000000002  X15 
0x0000000000000000
 X16 0x000000023BD30280  X17 0x000000000066BD90  X18 0x0000000000000000  X19 
0x000000023F2C0740
 X20 0x00000002325D3E27  X21 0x000000000000FFFF  X22 0x000000000000000D  X23 
0x000000000010FFFF
 X24 0x000000000000D800  X25 0x000000023F2C0AA8  X26 0x00000002325E2000  X27 
0x0000000000000002
 X28 0x0000000000000018   FP 0x000000023F2C06F0   LR 0x000000023248EA70

  V0 0x0000000000000000 0000000000000000   V1 0xFFFFFF80FFFFFFD0 
000000023F2C0800
  V2 0x0000000000000000 0000000000000000   V3 0x0000000000000000 
0000000000000000
  V4 0x0000000000000000 0000000000000000   V5 0x0000000000000000 
0000000000000000
  V6 0x0000000000000000 0000000000000000   V7 0x0000000000000000 
0000000000000000
  V8 0x0000000000000000 0000000000000000   V9 0x0000000000000000 
0000000000000000
 V10 0x0000000000000000 0000000000000000  V11 0x0000000000000000 
0000000000000000
 V12 0x0000000000000000 0000000000000000  V13 0x0000000000000000 
0000000000000000
 V14 0x0000000000000000 0000000000000000  V15 0x0000000000000000 
0000000000000000
 V16 0x0000000000000000 0000000000000000  V17 0x0000000000000000 
0000000000000000
 V18 0x0000000000000000 0000000000000000  V19 0x0000000000000000 
0000000000000000
 V20 0x0000000000000000 0000000000000000  V21 0x0000000000000000 
0000000000000000
 V22 0x0000000000000000 0000000000000000  V23 0x0000000000000000 
0000000000000000
 V24 0x0000000000000000 0000000000000000  V25 0x0000000000000000 
0000000000000000
 V26 0x0000000000000000 0000000000000000  V27 0x0000000000000000 
0000000000000000
 V28 0x0000000000000000 0000000000000000  V29 0x0000000000000000 
0000000000000000
 V30 0x0000000000000000 0000000000000000  V31 0x0000000000000000 
0000000000000000

  SP 0x000000023F2C06F0  ELR 0x000000023248E9B4  SPSR 0xA00002C9  FPSR 
0x00000000
 ESR 0x96000006          FAR 0x0000000000000008

 ESR : EC 0x25  IL 0x1  ISS 0x00000006

Data abort: Translation fault, second level

Stack dump:
  000023F2C05F0: 000000023F2C06A0 000000023BD31FF4 0000000238820000 
0000000000000002
  000023F2C0610: 000000023B1E2000 000000023FFF9000 0000000000000E20 
00000000001FFFFF
  000023F2C0630: 0000000238850000 0000000000000001 0000000000000003 
000000023FFF9E20
  000023F2C0650: 000000000000070C 0000000000000000 000000023F2C06D0 
000000023F2DD644
  000023F2C0670: 000000023F2C07C8 000000023F2C095F 0000000000000001 
000000023BD358ED
  000023F2C0690: 000000000000070C 0000000000000000 000000023F2C0750 
000000023BD31FF4
  000023F2C06B0: 0000000238820000 0000000000000001 000000023FFF9000 
000000023FFFA000
  000023F2C06D0: 000000023F2C07F0 000000023F2C2534 0000000000000001 
0000000000000000

000023F2C06F0: 000000023F2C0700 000000023248EA70 000000023F2C0840 
000000023248EBA8

  000023F2C0710: 00000002325D42BF 000000023B17E918 00000002325E2000 
0000000232489978
  000023F2C0730: 000000023F2C0AB0 00000002325E2000 0020004900460045 
0062007500740073
  000023F2C0750: 000000000020003A 0000000000000001 000000023F2C0860 
0000000000000001
  000023F2C0770: 0000000000000002 00000000000000FF 0000000000000000 
0000007F00000000
  000023F2C0790: 000000023F2C07F0 000000023F2C2548 000000023F2C07C0 
000000023F2DEDD8
  000023F2C07B0: 000000023F2C088D 000000023F2EA000 000000023F2C07F0 
000000023F2C2548
  000023F2C07D0: 0000000000000001 0000000000000000 000000023F2EB000 
000000023F2EA000
ASSERT [ArmCpuDxe] 
/home/gavin/sandbox/CCA/edk2/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(343):
 ((BOOLEAN)(0==1))

---> VirtPkg/Library/ArmPlatformLibQemu/IdMap.S

   .align    12
   .globl    idmap
   .globl    uart_pte
   .section  ".data.idmap", "aw", %progbits
   .align    12

# source edksetup.sh; export GCC5_AARCH64_PREFIX=
# make -j -C BaseTools; \                                               <<< 
Failed on the first attempt
   build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
    :
WriteSections64(): 
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
 AARCH64 small code model requires identical ELF and PE/COFF section offsets 
modulo 4 KB.
make: *** [GNUmakefile:405: 
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/OUTPUT/ArmPlatformPrePeiCore.efi]
 Error 2

# make -j -C BaseTools; \                                              <<< 
Succeed on the second attempt
   build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
    :
Generating FVMAIN FV
######
Fd File Name:QEMU_VARS 
(/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/QEMU_VARS.fd)
    :
- Done -
Build end time: 21:04:05, Jun.04 2024
Build total time: 00:00:06

After that, I'm unable to start the guest with the edk2 image successfully.

host# # mount | grep 9p
shr0 on /mnt/shr0 type 9p (rw,relatime,access=client,trans=virtio)
host# cat ./realm.sh
#!/bin/sh

SHR_DIR="/mnt/shr0"

qemu-system-aarch64 -accel kvm                              \
-machine virt,gic-version=3,confidential-guest-support=rme0 \
-cpu host -smp 2 -m 512M                                    \
-object 'rme-guest,id=rme0,measurement-algo=sha512'         \
-monitor none -serial mon:stdio -nographic                  \
-bios ${SHR_DIR}/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/QEMU_EFI.fd 
\
-kernel ${SHR_DIR}/linux/arch/arm64/boot/Image              \
-initrd ${SHR_DIR}/buildroot/output/images/rootfs.cpio      \
-append 'console=ttyAMA0'

host# ./realm.sh
UEFI firmware (version  built at 19:56:47 on Jun  4 2024)
add-symbol-file /home/gavin/sandbox/C                              <<< I don't 
see more output after it


I'm guessing in this case the firmware was corrupted because GenFw fails the
first time and never generated a complete binary



Yeah, the binary isn't reliable in this case.



Note that the guest edk2 is optional and experimental, you can use direct
kernel boot to get a working demo quicker.



I never did this before. Could you please provide the detailed steps on this?


Removing the -bios parameter to QEMU should be enough. You can also add
'earlycon' to -append to show early boot errors.



I didn't get a chance to try this yet since the host can't be brought up now.
I will try this later. I originally thought some sort of boot wrapper is needed
so that the kernel image has the capability to boot itself. For example, Mark
Rutland's boot wrapper [1] can be leveraged in this case. I don't think Image 
has
the capability to boot itself, right?

[1] 
https://git.kernel.org/pub/scm/linux/kernel/git/mark/boot-wrapper-aarch64.git

Thanks,
Gavin
reply via email to
[Prev in Thread] Current Thread [Next in Thread]