Re: [Qemu-devel] Booting kernels with PVHVM documentation?

[Alex Bennée]

Liam Merwick <address@hidden> writes:

> On 06/03/2019 17:51, Alex Bennée wrote:
>>
>> Hi,
>>
>> I've been looking at using PVH as an alternative to a long bios boot
>> sequence to boot some x86_64 test kernels for tests/tcg. I'm finding it
>> hard to piece together all the bits but I naively thought it would just
>> be a case of adding a few ELF NOTES to my boot.S with something like:
>>
>>            ELFNOTE(Xen, XEN_ELFNOTE_VIRT_BASE,      _ASM_PTR 0x100000)
>>            ELFNOTE(Xen, XEN_ELFNOTE_ENTRY,          _ASM_PTR _start)
>>            ELFNOTE(Xen, XEN_ELFNOTE_PHYS32_ENTRY,   _ASM_PTR 0)    /* entry 
>> == virtbase */
>>            ELFNOTE(Xen, XEN_ELFNOTE_PADDR_OFFSET,   _ASM_PTR 0)
>>
>>            .code64
>>            .section .text
>>            /* Kernel Entry Point */
>>    .global _start
>>    _start:
>>            // Setup stack ASAP
>>            movq $stack_end,%rsp
>>
>> However I'm running into lots of head scratching as the get_elf_note
>> code seems to skip over the notes before failing:
>>
>>    ./qemu-system-x86_64 -monitor none -display none \
>>      -chardev stdio,id=out -device isa-debugcon,chardev=out \
>>      -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./tests/hello
>>    load_elf64: processing hdr:0 of type 1
>>    load_elf64: processing hdr:1 of type 4
>>    get_elf_note_type64: looking for type 18, first is 3
>>    get_elf_note_type64: 4/20
>>    get_elf_note_type64: offset is 36
>>    get_elf_note_type64: note is 0
>>    get_elf_note_type64: 0/123713
>>    get_elf_note_type64: offset is 123728
>>    load_elf64: processing hdr:2 of type 1685382481
>>    qemu-system-x86_64: Error loading uncompressed kernel without PVH ELF Note
>
>
>
> What does 'readelf -Wn ./tests/hello' or 'readelf -p .notes
> ./tests/hello' show?

$ readelf -Wn tests/hello

Displaying notes found in: .note.gnu.build-id
  Owner                 Data size       Description
  GNU                  0x00000014       NT_GNU_BUILD_ID (unique build ID 
bitstring)         Build ID: 919289306d162384e3c8a38441088cf6c511f623

Displaying notes found in: .note.Xen
  Owner                 Data size       Description
  Xen                  0x00000008       Unknown note type: (0x00000003)    
description data: 00 00 10 00 00 00 00 00
  Xen                  0x00000008       NT_VERSION (version)       description 
data: 40 00 10 00 00 00 00 00
  Xen                  0x00000008       Unknown note type: (0x00000012)    
description data: 00 00 00 00 00 00 00 00
  Xen                  0x00000008       Unknown note type: (0x00000004)    
description data: 00 00 00 00 00 00 00 00

$ readelf -p .notes tests/hello
readelf: Warning: Section '.notes' was not dumped because it does not exist!

>>
>> So I thought I'd go back to the Linux kernel and see if I could get it
>> to boot up. So I built an x86_64 kernel with:
>>
>>    CONFIG_XEN_PVHVM=y
>>    CONFIG_XEN_PVHVM_SMP=y
>>    CONFIG_XEN_PVH=y
>>    CONFIG_PVH=y
>>
>> And tried to boot that, it certainly gets a lot further but in detecting
>> the note 18 it's looking for but then doesn't provide any output. So I
>> started digging around the patches and saw talk of a PVH option ROM
>> which does all the x86 mode escalation before booting the kernel.
>> However I was unable to find any documentation about if I should be
>> adding this manually to my command line or if it is auto-magiced into
>> place. So I have a number of questions:
>>
>>    * what's the canonical command line for booting a Linux PVHVM kernel?
>
> I had been using '-bios ./qboot.bin' (using
> https://github.com/bonzini/qboot/pull/17/files) but Stefano's changes
> mean that isn't necessary. Here's a command line I used

I think they are all in.

>
> sudo $QEMU \
>       -name testvm01 \
>       -machine
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit \

Hmm if I use the q35 machine type I might have to rethink the approach
for getting output and results as I'm currently using the isa debugcon
and debug exit devices.

I wonder how much framework I would need for a minimal virtio or PV
serial implementation? Currently the ISA stuff is dumb as bricks and
simple:

        /* Output a single character to serial port */
        .global __sys_outc
__sys_outc:
        pushl %ebp
        movl %esp, %ebp
        out %al,$0xE9
        movl %ebp, %esp
        popl %ebp
        ret

and:

        /* output any non-zero result in eax to isa-debug-exit device */
        test %al, %al
        jz 1f
        out %ax, $0xf4

1:      /* QEMU ACPI poweroff */
        mov $0x604,%edx
        mov $0x2000,%eax
        out %ax,%dx
        hlt
        jmp 1b



>       -cpu host \
>       -m 1024,maxmem=20G,slots=2 \
>       -smp 1 \
>       -no-user-config \
>       -nodefaults \
>       -object
> memory-backend-file,id=mem0,share,mem-path=$IMAGE,size=235929600 \
>       -device nvdimm,memdev=mem0,id=nv0 \
>       -append 'console=ttyS0,115200,8n1 root=/dev/pmem0p1 panic=1 rw
> tsc=reliable no_timer_check noreplace-smp
> init=/usr/lib/systemd/systemd rootfstype=ext4 rcupdate.rcu_expedited=1
> reboot=f' \
>       -no-reboot \
>       -serial mon:stdio \
>       -kernel $KERNEL
>
>
>>    * should this work in TCG as well?
>>    * are they any special linker rules required for the Xen.notes?
>>
>> And finally:
>>
>>    * is this idea of mine a weird abuse of the PVHVM boot protocol or
>>      does it make sense?
>>


--
Alex Bennée