[Qemu-devel] QEMU+Linux ARMv7A current state

[Peter Crosthwaite]

Hi,

I have done an audit of the ARMv7 boards to see what can boot a
vanilla linux kernel. The basic approach is to build ARM
multi_v7_defconfig kernel and boot QEMU using the DTBs built out by
the kernel. The intersection of what mainline Linux has a DTB for and
what QEMU models is tested. The boards that do/should work include:

    vexpress-a9
    vexpress-a15
    smdkc210 (exynos)
    xilinx-zynq-a9
    cubieboard (allwinner)
    highbank
    midway
    virt

Have I missed anything?

The basic rule is, no source code, config, or DTB edits allowed (i.e.
what is a day0 user of QEMU+Linux going to experience?).

So going board-by-board, these are the issues.

highbank/midway:

The kernel expects a secure monitor to be present which does at-least
two things:

* PSCI
* Some cache-maintainance ops.

QEMU has PSCI support, but the 0.1 command codings specified in the
DTB are different to QEMUs 0.1 encodings. So I switched monitor mode
for the CPUs on (it was conservatively set to off by the original
authors of the QEMU ARM monitor support) and turned PSCI on. I had to
write some patches to remap the PSCI encodings to highbank's custom
ones and then the kernel at least entry-points all 4 cpus. It then
deadlocks on something that I am yet to figure out. So the PSCI SMP
stuff is shelved.

The cache-maintenance is a bigger issue, that stops even a single core
boot. The kernel cache controller driver is doing an smc that QEMU
goes haywire on (as there simply is no firmware to catch the smc). But
caches in QEMU are a nop, so I wrote some monitor firmware that just
erets the smc without action and it works.

I am using Sata (sysbus-ahci) for the boot media which works straight
out of the box.

So single core highbank works with the dummy monitor firmware and the
monitor mode restrictor removed.

cubieboard:

The kernel expects a system level clock controller device that isn't
modelled. A dummy model which hardcodes all registers to a know reset
value and just lets the kernel read and write whatever it wants
resolves. I'm not 100% sure whether this is needed but the kernel
prints multiple errors with many pages of traces without. So it is at
least needed to reduce the dmesg noise.

QEMU cubieboard has no usable storage media, but the real hardware
does have AHCI sata. I added sysbus-ahci at the right place but turns
out the SATA controller has some custom power/clock (not really
sure??) registers specific to this SoC. It sets/clears bits then polls
them back expecting them to change to the other value asynchronously.
The kernel device probe then times-out. So I subclassed sysbus-ahci
and added the missing registers and forced the polled registers to the
"I'm done" state. It works.

I am using meta-sunxi Yocto-layer to build out the allwinner custom
kernel/rootfs etc, and with the clock and Sata changes I get a boot.
But when I change to the unedited kernel+dtb+rootfs I get stuck. RTC
messages are around the point of failure which is not modelled in
QEMU, so that is suspect.

So with a dummy clock-controller and a slightly modified Sata
controller (as a new device) cubieboard progresses, but it is still
stuck on something that looks RTC related.

xilinx-zynq-a9:

The kernel wants to use cpufreq to set the CPU frequency. The kernel
assumes ownership of the CPU clock divider but not the PLLs. QEMU uses
the HW default for the silicon (x26) for the PLL which is incompatible
with the dtb-set CPU frequency. The kernel boot asserts this and
crashes. The ideal solution is to provide some sort of pre-boot
firmware to set it to a compatible value. In real HW, this is the FSBL
bootloader.

TBH I think this is a kernel bug, as since cpufreq can be deconfigured
it is ultimately optional. This means it shouldn't be a showstopper on
a boot. The kernel should just switch off cpufreq and proceed and
behave as if the feature was never configured in the kernel.

There is also a critical bug in the SLCR model (my bad) where write to
registers are ignored. I have a patch.

More reading about the SLCR issues here (thanks to Guenter for getting
the ball rolling here):

https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg03374.html

I have a patch with sets up the needed SLCR values from software but
it needs more work.

The kernel also expect the ADC device which is absent. Thanks to
Guenter who patched this in which is under review:

https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg03375.html

I am using SD card for the rootfs media. SD has a bug where it cannot
handle a backing file of a non-round size. I am just passing the ext3
image in unchanged (so unpadded) to QEMU and QEMU rounds down the
image to the SD block size (512k!!), so I am losing us to 512k off the
end of my ext3 filesystem. I have a hacky patch that changes this to
round-up.

With the minimal firmware, slcr bugfix, Guenters ADC, the SD hack a
vanilla Zynq will be in good shape.

virt:

Virt largely works, but there are no immediately obvious
storage/network options that are supported by the kernel. So I have to
make some exceptions to the kernel config rule, that is, I add:

CONFIG_VIRTIO_PCI=y"
CONFIG_VIRTIO_BLK=y"
CONFIG_VIRTIO_NET=y"

To get disks and network for the virt board.

Could we add these configs to mainline Linux's defconfig? Note, we
don't need to add a DTS for this board, as QEMU's bootloader will
generate and provide it for us.

There is the known highmem PCI issue that is currently under discussion:

https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg06529.html

So aside from the missing VIRTIO kernel configs and highmem bug, virt
is in good shape.

vexpress:

vexpress boots up to rootfs probing, however the only storage media
that seems to be supported is flash, which doesn't have the needed
kernel configs to boot from (not 100% sure on that but I gave up
quickly). What are people who use this board in real hardware doing?
It may just work best with initrd style boots.

exynos:

QEMU models an Exynos skdkc210 but there is only a 310 available in
QEMU dtbs. QEMU exynos however does not model a lot on the board level
so we can largely get away with booting 310 dtb on 210.

Similar to vexpress, the exynos board has a lack of usable boot media.
The board does model USB (EHCI), however the DTB disables it as the
smdk310 board does not support it. So the alternative would be to use
one of the more full-featured exynos DTBs  (with way more peripherals)
from mainline Linux, but the kernel is liable to then probe all sorts
of missing hardware.

Exynos SDHCI support was on list for a few revs and quite some time,
merging it and using SD may solve the problem.

I have not attempted the SMP support.

Unlike all other boards, Exynos is quite consuming of RAM. All other
boards work with Yocto's default of 128MB but Exynos crashes unless
you jack it up. So users need to be careful of the -m option.

realview:

Realview is not tested, but I discuss it here as it is a bit of a red
herring. The kernel has dtb support for the realview-1176 board,
however this is not modelled by QEMU. I creating a Realview variant
with 1176 in QEMU, however the memory maps of all the peripherals were
mismatched.

Turns, out, QEMU is is modelling the realview FPGA emulation platform
which has a different peripheral map to the 1176 realview board
proper. So QEMU doesn't support the realview boards, just the FPGA
emulation platform, for which I can't find a kernel dtb. Should we
upstream one with some dtsi's for the CPU variation supported in QEMU?

So Realview does not work within my testing parameters with no
immediate prospects.

how I am testing:

So this is all powered by the Yocto project (Poky). I got some good
help from Nathan Rossi. I have a poky fork which changes the qemuarm
target to build my mainline (4.2.1) kernel (multi_v7_defconfig +
VIRTIO), all the DTBs and a usable rootfs. You can then specify the
type of machine (from the list above) to yocto's runqemu command. The
command sets up boot media and network automatically.

To use it:

git clone https://github.com/pcrost/poky.git
cd poky
source ./oe-init-build-env
MACHINE=qemuarm bitbake core-image-minimal #this takes a while
QEMUBIN=/path/to/qemu-arm MACHINE_SUBTYPE=virt runqemu qemuarm slirp

The QEMUBIN is optional as Yocto does build QEMU for you, but this
lets you BYO QEMU if you are doing qemu development outside of the
Yocto flow. Change MACHINE_SUBTYPE to one of the supported boards to
see results. I suggest starting with virt. Skip cubieboard, that
assumes my SATA patches. Everything else you will see varying degrees
of success. Pass qemuparams="-m 256" to runqemu for the smdkc200
(Exynos) board. Highbank and Midway will blank screen as the
outstanding issues happen pre-UART. A meaningful logbuf can be pulled
from RAM over the monitor. Useful instructions here:

http://www.wiki.xilinx.com/QEMU-Linux+Kernel+logbuf+Extraction

Some QEMU patches to follow to fix a few of the things mentioned.

We should get this into regression testing. Yocto has some automated
testing features in it's own right that we should be able to leverage.
Yet to investigate (Richard pasted some stuff on an earlier thread).

The exercise should be redone for ARM64, ARMv5 and ARMv6 and ARMv7M,
then other arches (many of which Poky supports).

Regards,
Peter