Re: [Qemu-devel] [PATCH 00/10] bundle edk2 platform firmware with QEMU

[Daniel P . Berrangé]

On Tue, Mar 12, 2019 at 04:40:45PM +0100, Laszlo Ersek wrote:
> On 03/11/19 11:35, Daniel P. Berrangé wrote:
> > On Sun, Mar 10, 2019 at 12:21:55PM +0100, Philippe Mathieu-Daudé wrote:
> >> On 3/10/19 4:56 AM, Michael S. Tsirkin wrote:
> >>> On Sat, Mar 09, 2019 at 01:48:16AM +0100, Laszlo Ersek wrote:
> >>>> Repo:   https://github.com/lersek/qemu.git
> >>>> Branch: edk2_build
> >>>>
> >>>> This series advances the roms/edk2 submodule to the "edk2-stable201903"
> >>>> release, and builds and captures platform firmware binaries from that
> >>>> release. At this point they are meant to be used by both end-users and
> >>>> by Igor's ACPI unit tests in qtest ("make check").
> >>>>
> >>>> Previous discussion:
> >>>>
> >>>>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
> >>>>   http://mid.mail-archive.com/address@hidden
> >>>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
> >>
> >> There David raised a concern about "[adding] ~206 MB of binaries to the
> >> pc-bios directory". I'm also worried.
> >>
> >> GitHub kindly suggest to use git-lfs. It is an extra dependency I'd
> >> rather strongly avoid (because we support a wide range of host OS, each
> >> using a wide types of filesystems).
> >>
> >> What about storing those binaries on a file server (http/ftp) altogether
> >> with a file containing its hashed digest (SHA1/SHA256)? Then we already
> >> have all the required tools to fetch and verify those blob roms with the
> >> build system.
> >> Or we could store the hashes in the QEMU repository too.
> > 
> > A simple approach would simply be to 'xz' compress all the edk*.fd
> > files but still store them in git. They're already opaque binary
> > files, so replacing one binary format with another binary format
> > is no big deal IMHO
> > 
> > $ ls -alsh edk2-*
> > 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 
> > edk2-aarch64-code.fd.xz
> > 2.1M -rw-rw-r--. 1 berrange berrange  64M Mar 11 10:29 edk2-arm-code.fd
> > 772K -rw-rw-r--. 1 berrange berrange  64M Mar 11 10:29 edk2-arm-vars.fd
> > 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-i386-code.fd
> > 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 
> > edk2-i386-secure-code.fd
> > 528K -rw-rw-r--. 1 berrange berrange 528K Mar 11 10:29 edk2-i386-vars.fd
> >  12K -rw-rw-r--. 1 berrange berrange  11K Mar 11 10:29 edk2-licenses.txt
> > 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-x86_64-code.fd
> > 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 
> > edk2-x86_64-secure-code.fd
> > 
> > This gives a 50% disk space saving over the sparse size:
> > 
> > $ ls -alsh edk2-*
> > 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 
> > edk2-aarch64-code.fd.xz
> > 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-arm-code.fd.xz
> >  12K -rw-rw-r--. 1 berrange berrange 9.8K Mar 11 10:29 edk2-arm-vars.fd.xz
> > 1.6M -rw-rw-r--. 1 berrange berrange 1.6M Mar 11 10:29 edk2-i386-code.fd.xz
> > 1.8M -rw-rw-r--. 1 berrange berrange 1.8M Mar 11 10:29 
> > edk2-i386-secure-code.fd.xz
> > 4.0K -rw-rw-r--. 1 berrange berrange  320 Mar 11 10:29 edk2-i386-vars.fd.xz
> >  12K -rw-rw-r--. 1 berrange berrange  11K Mar 11 10:29 edk2-licenses.txt
> > 1.6M -rw-rw-r--. 1 berrange berrange 1.6M Mar 11 10:29 
> > edk2-x86_64-code.fd.xz
> > 1.9M -rw-rw-r--. 1 berrange berrange 1.9M Mar 11 10:29 
> > edk2-x86_64-secure-code.fd.xz
> > 
> > So about 9 MB compressed, instead of 20MB for the uncompressed sparse
> > files, which is on a par with the existing ROM sizes.
> > 
> > We would then need a "make" rule that runs  unxz to "build" the firmware
> > files. Wouldn't need to be more complicated that this:
> > 
> >    edk2-%.fd: edk2-%.fd.xz
> >          unzx -c $< > $@
> > 
> >    CLEANFILES += $(wildcard edk2*.fd)
> 
> The problem with this idea is that such *.xz files are not directly
> consumeable by Igor's ACPI regression test cases in qtest. The above
> approach is suitable for "make install" only, not for "make check", AIUI.

I was suggesting that this rule to uncompress is wired into the
default "make" target actually. Probably by just marking them
as generated files, so they are always built. This will make
it work for tests, as well as developers running from the source
tree.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|