Re: [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"

[Cleber Rosa]

On Tue, Sep 10, 2019 at 09:41:20AM +0100, Daniel P. Berrangé wrote:
> On Tue, Sep 03, 2019 at 08:52:18PM -0400, Cleber Rosa wrote:
> > The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
> > from tests/acceptance/linux_initrd.py, is currently failing to fetch
> > the "vmlinuz" file.  The reason for the failure is that the Fedora
> > project retires older versions from the "dl.fedoraproject.org" URL,
> > and keeps them in "archives.fedoraproject.org".  As an added note,
> > that test uses a Fedora 28 image, because of the specific Linux kernel
> > version requirements of the test.
> > 
> > For the sake of stability, let's use URLs from the archived and
> > supposedely ever stable URLs.  The good news is that the currently
> > supported versions are also hosted on the later.  This change limits
> > itself to change the URLs, while keeping the fetched files the same
> > (as can be evidenced by the unchanged hashes).
> 
> The download.fedoraproject.org site we're (mostly) currently using
> is serviced by the Fedora mirrors which is very desirable as it
> spreads the load.
> 
> The archive.fedoraproject.org site is the master Fedora hosting
> server(s). dl.fedoraproject.org is the same master hosting service
> that is intended for use by the mirror sites to sync their content
> from.  Projects really shouldn't use either of these URLs for getting
> any content which is available via the mirror service as it places
> uncessary load on the Fedora master servers.
> 
> > Documentation and the "vm tests" fedora definition were also updated.
> > 
> > Signed-off-by: Cleber Rosa <address@hidden>
> > ---
> >  qemu-doc.texi                          |  6 +++---
> >  tests/acceptance/boot_linux_console.py | 25 +++++++++++++++----------
> >  tests/acceptance/linux_initrd.py       |  5 +++--
> >  tests/vm/fedora                        |  2 +-
> >  4 files changed, 22 insertions(+), 16 deletions(-)
> > 
> > diff --git a/qemu-doc.texi b/qemu-doc.texi
> > index 577d1e8376..37795f86fb 100644
> > --- a/qemu-doc.texi
> > +++ b/qemu-doc.texi
> > @@ -440,15 +440,15 @@ of <protocol>.
> >  
> >  Example: boot from a remote Fedora 20 live ISO image
> >  @example
> > -qemu-system-x86_64 --drive 
> > media=cdrom,file=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> > +qemu-system-x86_64 --drive 
> > media=cdrom,file=https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> >  
> > -qemu-system-x86_64 --drive 
> > media=cdrom,file.driver=http,file.url=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> > +qemu-system-x86_64 --drive 
> > media=cdrom,file.driver=http,file.url=http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
> >  @end example
> >  
> >  Example: boot from a remote Fedora 20 cloud image using a local overlay for
> >  writes, copy-on-read, and a readahead of 64k
> >  @example
> > -qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"http",, 
> > "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2";,,
> >  "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
> > +qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"http",, 
> > "file.url":"http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2";,,
> >  "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
> >  
> >  qemu-system-x86_64 -drive 
> > file=/tmp/Fedora-x86_64-20-20131211.1-sda.qcow2,copy-on-read=on
> >  @end example
> 
> Nothing quite says "cutting edge" like examples pointing to 5+ years old
> software. Or to put it another way, we should update this to point to a
> modern OS release. ie Fedora 30.
>

Hi Daniel,

You're right, we should, and that's an obvious next step.  I tried to
keep this patch within the context of fixing the test regressions, and
making the overall experience stable.

> These should also be changed to download instead of dl.
>

I disagree, specially here, and believe that we shouldn't use the
"archive" site.  Having documentation that is accurate and whose
examples can be reproduced without caveats is a nice usability
touch.

I understand your worries about the eventual load on the Fedora
server.  While I'm certainly not qualified to speak for the Fedora
project, it'd be natural that the project would eventually cope
with the additional load, if that's actually significant.

> 
> > diff --git a/tests/acceptance/boot_linux_console.py 
> > b/tests/acceptance/boot_linux_console.py
> > index 2504ef0150..8a9a314ab4 100644
> > --- a/tests/acceptance/boot_linux_console.py
> > +++ b/tests/acceptance/boot_linux_console.py
> > @@ -76,8 +76,9 @@ class BootLinuxConsole(Test):
> >          :avocado: tags=arch:x86_64
> >          :avocado: tags=machine:pc
> >          """
> > -        kernel_url = 
> > ('https://download.fedoraproject.org/pub/fedora/linux/'
> > -                      
> > 'releases/29/Everything/x86_64/os/images/pxeboot/vmlinuz')
> > +        kernel_url = 
> > ('https://archives.fedoraproject.org/pub/archive/fedora'
> > +                      
> > '/linux/releases/29/Everything/x86_64/os/images/pxeboot'
> > +                      '/vmlinuz')
> >          kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
> >          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> >  
> > @@ -250,8 +251,9 @@ class BootLinuxConsole(Test):
> >          :avocado: tags=arch:aarch64
> >          :avocado: tags=machine:virt
> >          """
> > -        kernel_url = 
> > ('https://download.fedoraproject.org/pub/fedora/linux/'
> > -                      
> > 'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz')
> > +        kernel_url = 
> > ('https://archives.fedoraproject.org/pub/archive/fedora'
> > +                      
> > '/linux/releases/29/Everything/aarch64/os/images/pxeboot'
> > +                      '/vmlinuz')
> >          kernel_hash = '8c73e469fc6ea06a58dc83a628fc695b693b8493'
> >          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> >  
> > @@ -271,8 +273,9 @@ class BootLinuxConsole(Test):
> >          :avocado: tags=arch:arm
> >          :avocado: tags=machine:virt
> >          """
> > -        kernel_url = 
> > ('https://download.fedoraproject.org/pub/fedora/linux/'
> > -                      
> > 'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz')
> > +        kernel_url = 
> > ('https://archives.fedoraproject.org/pub/archive/fedora'
> > +                      
> > '/linux/releases/29/Everything/armhfp/os/images/pxeboot'
> > +                      '/vmlinuz')
> >          kernel_hash = 'e9826d741b4fb04cadba8d4824d1ed3b7fb8b4d4'
> >          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> >  
> > @@ -318,8 +321,9 @@ class BootLinuxConsole(Test):
> >          :avocado: tags=arch:s390x
> >          :avocado: tags=machine:s390_ccw_virtio
> >          """
> > -        kernel_url = 
> > ('https://download.fedoraproject.org/pub/fedora-secondary/'
> > -                      'releases/29/Everything/s390x/os/images/kernel.img')
> > +        kernel_url = ('https://archives.fedoraproject.org/pub/archive'
> > +                      
> > '/fedora-secondary/releases/29/Everything/s390x/os/images'
> > +                      '/kernel.img')
> >          kernel_hash = 'e8e8439103ef8053418ef062644ffd46a7919313'
> >          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> >  
> > @@ -360,8 +364,9 @@ class BootLinuxConsole(Test):
> >          :avocado: tags=arch:ppc64
> >          :avocado: tags=machine:pseries
> >          """
> > -        kernel_url = 
> > ('https://download.fedoraproject.org/pub/fedora-secondary/'
> > -                      
> > 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz')
> > +        kernel_url = ('https://archives.fedoraproject.org/pub/archive'
> > +                      '/fedora-secondary/releases/29/Everything/ppc64le/os'
> > +                      '/ppc/ppc64/vmlinuz')
> >          kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
> >          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> 
> Don't change these URLs. We can update to Fedora 30 though if you want
> to.
> 

I believe your suggestion to not change the URLs is based solely on your
perspective on the load on the "archives" server, right?

You should know that there's a number of counter points.  One is that
these tests are used on environments, which are maintained for a much
longer time than the lifespan of the currently maintained Fedora
versions.  We really need to make it stable and reliable, and IMO it
should be done upstream, for the benefit of all.

Best,
- Cleber.

> >  
> > diff --git a/tests/acceptance/linux_initrd.py 
> > b/tests/acceptance/linux_initrd.py
> > index 23be5a63aa..c61d9826a4 100644
> > --- a/tests/acceptance/linux_initrd.py
> > +++ b/tests/acceptance/linux_initrd.py
> > @@ -54,8 +54,9 @@ class LinuxInitrd(Test):
> >          QEMU has supported up to 4 GiB initrd for recent kernel
> >          Expect guest can reach 'Unpacking initramfs...'
> >          """
> > -        kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
> > -                      'Everything/x86_64/os/images/pxeboot/vmlinuz')
> > +        kernel_url = 
> > ('https://archives.fedoraproject.org/pub/archive/fedora'
> > +                      
> > '/linux/releases/28/Everything/x86_64/os/images/pxeboot/'
> > +                      'vmlinuz')
> >          kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
> >          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> >          max_size = 2 * (1024 ** 3) + 1
> > diff --git a/tests/vm/fedora b/tests/vm/fedora
> > index e8fa5bf0d2..7fec1479fb 100755
> > --- a/tests/vm/fedora
> > +++ b/tests/vm/fedora
> > @@ -23,7 +23,7 @@ class FedoraVM(basevm.BaseVM):
> >      name = "fedora"
> >      arch = "x86_64"
> >  
> > -    base = "http://dl.fedoraproject.org/pub/fedora/linux/releases/30/";
> > +    base = 
> > "https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/";
> >      link = base + 
> > "Server/x86_64/iso/Fedora-Server-netinst-x86_64-30-1.2.iso"
> >      repo = base + "Server/x86_64/os/"
> >      full = base + "Everything/x86_64/os/"
> 
> This should be changed to download instead of dl.
> 
> > -- 
> > 2.21.0
> > 
> > 
> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|