Re: [Qemu-devel] [PATCH] Fedora images: use URLs from stable "archives.fedoraproject.org"

[Philippe Mathieu-Daudé]

On 9/4/19 2:36 PM, Wainer dos Santos Moschetta wrote:
> Hi Cleber,
> 
> On 09/03/2019 09:52 PM, Cleber Rosa wrote:
>> The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test,
>> from tests/acceptance/linux_initrd.py, is currently failing to fetch
>> the "vmlinuz" file.  The reason for the failure is that the Fedora
>> project retires older versions from the "dl.fedoraproject.org" URL,
>> and keeps them in "archives.fedoraproject.org".  As an added note,
>> that test uses a Fedora 28 image, because of the specific Linux kernel
>> version requirements of the test.
>>
>> For the sake of stability, let's use URLs from the archived and
>> supposedely ever stable URLs.  The good news is that the currently
>> supported versions are also hosted on the later.  This change limits
>> itself to change the URLs, while keeping the fetched files the same
>> (as can be evidenced by the unchanged hashes).
>>
>> Documentation and the "vm tests" fedora definition were also updated.
>>
>> Signed-off-by: Cleber Rosa <address@hidden>
>> ---
>>   qemu-doc.texi                          |  6 +++---
>>   tests/acceptance/boot_linux_console.py | 25 +++++++++++++++----------
>>   tests/acceptance/linux_initrd.py       |  5 +++--
>>   tests/vm/fedora                        |  2 +-
>>   4 files changed, 22 insertions(+), 16 deletions(-)
>>
>> diff --git a/qemu-doc.texi b/qemu-doc.texi
>> index 577d1e8376..37795f86fb 100644
>> --- a/qemu-doc.texi
>> +++ b/qemu-doc.texi
>> @@ -440,15 +440,15 @@ of <protocol>.
>>     Example: boot from a remote Fedora 20 live ISO image
>>   @example
>> -qemu-system-x86_64 --drive
>> media=cdrom,file=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
>>
>> +qemu-system-x86_64 --drive
>> media=cdrom,file=https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
>>
>>   -qemu-system-x86_64 --drive
>> media=cdrom,file.driver=http,file.url=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
>>
>> +qemu-system-x86_64 --drive
>> media=cdrom,file.driver=http,file.url=http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly
>>
>>   @end example
>>     Example: boot from a remote Fedora 20 cloud image using a local
>> overlay for
>>   writes, copy-on-read, and a readahead of 64k
>>   @example
>> -qemu-img create -f qcow2 -o
>> backing_file='json:@{"file.driver":"http",,
>> "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2";,,
>> "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
>> +qemu-img create -f qcow2 -o
>> backing_file='json:@{"file.driver":"http",,
>> "file.url":"http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2";,,
>> "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2
>>     qemu-system-x86_64 -drive
>> file=/tmp/Fedora-x86_64-20-20131211.1-sda.qcow2,copy-on-read=on
>>   @end example
>> diff --git a/tests/acceptance/boot_linux_console.py
>> b/tests/acceptance/boot_linux_console.py
>> index 2504ef0150..8a9a314ab4 100644
>> --- a/tests/acceptance/boot_linux_console.py
>> +++ b/tests/acceptance/boot_linux_console.py
>> @@ -76,8 +76,9 @@ class BootLinuxConsole(Test):
>>           :avocado: tags=arch:x86_64
>>           :avocado: tags=machine:pc
>>           """
>> -        kernel_url =
>> ('https://download.fedoraproject.org/pub/fedora/linux/'
>> -                     
>> 'releases/29/Everything/x86_64/os/images/pxeboot/vmlinuz')
>> +        kernel_url =
>> ('https://archives.fedoraproject.org/pub/archive/fedora'
>> +                     
>> '/linux/releases/29/Everything/x86_64/os/images/pxeboot'
>> +                      '/vmlinuz')
>>           kernel_hash = '23bebd2680757891cf7adedb033532163a792495'
>>           kernel_path = self.fetch_asset(kernel_url,
>> asset_hash=kernel_hash)
>>   @@ -250,8 +251,9 @@ class BootLinuxConsole(Test):
>>           :avocado: tags=arch:aarch64
>>           :avocado: tags=machine:virt
>>           """
>> -        kernel_url =
>> ('https://download.fedoraproject.org/pub/fedora/linux/'
>> -                     
>> 'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz')
>> +        kernel_url =
>> ('https://archives.fedoraproject.org/pub/archive/fedora'
>> +                     
>> '/linux/releases/29/Everything/aarch64/os/images/pxeboot'
>> +                      '/vmlinuz')
>>           kernel_hash = '8c73e469fc6ea06a58dc83a628fc695b693b8493'
>>           kernel_path = self.fetch_asset(kernel_url,
>> asset_hash=kernel_hash)
>>   @@ -271,8 +273,9 @@ class BootLinuxConsole(Test):
>>           :avocado: tags=arch:arm
>>           :avocado: tags=machine:virt
>>           """
>> -        kernel_url =
>> ('https://download.fedoraproject.org/pub/fedora/linux/'
>> -                     
>> 'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz')
>> +        kernel_url =
>> ('https://archives.fedoraproject.org/pub/archive/fedora'
>> +                     
>> '/linux/releases/29/Everything/armhfp/os/images/pxeboot'
>> +                      '/vmlinuz')
>>           kernel_hash = 'e9826d741b4fb04cadba8d4824d1ed3b7fb8b4d4'
>>           kernel_path = self.fetch_asset(kernel_url,
>> asset_hash=kernel_hash)
>>   @@ -318,8 +321,9 @@ class BootLinuxConsole(Test):
>>           :avocado: tags=arch:s390x
>>           :avocado: tags=machine:s390_ccw_virtio
>>           """
>> -        kernel_url =
>> ('https://download.fedoraproject.org/pub/fedora-secondary/'
>> -                     
>> 'releases/29/Everything/s390x/os/images/kernel.img')
>> +        kernel_url = ('https://archives.fedoraproject.org/pub/archive'
>> +                     
>> '/fedora-secondary/releases/29/Everything/s390x/os/images'
>> +                      '/kernel.img')
>>           kernel_hash = 'e8e8439103ef8053418ef062644ffd46a7919313'
>>           kernel_path = self.fetch_asset(kernel_url,
>> asset_hash=kernel_hash)
>>   @@ -360,8 +364,9 @@ class BootLinuxConsole(Test):
>>           :avocado: tags=arch:ppc64
>>           :avocado: tags=machine:pseries
>>           """
>> -        kernel_url =
>> ('https://download.fedoraproject.org/pub/fedora-secondary/'
>> -                     
>> 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz')
>> +        kernel_url = ('https://archives.fedoraproject.org/pub/archive'
>> +                     
>> '/fedora-secondary/releases/29/Everything/ppc64le/os'
>> +                      '/ppc/ppc64/vmlinuz')
>>           kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77'
>>           kernel_path = self.fetch_asset(kernel_url,
>> asset_hash=kernel_hash)
>>   diff --git a/tests/acceptance/linux_initrd.py
>> b/tests/acceptance/linux_initrd.py
>> index 23be5a63aa..c61d9826a4 100644
>> --- a/tests/acceptance/linux_initrd.py
>> +++ b/tests/acceptance/linux_initrd.py
>> @@ -54,8 +54,9 @@ class LinuxInitrd(Test):
>>           QEMU has supported up to 4 GiB initrd for recent kernel
>>           Expect guest can reach 'Unpacking initramfs...'
>>           """
>> -        kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/'
>> -                      'Everything/x86_64/os/images/pxeboot/vmlinuz')
>> +        kernel_url =
>> ('https://archives.fedoraproject.org/pub/archive/fedora'
>> +                     
>> '/linux/releases/28/Everything/x86_64/os/images/pxeboot/'
>> +                      'vmlinuz')
>>           kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a'
>>           kernel_path = self.fetch_asset(kernel_url,
>> asset_hash=kernel_hash)
>>           max_size = 2 * (1024 ** 3) + 1
>> diff --git a/tests/vm/fedora b/tests/vm/fedora
>> index e8fa5bf0d2..7fec1479fb 100755
>> --- a/tests/vm/fedora
>> +++ b/tests/vm/fedora
>> @@ -23,7 +23,7 @@ class FedoraVM(basevm.BaseVM):
>>       name = "fedora"
>>       arch = "x86_64"
>>   -    base = "http://dl.fedoraproject.org/pub/fedora/linux/releases/30/";
>> +    base =
>> "https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/";
>>
>>       link = base +
>> "Server/x86_64/iso/Fedora-Server-netinst-x86_64-30-1.2.iso"
>>       repo = base + "Server/x86_64/os/"
>>       full = base + "Everything/x86_64/os/"
> 
> Currently it has just a few test cases that boot a Linux kernel. At this
> point maintain the urls to fetch the files across a few source file is
> manageable but it can get complex as it increases in number of tests.
> While ago I started to work on the introduction of a "kernel provider"
> mechanism to replace this "fetch a kernel" boilerplate. Unfortunately I
> didn't time to complete its implementation, now If you think it is
> valuable then I can manage to resume the work. Find the draft
> implementation here:
> https://github.com/wainersm/qemu/tree/acceptance_boot_linux

I remember and liked it, maybe you should start with a post asking the
community if the effort is worthwhile and would be useful, and what
combinations (distrib{name/version} kernel{version} arch
profile{debug,lpae,...} ...) are expected.

> 
> An alternate approach would be to extend Avocado by mimic the
> avocado.utils.vmimage module [1], but rather have kernel providers.
> 
> [1]
> https://avocado-framework.readthedocs.io/en/71.0/api/utils/avocado.utils.html#module-avocado.utils.vmimage
> 
> 
> Anyway, the changes proposed on this patch looks good to me.
> 
> Reviewed-by: Wainer dos Santos Moschetta <address@hidden>

Ditto:

Reviewed-by: Philippe Mathieu-Daudé <address@hidden>