Re: [PATCH v3 2/5] tests/acceptance/boot_linux_console: remove Armbian 19.11.3 bionic test for orangepi-pc machine

[Willian Rampazzo]

Hi Philippe,

On Mon, Mar 22, 2021 at 1:59 PM Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
>
> On Mon, Mar 22, 2021 at 5:54 PM Philippe Mathieu-Daudé <f4bug@amsat.org> 
> wrote:
> >
> > Hi Willian,
> >
> > On 3/8/21 9:44 PM, Willian Rampazzo wrote:
> > > On Mon, Mar 8, 2021 at 5:41 PM Willian Rampazzo <wrampazz@redhat.com> 
> > > wrote:
> > >>
>
> > >>>>>> -    def do_test_arm_orangepi_uboot_armbian(self, image_path):
> > >>>>>> +    @skipUnless(os.getenv('ARMBIAN_ARTIFACTS_CACHED'),
> > >>>>>> +                'Test artifacts fetched from unreliable 
> > >>>>>> apt.armbian.com')
> > >>>>>> +    @skipUnless(os.getenv('AVOCADO_ALLOW_LARGE_STORAGE'), 'storage 
> > >>>>>> limited')
> > >>>>>> +    def test_arm_orangepi_bionic_20_08(self):
> > >>>>>> +        """
> > >>>>>> +        :avocado: tags=arch:arm
> > >>>>>> +        :avocado: tags=machine:orangepi-pc
> > >>>>>> +        :avocado: tags=device:sd
> > >>>>>> +        """
> > >>>>>> +
> > >>>>>> +        # This test download a 275 MiB compressed image and expand 
> > >>>>>> it
> > >>>>>> +        # to 1036 MiB, but the underlying filesystem is 1552 MiB...
> > >>>>>> +        # As we expand it to 2 GiB we are safe.
> > >>>>>> +
> > >>>>>> +        image_url = ('https://dl.armbian.com/orangepipc/archive/'
> > >>>>>> +                     
> > >>>>>> 'Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz')
> > >>>>>
> > >>>>> The Armbian 20 is not available on this link anymore. I can browse 
> > >>>>> just 21.
> > >>>>
> > >>>> Cat-and-mouse game *sigh*.
> >
> > 2021-03-22 17:18:10,701 download         L0067 INFO | Fetching
> > https://archive.armbian.com/orangepipc/archive/Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz
> > -> /home/phil/avocado/da
> > ta/cache/by_location/f2eb27a12b81ce15e93f340fabbced2136af1caa/Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz.nfwdzivg
> > 2021-03-22 17:18:11,263 asset            L0136 INFO | Temporary asset
> > file unavailable due to failed download attempt.
> > 2021-03-22 17:18:11,263 asset            L0368 ERROR| URLError: <urlopen
> > error [Errno 113] No route to host>
> >
> > It might be a temporary problem, but looking long term the
> > current setup doesn't scale IMHO.
>
> Also I just noticed I *do* have the image cached, even twice:
>
> $ find /home/phil/avocado/data/cache/by_location/ -name
> Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz -ls
>   131661 280768 -rw-rw-r--   1 phil     phil     287501560 Oct 23
> 12:45 
> /home/phil/avocado/data/cache/by_location/cac379e6a4480624b000dfde1b5d935454254ce2/Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz
>   131761 280776 -rw-rw-r--   1 phil     phil     287501560 Feb 20
> 00:39 
> /home/phil/avocado/data/cache/by_location/aac619d9e7ea3726ebc86c159fddfb9d3f6274a5/Armbian_20.08.1_Orangepipc_bionic_current_5.8.5.img.xz
>
> But the hash is different.
>

You have 2 images because you fetched them from different locations.
When you fetch by location, the assets go to the directory
representing the hash of the location, This allows, for example, the
fetch of multiple initrd or vmlinuz files from different locations,
representing different files, but with the same name. Also, this was
one of your requests some time ago :)

What Beraldo worked on in the last week was a way to remove "unused"
files. Your cache is a good example of files that are not necessary
anymore because now you are using a different location to fetch them.

> Cleber, why not use the blob hash as a cache key? It is supposed to be
> constant...
>

You can do that by setting the test to download the file by name, and
not by location.

> Cc'ing Beraldo because I guess remember he worked on improving Avocado cache.
>
> > >>> Just to clarify here: in this patch I wanted to only make the change to 
> > >>> remove the test for the 19.11.3 image.
> > >>> And in the 3rd patch (tests/acceptance/boot_linux_console: change URL 
> > >>> for test_arm_orangepi_bionic_20_08)
> > >>> do the update for to correct the 20.08 link. So each patch fixes one 
> > >>> problem at a time.
>