discuss-gnuradio
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Discuss-gnuradio] [UHD] Notes on FPGA images packaging, and packaging i


From: Martin Braun
Subject: [Discuss-gnuradio] [UHD] Notes on FPGA images packaging, and packaging in general
Date: Wed, 7 Mar 2018 17:12:54 -0800
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0

Hi all,

we've had a bunch of great discussions on the packaging of FPGA images
(binaries) and on packaging UHD in general. Since there were many
different discussions, I'd like to summarize them all in one big email.
Thanks to Rick, Maitland, and Phil for providing feedback.

If you just use UHD via git, none of this affects you.

## Release tags

Going forward, we'll change the format of the release tags. Before, it
was release_003_011_000_000 and so on. Now, it'll be v3.11.0.0, which is
a much more standard format. Personally, I'm happy to make this change
-- we were hesitant to change it for consistency's sake, but it seems no
one actually cares about the old format anyway. If you're pulling from
github, this means you get nice, reproducible URLs like this:
https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0

...and the tarball for the sources:
https://github.com/EttusResearch/uhd/archive/v3.11.0.0.tar.gz

...and so on. I'd be interested to see who cares about the old tag
format. For now, we'll double-tag (and we might re-tag old versions,
too, if there's interest) to keep things easy.

## Getting FPGA images for releases

We have two paths here. First, we have the uhd_images_downloader format
(I'll talk more about that below). You can grab images from the
images/manifest.txt file, but I understand that's cumbersome.

So, on top of the uhd_images_downloader format, we're considering
bringing back an archive per release, but we're still figuring out the
details of this.

We have for the sake of feedback uploaded an archive here:
https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0

I'd love to know if this format is suitable for people, and if things
are missing. If this is all everyone needs, we'll find a way to automate
this and make it error-proof.

However, this means that you will be downloading *all* default images
(currently, that's over 200 MB after unpacking), some of which rarely or
never change (e.g., the USRP1 image will always be the same).
If you want to tailor the images you download (e.g., you might only care
about the embedded images), pulling the individual packages might be a
better option. The problem is, we don't have an easy way of identifying
which ones there are without requiring to run a Python script at package
build time.
Something like this works, if you have access to the UHD source code
(bash code):

SRC_URI=http://files.ettus.com/binaries/cache/`cat images/manifest.txt |
grep -v '#' | awk -F' ' '{print $3}' | grep b200mini`

...and it would limit the images to the b200mini images, but it's not
exactly handy. If people have suggestions, I'd be curious to hear them.

## FPGA Versions

There was a question why we don't version our FPGA images (e.g., why
does the X300 image not have version 1.2.3, and we say this version
works with UHD 3.11.0.0). There are several problems with this: First,
we have over 40 FPGA images (not all are packaged in the default image).
Versioning them all correctly is already a hard logistical issue.
Second, how do we create meaningful version numbers? The FPGA images
consist of many subcomponents, all of which would need to go into an
image version. The X300 and X310 images, for example, should always have
similar versions when they have similar behaviour. But what if we fix
something only for X300? Then, both the X310 and X300 images have 5
variants (XG, HG, XA, HA, and AA). What if we fix a 10GigE issue? How do
we keep the 10 version numbers for the {X300, X310} X {HG, XG, HA, AA,
XA} in sync? Then of course, we have components on all X-Series that we
also use on the E310. So that makes the version-number-space even more
higher-dimensional. And so on.
Finally, we consider the FPGA source code part of UHD. That means, for
version 3.11.0.0 of UHD, there is exactly 1 commit on the FPGA
repository that matches the source code. The images built off of this
commit are the ones that matter, and so they already have a version (the
UHD version, in this case, 3.11.0.0).
For any given UHD commit, the correct set of FPGA images is the one
tracked in images/manifest.txt.

## The new images downloader

If you're used to using uhd_images_downloader, nothing changes for you.
Here's the contents of an older email explaining the motivation and
changes in greater detail.
Note that we'll start using this tool to distribute even more things
going forward. We have a bunch of things where building binaries is
getting more and more complicated for end users, and we'll try and make
it easy to update devices.


```
Subject: [UHD] Changes to uhd_images_downloader
Date: Tue, 23 Jan 2018 15:33:14 -0800
From: Martin Braun <address@hidden>

For the next UHD version, we will be updating uhd_images_downloader. The
problem with the old uhd_images_downloader is that it refers to one big
zip file for all the images. As we add more and more devices, we not
only have to update the zip file more often, it also becomes bigger and
bigger. If you're running uhd_images_downloader on an E310, it's most
likely that you only want the E310 images, and not the megabytes of
other FPGA images.

The new tool fixes all those issues by breaking up the zip file into
more, smaller zip files. On top of that, the new tool has a bunch of
other improvements:

- It maintains an inventory of downloaded files, and skips downloading
them again if they didn't change. This means if we roll out a fix for
the X300 only, and you run uhd_images_downloader without any arguments,
it won't try and update anything else. Should you delete the inventory,
it'll simply get everything, like it did before.
- You can now filter by type. Use the -t switch to match against a
regular expression, e.g. '-t b200mini' will only download a b200mini file.
- We'll be adding more image types soon that are not FPGA images, such
as filesystem updates.
- 'uhd_images_downloader -l' will list all the targets you can download.

In general, this should improve things and make other things easier.
However, since we had to basically rewrite the tool, which had otherwise
matured, it's likely that the new tool has shortcomings. We'd like to
hear from you regarding those.

There are a few problems we are already tracking:
- We currently have no archives other than zip files. We're intending to
add .tar.xz in the future, and you will be able to use
uhd_images_downloader to fetch those (assuming your system can unzip xz
archives)
- The tool output changed, and we're still fine tuning the output.


If you're a package maintainer, you probably hard-coded the URL for the
FPGA images in your package creation script.
We track the URLs of the individual archives, along with their SHA256
hash, in a manifest that is checked into git:
https://github.com/EttusResearch/uhddev/blob/master/images/manifest.txt

The base URL for all those is http://files.ettus.com/binaries/cache, so
http://files.ettus.com/binaries/cache/usrp1/fpga-6bea23d/usrp1_usrp1_fpga_default.zip
will fetch the USRP1 FPGA images.

We're happy for any feedback -- ideally before we release the next version.

Cheers,
Martin
```



Hopefully this is helpful. Again, I'd like to thank everyone for
providing feedback.

Cheers,
Martin





reply via email to

[Prev in Thread] Current Thread [Next in Thread]