discuss-gnuradio
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Discuss-gnuradio] [UHD] Notes on FPGA images packaging, and packagi


From: Martin Braun
Subject: Re: [Discuss-gnuradio] [UHD] Notes on FPGA images packaging, and packaging in general
Date: Thu, 8 Mar 2018 12:49:53 -0800
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0

On 03/08/2018 11:33 AM, Sid Hayn wrote:
> https://github.com/EttusResearch/uhddev/blob/master/images/manifest.txt
> 
> This address is invalid, assuming you meant this one?
> 
> https://github.com/EttusResearch/uhd/blob/master/images/manifest.txt

Yes! Thanks for pointing that out.

-- M
> 
> Thanks,
> Zero
> 
> On Wed, Mar 7, 2018 at 8:12 PM, Martin Braun <address@hidden
> <mailto:address@hidden>> wrote:
> 
>     Hi all,
> 
>     we've had a bunch of great discussions on the packaging of FPGA images
>     (binaries) and on packaging UHD in general. Since there were many
>     different discussions, I'd like to summarize them all in one big email.
>     Thanks to Rick, Maitland, and Phil for providing feedback.
> 
>     If you just use UHD via git, none of this affects you.
> 
>     ## Release tags
> 
>     Going forward, we'll change the format of the release tags. Before, it
>     was release_003_011_000_000 and so on. Now, it'll be v3.11.0.0, which is
>     a much more standard format. Personally, I'm happy to make this change
>     -- we were hesitant to change it for consistency's sake, but it seems no
>     one actually cares about the old format anyway. If you're pulling from
>     github, this means you get nice, reproducible URLs like this:
>     https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0
>     <https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0>
> 
>     ...and the tarball for the sources:
>     https://github.com/EttusResearch/uhd/archive/v3.11.0.0.tar.gz
>     <https://github.com/EttusResearch/uhd/archive/v3.11.0.0.tar.gz>
> 
>     ...and so on. I'd be interested to see who cares about the old tag
>     format. For now, we'll double-tag (and we might re-tag old versions,
>     too, if there's interest) to keep things easy.
> 
>     ## Getting FPGA images for releases
> 
>     We have two paths here. First, we have the uhd_images_downloader format
>     (I'll talk more about that below). You can grab images from the
>     images/manifest.txt file, but I understand that's cumbersome.
> 
>     So, on top of the uhd_images_downloader format, we're considering
>     bringing back an archive per release, but we're still figuring out the
>     details of this.
> 
>     We have for the sake of feedback uploaded an archive here:
>     https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0
>     <https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0>
> 
>     I'd love to know if this format is suitable for people, and if things
>     are missing. If this is all everyone needs, we'll find a way to automate
>     this and make it error-proof.
> 
>     However, this means that you will be downloading *all* default images
>     (currently, that's over 200 MB after unpacking), some of which rarely or
>     never change (e.g., the USRP1 image will always be the same).
>     If you want to tailor the images you download (e.g., you might only care
>     about the embedded images), pulling the individual packages might be a
>     better option. The problem is, we don't have an easy way of identifying
>     which ones there are without requiring to run a Python script at package
>     build time.
>     Something like this works, if you have access to the UHD source code
>     (bash code):
> 
>     SRC_URI=http://files.ettus.com/binaries/cache/`cat
>     <http://files.ettus.com/binaries/cache/`cat> images/manifest.txt |
>     grep -v '#' | awk -F' ' '{print $3}' | grep b200mini`
> 
>     ...and it would limit the images to the b200mini images, but it's not
>     exactly handy. If people have suggestions, I'd be curious to hear them.
> 
>     ## FPGA Versions
> 
>     There was a question why we don't version our FPGA images (e.g., why
>     does the X300 image not have version 1.2.3, and we say this version
>     works with UHD 3.11.0.0). There are several problems with this: First,
>     we have over 40 FPGA images (not all are packaged in the default image).
>     Versioning them all correctly is already a hard logistical issue.
>     Second, how do we create meaningful version numbers? The FPGA images
>     consist of many subcomponents, all of which would need to go into an
>     image version. The X300 and X310 images, for example, should always have
>     similar versions when they have similar behaviour. But what if we fix
>     something only for X300? Then, both the X310 and X300 images have 5
>     variants (XG, HG, XA, HA, and AA). What if we fix a 10GigE issue? How do
>     we keep the 10 version numbers for the {X300, X310} X {HG, XG, HA, AA,
>     XA} in sync? Then of course, we have components on all X-Series that we
>     also use on the E310. So that makes the version-number-space even more
>     higher-dimensional. And so on.
>     Finally, we consider the FPGA source code part of UHD. That means, for
>     version 3.11.0.0 of UHD, there is exactly 1 commit on the FPGA
>     repository that matches the source code. The images built off of this
>     commit are the ones that matter, and so they already have a version (the
>     UHD version, in this case, 3.11.0.0).
>     For any given UHD commit, the correct set of FPGA images is the one
>     tracked in images/manifest.txt.
> 
>     ## The new images downloader
> 
>     If you're used to using uhd_images_downloader, nothing changes for you.
>     Here's the contents of an older email explaining the motivation and
>     changes in greater detail.
>     Note that we'll start using this tool to distribute even more things
>     going forward. We have a bunch of things where building binaries is
>     getting more and more complicated for end users, and we'll try and make
>     it easy to update devices.
> 
> 
>     ```
>     Subject: [UHD] Changes to uhd_images_downloader
>     Date: Tue, 23 Jan 2018 15:33:14 -0800
>     From: Martin Braun <address@hidden
>     <mailto:address@hidden>>
> 
>     For the next UHD version, we will be updating uhd_images_downloader. The
>     problem with the old uhd_images_downloader is that it refers to one big
>     zip file for all the images. As we add more and more devices, we not
>     only have to update the zip file more often, it also becomes bigger and
>     bigger. If you're running uhd_images_downloader on an E310, it's most
>     likely that you only want the E310 images, and not the megabytes of
>     other FPGA images.
> 
>     The new tool fixes all those issues by breaking up the zip file into
>     more, smaller zip files. On top of that, the new tool has a bunch of
>     other improvements:
> 
>     - It maintains an inventory of downloaded files, and skips downloading
>     them again if they didn't change. This means if we roll out a fix for
>     the X300 only, and you run uhd_images_downloader without any arguments,
>     it won't try and update anything else. Should you delete the inventory,
>     it'll simply get everything, like it did before.
>     - You can now filter by type. Use the -t switch to match against a
>     regular expression, e.g. '-t b200mini' will only download a b200mini
>     file.
>     - We'll be adding more image types soon that are not FPGA images, such
>     as filesystem updates.
>     - 'uhd_images_downloader -l' will list all the targets you can download.
> 
>     In general, this should improve things and make other things easier.
>     However, since we had to basically rewrite the tool, which had otherwise
>     matured, it's likely that the new tool has shortcomings. We'd like to
>     hear from you regarding those.
> 
>     There are a few problems we are already tracking:
>     - We currently have no archives other than zip files. We're intending to
>     add .tar.xz in the future, and you will be able to use
>     uhd_images_downloader to fetch those (assuming your system can unzip xz
>     archives)
>     - The tool output changed, and we're still fine tuning the output.
> 
> 
>     If you're a package maintainer, you probably hard-coded the URL for the
>     FPGA images in your package creation script.
>     We track the URLs of the individual archives, along with their SHA256
>     hash, in a manifest that is checked into git:
>     https://github.com/EttusResearch/uhddev/blob/master/images/manifest.txt
>     <https://github.com/EttusResearch/uhddev/blob/master/images/manifest.txt>
> 
>     The base URL for all those is http://files.ettus.com/binaries/cache
>     <http://files.ettus.com/binaries/cache>, so
>     
> http://files.ettus.com/binaries/cache/usrp1/fpga-6bea23d/usrp1_usrp1_fpga_default.zip
>     
> <http://files.ettus.com/binaries/cache/usrp1/fpga-6bea23d/usrp1_usrp1_fpga_default.zip>
>     will fetch the USRP1 FPGA images.
> 
>     We're happy for any feedback -- ideally before we release the next
>     version.
> 
>     Cheers,
>     Martin
>     ```
> 
> 
> 
>     Hopefully this is helpful. Again, I'd like to thank everyone for
>     providing feedback.
> 
>     Cheers,
>     Martin
> 
> 
> 
>     _______________________________________________
>     Discuss-gnuradio mailing list
>     address@hidden <mailto:address@hidden>
>     https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>     <https://lists.gnu.org/mailman/listinfo/discuss-gnuradio>
> 
> 




reply via email to

[Prev in Thread] Current Thread [Next in Thread]