Re: ‘guix publish’ now compresses archives

[Tomáš Čech]

On Tue, Jul 19, 2016 at 12:22:42AM +0200, Ludovic Courtès wrote:
> Hi!
>
> ‘guix publish --compression’ can now compress archives in gzip format,
> using zlib, which makes it much more usable in real life.  See commit
> 4a1fc562ae5eedf40f6ae4eabe30580b0983b8f6.
>
> ‘guix publish’ is cache-friendly: it uses /nar/gzip URLs for gzipped
> archives, and /nar for uncompressed archives.  That way, even if ‘guix
> publish’ is restarted with different compression parameters, previously
> announced /nar/* URLs remain valid.
>
> Please try it and report back.
>
> To recap, the difficulty with all this is that we couldn’t just call out
> the ‘gzip’ command because ‘guix publish’ uses threads, and threads and
> fork(2) don’t go together well.  So we needed zlib bindings, and when
> discussing it earlier with Dave, I thought we’d have to wrap zlib’s
> low-level API, which is painful, so we were sad and all.  Next we had
> this hydra.gnu.org outage, which entailed more sadness.
>
> Then I realized that zlib has this high-level ‘gz’ API, which is easy
> and does what we need; hence, (guix zlib) provides bindings to that.
> The only limitation is that it needs a file descriptor as its source or
> sink and cannot compress into memory.  That’s enough for ‘guix publish’
> though.
>
> Comments welcome!

When I saw archive improvement I remembered I have a comment about
that (but not associated to compression).

Recently when Snappy and Flatpak news flew around the world I saw some
people were thinking about Guix as possible and cleaner alternative to
new package types.

As I am happy to see enthusiasm about Guix, I don't think that current
archive format is usable for distribution of binary packages of
whatever origin. Problem is that we have package definition as part of
package manager GIT snapshot or local files but this information is
not distributed along with the package archive.

If this area is interesting for us, we can do better. It would be
great to be able to verify such archive, maintain the reproducibility
or provide recipe how to bake missing dependencies.

I was thinking that we could (and should) add some optional support
for metadata which could describe the origin of the package. It could
contain:
- Guix GIT revision used
- differences against GIT revision
- local package recipe
- exported dependency tree of package recipes
- optionally URL where missing dependencies can be found

What do you think?

S_W

signature.asc
Description: Digital signature