Re: Go packaging

[Pjotr Prins]

Thanks Leo for the explanation. Now I understand why Go programs, such
as the IPFS implementation, have so many dependencies... What I
understand now is that packages get built 'lazily' and there is really
no way to force a build - other than running the target software. I
noticed the hash values in dependencies, so these are git checkouts
(i.e., they are versioned, but not in the standard sense).

It is an interesting approach - because it does guarantee people are
using the same (deep) software stack. I noticed that too:

On Tue, Oct 03, 2017 at 11:15:04AM -0400, Leo Famulari wrote:
> Based on my work creating a go-build-system and packaging a non-trivial
> Go application [0], I want to start a discussion on how we can
> efficiently package Go software in Guix.
> 
> Go software is developed rather differently from most of what we
> package, and I think our package abstraction does not fit Go libraries
> well.
> 
> The primary differences are 0) Go libraries are typically bundled as
> source code, 1) Go software is typically unversioned, 2) static archives
> (.a) are the norm, and 3) Go libraries are expected to be built as part
> of the build process of the calling application. That is, there is no
> standard way to build an entire Go library on its own.
> 
> Elaboration:
> 
> 0,1) The problem with every application bundling unversioned libraries
> is that we need to package a different Git commit of the library for
> every application we package, or risk our packages not working. Adding
> package variants per-version is a bit messy currently.
> 
> 3) As an example of how Go libaries are built piecemeal, take the core
> networking library, 'golang.org/x/net' [1]. It includes dozens of
> submodules such as bpf, icmp, ipv4, ipv6, etc. There is no way to build
> all these submodules with a single command. Instead, each one is built
> when it is needed during the build process of the calling application.
> There are no build scripts. The library compilation process is
> standardized as, for example, `go install golang.org/x/net/ipv4`.
> 
> This means that the entire networking library would consist of several
> dozen Guix packages, multiplied by the number of different Git commits
> required by the calling applications. It's unreasonable, in my opinion.
> 
> My suggestion is that we have two layers of Go library packages: a) a
> template layer that includes the source URI, unpack-path, and other
> package metadata, and b) a layer to create instances of the package
> within the inputs field of the calling package.
> 
> Perhaps the instantiation layer could look like this in practice:
> 
> (define-public my-go-program
>   [...]
>   (inputs
>    `(("golang.org/x/net"
>       ,(go-package golang-org-x-net
>         (version "ffcf1bedda")
>       (import-paths '("golang.org/x/net/ipv4"
>                       "golang.org/x/net/context"
>                       "golang.org/x/net/ipv6"
>                       "golang.org/x/net/internal/iana")))))))

When I read the package.json file it includes things like:

  "gxDependencies": [
    {
      "hash": "QmSpJByNKFX1sCsHBEp3R73FL4NF6FnQTEGyNAXHm2GS52",
      "name": "go-log",
      "version": "1.2.0"
    },
    {
      "author": "whyrusleeping",
      "hash": "QmZfwmhbcgSDGqGaoMMYx8jxBGauZw75zPjnZAyfwPso7M",
      "name": "go-libp2p-secio",
      "version": "1.1.8"
    },
    {
      "author": "whyrusleeping",
      "hash": "QmaPbCnUMBohSGo3KnxEa2bHqyJVVeEEcwtqJAYxerieBo",
      "name": "go-libp2p-crypto",
      "version": "1.5.0"
    },
pac

indeed, there are even two versions in there for 'whyrusleeping'
deep dependencies ;). (would that be a reference to Ruby's Why?).

On the surface, similar to Rubygems, I think it is no problem to
distribute source packages from Guix and have them compile on the fly.
In a way that is also an interesting model for late optimizations -
something we are lacking in our current infrastructure. What I think
we should do is import above json file and generate GNU binary
packages that are GO source bundles. Provided GO can use a live build
directory outside the store it will only compile bundles once, on
demand. Ruby 'compiles' or interprets every time, so that is one up on
Go ;). That target directory would be mutable, so that is a downside,
a potential security risk.

What you are saying is that, inside the build system, we pull in all
packages as sources and do a complete compile - which flies in the
face of how dependencies are built independently today. One question
is, can you force that full compilation? 

We ought to have a look at how Nix packaged Go builds because they are
already have a solution. Be interesting to see if they found a way to
compile packages 'greedily', the way Python does it.

Pj.