[Mingw-cross-env-list] Concept for MXE Binary Packages

[Volker Grabsch]

Dear MXE team,

Volker Grabsch schrieb:
> I believe it is time to push this forward.  Although I can't
> contribute much "manpower", I can at least try to provide a concept.
> I will write that down in a separate email.  Stay tuned. :-)

Okay, so here are my thoughts on that topic.  Please state your
opinion on the topics and options.  Any comments and discussion are
welcome.

The goals is to finally come up with a good consensus in the team.


1) Distribution independence
----------------------------

The early versions of MXE (formerly "win32-cross") started as Debian
source packages which were meant to run through dpkg-cross.  I stopped
that route for two reasons: Frist, dpkg-cross didn't get along with
Windows/MinGW specifics very well.  Second, the userbase was too
limited.  There's not just Debian out there.  I then decided to
rewrite it as shell script (and later as Makefile, as we know it), and
finally the userbase grew, I found more support and a real community
formed around the project.

To avoid repeating that mistake, I propose to establish a
distribution-independent approach first, and a separate
distribution-specific layer for convenience.


2) Binary dependencies
----------------------

Roughly speaking, most of our build results are independent of the
build OS:

- Windows libraries (cross-compiled)
- Windows executables (cross-compiled)
- Scripts (shell, Python, etc.)
- Configuration files/scripts (for pkgconf, cmake, qmake, etc.)

However, we also produce quite a few binaries which are depending
of the build OS:

- Cross compiler and binutils
- Code generators of various libraries
- (something else?)

We'll have to investigate how deeply these are tied to the build
system.

2a) Is it just the GLIBC_xxx version?

Then, we may be able to produce binaries which run on all latest
versions of the distros.

2b) Or, do we also have dependencies to specific Libstdc++ versions?
2c) Maybe other dependencies?

In those cases, we'll either have to clean up things (e.g. compiling
libstdc++ statically into our binaries), or we'll have to provide lots
of variants even among a single distro.


3) Simplicity at what level?
----------------------------

Assuming we want to keep things simple, there are conflicting goals
where we have to sacrifice simplicity of some aspects for the
simplicity of other aspects.

Among those decisions, I believe the following one is the most
important (and probably the most controversial):

3a) Don't modify recipes, depend on complex package manager

3b) Modify receipes, make package management trivial

Regarding 3a),

we would continue to install each package into our large usr/ tree,
possibly overwriting each other's files, especially when installing a
new version of an already built package.  However, we wouldn't care
because we'd let a package manager handle that mess.  For producing
clean packages, we'd have to keep several limitations in mind:

- Don't build two packages at once.  Other, the package manager can't
  decide safely which files came from which package.

- When building a new version of a package, uninstall the old version
  first.  Otherwise, the package manager can't decide safely which
  files were overwritten with identical contents.

- To benefit from auto-cleanup during development, the package manager
  is needed even there, i.e. while testing and modifying package
  receipes.

Regarding 3b),

we would cleanup our package receipes to install each package into a
separate folder.  Probably we would even create a separate folder for
each package version (libmng-2.0.1/, libmng-2.0.2/, etc.)  Combination
of these locations would happen via symlinks.  This is essentially
what distribution-independent package managers like NPM or GEM do, to
keep things simple.

However, our current build doesn't have this cleanly separated
structure, which leads to the following issues with this approach:

- We have to convert the packages almost certainly by hand, thus the
  set of binary packages will be incomplete for the first weeks.

- We need a transition plan such that "clean" and "old" packages can
  remain side-by-side for some time.

- The packages must be combined somehow, either via environment
  variables (PATH, PKG_CONFIG_*, etc.) or via symlinks (e.g. having a
  bin/, lib/ and lib/pkgconfig/ folders full of automatically
  generated symlinks).


4) Which package manager?
-------------------------

First of all, I'm using Debian and like dpkg/APT very much.  So I
personally wouldn't mind any Debian-specific approach.  However, there
are other distributions out there (see "1)").

We could probably concentrate on simple binary packages first (as
simple as TarXZ?), and generate Debian or RPM packages from that.
This is especially easy when going the route described in 3b).

Or, we could decide for one package manager, build everything for that
one, and try to convert to other packages.  That would probably the
simpler route when going 3a).  We could decide on Debian and convert
to RPM via "alien".  We could also decide for a more neutral build
system such as Guix, and convert to Debian or RPM from that.


5) Fully reproducible builds?
-----------------------------

We could aim for fully reproducible builds, that is, where everyone
can reproduce the binary package byte-for-byte if they wish.  In the
global community, this is an important goal to strive for, and there
are good reasons Debian, Guix and others are putting much effort into
that one.

However, I'm not sure if this is worthwhile for a relatively small
community such as MXE.  On the one hand, we depend on trust more than
others, as there are fewer people needed to compromise us.  On the
other hand, our resources are limited, so we shouldn't go down every
paranoid route.

Using approach 3a) or, mostly equivalently, deciding to use Guix as
package manager, we could strive for fully reproducible builds.  The
initial effort would be bigger, but then the goal is within reach.

If we decide against fully reproducible builds, our initial effort may
become much lower.  But changing our minds afterwards may become
prohibitively expensive - so expensive that we probably won't be able
to do that later on, even if we wanted.



That's it from me.  Please state your opinion on the topics and
options.  Also, please add options in case I overlooked some.  And
raise important questions that I forgot.


Thanks,
Volker

-- 
Volker Grabsch
---<<(())>>---