Re: [lmi] wxWidgets submodules and install_wx.make

[Greg Chicares]

On 2018-02-02 17:11, Vadim Zeitlin wrote:
> 
>  As alluded to in a previous message, there is a problem with using
> install_wx.make with the latest versions of wxWidgets because downloading
> wxWidgets archive doesn't include the submodules and even though lmi
> doesn't need all of them (and so anybody working on it would profit from
> having a smaller local working copy not including libjpeg or libtiff), it
> does need some of them (expat, libpng, zlib).

This used to work, so what has changed?

Here's some background information. Originally, 'install_wx.make'
downloaded and built only wxwidgets.org release tarballs. Then it
gained the optional ability to get the sources by SHA1 instead:

  commit 27f633e1ba1017c4e3ca8c32750f2de3d5b53a6a
  Date:   2015-08-06T15:29:21+00:00
  Retrieve wx archives from github by sha1sum

Later, we removed its ability to get tarballs via ftp:

  commit b585cc2beed0e2ab029eb4a7e898dfd9b00f5512
  Date:   2017-05-19T21:03:50+00:00
  Remove obsolete ftp.wxwidgets.org support

and perhaps that commit also removed any ability to download and
build from release tarballs (though I'm not sure about that).

Now, AFAICS, we're using special tarballs that github creates
dynamically for a given SHA1 sum (as opposed to the classic
tarballs that used to be provided via ftp, and are apparently
still offered on github, e.g.:
  
https://github.com/wxWidgets/wxWidgets/releases/download/v3.1.0/wxWidgets-3.1.0.tar.bz2
). IIRC, we reasoned that those classic tarballs corresponded
exactly to some SHA1, so they're subsumed into the method of
downloading by SHA1.

Is it the case that the dynamically-generated tarballs that we've
used exclusively for the last several years formerly contained
"submodules" like expat, but now exclude them? Is that the issue?
Did github change its snapshotting process? Did wx restructure
its repository?

>  I'm not sure what would be your preferred solution to this problem. At the
> top level of the decision tree is the question of whether lmi needs to
> continue to use 3rd party libraries bundled with wxWidgets[*] or if it can
> use them directly. IMO it's not impossible that using some official builds
> provided by MinGW community could be a better solution, in theory, but I'm
> not sure if we want to do it right now. So I'll assume we do want to
> continue to use the versions bundled with wxWidgets, but please correct me
> if I'm wrong.

Correct. The MinGW-w64 community seems to be a loose confederation
of people who work on the gcc sources (under the supervision of the
gcc steering committee, which I think is pretty rigorous) but do not
release binaries, and others who build and release binaries that are
hosted at the project's sf.net site (but whose work isn't supervised
by anyone AFAICT). Now, if those packagers also distribute some
third-party libraries, who's accountable, and where's the rigor? And,
last time I looked, some packagers were identified only by nicknames,
so I'm not satisfied that any identifiable person is responsible.

I consider wx-built binaries trustworthy. If we could no longer use
them due to some technical issue, we should build them ourselves from
their original sources; but that requires work that I'd rather avoid.

>  If we do this, the only solution I see is to download all of the archives
> from the corresponding submodule repositories, e.g.
> https://github.com/wxWidgets/libexpat/tree/wx, and unzip them in the right
> place. This shouldn't be difficult to do, but it's not completely trivial
> to do it using shell fragments in a makefile neither, so I wonder if we

[...At this point I was wondering why we couldn't just use git? Oh, because
'install_wx.make' doesn't actually invoke git: instead, it gets a tarball
from github corresponding to a SHA. And of course github might withdraw
that facility at any time. Well, GMTA, because you go on to say...]

> really want to do this or if we could just use "git clone" instead? With
> "--depth=1", it shouldn't take that much longer than downloading a ZIP (but
> I didn't measure this), but we could use --recurse-submodules option to get
> all the submodules for us. And, of course, using Git would make things
> simpler for me when testing lmi with newer wxWidgets versions too, which is
> nice even if it's not, and shouldn't be, a preponderant factor in the
> decision.

A moderate change either way in bandwidth or download time doesn't matter
much. Completeness and robustness are crucial, so "git clone" is the only
candidate for the One True Way.

>  Of course, all this is only really a problem when using arbitrary
> wxWidgets commits and wouldn't be such a problem if we used only the
> official releases, which will continue to include submodules in them. But I
> think it could be useful to continue to allow using non-released versions
> of wxWidgets, even if it's less simple now than it was before.

We've found it essential to use non-released versions corresponding
to arbitrary SHA1s; let's assume that's a firm requirement.

>  Please let me know if you have any thoughts about this,

At this point, 'install_wx.make' uses tarballs for historical reasons
only; we can simply use 'git clone' once and something like 'git pull'
subsequently (and we even know that those commands can be used across
a severe corporate firewall). We don't need to validate downloads with
md5sums because git's SHA1 self-validation is stronger. We don't need
tools like wget, tar, etc.

I've never used git submodules before, but initializing and updating
them is easy with '--recurse-submodules', and I'm sure you can easily
explain any extra commands required to build them.

Can we conveniently make wxPdfDoc behave as a submodule of wxWidgets,
locally, so that updating and building it would become as automatic
as building, say, libpng as a submodule?

I take it that our accustomed way of upgrading wx no longer works;
yet we might need to upgrade at any time, so we'd better rewrite
'install_wx.make' pretty soon.