Re: [lmi] Cygwin build failure

[Greg Chicares]

[quoting personal email by prior permission]

On 2019-04-30 00:00, Vadim Zeitlin wrote:
> On Mon, 29 Apr 2019 18:56:42 +0000 Greg Chicares <address@hidden> wrote:
> 
> GC> Thanks. I'm haunted by a fear that I've overlooked something
> GC> that's important for cygwin.
> 
>  I'm afraid this is indeed the case, as Ilya's attempts to build it have
> run into a pretty obvious problem in almost the very beginning: the scripts
> try to build both 32 and 64 bit versions, but only 32 bit compiler is
> available, so this is never going to work. Clearly, if we want to build for
> both architectures, we need to install both compilers. And equally clearly,
> we can't install them both to the same directory, so more changes to the
> makefiles are needed, but first we need to decide where should we install
> them

Thank you and Ilya for finding this defect.

The system-test results (with many proprietary testdecks) differ for
32- vs. 64-bit builds. The differences are pretty small, and I think
we'll get comfortable with them, but that'll take some time. We'll
also want to perform some manual acceptance tests and compare the
speed of various operations on corporate machines. We'll also have
to work around any issues with corporate "security" software. All
of this will take time and effort, so we'll want to have both 32-
and 64-bit builds available for a while. (You won't, but we will.)

But they don't both have to be available in the same cygwin instance.
Cygwin allows multiple installations on the same machine (we'll want
two, but you'll want only one, I think). In light of your findings,
I think it makes sense to change the strategy as follows:

(1) Test the recent lmi trunk changes for 32-bit builds only. In the
office, we can do this in the 32-bit cygwin instance we've been
using, without performing any cygwin update there. IIRC, you're
both using 64-bit cygwin only, so you could do it in your existing
cygwin installation. This subtask involves only pulling lmi trunk
and then rebuilding from scratch, with 'install_msw.sh', which I'll
change, as you recommend...

> or perhaps return on the decision to build both versions at once
> because the overhead of doing this is even more significant than I thought
> in terms of both time (downloading gcc is never going to be instantaneous)
> and disk space (it's an extra half a gigabyte after decompressing).

...so that it builds only the 32-bit version.

This is already a non-trivial step because...

>  Moreover, even 32 bit build fails with 8.1.0, which is the version
> currently installed by install_mingw.make, because currently the compiler
> is not installed in the correct directory and we end up with
> /MinGW_/mingw32/bin/gcc.exe instead of the expected /MinGW_/bin/gcc.exe.

What can have gone wrong?

$git log --oneline -- install_mingw.make
9afb0922 Rename "scratch" directories
74fe75e2 Upgrade to gcc-8.x
bcdd1c61 Terminate all files with a single (not double) newline
53cea856 Update copyright notices
e3bbb946 Upgrade to gcc-7.3
45cbda55 Experimentally use 'cp' where 'mv' fails

It certainly worked as of the last commit shown, 45cbda55; we know that
because that change was a required workaround for a cygwin problem.

The other commits since then seem innocent. Only the most recent one,
9afb0922, looks like it could possibly be wrong, but I don't see
anything in it that could account for the observed anomaly.

> I'm rather puzzled about how this could ever work because it seems that we
> have "$(CP) --archive $(ad_hoc_dir)/mingw32 $(prefix)" command in
> install_mingw.make since 3e435a18e5 from more than 3 years ago, but this
> command doesn't do the right thing. Of course, this assumes I understand
> correctly what the "right thing" is, but looking at install_wx.sh and
> install_libxml2_libxslt.make it does seem like the compiler is expected to
> be in /MinGW_/bin and not some other directory. Although this might need to
> be changed in the light of above, i.e. maybe we should rather use
> /MinGW_/mingw{32,64}/bin/gcc.exe instead. But I really don't know how do
> you plan to address this.

Is it possible that the MinGW-w64 project has changed the structure of
the archives it distributes? Maybe they've inserted an extra subdirectory.
I'll grab the files and take a look.

>  Finally, and this is a very minor nitpick compared to everything else, but
> I wonder if we could use "wget --no-verbose" option to get rid of the
> progress bar in the log. It takes ~1000 lines for the compiler download and
> is just annoying to navigate (I _could_ define a fold in Vim which would
> conceal wget progress, but it's easier for me to lobby you to add the wget
> option than to do this).

I had removed that option in commit 5b6ed077a0b:

+# New spelling '--no-verbose' has replaced original '--non-verbose':
+#   http://sourceware.org/ml/cygwin-apps/2005-10/msg00140.html
+# However, don't use
+#   WGETFLAGS := '--no-verbose --timestamping'
+# because, as this is written in 2007-08, sourceforge's mirror system
+# is behaving in anomalous ways that '--no-verbose' would mask.

It appears that 'wget' is among the few things that do work in this
makefile, so I'll add that option as you suggest.

In practice, sf.net might be among the websites filtered by corporate
"security" software, and the workaround is to download it on a personal
machine and then use an approved "secure file sharing" system to get it
into /cache_for_lmi/downloads/ once and for all.

>  Please let me know if you'd like me to come up with some solution for
> fixing the major problems above or if you'd prefer to do it yourself.

It's too late tonight, so I'll do it tomorrow.

I wrote (1) above, but it'll be a while until we get to (2), which
would be a 64-bit build of lmi, in a separate cygwin installation.
The details vary by continent; I'll use capital letters to signify
discrete steps, and '-' for steps that require no action:

   USA  Eurasia  
  ----- -------
    A      -     install 64-bit cygwin
    -      B     debug lmi trunk
    C      -     upgrade to 32-bit gcc-8 using debugged lmi trunk
    -      D     switch from 32- to 64-bit compiler
    E            acceptance test for 64-bit builds
    F      F     replace 32- with 64-bit builds

AIUI, you already have a 64-bit cygwin instance, in which gcc-8
is non-functionally installed, so you don't need gcc-7 there, and
you don't need a 32-bit cygwin instance: you'll only need one
cygwin instance with one gcc-8 wordsize at any one time.

On the other side of the ocean, we'll want two cygwin instances,
at least for a while; the old 32-bit one with gcc-7 can be
removed after step F, when a new 32-bit cygwin instance with
gcc-8 takes its place.