Re: [lmi] Cygwin build failure

[Greg Chicares]

[TL;DR: what does 'cp -a /directory/that/exists /another/directory' do?]

On 2019-04-30 03:36, Vadim Zeitlin wrote:
> On Tue, 30 Apr 2019 01:59:20 +0000 Greg Chicares <address@hidden> wrote:
> 
> [...speaking of 32- and 64-bit builds coexistence...]
> 
> GC> But they don't both have to be available in the same cygwin instance.
> GC> Cygwin allows multiple installations on the same machine
> 
>  Interesting, I didn't know this.

That was a major enhancement at the end of 2009:

https://cygwin.com/ml/cygwin-announce/2009-12/msg00027.html
| Date: Wed, 23 Dec 2009 10:56:07 +0100
| - Multiple Cygwin installations can co-exist on a machine, as long
|  as you keep them separate.

https://cygwin.com/faq/faq.html#faq.using.multiple-copies
| you can install two or more separate Cygwin distros on a single machine
| ... a clean separation requires that you don't try to run executables of one
| Cygwin installation from processes running in another Cygwin installation

> But even if it's supported, I'm not sure
> what do we gain from having 2 Cygwin installations?

If you gain nothing, don't do it yourself.

In our office, we've been using 32-bit cygwin since the end of the last
century. These days, when a 32-bit problem is reported on the cygwin
mailing list, the virtually automatic reply is: upgrade to a 64-bit
cygwin version and see if that fixes the problem. They're still
supporting their 32-bit legacy system for now, but recommending that
everyone upgrade to 64-bit cygwin. We'd like to do that as well, before
they withdraw support for the 32-bit system we're using.

It makes sense for us to create a new 64-bit cygwin installation now.
Until we're sure it's working properly, we'll keep the current 32-bit
cygwin installation for production, because it's known to be stable.
That's why we need to have two separate cygwin installations at the
same time.

Going forward, we might want to have two separate 64-bit cygwin
installations for much the same reason that I have multiple chroots:
changes in an experimental environment cannot affect a stable
production environment. Alternatively, we might want two different
lmi "prefixes" in a single cygwin installation. We can't say today
which of these methods would suit us best for future needs. But
for our immediate need right now, we definitely want to add a
64-bit cygwin installation to our existing 32-bit one.

> It seems like an
> overkill if the goal is just to avoid the /MinGW_ conflict.

The motivation is to preserve a working development environment while
creating a new development environment:
 - upgrade to 64-bit cygwin
 - upgrade to gcc-8
 - upgrade to 64-bit lmi
Each of those three upgrades is worthwhile in itself.

> It could be
> advantageous from the point of view of being simply able to remove the 32
> bit Cygwin version after switching to 64 bit one, but OTOH continuing to
> have just a single version of Cygwin would seem to be even more
> advantageous because nothing would need to be removed in the first place
> because everybody could just continue using whichever version of Cygwin
> they already have. Or am I wrong in assuming that we can use 32 bit Cygwin
> for 64 bit builds as well? I admit I haven't tested it myself, but AFAICS
> it should work, shouldn't it?

You can build binaries for either word size in a cygwin installation
of either word size.

>  My main concern here is that even having 2 Cygwin installations is
> perfectly supported, it must still be much rarer than having one of them
> and so chances of having some weird problems (which are unfortunately not
> unheard of with Cygwin) with them are higher. And then there is also the
> issue of human confusion, i.e. it must be relatively easy to accidentally
> run 32-bit version instead of 64-bit one or vice versa, while having only a
> single version is much less error-prone.

The cygwin maintainers have multiple installations. The main reason they
changed cygwin to permit such multiplicity in 2009 was for their own
convenience. Any problem with multiple installations would be a major
issue for the maintainers themselves and would draw their focused effort.

There was a period when 64-bit cygwin was so new that it didn't have as
many packages available as the 32-bit version, so many users stayed with
the 32-bit version. But now the 64-bit system is the primary focus, and
there's no reason to use the 32-bit version; eventually it'll wither.

It does require discipline to maintain production and experimental
systems concurrently for a transition period. But no other option makes
sense:
 - stay in a legacy environment until it crumbles
 - throw away a stable environment and jump to an untested one

> GC> >  Moreover, even 32 bit build fails with 8.1.0, which is the version
> GC> > currently installed by install_mingw.make, because currently the 
> compiler
> GC> > is not installed in the correct directory and we end up with
> GC> > /MinGW_/mingw32/bin/gcc.exe instead of the expected /MinGW_/bin/gcc.exe.
> GC> 
> GC> What can have gone wrong?
> GC> 
> GC> $git log --oneline -- install_mingw.make
> GC> 9afb0922 Rename "scratch" directories
> GC> 74fe75e2 Upgrade to gcc-8.x
> GC> bcdd1c61 Terminate all files with a single (not double) newline
> GC> 53cea856 Update copyright notices
> GC> e3bbb946 Upgrade to gcc-7.3
> GC> 45cbda55 Experimentally use 'cp' where 'mv' fails
> GC> 
> GC> It certainly worked as of the last commit shown, 45cbda55; we know that
> GC> because that change was a required workaround for a cygwin problem.
> GC> 
> GC> The other commits since then seem innocent. Only the most recent one,
> GC> 9afb0922, looks like it could possibly be wrong, but I don't see
> GC> anything in it that could account for the observed anomaly.
> 
>  Me neither, but the copy command copies (recursively) "mingw32"
> subdirectory from /MinGW_/ad_hoc to /MinGW_ itself which seems just as
> incorrect now as in 45cbda55. In fact, I now wonder if the workaround of
> replacing "mv" with "cp" could be just due to using the wrong directory
> name in 8a7d87dbdf? I.e. my hypothesis is that this never worked since
> switching to MinGW-w64 toolchain in that commit and that the workaround
> didn't work neither and that the files just were copied/moved manually to
> fix the problem instead. I realize that this seems rather implausible, but
> I simply don't see anything else.

8a7d87db was 2016-11-01T21:56:31+00:00 Replace 'echo -e' with 'printf', for 
portability
45cbda55 was 2018-03-06T13:20:49+00:00 Experimentally use 'cp' where 'mv' fails

I started moving from cygwin VMs to wine chroots late in 2016,
but I was still using cygwin as late as 2017-08:

https://lists.nongnu.org/archive/html/lmi/2017-08/msg00019.html
| Sun, 20 Aug 2017 18:26:01 +0000
| [...] the experimental workaround I committed,
| which I'm testing now by rebooting my VM and running
|
|   $cd /tmp
|
|   $logfile=log-`date -u +'%Y%m%dT%H%MZ'`; echo "Log file is 
| '$logfile.tar.bz2'."; ./install_msw.sh >$logfile 2>&1; tar -cjf 
| $logfile.tar.bz2 $logfile

so I'm sure that I personally tested everything then, and it's just
not possible that multiple people used manual workarounds at that time
when the email record deliberately shows a build from scratch.

But I was no longer using cygwin as of a few months later:

  https://lists.nongnu.org/archive/html/lmi/2017-11/msg00019.html
| What are the exact consequences, in a world where:
| - Kim uses msw, and I do not

and then, at the time of commit 45cbda55:

  https://lists.nongnu.org/archive/html/lmi/2018-03/msg00017.html
| mv scratch/mingw32 /MinGW_
| mv: cannot move 'scratch/mingw32' to '/MinGW_': Permission denied
[...]
| This 'mv' command being all that stood between us and success,
| we tried brute force:
|   cp -a scratch/mingw32 /MinGW_
| and that worked. How can 'cp' succeed where 'mv' failed?

so that step was performed manually, but it "worked", so that
doesn't seem to explain what you're seeing now. I wonder if the
problem might lie here, where a '7-Zip' alternative was added:

26b625b5 2017-08-19T14:28:42+00:00 Upgrade to gcc-6.3.0

As stated above, I rebuilt everything from scratch one day later
using cygwin in a VM, but I can't be sure whether that rebuild
followed the 'bsdtar' path or the '7-Zip' alternative. Perhaps
they aren't equivalent.

Using the patch below [0] in my chroot today, everything seems
to work as expected--e.g.:

$ls -o /opt/lmi/MinGW_8/bin/i686-w64-mingw32-gcc-8.1.0.exe
-rwxr-xr-x 1 greg 1862656 May 12  2018 
/opt/lmi/MinGW_8/bin/i686-w64-mingw32-gcc-8.1.0.exe

I'm certainly using the 'bsdtar' path because my chroot has no
"c:/Program\ Files/7-Zip/7z" to fall back on.

I suspect that the anomaly stems from one of three causes: either
 - the '7-Zip' command doesn't do what 'bsdtar' does, and you're
   using the '7-Zip' command; or
 - cygwin 'bsdtar' doesn't do what debian 'bsdtar' does, which
   seems unthinkable; or
 - the 'cp' and 'mv' commands don't do the same thing.
I'm often surprised by the result of copying directories with 'cp':
sometimes the contents are copied, and sometimes the directory (with
its contents) is copied instead. I've never found a good explanation.
To test the third possibility, I reran with the second patch below [1],
and the results are the same; that seems to rule out that explanation.
Or does it? Might 'cp' or 'mv' do different things depending on whether
or not the destination directory already exists? Do you know a way to
write those commands that would avoid any ambiguity?

Can you say which command ('bsdtar' or '7-Zip') is actually performing
the extraction on your machine, and exactly what it does? Here's what
I see (this copy-paste block is probably uninteresting, but the one
that follows it could be extremely interesting, so feel free to skip
ahead to that part):

/opt/lmi/src/lmi[0]$rm -rf /opt/lmi/MinGW_8                                  
/opt/lmi/src/lmi[0]$rm -rf /tmp/mingw8-ad_hoc                                
/opt/lmi/src/lmi[0]$          
/opt/lmi/src/lmi[0]$ls /tmp/mingw8-ad_hoc 
ls: cannot access '/tmp/mingw8-ad_hoc': No such file or directory
/opt/lmi/src/lmi[2]$ls -d /tmp/
/tmp/
/opt/lmi/src/lmi[0]$ls -d /tmp/mingw8-ad_hoc
ls: cannot access '/tmp/mingw8-ad_hoc': No such file or directory
/opt/lmi/src/lmi[2]$ls /tmp/mingw8-ad_hoc   
ls: cannot access '/tmp/mingw8-ad_hoc': No such file or directory
/opt/lmi/src/lmi[2]$
/opt/lmi/src/lmi[2]$bsdtar --extract --keep-old-files 
--directory=/tmp/mingw8-ad_hoc 
--file=/cache_for_lmi/downloads/i686-8.1.0-release-win32-sjlj-rt_v6-rev0.7z
bsdtar: could not chdir to '/tmp/mingw8-ad_hoc'
/opt/lmi/src/lmi[1]$mkdir /tmp/mingw8-ad_hoc
/opt/lmi/src/lmi[0]$bsdtar --extract --keep-old-files 
--directory=/tmp/mingw8-ad_hoc 
--file=/cache_for_lmi/downloads/i686-8.1.0-release-win32-sjlj-rt_v6-rev0.7z
/opt/lmi/src/lmi[0]$ls /tmp/mingw8-ad_hoc/bin     
ls: cannot access '/tmp/mingw8-ad_hoc/bin': No such file or directory
/opt/lmi/src/lmi[2]$ls -d /tmp/mingw8-ad_hoc/mingw32/bin
/tmp/mingw8-ad_hoc/mingw32/bin

[COMEFROM "skip ahead to that part"]

And here's the thing about 'cp' that confuses me, which looks like
it might explain the anomaly you report:

/opt/lmi/src/lmi[0]$rm -rf /opt/lmi/does_not_exist
/opt/lmi/src/lmi[0]$mkdir /opt/lmi/exists
/opt/lmi/src/lmi[0]$
/opt/lmi/src/lmi[0]$cp --archive /tmp/mingw8-ad_hoc/mingw32 
/opt/lmi/does_not_exist 
/opt/lmi/src/lmi[0]$ls -o 
/opt/lmi/does_not_exist/bin/i686-w64-mingw32-gcc-8.1.0.exe 
-rwxr-xr-x 1 greg 1862656 May 12  2018 
/opt/lmi/does_not_exist/bin/i686-w64-mingw32-gcc-8.1.0.exe
/opt/lmi/src/lmi[0]$
/opt/lmi/src/lmi[0]$cp --archive /tmp/mingw8-ad_hoc/mingw32 /opt/lmi/exists     
    
/opt/lmi/src/lmi[0]$ls -o /opt/lmi/exists/bin/i686-w64-mingw32-gcc-8.1.0.exe
ls: cannot access '/opt/lmi/exists/bin/i686-w64-mingw32-gcc-8.1.0.exe': No such 
file or directory
/opt/lmi/src/lmi[2]$ls -o 
/opt/lmi/exists/mingw32/bin/i686-w64-mingw32-gcc-8.1.0.exe 
-rwxr-xr-x 1 greg 1862656 May 12  2018 
/opt/lmi/exists/mingw32/bin/i686-w64-mingw32-gcc-8.1.0.exe

Bonus question: what does
        $(MKDIR) --parents $(prefix)
        $(RM) --force --recursive $(prefix)
in that makefile do? I'm not sure, but I suppose mkdir is used
in an attempt to make the rm command safer. If this is a good
idiom, then how should we document it so that I can understand it?
Otherwise, what should we replace it with?

---------

[0] "Using the patch below"

My chroot of course doesn't have a cygwin mount table, which would map
'/MinGW_' to some physical directory in '/opt/lmi/' (to avoid writing
to the msw root directory). Therefore, I've changed 'prefix' directly
here. I also added '8' to make the directories unique, and manually
reverted the 'cp' replacement for the original 'mv' command.

---------8<--------8<--------8<--------8<--------8<--------8<--------8<-------
diff --git a/install_mingw.make b/install_mingw.make
index e37f9669..c75d2f51 100644
--- a/install_mingw.make
+++ b/install_mingw.make
@@ -42,11 +42,11 @@ file_list  = $($(version))
 #   http://article.gmane.org/gmane.comp.gnu.mingw.user/14748
 #     [2005-01-17T18:15:26Z from Aaron W. LaFramboise]
 
-prefix    := /MinGW_
+prefix    := /opt/lmi/MinGW_8
 
 cache_dir := /cache_for_lmi/downloads
 
-ad_hoc_dir := $(prefix)/ad_hoc
+ad_hoc_dir := /tmp/mingw8-ad_hoc
 
 # In the past, it seemed necessary to specify a mirror, e.g.:
 #  mirror := http://easynews.dl.sourceforge.net/sourceforge/mingw
@@ -107,8 +107,12 @@ ad_hoc_dir_exists = \
 
 .PHONY: all
 all: $(file_list)
-       $(CP) --archive $(ad_hoc_dir)/mingw32 $(prefix)
-       $(RM) --force --recursive $(ad_hoc_dir)
+#      $(CP) --archive $(ad_hoc_dir)/mingw32 $(prefix)
+#      $(RM) --force --recursive $(ad_hoc_dir)
+       $(MV) $(ad_hoc_dir)/mingw32 $(prefix)
+#      $(RMDIR) $(ad_hoc_dir)
+#      $(CP) --archive $(ad_hoc_dir)/mingw32 $(prefix)
+#      $(RM) --force --recursive $(ad_hoc_dir)
 
 $(file_list): initial_setup
 
--------->8-------->8-------->8-------->8-------->8-------->8-------->8-------

Here's the output produced by that modified makefile:

type "wget" >/dev/null || { printf '%b' "\nError: Unable to find 'wget', which 
is required for" "\nautomated downloads. Download it manually from the MinGW" 
"\nsite and install it on your PATH." "\n"      && false; }
[ ! -e /opt/lmi/MinGW_8     ]    || { printf '%b' "\nError: Prefix directory 
'/opt/lmi/MinGW_8' already exists." "\nIt is generally unsafe to install one 
version of a program" "\non top of another. Probably you ought to rename the 
old" "\nversion in order to preserve it; if not, then remove it." "\n"     && 
false; }
[ ! -e /tmp/mingw8-ad_hoc ]    || { printf '%b' "\nError: Nonce directory 
'/tmp/mingw8-ad_hoc' already exists." "\nProbably it is left over from a 
previous failure." "\nJust remove it unless you're sure you want whatever 
files" "\nit might contain." "\n" && false; }
mkdir --parents /opt/lmi/MinGW_8
rm --force --recursive /opt/lmi/MinGW_8
mkdir --parents /tmp/mingw8-ad_hoc
cd /cache_for_lmi/downloads && [ -e i686-8.1.0-release-win32-sjlj-rt_v6-rev0.7z 
] || wget  
http://downloads.sourceforge.net/mingw-w64/i686-8.1.0-release-win32-sjlj-rt_v6-rev0.7z
cd /cache_for_lmi/downloads && echo "28ec1e65ab85a9e1043998516045ab62 
*i686-8.1.0-release-win32-sjlj-rt_v6-rev0.7z" | md5sum --check
i686-8.1.0-release-win32-sjlj-rt_v6-rev0.7z: OK
bsdtar --extract --keep-old-files --directory=/tmp/mingw8-ad_hoc 
--file=/cache_for_lmi/downloads/i686-8.1.0-release-win32-sjlj-rt_v6-rev0.7z \
  || c:/Program\ Files/7-Zip/7z x `cygpath -w 
/cache_for_lmi/downloads/i686-8.1.0-release-win32-sjlj-rt_v6-rev0.7z` 
-o/tmp/mingw8-ad_hoc
mv /tmp/mingw8-ad_hoc/mingw32 /opt/lmi/MinGW_8

As expected, there's no '/mingw32/' component on the resulting path:

$ls -o /opt/lmi/MinGW_8/bin/i686-w64-mingw32-gcc-8.1.0.exe
-rwxr-xr-x 1 greg 1862656 May 12  2018 
/opt/lmi/MinGW_8/bin/i686-w64-mingw32-gcc-8.1.0.exe

[1] "I reran with the second patch below"

Same as HEAD, except with the same directory changes as [0] above.

---------8<--------8<--------8<--------8<--------8<--------8<--------8<-------
diff --git a/install_mingw.make b/install_mingw.make
index e37f9669..b3c2499c 100644
--- a/install_mingw.make
+++ b/install_mingw.make
@@ -42,11 +42,11 @@ file_list  = $($(version))
 #   http://article.gmane.org/gmane.comp.gnu.mingw.user/14748
 #     [2005-01-17T18:15:26Z from Aaron W. LaFramboise]
 
-prefix    := /MinGW_
+prefix    := /opt/lmi/MinGW_8
 
 cache_dir := /cache_for_lmi/downloads
 
-ad_hoc_dir := $(prefix)/ad_hoc
+ad_hoc_dir := /tmp/mingw8-ad_hoc
 
 # In the past, it seemed necessary to specify a mirror, e.g.:
 #  mirror := http://easynews.dl.sourceforge.net/sourceforge/mingw
--------->8-------->8-------->8-------->8-------->8-------->8-------->8-------

Then I removed the two applicable directories and ran the same
commands as in [0] above, with the same outcome on the last line:

/opt/lmi/src/lmi[0]$rm -rf /opt/lmi/MinGW_8 
/opt/lmi/src/lmi[0]$rm -rf /tmp/mingw8-ad_hoc 
/opt/lmi/src/lmi[0]$make -f install_mingw.make                               
type "wget" >/dev/null || { printf '%b' "\nError: Unable to find 'wget', which 
is required for" "\nautomated downloads. Download it manually from the MinGW" 
"\nsite and install it on your PATH." "\n"      && false; }
[ ! -e /opt/lmi/MinGW_8     ]    || { printf '%b' "\nError: Prefix directory 
'/opt/lmi/MinGW_8' already exists." "\nIt is generally unsafe to install one 
version of a program" "\non top of another. Probably you ought to rename the 
old" "\nversion in order to preserve it; if not, then remove it." "\n"     && 
false; }
[ ! -e /tmp/mingw8-ad_hoc ]    || { printf '%b' "\nError: Nonce directory 
'/tmp/mingw8-ad_hoc' already exists." "\nProbably it is left over from a 
previous failure." "\nJust remove it unless you're sure you want whatever 
files" "\nit might contain." "\n" && false; }
mkdir --parents /opt/lmi/MinGW_8
rm --force --recursive /opt/lmi/MinGW_8
mkdir --parents /tmp/mingw8-ad_hoc
cd /cache_for_lmi/downloads && [ -e i686-8.1.0-release-win32-sjlj-rt_v6-rev0.7z 
] || wget  
http://downloads.sourceforge.net/mingw-w64/i686-8.1.0-release-win32-sjlj-rt_v6-rev0.7z
cd /cache_for_lmi/downloads && echo "28ec1e65ab85a9e1043998516045ab62 
*i686-8.1.0-release-win32-sjlj-rt_v6-rev0.7z" | md5sum --check
i686-8.1.0-release-win32-sjlj-rt_v6-rev0.7z: OK
bsdtar --extract --keep-old-files --directory=/tmp/mingw8-ad_hoc 
--file=/cache_for_lmi/downloads/i686-8.1.0-release-win32-sjlj-rt_v6-rev0.7z \
  || c:/Program\ Files/7-Zip/7z x `cygpath -w 
/cache_for_lmi/downloads/i686-8.1.0-release-win32-sjlj-rt_v6-rev0.7z` 
-o/tmp/mingw8-ad_hoc
cp --archive /tmp/mingw8-ad_hoc/mingw32 /opt/lmi/MinGW_8
rm --force --recursive /tmp/mingw8-ad_hoc
/opt/lmi/src/lmi[0]$
/opt/lmi/src/lmi[0]$ls -o /opt/lmi/MinGW_8/bin/i686-w64-mingw32-gcc-8.1.0.exe 
-rwxr-xr-x 1 greg 1862656 May 12  2018 
/opt/lmi/MinGW_8/bin/i686-w64-mingw32-gcc-8.1.0.exe