Re: [lmi] Directory structure for multiple-architecture builds

[Greg Chicares]

On 2019-04-02 02:22, Vadim Zeitlin wrote:
> On Mon, 1 Apr 2019 18:47:49 +0000 Greg Chicares <address@hidden> wrote:
[...]
> GC>   http://git.savannah.nongnu.org/cgit/lmi.git/commit/?h=odd/multiarch
[...]
> - I actually don't remember why had I written that I preferred using
>   symlinks before, but now I have serious doubts about this. As long as
>   we have this PERFORM variable anyhow, what prevents us from determining
>   the right PATH/WINEPATH combination once and putting it inside this
>   variable?

$PERFORM is either 'wine' or '', as in
  "$PERFORM" lmi_wx_shared.exe
Perhaps you mean to write $LMI_HOST, which is an autotools '--host'
string? Or perhaps not...

[I've made it extremely confusing by using $LMI_HOST in some places
as though it were a '--build' string, e.g., where it's (ab)used to
set $PERFORM:

  case "$LMI_HOST" in
    ("i686-w64-mingw32" | "x86_64-w64-mingw32")
        PERFORM=wine
        ;;
    (*)
        PERFORM=
        ;;
  esac

which literally says: if we're compiling an i686 or x86_64 binary,
then the build platform must be GNU/Linux, so we must use 'wine';
otherwise, the build platform is cygwin, so we must not use 'wine'.
I'll have to add something like LMI_CROSS_MSW for that purpose.]

Indeed the modified build system in branch 'odd/multiarch uses
$LMI_HOST to determine the '--host' architecture, e.g., in
'install_msw.sh':
  for LMI_HOST in i686-w64-mingw32 x86_64-w64-mingw32
which builds lmi and its dependent libraries for each arch.

However, just setting a single environment variable cannot by itself
do everything we might want: $PATH and $WINEPATH must be adapted
(as well as architecture-dependent symlinks, if we choose to use any).
Unless...unless we create a highly magical string into which all
those things can be subsumed, e.g.:
  MAGIC="PATH=/some/paths WINEPATH=whatever HOST=gnu_host_string 
PERFORM=whatever"
and then write "$MAGIC" before every command that needs it, which
could include many that don't require "$PERFORM" today. For example,
if we further expand MAGIC:
  MAGIC_SQUARED="$MAGIC prefix=/opt/lmi CC=gcc CXX=g++ ..."
then we might be able to move much logic out of makefiles and shell
scripts, into a single environment variable. But at what point does
that become starkly worse than what it replaced?

Or would adding {$HOST, $PATH, $WINEPATH} to the existing $PERFORM
be enough to accomplish our purposes? We'd have to take care to
write it in front of commands we enter manually such as:
-  wine ./lmi_wx_shared --ash_nazg --data_path=/opt/lmi/data
+  $PERFORM ./lmi_wx_shared --ash_nazg --data_path=/opt/lmi/data
but that's not so terrible, as I already must remember to write
"wine" there. And maybe that could be replaced by some hypothetical
script whose name is easier to type, such as 'do':
  do ./lmi_wx_shared --ash_nazg --data_path=/opt/lmi/data
and thus [obligatory Molière quote elided] it's something I've
already been doing without realizing it (though everyone else would
have to start speaking prose, too).

Whether it's $DO in the environment or a 'do' script (reachable on
an invariant part of $PATH), the contents are the same, as are the
circumstances in which it's necessary, so the choice would come
down to convenience--if that's the method we choose.

But first let's establish the goal, as to which I think we agree:

>   [...] It looks like this ought to make everything work and we
>   wouldn't need to update the symlinks. In particular, I'd really like to
>   have both 32- and 64-bit versions on the same machine in parallel, which
>   is not really compatible with needing /opt/lmi/bin symlink to point to
>   just one of them.

Yes, that's exactly my goal: to have
  {32,64}-bit X {linux, msw} X {gcc, clang} X ...
builds coexisting simultaneously; and to switch among them easily
and reliably; and repeatedly, and often. Here are some use cases:

- Once we've got i386 and x86_64 builds working, we'll want to
switch back and forth (e.g., for testing) many times a day, or
even many times an hour.

- Even when we've dropped i386, we'll often want to test a new
compiler version against an old one.

- Similarly, we could test different versions of a library such
as wx. We've done such testing in the past by archiving an old
version in a special subdirectory, and building with a new version
in our main lmi tree; but if we want to make source changes, then
rebuilding the archived version can be bothersome.

- If I get a puzzling diagnostic from gcc, I might want to try
compiling with clang. Or maybe someday I'll compile with clang
first, for development, then with gcc, for distribution.

I'm confident that Kim has other use cases, which I'm just not
knowledgeable enough to describe in detail; and probably others
will arise that we haven't yet imagined.

And here are some things that aren't quite use cases, but maybe
"desiderata":

- We shouldn't need to spell out an architecture whenever we
  run a command. I.e., our custom is to navigate thus
    cd /opt/lmi/bin
  and then run commands such as
    wine ./lmi_wx_shared
  but it would be less convenient to have to navigate thus
    cd /opt/lmi/x86_64-w64-mingw32/bin
  or alternatively to enter commands such as
    wine /opt/lmi/bin/x86_64-w64-mingw32/lmi_wx_shared

- We must retain the ability to work with multiple terminals.
  For instance, I just run
    konsole --tabs-from-file ~/konsole_tabs &
  to get five predefined terminal tabs:
    vim -p /opt/lmi/src/lmi/tabs/**/startup*
  and I want to do
    /opt/lmi/src/lmi[0]$make install
  in the "Build" tab to build with my latest code change, then
    /opt/lmi/bin[0]$wine ./lmi_wx_shared
  in the "Run" tab to run lmi with that code change. That is,
  I really want all these tabs to use the same architecture.

What we have today is a set of variables in the environment:

 - $PERFORM for running 'wine' wherever appropriate
 - $LMI_HOST to select an architecture
     (we'll stop using it to set $PERFORM because that's nasty)
 - $PATH to find libraries, both when building and running lmi
 - $WINEPATH to find libraries, similarly, under 'wine'

and what we want is a method to choose one at a time among many
configurations--one that supports frequent switching robustly.
Candidates:

(1) [not recommended] Set all of the variables above, compatibly,
all the time, by hand. This is fragilissimo: after trying it,
I thought symlinks were much easier.

(2) One environment variable (benign version): $PERFORM, caffeinated.
  MAGIC="PATH=/some/paths WINEPATH=whatever HOST=gnu_host_string 
PERFORM=whatever"
I already use 'wine' wherever necessary on the command line,
and makefiles use $PERFORM wherever necessary to support my
work in a chroot; everyone else would have to use it, too
(so it imposes a new inconvenience on everyone but me).
But this alone wouldn't DTRT for
  make install
or other such commands.

(3) One environment variable to bring them all and in the
darkness bind them (maleficent version): $PERFORM on crack.
We'd have to type it before 'make', e.g.:
  "$INCANTATION" make install
but it would be robust...unless we forget to use it, and type
  make install
which could break everything. And the 'do' script hypothesized
above, on crack, would be just as evil.

(4) A different sort of script--call it 'set_architecture.sh',
for instance. It would set all appropriate environment variables
consistently (so we'd have to 'source' it rather than run it).
We'd run it each time the architecture is to be switched. This
is a disciplined version of (1), without the inconvenience of
(2) or (3).

(5) [branch 'odd/multiarch'] Symlinks:
  ln -sfn /opt/lmi/"$HOST_ARCHITECTURE"/local /opt/lmi/local
  ln -sfn /opt/lmi/"$HOST_ARCHITECTURE"/bin /opt/lmi/bin
This has the unique advantage that it works across terminals.
It also seems to be the only one so far that supports this
convenient usage:
  cd /opt/lmi/bin
  [wine] ./lmi_wx_shared
(without changing $PATH, it makes $PATH point elsewhere).

(6) Let's step back and look at this from another angle. What
exactly do we have in /opt/lmi/bin/ that we want to run as ./foo
when that's our current working directory? `ls /opt/lmi/bin` finds:

DLLs not directly invokable:
  skeleton.dll, wx_new.dll, libantediluvian.dll, liblmi.dll

never run at the command line:
  antediluvian_cgi.exe, antediluvian_cli.exe
  bcc_ar.exe, bcc_cc.exe, bcc_ld.exe, bcc_rc.exe
  elapsed_time.exe

rarely run at the command line:
  generate_passkey.exe
  ihs_crc_comp.exe
  product_files.exe

these are the only ones we really care about:
  lmi_cli_shared.exe
  lmi_wx_shared.exe
  rate_table_tool.exe
  wx_test.exe

...so there are only four special cases, all similar; can they be
treated otherwise? e.g., via four nearly-identical shell scripts?
which could even subsume variations across architecture etc.:

  ./lmi_cli_shared.sh i686
  # argument matches 'case "$1" in ("i686*")'
  # script sets $PATH etc.

  ./lmi_wx_shared.sh 64 safestdlib
  # argument 1: maybe 'case...("64")' to avoid typing "x86_64"
  # argument 2: $(build_type) defined in 'workhorse.make'

And these scripts would reside in /opt/lmi/src/lmi , not in
/opt/lmi/bin : we wouldn't need that bin/ subdirectory any more;
that would take some getting used to, but might be liberating.

(7) As an alternative to (6), we could write a 'switch_arch.sh'
script that just establishes a symlink for each of those binaries.
I'm not sure that's much different from (5), unless file symlinks
are generally preferable to directory symlinks for some reason I'm
not aware of. But this seems to be a popular approach for 'install'
targets in autotoolized libraries. And, like (5), it transparently
affects already-running terminals, which from my POV is good.

IOW, even if environment variables seem simple and friendly, and
symlinks seem otherwise, one real advantage of (5) and (7) is that
the choice of architecture is expressed in the filesystem shared
by all terminals, rather than in their independent environments.

[...turning to a different subject...]

> GC> IOW, perhaps I would have gotten a message like
> GC>   i686-whatever-lmi.exe cannot find i686-whatever-xml2.dll
> GC> instead of
> GC>   whatever-lmi cannot find xml2.dll
> GC> when there's obviously an xml2.dll on $PATH, or on $WINEPATH, or
> GC> even in one directory reachable through $PATH and another reachable
> GC> through $WINEPATH.
> 
>  I sympathize as I, of course, have made the same blunder too in the past
> (and more times than I care to admit). But IME it's actually easier to
> debug this with Wine than with native MSW, as its debug messages clearly
> indicate the problem. So in some sense you were lucky.

I agree that 'wine' diagnostics indicate the problem, but IMO their
clarity rivals g++'s. Here's a problem I just resolved. I had two
terminals, both "chrooted to the same chroot" if that expression
is sensible (the first command I ran after opening each one was
'schroot' with identical parameters). In one of them, I could run
a 64-bit msw build of lmi successfully, but not in the other. Here
are the crucial parameters of the one that worked:

/opt/lmi/src/lmi[0]$readlink /opt/lmi/bin                                       
           
/opt/lmi/x86_64-w64-mingw32/bin
/opt/lmi/src/lmi[0]$readlink /opt/lmi/local                                     
           
/opt/lmi/x86_64-w64-mingw32/local
/opt/lmi/src/lmi[0]$echo $PATH
/opt/lmi/x86_64-w64-mingw32/local/bin:/opt/lmi/x86_64-w64-mingw32/local/lib:/usr/local/bin:/usr/bin:/bin
/opt/lmi/src/lmi[0]$echo $WINEPATH
Z:\opt\lmi\local\bin;Z:\opt\lmi\local\lib;Z:\opt\lmi\third_party\bin

The other had exactly the same parameters except for this one:

/opt/lmi/bin[0]$echo $WINEPATH 
Z:\opt\lmi\x86_64-w64-mingw32\local\bin;Z:\opt\lmi\x86_64-w64-mingw32\local\lib:Z:\opt\lmi\third_party\bin

but here's what happened when I tried to start lmi (with exactly
the same command that succeeded in the other terminal):

/opt/lmi/bin[0]$wine ./lmi_wx_shared --ash_nazg --data_path=/opt/lmi/data
002a:err:module:import_dll Library 
wxmsw312u_gcc_x86_64-w64-mingw32-gcc-8.2-win32-f741031e69de73d5816cc56e99c9beba3ac820de.dll
 (which is needed by L"Z:\\opt\\lmi\\bin\\lmi_wx_shared.exe") not found
002a:err:module:import_dll Library 
wxmsw312u_gcc_x86_64-w64-mingw32-gcc-8.2-win32-f741031e69de73d5816cc56e99c9beba3ac820de.dll
 (which is needed by L"Z:\\opt\\lmi\\bin\\skeleton.dll") not found
002a:err:module:import_dll Library 
wxmsw312u_gcc_x86_64-w64-mingw32-gcc-8.2-win32-f741031e69de73d5816cc56e99c9beba3ac820de.dll
 (which is needed by 
L"Z:\\opt\\lmi\\x86_64-w64-mingw32\\local\\bin\\libwxcode_mswu_pdfdoc-3.1-0.dll")
 not found
002a:err:module:import_dll Library libwxcode_mswu_pdfdoc-3.1-0.dll (which is 
needed by L"Z:\\opt\\lmi\\bin\\skeleton.dll") not found
002a:err:module:import_dll Library skeleton.dll (which is needed by 
L"Z:\\opt\\lmi\\bin\\lmi_wx_shared.exe") not found
002a:err:module:attach_dlls Importing dlls for 
L"Z:\\opt\\lmi\\bin\\lmi_wx_shared.exe" failed, status c0000135

Trying to use those diagnostics to fix the problem, I search for the
crucial missing DLL:

/opt/lmi/src/lmi[0]$ls /opt/lmi/**/wxmsw312u_gcc_x86_64*
/opt/lmi/x86_64-w64-mingw32/local/lib/wxmsw312u_gcc_x86_64-w64-mingw32-gcc-8.2-win32-f741031e69de73d5816cc56e99c9beba3ac820de.dll
/opt/lmi/x86_64-w64-mingw32/wx-scratch/lmi-gcc-8.2-win32/lib/wxmsw312u_gcc_x86_64-w64-mingw32-gcc-8.2-win32-f741031e69de73d5816cc56e99c9beba3ac820de.dll

Ignoring the second one (it's in a 'wx-scratch' directory that's
never used after wx installed from it), I reason that this directory:
  /opt/lmi/x86_64-w64-mingw32/local/lib/
should probably be on $PATH, as it is:
  
/opt/lmi/x86_64-w64-mingw32/local/bin:/opt/lmi/x86_64-w64-mingw32/local/lib:/usr/local/bin:/usr/bin:/bin
  required: ------------------------->  /opt/lmi/x86_64-w64-mingw32/local/lib/ 
<---
and also on $WINEPATH...yet it already is:
  
Z:\opt\lmi\x86_64-w64-mingw32\local\bin;Z:\opt\lmi\x86_64-w64-mingw32\local\lib:Z:\opt\lmi\third_party\bin
  required: ----------------------------->  
/opt/lmi/x86_64-w64-mingw32/local/lib/ <---

But $WINEPATH is tricky. Somewhere, sometime, I had formed the impression
that 'wine' wouldn't read through a GNU/Linux symlink; but in this case
it seems as though $WINEPATH requires the symlink and can't use the full,
canonical filepath. OTOH, it's not unlikely that I'm missing something.