Re: [lmi] Directory structure for multiple-architecture builds

[Greg Chicares]

On 2019-04-05 03:08, Vadim Zeitlin wrote:
> On Thu, 4 Apr 2019 23:47:05 +0000 Greg Chicares <address@hidden> wrote:
> 
> GC> On 2019-04-02 02:22, Vadim Zeitlin wrote:
> GC> > On Mon, 1 Apr 2019 18:47:49 +0000 Greg Chicares <address@hidden> wrote:
> GC> [...]
> GC> > GC>   
> http://git.savannah.nongnu.org/cgit/lmi.git/commit/?h=odd/multiarch
> GC> [...]
> GC> > - I actually don't remember why had I written that I preferred using
> GC> >   symlinks before, but now I have serious doubts about this. As long as
> GC> >   we have this PERFORM variable anyhow, what prevents us from 
> determining
> GC> >   the right PATH/WINEPATH combination once and putting it inside this
> GC> >   variable?
> GC> 
> GC> $PERFORM is either 'wine' or '', as in
> GC>   "$PERFORM" lmi_wx_shared.exe
> 
>  Yes, so if we set it to "PATH=... WINEPATH=..." it should work, shouldn't
> it? BTW, just to be clear, I was only speaking about using this variable in
> the makefiles, not interactively.

In the 'odd/multiarch' branch, I've retained $PERFORM in its original
  {"wine", ""}
meaning (which might no longer be necessary), but added a 'set_arch.sh'
script, documented thus:
  # Set $PATH, $WINEPATH, and $PERFORM based on $LMI_HOST.

> GC> And maybe that could be replaced by some hypothetical
> GC> script whose name is easier to type, such as 'do':
> GC>   do ./lmi_wx_shared --ash_nazg --data_path=/opt/lmi/data
[...]
>  Indeed, many other people already do do something like this.

Actually, I've committed two scripts: 'set_arch.sh', and 'run.sh'
(akin to the hypothetical 'do' above); and, having done that, I've
found no use for 'run.sh', which is just:
  . ./set_arch.sh
  exec "$@"
Rather than prefixing every command with 'run.sh', it seems more
convenient to ensure that 'set_arch.sh' has been sourced into the
main shell's environment (e.g., in ~/.zshrc, and also in the
'install_msw.sh' do-everything-from-scratch-for-msw script),
switching context whenever desired thus:
  LMI_HOST=i686-w64-mingw32 ; . ./set_arch.sh
  LMI_HOST=x86_64-w64-mingw32 ; . ./set_arch.sh

> GC> Yes, that's exactly my goal: to have
> GC>   {32,64}-bit X {linux, msw} X {gcc, clang} X ...
> GC> builds coexisting simultaneously;
> 
>  Perfect; we are in absolute agreement then.
> 
> GC> And here are some things that aren't quite use cases, but maybe
> GC> "desiderata":

Maybe some of the things I listed aren't so desirable after all...

> GC> - We shouldn't need to spell out an architecture whenever we
> GC>   run a command. I.e., our custom is to navigate thus
> GC>     cd /opt/lmi/bin
> GC>   and then run commands such as
> GC>     wine ./lmi_wx_shared
> GC>   but it would be less convenient to have to navigate thus
> GC>     cd /opt/lmi/x86_64-w64-mingw32/bin
> GC>   or alternatively to enter commands such as
> GC>     wine /opt/lmi/bin/x86_64-w64-mingw32/lmi_wx_shared
> 
>  Sorry if I'm just stating the obvious, but there is no escaping the fact
> that you will need to specify one of N builds of lmi to use. It can be done
> either globally, i.e. by setting PATH global variable (likely by sourcing
> some shell script taking care of doing it and everything else properly), or
> for each command, by specifying the path, or at least a subset of the path,
> in it.

It's been my general practice to run lmi binaries by specifying explicit
pathnames, and to avoid relying much on environment variables such as $PATH.
Thus, to run lmi, I've always either navigated to a bin/ directory and typed
  wine ./lmi_wx_shared ...
or, from any other directory,
  wine /opt/lmi/bin/lmi_wx_shared ...
so it's against old habit to type something like
  wine /opt/lmi/i686-w64-mingw32/bin/lmi_wx_shared
or
  cd /opt/lmi/i686-w64-mingw32/bin/
instead. However:

 - I didn't type 'wine' at the beginning, either, until a few years ago when
I moved all my development to GNU/Linux exclusively--so it's not impossible
to acquire new habits; and
  wine /opt/lmi/i <TAB>
autocompletes nicely.

 - Actually, I don't often type commands--copying and pasting them from text
files is easier. And globally inserting "$LMI_HOST/' into a few dozen such
commands is easy--it's already done in this branch's 'gwc/develop*.txt'.
Once I'd done that, I didn't really wish for anything else.

 - The change can be made invisible with a few symlinks. You had [below]
urged that symlinks be at most optional, and I've come around to the same
viewpoint--I'm not sure I'd use them at all now. But I've been switching
between architectures often, whereas you or, say, Kim might switch rarely.
Commands to create symlinks could easily be added to 'set_arch.sh',
conditioned on some $USE_SYMLINKS option; the advantage is that old
commands would continue to work transparently for a single architecture.

 - Scripts could achieve much the same thing as symlinks (i.e., (6) below).

> GC> - We must retain the ability to work with multiple terminals.
> GC>   For instance, I just run
> GC>     konsole --tabs-from-file ~/konsole_tabs &
> GC>   to get five predefined terminal tabs:
> GC>     vim -p /opt/lmi/src/lmi/tabs/**/startup*
> GC>   and I want to do
> GC>     /opt/lmi/src/lmi[0]$make install
> GC>   in the "Build" tab to build with my latest code change, then
> GC>     /opt/lmi/bin[0]$wine ./lmi_wx_shared
> GC>   in the "Run" tab to run lmi with that code change. That is,
> GC>   I really want all these tabs to use the same architecture.
> 
>  I'd say that this is a counter goal

Yes. I had to get some experience with different methods before I could
see this clearly.

> GC> What we have today is a set of variables in the environment:
> GC> 
> GC>  - $PERFORM for running 'wine' wherever appropriate
> GC>  - $LMI_HOST to select an architecture
> GC>      (we'll stop using it to set $PERFORM because that's nasty)
> GC>  - $PATH to find libraries, both when building and running lmi
> GC>  - $WINEPATH to find libraries, similarly, under 'wine'
> GC> 
> GC> and what we want is a method to choose one at a time among many
> GC> configurations--one that supports frequent switching robustly.
> GC> Candidates:
> GC> 
> GC> (1) [not recommended] Set all of the variables above, compatibly,
> GC> all the time, by hand.
[...]
> GC> (2) One environment variable [...something like...]
> GC>   MAGIC="PATH=/some/paths WINEPATH=whatever HOST=gnu_host_string 
> PERFORM=whatever"
[...]
> GC> (3) One environment variable [...a different variation...]
[...]
> GC> (4) A different sort of script--call it 'set_architecture.sh',
> GC> for instance. It would set all appropriate environment variables
> GC> consistently (so we'd have to 'source' it rather than run it).
> GC> We'd run it each time the architecture is to be switched. This
> GC> is a disciplined version of (1), without the inconvenience of
> GC> (2) or (3).
> 
>  I think this is the best approach for interactive use.

Yes--certainly, for anyone who switches architectures often.

> I am not sure if
> this should be used in the makefiles. At the very least, running "make
> HOST=xxx" should complain if "xxx" is different from the host set in the
> environment.

Since I began switching architecture via these commands only:
  LMI_HOST=i686-w64-mingw32 ; . ./set_arch.sh
  LMI_HOST=x86_64-w64-mingw32 ; . ./set_arch.sh
I haven't had any problems. I.e., if none of those environment variables
is ever set except by this method, then inconsistency cannot arise.

I'm not sure we need to use 'set_arch.sh' in any makefile, as it is
set by 'install_msw.sh' (and 'gwc/.zshrc').

> GC> (5) [branch 'odd/multiarch'] Symlinks:
[...]
>  Of course it shouldn't be forbidden to create such symlinks if you'd like
> to. But I just hope it won't be required to create them and that some way
> of running lmi without them would also be available.

Agreed.

> GC> (6) Let's step back and look at this from another angle. What
> GC> exactly do we have in /opt/lmi/bin/ that we want to run as ./foo
> GC> when that's our current working directory?
[...maybe half a dozen programs at most...]
[...we could provide a 'cover' script for each...]
> GC> which could even subsume variations across architecture etc.:
> GC> 
> GC>   ./lmi_cli_shared.sh i686
> 
>  Is this really better than
> 
>       $ lmi_run lmi_cli_shared i686
> 
> ? The advantage of a single lmi_run script is that there is just one of it,
> and not 4 (or, tomorrow, 5...).

Okay, 'run.sh' (already in this branch) or something like it seems
preferable to a collection of several distinct scripts.

I tend to think, though, that anyone who works primarily with
whatever architecture we're using in production at the time would
be better served by setting symlinks, because they're "sticky":
you choose an architecture once, and it applies thereafter until
you explicitly change it.

> GC> (7) As an alternative to (6), we could write a 'switch_arch.sh'
> GC> script that just establishes a symlink for each of those binaries.
> GC> I'm not sure that's much different from (5), unless file symlinks
> GC> are generally preferable to directory symlinks for some reason I'm
> GC> not aware of. But this seems to be a popular approach for 'install'
> GC> targets in autotoolized libraries. And, like (5), it transparently
> GC> affects already-running terminals, which from my POV is good.
> 
>  From my POV it's bad and I indeed don't see any advantage to this approach
> compared to (5).

Agreed, but let me emphasize that this (7), is really (5) implemented
in terms of symlinks--so, even though I suggested a similar script name
above, it's radically different from 'set_arch.sh' in the branch, which
only sets environment variables.

>  To summarize, my preference for a "global" solution would be to have a
> "lmi_setenv" script that would be sourced to set up everything correctly

I believe that describes 'set_arch.sh'.

> (after possibly checking for some common problems?)

Given the description of 'set_arch.sh':
  # Set $PATH, $WINEPATH, and $PERFORM based on $LMI_HOST.
if it's always used by cutting and pasting (or recalling) a command like
  LMI_HOST=x86_64-w64-mingw32 ; . ./set_arch.sh
then what problem can arise? The only one I see is specifying an invalid
triplet, which could be handled by

  case "$LMI_HOST" in
    (...known triplets) ;;
    (*) print "Error..."; exit 9;

I've found it so unpleasant to set $WINEPATH at the command line that
I'm pretty sure I'll never try it again--so it doesn't take a lot of
self-discipline to avoid switching architecture manually.

> and for a "local", i.e.
> per-command, solution it would be to have a "lmi_run" helper script. But,
> of course, I'm not at all representative of the target demographic here,
> and will find some way to do what I need to do anyhow, so please feel
> completely free to ignore my preferences.

Our monthly releases for msw users copy all required libraries (and even
data files) into a single directory, and provide a single "shortcut" with
the full path to our GUI binary. No $PATH is required (and the corporation
probably wouldn't let us set one anyway), and I have no ambition to try
symlinks under a variety of msw versions.

So you're either one-third of the target demographic, or three-fifths if
we count all the TT-Solutions developers--and you're the most experienced
one, so your recommendations are more than mere personal preferences. Kim
and I might want two different solutions anyway, and I don't see any
problem accommodating whichever you like best, given that you don't favor
the options above that would clutter the repository with lots of redundant
scripts.

BTW, could you please take a look at the way $build_dir is set in the
'odd/multiarch' branch's 'GNUmakefile' and let me know your thoughts?
We've always had a single 'build/' subdirectory, so that deleting it
(recursively) was a good implementation of 'make clobber'. Here, I did
something different--keeping 'build/' under each architecture--but I'm
having second thoughts about that. My motivation was that writing
something like
  exec_prefix=$prefix/$LMI_HOST
  build_dir=$exec_prefix/../build/$LMI_HOST
or, better,
  build_dir=$prefix/build/$LMI_HOST
to insert the string "build/" in the middle of exec_prefix seemed ugly.
But now it seems that what I did instead required much greater ugliness
in order to write a 'clobber:' target that is less than satisfactory.
And I'm already synthesizing a similar hierarchy
  logdir=/tmp/lmi/"$LMI_HOST"/logs
in 'nychthemeral_test.sh' anyway, directly rather than through some
less robust technique like
  ${prefix##opt}tmp
so maybe
  build_dir=$prefix/build/$LMI_HOST
really isn't objectionable after all.