Re: path separator (was: target triplet)

[Keith MARSHALL]

Bob Rossi wrote, quoting Ralf Wildenhues, quoting me:
>>> http://article.gmane.org/gmane.comp.gnu.mingw.msys/3822
>>
>> Thanks for the pointer, I was not fully aware of the limitation that
>> $PATH_SEPARATOR is useful for $build only, not for $host.  OK to apply
>> the patch at the end to document this?
>
> I have a question in regards to this. I'm building with mingw. Now, I 
> can run my application in msys or in the cmd window. I was assumming
> that if my application is running in msys that I would want to use the :
> separator and if it was running in the cmd window I'd want to use the ;
> separator. Is this incorrect?

Hi Bob,

You had to ask :)  With advance apologies for the long dissertation,
hopefully the following addresses your concerns.

You've got to consider what your app is going to see, after it receives
control from `exec'.  Let's consider three cases; the native cmd.exe case
and the MSYS sh.exe cases, which you mention, and also the Cygwin bash.exe
case, which has some associated relevance.

First, the cmd.exe case.

cmd.exe is a really dumb shell; it gives your app an absolute minimum of
assistance, to get it running; this is restricted to:

1) Create a *verbatim* copy of the environment, to pass to your process.

2) Allocate process memory, and load the initial process image from your
   app.exe file.

3) Copy your command line, *verbatim* and unparsed, (except for the 
initial
   identification of the app.exe file name), into the process memory 
space,
   at a predefined address.

4) Set the CPU's instruction pointer to the entry point of your program,
   (at which point, your app begins running).

Note that this *doesn't* include any of the argc/argv setup that you would
normally expect from any Unixy shell; your app has to do this for itself,
but you don't normally have to worry about it, because appropriate setup
code is supplied by the compiler's runtime library, in the form of init
code to run before your `main' function gets control.

Note also that your app is using the *native* Win32 API for file system
access; thus all file name references and path specifications supplied to
it, either on the command line or in the environment, *must* be in native
format, i.e. with appropriate leading `d:' drive specifiers, (or UNC specs
for network paths), and with `;' as the `PATH_SEPARATOR'.  A commonly held
*misconception* is that you also need to use '\\' instead of '/' as the
`DIR_SEPARATOR'.  This is *absolutely* *not* the case; you can safely
ignore anyone who tries to tell you otherwise.  The *only* instance in
which you *do* need to use '\\' as the `DIR_SEPARATOR', is in the argument
string to a spawned, (or execed), brain-dead subprocess which parses it's
arguments according to CP/M conventions; (do note that most standard Win32
commands, i.e. those supplied with the OS, including cmd.exe itself, fall
into this category).

Ok, let's leave the MSYS sh.exe case for now, (we'll come back to it 
later),
and briefly consider the Cygwin bash.exe case.

With Cygwin, you are running a POSIX emulation layer on top of the Win32 
OS,
so your app gets all of the initialisation support you'd expect from a 
true
POSIX system.  There is no need for us to concern ourselves with the 
details
of how this happens; some very clever people have set it up for us, so 
that
it just works.  The aspects of it that are relevant to this discussion 
are:

1) The environment that is passed to your app is again a *verbatim* copy 
of
   that which was defined in the bash shell.

2) When you access command line arguments in argv, they are again 
*verbatim*
   copies of what you typed on the command line.

So, what's the difference from the cmd.exe case?  The *significant* 
difference
is that when your app refers to any file system entity, the reference no 
longer
needs to be specified in native Win32 format; it can just as well be in 
Cygwin's
POSIX compatible format, and the emulation layer will take care of mapping 
it to
the proper file system entity, without you needing to even know or care 
how.

So finally, to get back to that MSYS sh.exe case.

This falls somewhere between the cmd.exe and the Cygwin bash.exe cases. In
fact, MSYS is a specialised modification of an early version of Cygwin 
itself,
so when you are running MSYS sh.exe, you are, technically sitting on top 
of a
limited Cygwin emulation layer.  The important distinction lies in how the 
two
shells, Cygwin's bash.exe and MSYS sh.exe, exec processes:

1) Cygwin's bash.exe assumes that the execed process will know how to use 
the
   Cygwin POSIX emulation layer to map Cygwin specific entity references 
to
   their native counterparts; if this isn't the case, as in the case of a 
Win32
   native app, then the onus falls squarely on the user to ensure that all
   references, and file system references, are specified in the proper 
*native*
   format; there is no assistance provided to perform the translation.

2) Conversely, MSYS sh.exe accommodates the presumption that the majority 
of
   execed processes will be *native* Win32 processes, (as is the case of 
apps
   you build with MinGW, or with any other native Win32 compiler).  Since
   such processes will not know how to use the MSYS implementation of 
Cygwin's
   POSIX emulation layer, MSYS sh.exe performs some extra initialisation 
work,
   beyond that performed by either cmd.exe or Cygwin's bash, to make it 
appear
   to the execed process, as if it had been invoked by cmd.exe.

Do note that there are a few programs supplied with MSYS, which are not 
native
Win32 programs.  These are, essentially Cygwin programs, although we do 
prefer
to call them MSYS programs.  The MSYS modifications to the original Cygwin 
base
included some magic to identify these special MSYS programs, and to cause 
sh.exe
to exec them directly, just as Cygwin's bash.exe would exec a Cygwin app; 
these
then receive verbatim copies of the environment and arguments, just as 
their
Cygwin counterparts would, and they rely on the MSYS variant of the Cygwin 
POSIX
emulation layer to map all POSIX style entity references to their native 
form.

OTOH, all programs which are *not* identified by MSYS sh.exe as MSYS 
specials,
receive the following preparation for exec:

1) The environment is copied, and transformed from its emulated POSIX 
format,
   as seen by sh.exe, to a cmd.exe compatible format; in particular, this 
means
   that all path names present in the environment in POSIX emulated form 
have
   their drive specifiers appropriately transformed, and all multiple path
   references have the PATH_SEPARATOR transformed from ':' to ';'. 
(However,
   there is *no* transformation of '/' to '\\', even where it appears as a
   DIR_SEPARATOR).

2) The process memory is allocated, and the process image is loaded, just 
as
   it would be by cmd.exe

3) The command line is parsed, and all included references to file system 
paths
   are transformed as in the environment transformation; the command line 
is
   then reconstructed from the transformed argument list, and copied into 
the
   process memory space, to mimic cmd.exe behaviour.

4) The program is started, exactly as cmd.exe would do it.

Now, you've built your application with MinGW, so it's a *native* app; 
when it
sees a PATH_SEPARATOR *internally*, it must see a semicolon.  When you 
specify
that PATH_SEPARATOR in cmd.exe, then *you* must specify it as semicolon; 
cmd.exe
will not transform a colon to a semicolon for you, under any 
circumstances.  (It
will not transform '/' to '\\' either, but you don't care about that; 
cmd.exe
will gripe if you try to use '/' where you *need* '\\'; if it doesn't 
gripe,
then one's as good as the other).

OTOH, when you run your command under MSYS sh.exe, you will most likely 
use a
colon to specify PATH_SEPARATOR; (if you quote it appropriately, you can 
also
use a semicolon).  In either case, sh.exe will transform it to a 
semicolon, or
leave it as such, so all your app *ever* *sees* is PATH_SEPARATOR == ';'.

The thorny issue arises during the build phase of your app.  While it is 
generally
a bad idea to rely on hard coded paths, embedded in an app, maybe you just 
want to
supply a sane fallback in this way, and you would like to let autoconf 
pass in an
appropriate PATH_SEPARATOR, via an AC_SUBST into a generated header.  The 
problem
is that, when you use MSYS to host your MinGW build environment, autoconf 
sets a
PATH_SEPARATOR == ':', when you really need PATH_SEPARATOR == ';'.  The 
macro I
proposed previously works around this, by using knowledge imbued by the 
compiler
that it's targetting a native Win32 host, to correctly define an 
alternative
substitution variable.

HTH,
Keith.