Re: Handling of SHELL by GNU make: not POSIX compliant

[David Boyce]

Soren,

Sorry for the late and lengthy response. I'll address your points up top 
here because my response didn't interleave well with yours.

First, to your last point that the proposed change might not achieve "any 
practical improvement for anybody". Believe me, I don't have time to go 
looking for technical standards violations to quibble with. On the 
contrary, as with most bugs, I discovered this one only when it jumped up 
and bit me. I can't be the judge of how common the problem is or will be, 
but I assure you it did have a practical effect on me at least.

Next, regarding your Windows concerns: I don't know if there's a fixed GNU 
policy but as far as I know POSIX conformance is just not a concern on 
Windows. In particular the handling of SHELL is already a completely 
different case on Windows, for good reason as you note, so I see no reason 
the Windows behavior should change at all. When I argued for POSIX 
conformance I was thinking only of POSIX platforms. Sorry for not making 
that clear before.

However, let's go into that in some detail anyway. One of the Windows 
differences is that SHELL is inherited from the env, and if you're using 
that feature then "my" POSIX fix would have no effect even if implemented 
on Windows. So it only affects someone who might be setting SHELL 
_explicitly_ in a main Makefile and depending on its being automatically 
exported for use in sub-Makefiles. One might argue that even in this case, 
requiring

        export SHELL    := bash

would be both more intuitive and more self-documenting than

        SHELL   := bash

which codifies the hidden assumption that in GNU make SHELL, unlike any 
other make variable and unlike any other make program, will be 
automatically exported. However, I'm personally agnostic on the question of 
how SHELL is handled on Windows and will leave that to people using GNU 
make there.

It's worth reiterating the importance of POSIX on the Unix side, though. 
POSIX doesn't do invention, it describes existing behavior. Thus, 
"non-POSIX-conformant" is shorthand for "incompatible (in this way) with 
Solaris make, HPUX make, AIX make, and just about every other Unix make 
program". Which is how I ran across it in the first place and which is a 
pretty big deal, IMO.

Let me explain what I'm doing as an example of why it matters (note, this 
is Unix-specific). In my spare time, currently copious since I'm 
unemployed, I'm developing a build-auditing tool. The idea is you run a 
build under control of my program and it keeps a record of all files opened 
during that build. For anyone familiar with ClearCase the idea is to get 
something like its "config records" but without requiring the special 
ClearCase filesystem. Instead of communicating with the filesystem as 
ClearCase does, my tool inserts a shim into the shared-library call 
sequence using LD_PRELOAD and uses it to intercept and record uses of 
open() and related system calls. Consider it analogous to "truss -o 
/tmp/truss.out -f -t open,creat", more or less, but more portable and 
entirely in user space.

This is working pretty well. It would potentially let you replace or 
augment timestamp-based dependency analysis with checksum/CRC/MD5 
'fingerprints' in the way cook et al do. I also have a sample impementation 
using it to generate full autodependencies very much in the manner of "gcc 
-MD" except, of course, for being language independent and getting ALL the 
dependencies.

So next I come to the question of how best to actually integrate it with 
make. The checksum-based dependency stuff would require major surgery on 
the make program (any make, not necessarily GNU) and I'm not ready to even 
consider that. But the auto dependency generation can, in theory, be easily 
integrated by telling make to run all build commands under control of the 
auditor. Imagine I set

        SHELL := truss -o /tmp/truss.out -f -t open,creat

(of course this wouldn't actually work because make assumes $SHELL to be a 
single word and insists on invoking it with a -c flag, not to mention truss 
would keep overwriting its tmp file, but you get the idea). If I do the 
above in any regular make, all my build commands will be run under control 
of truss and I'll get audits in /tmp. Very nice. If I do it with GNU make, 
my shell EV will become "corrupted: and if any child process does anything 
with it I'll end up with truss trussing truss. Very much not good. It boils 
down to the Heisenbergian principle: any auditor which has a profound 
effect on the process it's auditing is useless. And inadvertently changing 
the SHELL EV for all children is pretty profound; as you say it's one of 
the major Unix system EV's.

I do have the workaround Paul suggested of saving the 'real' value of SHELL 
aside as REAL_SHELL and setting SHELL=$REAL_SHELL in the auditor program. 
This works and I've used it for my proof of concept but it's offensive.

IMO the real problem is the overloading of the name $SHELL by make some 3 
decades ago; there really is no correlation beween the concepts "what is my 
preferred shell?" and "what command to I want to use, within this 
particular Makefile, to control my build scripts?". The original author of 
make, observing that the answer to the latter question was most commonly 
/bin/sh, used SHELL for the macro that controlled it (early versions of 
make actually contained commented-out code to inherit SHELL, showing that 
he didn't realize his mistake till after the fact). The long-standing hack 
of not inheriting SHELL is really a workaround for the fact that he should 
have used, say, MAKECMD instead and had that default to /bin/sh without 
ever involving $SHELL.

FWIW GNU make already has something similar (MAKESHELL), though it's 
currently implemented only on Windows, but I'd argue that ideally the name 
wouldn't even contain the substring 'SHELL' because it implies that one 
must use a shell-like command. And I don't see why that constraint is 
necessary.

At the risk of being too wordy, let me pursue that angle a bit. Make 
novices often tend to act as if its only use is to compile C code and need 
to be reminded that it's really a general-purpose rules engine that can be 
used to manage anything involving targets, dependencies, and recipes. NIS 
databases are a classic example. But how general-purpose is it really? In 
the basic make rule:

targets: dependencies
        command ...

I can use absolutely any set of files for my targets and likewise for 
dependencies. But do I have the same range of options when it comes to 
'command'? No. I'm constrained to running '/bin/sh -c "command ..."'. I can 
replace /bin/sh with another program but (a) it must accept -c and -e flags 
with similar semantics to /bin/sh and (b) to do it with GNU make I have to 
blow away my $SHELL environment var. In a really ideal world, MAKECMD could 
be assigned an arg vector allowing,say, {} or % to represent the "command 
..." string but that's a pretty major change. Paul has said he's considered 
a SHELLFLAGS (aka MAKECMDFLAGS) which might be a reasonable compromise. 
Anyway, this is all blue sky stuff beyond the current question; I just 
bring it up to illustrate that make is noticeably less flexible than it 
could be when it comes to invoking recipes.

Summary time: my bottom line argument is that there is *no* logical 
correlation between a user's preferred shell and a Makefile's preferred 
command executor, and that GNU make should respect this boundary at least 
as much as POSIX does.

Sorry for the length.

-David Boyce

> I don't see this issue affecting me much anytime soon, and probably i
> ought to keep my mouth shut. But I found myself provoked in a wondering
> kind of way.
>
> I am "provoked" into having questions about the meaning of all this. It
> seems to me that "What You Expect (tm)" would be for any subprocess (sub
> shell, child process) to inherit the setting for "SHELL" that is in
> effect in the Makefile. I strain my meager experience to think of a
> situation in which you'd want child processes of 'make' to have a value
> of SHELL that's different in the environment of that process than it is
> in the value of the 'make' macro 'SHELL'. To what possible end would
> such an inconsistency be maintained? Can anyone give me an example?
>
> As a 'make' user on a platform that requires shell issues to be handled
> differently (historically always has -- Windows / DOS) I am also
> wondering how an attempt to change the behavior of 'make' in the manner
> David Boyce seems to be suggesting would impact me and those like me. It
> appears IMHO that we totally rely on 'make' setting SHELL in the
> environment of child processes since we cannot be guaranteed otherwise
> that SHELL will be useful or correct -- the native "shell"-like entity
> on our platforms being basically a big problem issue that many have
> sought to avoid by installing a foreign, ported Bourne-compatible shell
> (sh, bash, zsh, ksh).
>
> To sum up, my possibly naive understanding is that one would ordinarily
> expect the setting for a parameter of such a primary importance as SHELL
> to be propagated to all children -- and to their children, which is what
> we are talking about right? -- of the 'make' application. We who have
> any familiarity with 'make' at all understand how SHELL is already
> specially-treated in that there is a "no-inherit" policy on *nix-ish
> platforms and a differing "inherit" policy on DOS/Win -ish ones; and it
> seems to me that layering on additional complexity (that 'make's idea of
> SHELL and 'make's children's idea of SHELL would possibly differ) to
> that already complex situation simply makes things harder for users
> without achieving any practical improvement for anybody?
>
>    Regards,
>     Soren A
>
>
>
>
> _______________________________________________
> Make-alpha mailing list
> address@hidden
> http://mail.gnu.org/mailman/listinfo/make-alpha
>
> --
> -------------------------------------------------------------------------------
>  Paul D. Smith <address@hidden>          Find some GNU make tips at:
>  http://www.gnu.org                     http://make.paulandlesley.org
>  "Please remain calm...I may be mad, but I am a professional." --Mad 
> Scientist