Re: distclean-recursive broken?

[Ralf Corsepius]

On Mon, 2004-02-16 at 09:20, Alexandre Duret-Lutz wrote:
> [I've s/CONFIG_SUBDIRS/DIST_SUBDIRS/ your mail for clarity;
> there is no CONFIG_SUBDIRS variable.  Maybe that's where the
> confusion comes from?  DIST_SUBDIRS is not related to
> AC_CONFIG_SUBDIRS.]
> 
> >>> "Ralf" == Ralf Corsepius <address@hidden> writes:
> 
>  Ralf> On Sun, 2004-02-15 at 22:28, Alexandre Duret-Lutz wrote:
>  >> >>> "Ralf" == Ralf Corsepius <address@hidden> writes:
>  >> 
>  >> [...]
>  >> 
>  Ralf> SUBDIRS = sub1 @moresubdirs@
>  Ralf> DIST_SUBDIRS = sub1
>  >> >> 
>  >> >> Unfortunately we do not support such setup.  Right now
>  >> >> $(SUBDIRS) must always be a subset of $(DIST_SUBDIRS).
>  Ralf> The example below proves the contrary.
>  >> 
>  >> Additionally each directory listed in DIST_SUBDIRS or SUBDIRS
>  >> should always have a Makefile.
> 
>  Ralf> I think you are missing the point: I am running "make distcheck". 
> 
> I confess I fail to see why this is relevant.  distcheck just
> runs some ordinary commands a user may want to run.
> 
>     ./configure [any option]
>     make dist
>     make distclean
OK. This is equally broken:

With my autobug-0.8 example:

./configure --enable-sub1 -> recurses into SUBDIRS
make -> recurses into SUBDIRS
# make distclean -> recurses into CONFIG_SUBDIRS
Making distclean in sub2
make[1]: Entering directory `/users/corsepiu/autobug-0.8/sub2'
make[1]: *** No rule to make target `distclean'.  Stop.
make[1]: Leaving directory `/users/corsepiu/autobug-0.8/sub2'
make: *** [distclean-recursive] Error 1

=> "make distclean" only works iff SUBDIRS == CONFIG_SUBDIRS.

> is a legitimate user sequence. `make dist' should distribute all
> directories (all those listed in DIST_SUBDIRS, not only the
> subset to built with `make all' which is listed in SUBDIRS),
Please read what you wrote ;)

...'make dist' all those ... in DIST_SUBDIRS ...

=> "make dist" make should recurse into DIST_SUBDIRS.
ATM, however:
- "make dist" recurses into SUBDIRS!!!
- "make distclean-recursive" recurses into DIST_SUBDIRS.

==> "make dist" and "make distcheck" require DIST_SUBDIRS=$(SUBDIRS).

My conclusion: DIST_SUBDIRS is useless.

> this implies that `make dist' must recurse into these
> directories, hence ./configure must have created a Makefile in
> all of them unconditionally.  Since there is a Makefile in these
> distributed but otherwise not built directories, distclean must
> recurse into them to clean them.
> 
> If you want to convince yourself, look at test/subconf3.test
> (this is the example from the manual), and edit
> lib/am/subdirs.am so distclean recurses only in SUBDIRS, not
> DIST_SUBDIRS.  This ought to cause the test to fail because
> opt/Makefile is not cleaned.
> 
>  Ralf> "make distcheck" first recurses into SUBDIRS, but afterwards recurses
>  Ralf> into DIST_SUBDIRS on the "make distclean-recursive".
> 
> To remove the Makefile that each DIST_SUBDIRS directory must have.
Where do you expect these Makefiles to come from?

With autoconf/automake based configurations, these Makefile's stem from
preceding configure runs:

** configure ... recurses into SUBDIRS **

Therefore, with conditional SUBDIRS, there cannot exist any Makefile's
in all DIST_SUBDIRS outside of SUBDIRS, no matter if SUBDIRS is a sub-
or superset of DIST_SUBDIRS.

>  Ralf> Without any doubt, this behavior is broken: "make distcheck" must run
>  Ralf> distclean-recursive on *same set* of directories it ran configure
>  Ralf> beforehand - ATM, it doesn't. 
> 
> If there is a directory `make distclean' must not recurse into
> because there is no Makefile, then `make dist' cannot recurse
> into this directory either, hence this directory must not be
> listed in DIST_SUBDIRS.
In this case, there is not any need to have it in SUBDIRS.

> Or to explain it in the other way: DIST_SUBDIRS contains the
> directories `make dist' must recurse into.  Since `make dist'
> must recurse in these directories, they must have a Makefile
> (i.e. be configured).
So you are proposing to set 

SUBDIRS = @subdirs@
DIST_SUBDIRS = @subdirs@

.. and yes, this seems to work for me.

This is not an actual surprize to me, because this sets
SUBDIRS == DIST_SUBDIRS.

>   Since they have a Makefile `make
> distclean' must also recurse into them to remove it.
Wrong.

configure recurses into SUBDIRS. If SUBDIRS is a subset of DIST_SUBDIRS,
there don't exist any Makefile's make could invoke on
"make distclean-recursive".

> DIST_SUBDIRS is tied to `make dist', not to AC_CONFIG_SUBDIRS.
Then the recursive configuration step which is part of "make dist"
should recurse into DIST_SUBDIRS. ATM, it recurses into SUBDIRS.

> Sure, you could fix your problem this by conditionally omitting
> unconfigured directories from DIST_SUBDIRS and SUBDIRS.
No. I can fix the problem by only recursing into SUBDIRS in
distclean-recursive or be changing "make distcheck" to recurse into
DIST_SUBDIRS only.

>   But the
> consequence is that these directories would not always be
> distributed.
Yes, exactly this is my intention.

I have config-subdirs, configure in can only be run conditionally.
Therefore, I am dynamically setting up SUBDIRS and have to exclude these
directories from CONFIG_SUBDIRS, but ...

... I can add a dist-hook: to let "make dist" add them to tarball.

... I can apply DISTCHECK_CONFIGURE_ARGS to run distcheck selectively on
directories on demand.

I.e. I am using
* "make dist" to cut a tarball, disregarding automake's dist-packaging.
* "make distcheck DISTCHECK_CONFIGURE_ARGS=..." to check selectively
check for package correctness.

>   (`make dist' would be complete only in some kind
> of "full" configuration.)  I know some people do DIST_SUBDIRS =
> $(SUBDIRS), precisely to support conditional AC_CONFIG_SUBDIRS.
Exactly, that's what I do.

> I do not recommend it because it's then easy to make incomplete
> distributions by mistake.
You should consider to change this attitude :(

There are occasions where conditional SUBDIRS are inevitable, for
example:

1. Some subdirs might be optional sub-packages. Presence of these
sub-packages can be detected by higher level configure scripts, autoconf
supports such conditionally presence of subdirs.

2. Some subdirs/sub-packages might not be compilable on certain hosts.
E.g. I have sub-packages, which can only be configured on certain
architectures. Your perspective would require me to add
AM_CONDITIONAL([VALIDARCH],[test "$host_cpu" = "supported_cpu"]) to
those subpackages and to apply this conditional in all of these
packages' Makefile.ams.

3. VPATH/multiple builds: The build-directory tree does not necessarily
have to match with the source directory-tree.

Now combine all this and you have my actual situation.

>   Maybe one can combine this with a
> conditional dist-hook and be safe.
> 
> [...]
> 
>  Ralf> This also matches with autoconf.info:
>  Ralf> ------
>  Ralf> However `make dist' should always recurse into both `src/' and
>  Ralf> `opt/'.  Because `opt/' should be distributed even if it is not needed
>  Ralf> in the current configuration. This means `opt/Makefile' should be
>  Ralf> created unconditionally.  (1)
>  Ralf> ------
> 
> The intent of this paragraph is to explain why although it will
> be possible to omit some directories from SUBDIRS (so they are
> not built) they should always be 1) configured unconditionally
> and 2) listed in DIST_SUBDIRS, so that `make dist' distributes
> them.  The footnote should also put more emphasis on
> "unconditionally".
>
> The SUBDIRS/DIST_SUBDIRS distinction is introduced in the
> following paragraphs.  This one just prepares the reader to it.
> 
>  >> I see pretty well that you want conditional AC_DIST_SUBDIRS.
>  Ralf> IMO, one reason for not having them is design of "make distcheck".
> 
> Unfortunately I still don't see why this is a problem specific
> to `make distcheck' :(  Any suggestion?
The core of the problem is _one_ "make" operation recursing into two
different sets of subdirs: SUBDIRS and DIST_SUBDIRS.

AFAICT, this only works iff SUBDIRS == DIST_SUBDIRS.

>  >> The SUBDIRS/DIST_SUBDIRS
>  >> distinction is not meant to solve this problem.
>  Ralf> I disagree - It does.
> 
>  Ralf> As the citation from automake.info says, the set of SUBDIRS to be
>  Ralf> recursed by "make dist" must match the default set of SUBDIRS to be
>  Ralf> recursed into.
> 
> I'm really surprised you read it this way.  Maybe it helps to
> quote the previous paragraphs too:
> 
Frankly speaking, IMO, this paragraph is a masterpiece of "saying
nothing with a lot of words". Instead of giving such prosaic
explanation, it should be condensed to 
"cooking recipes".

> |    To illustrate how this works, let's assume we have two directories
> | `src/' and `opt/'.  `src/' should always be built, but we want to
> | decide in `./configure' whether `opt/' will be built or not.  (For this
> | example we will assume that `opt/' should be built when the variable
> | `$want_opt' was set to `yes'.)
> | 
> |    Running `make' should thus recurse into `src/' always, and then
> | maybe in `opt/'.
> | 
> |    However `make dist' should always recurse into both `src/' and
> | `opt/'.  Because `opt/' should be distributed even if it is not needed
> | in the current configuration. This means `opt/Makefile' should be
> | created unconditionally.  (1)
> 
> This means that `make dist' will recurse in *more* directories
> than `make'.  That's the whole point of this entire section and
> of the DIST_SUBDIRS/SUBDIRS distinction.
> Is it clearer now?
No. IMO we are talking past each other.

>   Any suggestion how to word this more clearly?
No, there seem to be fundamental disagreements.

To me, the essentials are:
* Current automake does not support conditional subdirs.
* There exists an inconsistency/bug in automake: configure/make recurses
into SUBDIRS, "distclean-recursive" recurses into DIST_SUBDIRS.

Even more condensed: DIST_SUBDIRS must be identical to SUBDIRS, anything
else does not work.

Ralf