Re: Failing to use gnulib bootstrap in libtool

[Gary V. Vaughan]

Hi Bruno,

On 5 Sep 2010, at 17:14, Bruno Haible wrote:
>> The architecture of my current rewrite of gnulib bootstrap, moves all
>> the top-level code into functions ...  Then `bootstrap.conf' is sourced, 
>> where
>> the nodes of that "shell-function-require-tree" can be overwritten
>> with replacement functions to radically alter the behaviour of the
>> script if necessary...
> 
> Sounds very open, powerful, and easy to use. Great!

Thanks for the encouragement.  I'll post the complete script for comment as
soon as I'm ready to test with other gnulib bootsrap using packages, and
(pending feedback) I'll decompose my changes into logical bite-sized patches
for review and hopeful incorporation into upstream.  (I don't know whether
my gnulib CVS commit bit survived the migration to git, but it seems to
have worked for libtool and m4 so I can probably push the patches myself
after successful review). 

> So, there are two ways in which 'bootstrap' can organize the specification
> of the gnulib module names and other gnulib-tool arguments (--source-base,
> --m4-base, ...):
> 
>  A) The user specifies them in the file gnulib-cache.m4.
>     In this case:
>       - Neither the user nor the 'bootstrap' script must use the
>         --import option, because this would kill the contents of
>         gnulib-cache.m4. Instead the user must use --add-import,
>         --remove-import only, and the 'bootstrap' script must use
>         --update only.
>       - The gnulib-cache.m4 file is under version control.
> 
>  B) The user specifies them in a file other than gnulib-cache.m4,
>     such as bootstrap.conf.
>     In this case:
>       - The user must not invoke gnulib-tool directly, not with
>         --import, --add-import, --remove-import, because these
>         options would leave bootstrap.conf in an inconsistent state
>         with the rest of the package, and not --update because this
>         is bootstrap's job.
>       - The 'bootstrap' script must use --import always. It can also
>         use --update if it detects that bootstrap.conf has not changed;
>         but there is no benefit in doing this because it will end up
>         doing the same as the complete --import command; no speedup.
>       - The gnulib-cache.m4 file is not under version control; it's
>         a mere "output" file.
> 
> I'll let you decide which approach your 'bootstrap' script shall support.

This is much clearer, thanks.  I was under the impression from your
earlier message that you had considered option A to be brain-damaged,
but I'm glad I was mistaken.

> The advantage of A, compared to B, is:
>  - The user is able to use "gnulib-tool --add-import module" instead
>    of modifying a file.
> 
> The drawbacks of A, compared to B, is:
>  - The user has two different "input" files where he defines the
>    specification of the bootstrap process: bootstrap.conf and gnulib-cache.m4.
>    They use different syntaxes (shell variable assignments vs. m4 macro
>    invocations).
>  - The user can not add comments to gnulib-cache.m4, as they would be
>    erased the next time the file is written.
>  - Tool failure can accidentally overwrite gnulib-cache.m4.

Ack.

>>> The reason is to avoid mixing inputs and outputs, whereas your proposal
>>> amounts to introducing circular dependencies.
>> 
>> I don't think you can apply that argument to a cache file, which by its 
>> nature
>> is both an input and an output.
> 
> In general you would have to worry. With gnulib-cache.m4 things will probably
> be less severe: There will be no version control conflicts, because
> gnulib-cache.m4 is stable: When it is rewritten after using itself as input,
> it is unchanged.

Yes, that makes sense to me too.

>>> It is *very* important to design things so that every file is either input
>>> (maintained by the developers, manually) or output (generated by some 
>>> tools),
>>> but not both. If you don't follow this principle,
>>> - The file will be messed up and require manual correction when the
>>>   developer invoked the tools with wrong parametrizations, or when some
>>>   unintended failure happened.
>> 
>> Reading a cache file with sensible fall-back options does not have this
>> problem, and if some other tool messes up the cache, then it can easily
>> be deleted at the penalty of having to recalculate its contents again from
>> first principles.  Really, it's just a cache
> 
> No. In case A above, where gnulib-cache.m4 is "input" and "output", tool
> failure will erase the information it contains, with no way to recalculate it.
> (The user will have a backup in version control, and may pray that this
> backup is not too old.)

Also a good point.  But it looks as though the patches you applied to 
gnulib-tool
recently mitigate these concerns, so I won't worry about them any more.

>> [Really it's] a way to short circuit 
>> repeated identical calculations on consecutive runs.
> 
> You misunderstood gnulib-cache.m4: It is in no way a "short-circuit". Its
> presence does not cause any computation, file creation or anything else to
> be optimized away.

So I did.  I could have sworn that --import always copied everything anew,
and that --update always recopied only outdated files.  I see now that
gnulib-cache.m4 is only a cache of the options, and --update vs --import
has no effect on the amount of file copying that occurs.  Thanks for the
explanation.

>>> - Multiple developers working on one project will get confused about
>>>   changes that one developer made (unintentionally) and another developer
>>>   considers wrong.
>> 
>> The project will need a policy to avoid this.  core-utils, gettext and others
>> have a policy of not checking in gnulib-cache.m4
>> ... I don't see a problem with committing it, or not-
>> committing it as long as the developers on each project reach a consensus.
> 
> As explained above, this is not a per-project policy. Either a project is
> in case A, or it is case B. From that point on there is no choice any more.

I believe we're agreeing here too.  Only I didn't explain very clearly.  I
don't mean to imply that one can move back and forth easily, simply that
the project "M4" is in case A, and the project "coreutils" is in case B...
and both are perfectly fine as long as all developers on those projects
agree and understand the "policy" of remaining in case A (for M4) and case B
(for coreutils).

>>> - You will not be able to evolve the format of that file easily.
>> 
>> Easily averted by recalculating the values that you wanted to extract from 
>> the
>> cache-file, but which weren't available from the cache (because the cache is
>> missing, or has changed formats).
> 
> If the values can be recalculated, then you're in case B, and gnulib-cache.m4
> is mere "output", not "input"+"output".

I understand this too now, and agree also.

>>> 1. You must not use 'gnulib-tool --update'.
>> 
>> Huh?  That makes no sense to me!
> 
> I hope the doc makes it clear now?

It does.  Thanks for persevering.

>>> 2. You even have to remove gnulib-cache.m4 before running
>>>    'gnulib-tool --import ...'. Because when gnulib-cache.m4 is present,
>>>    'gnulib-tool --import ...' will add the specified modules but not throw
>>>    away modules that are listed in gnulib-cache.m4 and not listed on the
>>>    command lines. In other words, if you don't remove gnulib-cache.m4,
>>>    you are never able to de-import modules.
>> 
>> That is definitely a wrinkle.
> 
> It is fixed now, as of yesterday.

That's great!

>>>> 6. AC_CONFIG_AUX_DIR detection
>>>> ==============================
>>>> 
>>>> Why another bunch of forks?  Might as well get this data while running sed 
>>>> over
>>>> configure the first time, by adding the following to 
>>>> `extract_package_values':
>>>> 
>>>>   /AC_CONFIG_AUX_DIR(/         {
>>>>       s|^.*AC_CONFIG_AUX_DIR([['"$tab"' ]*\([^])]*\).*$|config_aux_dir=\1|
>>>>       p
>>>>   }
>>> 
>>> Yes, this does make sense. configure.ac is an input, bootstrap.conf is an 
>>> input;
>>> they do not need to both specify the same thing.
>> 
>> Better yet, I've written a shell function to extract these values from 
>> configure.ac
>> using autom4te traces, which is much less fragile
> 
> Nope, this is not better. You will or may run into error messages such as
>   m4: aclocal.m4:851: Cannot open m4/absolute-header.m4: No such file or 
> directory
>   autom4te: m4 failed with exit status: 1
> Recall that at the moment you need to know the config_aux_dir, you have an
> inconsistent state of {configure.ac,aclocal.m4,m4/*} files. The purpose of
> running 'bootstrap' is precisely to import the m4/* files and THEN to
> regenerate a good aclocal.m4. Which means, at this stage you *must*not* do
> anything that uses a potentially outdated aclocal.m4.

I can't reproduce that error, no matter what combination of macro definition
files I am missing:

  $ mkdir ~/works-for-me
  $ cd ~/works-for-me
  $ cat >configure.ac <<'EOF'
  AC_INIT([this project],[some version],address@hidden)
  AC_CONFIG_HEADERS([notexists.h])
  AC_CONFIG_AUX_DIR([build-aux])
  AC_CONFIG_MACRO_DIR([m4])
  AM_INIT_AUTOMAKE([foreign])
  AC_PROG_CC
  gl_EARLY
  LT_INIT
  gl_INIT
  AC_CONFIG_FILES([notexists/Makefile.am])
  EOF
  $ autom4te --language=Autoconf --trace=AC_INIT configure.ac
  configure.ac:1:AC_INIT:this project:some version:address@hidden
  $ ~/../Shared/gnulib.git/gnulib-tool --import dummy
  [[...]]
  $ cat m4/gnulib-tool.m4 m4/gnulib-cache.m4 \
  >   | autom4te --language=Autoconf --trace=gl_TESTS_BASE -
  stdin:87:gl_TESTS_BASE:tests
  $ cat >Makefile.am <<'EOF'
  ACLOCAL_AMFLAGS = -I m4
  SUBDIRS = lib .
  EOF
  $ autoreconf -fvi
  [[...]]
  $ rm -rf m4
  $ autom4te --language=Autoconf --trace=AC_CONFIG_AUX_DIR configure.ac 
  configure.ac:3:AC_CONFIG_AUX_DIR:build-aux

>>>> 10. slurp()
>>>> ==========
>>> ...
>>> I think it has/had two purposes:
>>> 1. It allowed Paul to use symbolic links or hard links to gnulib files,
>>>    at a time when gnulib-tool did not have options for symbolic links.
>>> 2. It prevents some files from being edited. Again, this dated from before
>>>    it become gnulib-tool standard to mark generated files with "DO NOT 
>>> EDIT!"
>> 
>> So, with a little work in existing gnulib bootstrap using projects, slurp 
>> can be removed?
> 
> I guess so, yes.

Excellent, that simplifies the rewritten bootstrap script considerably.

Cheers,
-- 
Gary V. Vaughan (address@hidden)

PGP.sig
Description: This is a digitally signed message part