Proposal to make better use of 5th arg to AC_CHECK_LIB and AC_SEARCH_LIBS

[Rainer Orth]

Back in may, I raised PR autoconf/176 against autoconf 2.49e (the problem
exists in all previous and current releases):

Consider the following configure.in:

        AC_INIT(check_lib, 0.1)
        AC_PROG_CC
        AC_CHECK_LIB(hesiod, hes_init,,, -lresolv)
        AC_OUTPUT(Makefile)

and Makefile.in:

        LIBS = @LIBS@

On platforms where both libhesiod and libresolv are used and needed to
successfully use hes_init, the current implementation of AC_CHECK_LIB will
detect that hes_init is present by trying to link with both -lhesiod and
-lresolv, but recording only -lresolv in LIBS.

This approach has two flaws in my opinion:

* It always tries to use -lresolv, while it may be necessary (or even
  present) only on some platforms.  This may even be harmful in some cases:
  consider Solaris 2 whose libsocket is always necessary for socket
  functions, compared to IRIX which has broken socket functions in
  libsocket which should be avoided.  Handling this with the current macro
  is impossible.

* It doesn't record its use of -lresolv at all, so there's no way to decide
  if it was needed to successfully use hes_init.

I'd like to propose the following behavior instead, which I'd consider much
more useful:

* It should first try to link *without* -lresolv and see if this succeeds.
  If this works, all is fine and LIBS can be set to just -lhesiod as
  before.

* If this first attempt fails, a second attempt should be made with *both*
  -lhesiod and -lresolv on the link line.  If this succeeds, both -lhesiod
  and -lresolv should be added to LIBS.

I've proposed a patch to implement this in the PR above, and a replacement
macro (AC_CHECK_LIB2) with this behavior has been successfully used in the
popular am-utils automounter for years.

The PR (and patch) had been rejected before on the basis that it differs
from documented autoconf behavior, but I hope to have demonstrated that the
current behavior is flawed and should be fixed, either by changing the
current macro, or, if backwards (bugwards?) compatibility is really an
issue here, by adding a new macro with the desired/more useful behavior.

        Rainer

-----------------------------------------------------------------------------
Rainer Orth, Faculty of Technology, Bielefeld University

Email: address@hidden