some new documentation

[Bruno Haible]

Hi,

When Jim mentioned issues about TESTS_ENVIRONMENT, I realized that the gnulib
documentation has no details about how a tests module normally looks like, and
even the doc about regular modules is scarce. In fact, the module description
details are undocumented!

This fixes it. More doc could be added about common gnulib idioms. This is
just the basics.


2010-01-24  Bruno Haible  <address@hidden>

        More documentation.
        * doc/gnulib.texi (Writing modules): New chapter.
        (Miscellaneous Notes): Move sections "Comments" and "Header files" to
        the new chapter.

--- doc/gnulib.texi.orig        Sun Jan 24 16:31:08 2010
+++ doc/gnulib.texi     Sun Jan 24 16:29:01 2010
@@ -53,6 +53,7 @@
 @menu
 * Introduction::
 * Invoking gnulib-tool::
+* Writing modules::
 * Miscellaneous Notes::
 * POSIX Substitutes Library::       Building as a separate substitutes library.
 * Header File Substitutes::         Overriding system headers.
@@ -113,38 +114,53 @@
 @include gnulib-tool.texi
 
 
address@hidden Miscellaneous Notes
address@hidden Miscellaneous Notes
address@hidden Writing modules
address@hidden Writing modules
+
+This chapter explains how to write modules of your own, either for your own
+package (to be used with gnulib-tool's @samp{--local-dir} option), or for
+inclusion in gnulib proper.
+
+The guidelines in this chapter do not necessarily need to be followed for
+using @code{gnulib-tool}.  They merely represent a set of good practices.
+Following them will result in a good structure of your modules and in
+consistency with gnulib.
 
 @menu
-* Comments::
+* Source code files::
 * Header files::
-* Out of memory handling::
-* Obsolete modules::
-* Library version handling::
-* Windows sockets::
-* Libtool and Windows::
-* License Texinfo sources::
-* Build robot for gnulib::
+* Implementation files::
+* Specification::
+* Module description::
+* Autoconf macros::
+* Unit test modules::
+* Incompatible changes::
 @end menu
 
 
address@hidden Comments
address@hidden Comments
-
address@hidden comments describing functions
address@hidden describing functions, locating
-Where to put comments describing functions: Because of risk of
-divergence, we prefer to keep most function describing comments in
-only one place: just above the actual function definition.  Some
-people prefer to put that documentation in the .h file.  In any case,
-it should appear in just one place unless you can ensure that the
-multiple copies will always remain identical.
address@hidden Source code files
address@hidden Source code files
+
+Every API (C functions or variables) provided should be declared in a header
+file (.h file) and implemented in one or more implementation files (.c files).
+The separation has the effect that users of your module need to read only
+the contents of the .h file and the module description in order to understand
+what the module is about and how to use it - not the entire implementation.
+Furthermore, users of your module don't need to repeat the declarations of
+the functions in their code, and are likely to receive notification through
+compiler errors if you make incompatible changes to the API (like, adding a
+parameter or changing the return type of a function).
 
 
 @node Header files
 @section Header files
 
+The .h file should declare the C functions and variables that the module
+provides.
+
+The .h file should be stand-alone.  That is, it does not require other .h files
+to be included before.  Rather, it includes all necessary .h files by itself.
+
 @cindex double inclusion of header files
 @cindex header file include protection
 It is a tradition to use CPP tricks to avoid parsing the same header
@@ -207,20 +223,402 @@
 is recommended to place the @code{#include} before the @code{extern
 "C"} block.
 
address@hidden Implementation files
address@hidden Implementation files
+
+The .c file or files implement the functions and variables declared in the
+.h file.
+
 @subheading Include ordering
 
-When writing a gnulib module, or even in general, a good way to order
-the @samp{#include} directives is the following.
+Every implementation file must start with @samp{#include <config.h>}.
+This is necessary for activating the preprocessor macros that are defined
+on behalf of the Autoconf macros.  Some of these preprocessor macros,
+such as @code{_GNU_SOURCE}, would have no effect if defined after a system
+header file has already been included.
+
+Then comes the @samp{#include "..."} specifying the header file that is
+being implemented.  Putting this right after @samp{#include <config.h>}
+has the effect that it verifies that the header file is self-contained.
+
+Then come the system and application headers. It is customary to put all the
+system headers before all application headers, so as to minimize the risk
+that a preprocessor macro defined in an application header confuses the system
+headers on some platforms.
+
+In summary:
 
 @itemize
address@hidden First comes the #include "..." specifying the module being 
implemented.
address@hidden Then come all the #include <...> of system or system-replacement 
headers,
address@hidden
+First comes #include <config.h>.
address@hidden
+Second comes the #include "..." specifying the module being implemented.
address@hidden
+Then come all the #include <...> of system or system-replacement headers,
 in arbitrary order.
address@hidden Then come all the #include "..." of gnulib and private headers, 
in
address@hidden
+Then come all the #include "..." of gnulib and application headers, in
 arbitrary order.
 @end itemize
 
 
address@hidden Specification
address@hidden Specification
+
+The specification of a function should answer at least the following
+questions:
address@hidden
address@hidden
+What is the purpose of the function?
address@hidden
+What are the arguments?
address@hidden
+What is the return value?
address@hidden
+What happens in case of failure? (Exit? A specific return value? Errno set?)
address@hidden
+Memory allocation policy: If pointers to memory are returned, are they freshly
+allocated and supposed to be freed by the caller?
address@hidden itemize
+
address@hidden specification
address@hidden comments describing functions
address@hidden describing functions, locating
+Where to put the specification describing exported functions? Three practices
+are used in gnulib:
+
address@hidden
address@hidden
+The specification can be as comments in the header file, just above the
+function declaration.
address@hidden
+The specification can be as comments in the implementation file, just above
+the function definition.
address@hidden
+The specification can be in texinfo format, so that it gets included in the
+gnulib manual.
address@hidden itemize
+
+In any case, the specification should appear in just one place, unless you can
+ensure that the multiple copies will always remain identical.
+
+The advantage of putting it in the header file is that the user only has to
+read the include file normally never needs to peek into the implementation
+file(s).
+
+The advantage of putting it in the implementation file is that when reviewing
+or changing the implementation, you have both elements side by side.
+
+The advantage of texinfo formatted documentation is that it is easily
+published in HTML or Info format.
+
+Currently (as of 2010), half of gnulib uses the first practice, nearly half
+of gnulib uses the second practice, and a small minority uses the texinfo
+practice.
+
+
address@hidden Module description
address@hidden Module description
+
+For the module description, you can start from an existing module's
+description, or from a blank one: @file{module/TEMPLATE} for a normal module,
+or @file{module/TEMPLATE-TESTS} for a unit test module.  Some more fields
+are possible but rarely used.  Use @file{module/TEMPLATE-EXTENDED} if you
+want to use one of them.
+
+Module descriptions have the following fields.  Absent fields are equivalent
+to fields with empty contents.
+
address@hidden @asis
address@hidden Description
+This field should contain a concise description of the module's functionality.
+One sentence is enough.  For example, if it defines a single function
address@hidden, the description can be @samp{frob() function: frobnication.}
+Gnulib's documentation generator will automatically convert the first part
+to a hyperlink when it has this form.
+
address@hidden Status
+This field is either empty/absent, or contains the word @samp{obsolete}.  In
+the latter case, @command{gnulib-tool} will, unless the option
address@hidden is given, omit it when it used as a dependency.  It is
+good practice to also notify the user about an obsolete module.  This is done
+by putting into the @samp{Notice} section (see below) text like
address@hidden module is obsolete.}
+
address@hidden Notice
+This field contains text that @command{gnulib-tool} will show to the user
+when the module is used.  This can be a status indicator like
address@hidden module is obsolete.} or additional advice.  Do not abuse this
+field.
+
address@hidden Applicability
+This field is either empty/absent, or contains the word @samp{all}.  It
+describes to which @code{Makefile.am} the module is applied.  By default,
+a normal module is applied to @address@hidden/Makefile.am}
+(normally @code{lib/Makefile.am}), whereas a module ending in @code{-tests}
+is applied to @address@hidden/Makefile.am} (normally
address@hidden/Makefile.am}).  If this field is @samp{all}, it is applied to
+both @code{Makefile.am}s.  This is useful for modules which provide
+Makefile.am macros rather than compiled source code.
+
address@hidden Files
+This field contains a newline separated list of the files that are part of
+the module.  @code{gnulib-tool} copies these files into the package that
+uses the module.
+
+This list is typically ordered by importance: First comes the header file,
+then the implementation files, then other files.
+
+It is possible to have the same file mentioned in multiple modules.  That is,
+if the maintainers of that module agree on the purpose and future of said
+file.
+
address@hidden Depends-on
+This field contains a newline separated list of the modules that are required
+for the proper working of this module.  @code{gnulib-tool} includes each
+required module automatically, unless it is specified with option
address@hidden or it is marked as obsolete and the option
address@hidden is not given.
+
+A test modules @code{foo-tests} implicity depends on the corresponding non-test
+module @code{foo}.  @code{foo} implicitly depends on @code{foo-tests} if the
+latter exists and if the option @code{--with-tests} has been given.
+
+Tests modules can depend on non-tests modules.  Non-tests modules should not
+depend on tests modules. (Recall that tests modules are built in a separate
+directory.)
+
address@hidden configure.ac-early
+This field contains @file{configure.ac} stuff (Autoconf macro invocations and
+shell statements) that are logically placed early in the @file{configure.ac}
+file: right after the @code{AC_PROG_CC} invocation.  This section is adequate
+for statements that modify @code{CPPFLAGS}, as these can affect the results of
+other Autoconf macros.
+
address@hidden configure.ac
+This field contains @file{configure.ac} stuff (Autoconf macro invocations and
+shell statements).
+
+It is forbidden to add items to the @code{CPPFLAGS} variable here, other than
+temporarily, as these could affect the results of other Autoconf macros.
+
+We avoid adding items to the @code{LIBS} variable, other than temporarily.
+Instead, the module can export an Autoconf-substituted variable that contains
+link options.  The user of the module can then decide to which executables
+to apply which link options.  Recall that a package can build executables of
+different kinds and purposes; having all executables link against all
+libraries is inappropriate.
+
+If the statements in this section grow larger than a couple of lines, we
+recommend moving them to a @code{.m4} file of their own.
+
address@hidden Makefile.am
+This field contains @code{Makefile.am} statements.  Variables like
address@hidden are transformed to match the name of the library
+being built in that directory.  For example, @code{lib_SOURCES} may become
address@hidden (for a plain library) or @code{libgnu_la_SOURCES}
+(for a libtool library).  Therefore, the normal way of having an
+implementation file @code{lib/foo.c} compiled unconditionally is to write
address@hidden
+lib_SOURCES += foo.c
address@hidden smallexample
+
address@hidden Include
+This field contains the preprocessor statements that users of the module
+need to add to their source code files.  Typically it's a single include
+statement.  A shorthand is allowed: You don't need to write the word
+``#include'', just the name of the include file in the way it will appear
+in an include statement.  Example:
address@hidden
+"foo.h"
address@hidden smallexample
+
address@hidden Link
+This field contains the set of libraries that are needed when linking
+libraries or executables that use this module.  Often this will be
+written as a reference to a Makefile variable.  Please write them
+one per line, so that @command{gnulib-tool} can remove duplicates
+when presenting a summary to the user.
+Example:
address@hidden
+$(POW_LIBM)
+$(LTLIBICONV) when linking with libtool, $(LIBICONV) otherwise
address@hidden smallexample
+
address@hidden License
+This field specifies the license that governs the source code parts of
+this module.  See @ref{Copyright} for details.
+
address@hidden Maintainer
+This field specifies the persons who have a definitive say about proposed
+changes to this module.  You don't need to mention email addresses here:
+they can be inferred from the @code{ChangeLog} file.
+
+Please put at least one person here.  We don't like unmaintained modules.
address@hidden table
+
+
address@hidden Autoconf macros
address@hidden Autoconf macros
+
+For a module @code{foo}, an Autoconf macro file @file{m4/foo.m4} is typically
+created when the Autoconf macro invocations for the module are longer than
+one or two lines.
+
+The name of the main entry point into this Autoconf macro file is typically
address@hidden  For modules outside Gnulib that are not likely to be moved
+into Gnulib, please use a prefix specific to your package: @code{gt_} for
+GNU gettext, @code{cu_} for GNU coreutils, etc.
+
+For modules that define a function @code{foo}, the entry point is called
address@hidden instead of @code{gl_FOO}.  For modules that provide a
+header file with multiple functions, say @code{foo.h}, the entry point is
+called @code{gl_FOO_H} or @code{gl_HEADER_FOO_H}.  This convention is useful
+because sometimes a header and a function name coincide (for example,
address@hidden and @code{fcntl.h}).
+
+For modules that provide a replacement, it is useful to split the Autoconf
+macro into two macro definitions: one that detects whether the replacement
+is needed and requests the replacement using @code{AC_LIBOBJ} (this is the
+entry point, say @code{gl_FUNC_FOO}), and one that arranges for the macros
+needed by the replacement code @code{lib/foo.c} (typically called
address@hidden).  The reason of this separation is
address@hidden
address@hidden
+to make it easy to update the Autoconf macros when you have modified the
+source code file: after changing @code{lib/foo.c}, all you have to review
+is the @code{Depends-on} section of the module description and the
address@hidden macro in the Autoconf macro file.
address@hidden
+The Autoconf macros are often large enough that splitting them eases
+maintenance.
address@hidden enumerate
+
+
address@hidden Unit test modules
address@hidden Unit test modules
+
+A unit test that is a simple C program usually has a module description as
+simple as this:
+
address@hidden
+Files:
+tests/test-foo.c
+tests/macros.h
+
+Depends-on:
+
+configure.ac:
+
+Makefile.am:
+TESTS += test-foo
+check_PROGRAMS += test-foo
address@hidden smallexample
+
+The test program @file{tests/test-foo.c} often has the following structure:
+
address@hidden
address@hidden
+First comes the obligatory @samp{#include <config.h>}.
+
address@hidden
+Second comes the include of the header file that declares the API being tested.
+Including it here verifies that said header file is self-contained.
+
address@hidden
+Then come other includes.  In particular, the file @file{macros.h} is often
+used here.  It contains a convenient @code{ASSERT} macro.
address@hidden itemize
+
+The body of the test, then, contains many @code{ASSERT} invocations.  When
+a test fails, the @code{ASSERT} macro prints the line number of the failing
+statement, thus giving you as a developer a idea which part of the test
+failed, even when you don't have access to the machine where the test failed
+and the reporting user cannot run a debugger.
+
+Sometimes it is convenient to write part of the test as a shell script.
+(For example, in areas related to process control or interprocess
+communication, or when different locales should be tried.) In these cases,
+the typical module description is like this:
+
address@hidden
+Files:
+tests/test-foo.sh
+tests/test-foo.c
+tests/macros.h
+
+Depends-on:
+
+configure.ac:
+
+Makefile.am:
+TESTS += test-foo.sh
+TESTS_ENVIRONMENT += FOO_BAR='@@FOO_BAR@@'
+check_PROGRAMS += test-foo
address@hidden smallexample
+
+Here, the @code{TESTS_ENVIRONMENT} variable can be used to pass values
+determined by @code{configure} or by the @code{Makefile} to the shell
+script, as environment variables.
+
+Regardless of the specific form of the unit test, the following guidelines
+should be respected:
+
address@hidden
address@hidden
+A test indicates success by exiting with exit code 0.  It should normally
+not produce output in this case.  (Output to temporary files that are
+cleaned up at the end of the test are possible, of course.)
address@hidden
+A test indicates failure by exiting with an exit code different from 0 and 77,
+typically 1.  It is useful to print a message about the failure in this case.
+The @code{ASSERT} macro already does so.
address@hidden
+A test indicates "skip", that is, that most of its interesting functionality
+could not be performed, through a return code of 77.  A test should also
+print a message to stdout or stderr about the reason for skipping.
+For example:
address@hidden
+  fputs ("Skipping test: multithreading not enabled\n", stderr);
+  return 77;
address@hidden smallexample
+Such a message helps detecting bugs in the autoconf macros: A simple message
address@hidden: test-foo} does not sufficiently catch the attention of the user.
address@hidden itemize
+
+
address@hidden Incompatible changes
address@hidden Incompatible changes
+
+Incompatible changes to Gnulib modules should be mentioned in Gnulib's
address@hidden file.  Incompatible changes here mean that existing source code
+may not compile or work any more.
+
+We don't mean changes in the binary interface (ABI), since
address@hidden
address@hidden
+Gnulib code is used in source-code form.
address@hidden
+The user who distributes libraries that contain Gnulib code is supposed to
+bump the version number in the way described in the Libtool documentation
+before every release.
address@hidden enumerate
+
+
address@hidden Miscellaneous Notes
address@hidden Miscellaneous Notes
+
address@hidden
+* Out of memory handling::
+* Obsolete modules::
+* Library version handling::
+* Windows sockets::
+* Libtool and Windows::
+* License Texinfo sources::
+* Build robot for gnulib::
address@hidden menu
+
+
 @node Out of memory handling
 @section Out of memory handling