Dangers using data from automatically generated makefiles

[gk]

Warning - this is a long post.
I finally understand thoroughly how make processes makefiles and a problem 
that may arise and drive you batty trying to figure it out.

MORAL of the story:

Never use a data from automatically generated makefiles as input to the 
automated generation of other makefiles!

Example:
%.phpx are php source files which produce output files using php 
command-line parser
%.html files are generated from %.html.phpx files
%.phpx.mkdo dependency makefiles are generated from %.phpx files
%.mko makefiles are generated from %.mks meta-makefiles

XMakefile.mks exports the following variable:
export XMAKE_LANG:=en

XMakefile.mk can be considered the main makefile (actually there are 
others, but they are not important to this example) which includes other 
makefiles in this order:
# .mko files, auto-generated from .mks files
-include $(XM_customAutoMakefiles)
# .mkdo files, auto-generated per XMExtension rules
-include $(XM_dependencyAutoMakefiles)


This is the trace of how make behaves when you try to build target:

xmake index.html

- INITIAL STATE

XMakefile.mks:
        export XMAKE_LANG:=es

index.html.phpx:
        includes navbar.html

navbar.html.phpx:
        includes file:
                'navbar_item_home'.$_ENV['XMAKE_LANG'];

navbar_item_home.en:
        contains the text: 'home'

navbar_item_home.es:
        contains the text: 'origen'

THESE FILES DO NOT EXIST YET:
        XMakefile.mko
        index.html.phpx.mkdo
        navbar.html.phpx.mkdo
        index.html
        navbar.html

- INITIAL BUILD

0. Makefile.mk is read, which FAILS to include non-existent makefiles in 
this order:
        include XMakefile.mko
        include index.html.phpx.mkdo navbar.html.phpx.mkdo

GENERATE included makefiles in REVERSE ORDER READ:

1. XMakefile.mko is generated as PREREQUISITE of navbar.html.phpx.mkdo

2. navbar.html.phpx.mkdo is generated
        NOTE: $_ENV['XMAKE_LANG'] is not set in makefile environment yet 
since XMakefile.mko is not yet included
                The default value 'en' for XMAKE_LANG will be used
        navbar.html : navbar.html.phpx.mkdo
        navbar.html.phpx.mkdo : navbar.html.phpx navbar_item_home.en 
XMakefile.mko

3. index.html.phpx.mkdo is generated
        index.html : index.html.phpx.mkdo
        index.html.phpx.mkdo : index.html.phpx navbar.html XMakefile.mko


RE-LOAD all makefiles, after generation of included makefiles
        NOTE: $XMAKE_LANG is now set to 'es' in makefile environment

GENERATE PREREQUISITES and TARGETS, IN ORDER

4. PREREQUISITE navbar.html is generated, using navbar_item_home.es:
        navbar = 'origen'

5. TARGET index.html is generated, using $_ENV['XMAKE_LANG'], including 
navbar.html
        lang = 'es'
        navbar = 'origen'

- EDIT XMakefile.mks:

        export XMAKE_LANG:=en

- SECONDARY BUILD

0. Makefile.mk is read, which includes makefiles in this order:
        include XMakefile.mko (old out of date version)
                XMAKE_LANG:=es
        include index.html.phpx.mkdo (old out of date version)
                index.html : index.html.phpx.mkdo
                index.html.phpx.mkdo : ... navbar.html XMakefile.mko
        include navbar.html.phpx.mkdo (old out of date version)
                navbar.html : navbar.html.phpx.mkdo
                navbar.html.phpx.mkdo : ... navbar_item_home.es XMakefile.mko

GENERATE included makefiles in REVERSE ORDER READ:

1. XMakefile.mko is updated as PREREQUISITE of navbar.html.phpx.mkdo
        NOTE: makefile is updated but NOT RE-LOADED!
        export XMAKE_LANG:=es
                OLD value is still in the current environment!

2. navbar.html.phpx.mkdo is updated  as PREREQUISITE of navbar.html (which 
is prerequisite of index.html.phpx.mkdo)

        navbar.html.phpx generates .mkdo file using OLD value:
                 $_ENV['XMAKE_LANG'] == 'es'
                navbar.html : navbar.html.phpx.mkdo
                navbar.html.phpx.mkdo : ... navbar_item_home.es XMakefile.mko

3. navbar.html is updated as PREREQUISITE of index.html
        Uses OLD $_ENV['XMAKE_LANG'] == 'es'
        => navbar_item_home.es
                navbar = 'origen'

        THIS IS THE CRUX OF THE PROBLEM:
                navbar.html is only generated now because it is an 
intermediate of another makefile:
                        index.html.phpx.mkdo
                HOWEVER:
                        navbar.html is also a TARGET which should be build 
in a correct up to date makefile context
                PROBLEM:
                        since navbar.html was a PREREQUISITE of another 
included makefile,
                        it will NEVER get built in the correct, up to date 
makefile environment!

4. index.html.phpx.mkdo is updated
        Uses OLD $_ENV['XMAKE_LANG'] == 'es'
        (no significant consequence in this example)

5. RE-LOAD makefiles, generate TARGETS:

Re-executing: /usr/local/bin/make -f 
/usr/local/apache/htdocs/common/xmake/config/XMakefile.mkh -r -d
GNU Make 3.80

...

No need to remake target 
`/usr/local/apache/htdocs/gembroker/www/dev/XMake/common/navbar.html'.
   THIS IS AN ERROR - FILE SHOULD BE OUT OF DATE!

No need to remake target 
`/usr/local/apache/htdocs/gembroker/www/dev/XMake/common/navbar.html.phpx.mkdo'.
   THIS IS AN ERROR - FILE SHOULD BE OUT OF DATE!

QUESTION:

        What is the MORAL of this story?
        I think it is this:
                Never use a data from an automatically generated makefile 
as input to the automated generation of other makefiles!

SOLUTION:
        export variables either:
                1. From OUTSIDE the makefile environment; i.e., from the shell
                OR
                2. From a MANUALLY generated makefile (such as 
XMakefile.mko) that will never be 'out of date'!


- Greg Keraunen
http://www.xmake.org