Re: [ft-devel] FreeType Amalgamation

[Antoine Leca]

[ If your reader did not already told you: warning, long post! ]

Vinnie wrote:
> I'm interested in building an "amalgamated" version of FreeType.
If I understand correctly your quest, Freetype 1 used to have such an
option ("single object file", a.k.a freetype.[co]); in fact it was/is
even the base option!
For Freetype 2 (b. 1999), David went away from this scheme, in my eyes
for the benefit of a much increased modularity; of course, for
"mainstream" compilations where all the stuff ends being part of a
shared library or DLL built by mostly automatic tools, there is not much
of a difference at the end; where it IS really different is for others
users of the library, particularly those who compile for embedded
targets, where the space is always a concern.

I do not know if it still useful. Voices?

>  Specifically, I would like to use an in-house tool to combine all the 
> FreeType headers and source files into a set consisting of a single .h and .c 
> (or a couple of .c files if they are too large). This amalgamation would be 
> much easier to use - instead of requiring a bulky Makefile or IDE-specific 
> project file, one could just add the .c to an existing project.
Not sure I understand all the point, but it seems to me an issue at
software engineering here.

About the .c first:

In the beginning, (static) libraries where invented in order to share
code between projects; to maximize the space efficiency, the pieces in
the libraries were cut in smallest chunks; and linkers were invoked to
select only the relevant parts into the final binary. This model also
proved very good at paralleling tasks, where the libraries are developed
by some team, with a clearly defined interface, and used by several
independent programs created by different teams.

Then came dynamic linking and shared libraries, and in this view the
cutting process became essentially obsolete: all the code is present on
secondary storage (the shared object), and the selection of the
necessary pieces is done as-needed, mainly by the demand-paging
functions of the operating system through the dynamic linker.

Are you saying the Freetype project should move away from that
cutting-in-pieces process?

Or is it just about merging a big number of independently kept .c and
internal .h into one physical file (like sqlite3 is doing)?
My guess is such a work should pretty easy to do for the .c, but more
complex for the headers internal to the library; it looks like to me as
standard 4.3BSD unifdef(1), expanded to deal with #include as well;
perhaps such a tool already exist. Another possibility is a sed script
along the lines of
    /^#include FT_FREETYPE_H/{
        d; r path/to/already/merged/freetype/freeetype.h; }
    /^#include FT_ERRORS_H/{
        d; r path/to/already/merged/freetype/fterrors.h; }
    /^#include FT_INTERNAL_OBJECTS_H/{
        d; r path/to/already/merged/freetype/internal/ftobjs.h; }
I have no idea if there are cyclic dependencies, however...

Then, about the .h:
First, I understand that from the external point of view, it is already
looks like
    #include <ft2build.h>
    #include FT_FREETYPE_H
Granted, they are two files (ft2build.h is really just a proxy for
config/ftheader.h, you could substitute one for the other) not one, but
it really look like to me what you are asking for.

If the point is about the headers for the optional parts (#include
FT_BITMAP_H, FT_CACHE_H, FT_GLYPH_H, FT_TRUETYPE_IDS_H, etc.) this is
really the same point as above: if modularity is dropped, all those
files will be integrated into FT_FREETYPE_H; again very much like as it
was for Freetype 1, or as it is for <windows.h> or <u.h> (Plan 9). An
important resulting difference here used to be compilation times --
which issue was "solved", particularly for Windows or Carbon, with
pre-compiled headers, a "feature" which is at odds with Freetype build
system... but I digress ;-)

>  For my purposes, there is value in being able to distribute a project that 
> uses FreeType, but has no external dependencies (i.e. someone does not need 
> to visit a separate source code repository to obtain freetype).
Freetype is basically free-standing, it does not need anything beyond a
C compiler and the most standard C functions; so a full copy of Freetype
tree will fit the purpose, it is just less practical than a couple of files.

The real issue with external dependencies is about upgrading. If we have
a look over the last 10 years of Freetype history as condensed into
docs/CHANGES, most version bumps are either bugfixes for some previously
introduced changes, or vulnerabilities fixes. Both are good reasons to
upgrade as soon as possible the version of Freetype in use. Having
Freetype provided in form of a few big flat files does not change
anything to the issue, upgrading is still recommended practice.
What is important on this respect OTOH, is that the amalgamationprocess
be realised by the Freetype project, and released through it: that way
any recommended upgrade could be made available with lowest delay. This
is in opposition to a in-house flattening
of the Freetype tree integrated into a bigger project, where any further
upgrade is then dependent on that developer willingness and availability...

>  True, this could be done by creating a deep clone of the entire FreeType 
> source tree but this is a bulky solution.
I am not sure this is as bulky as you seem to imply.
The issue as I see it is about how to build the code. In present days,
there are already several ways to do so:
- the fully automated way through autoconf and libtool
- the mostly automated way through GNU-make files (after initial config)
- the mostly automated way through Jam
- the pre-canned way of already provided "IDE projects" files
- the fully manual way (INSTALL.ANY part I)
- the flat-directory way (INSTALL.ANY part II)

To be honest I must agree it is a long time since I tried either of the
last two ways, so perhaps the necessary pre-configuration for them
(mainly setting up the include files) could be a bit complex nowadays.

I understand your proposal as adding a new way, based on a different
distribution consisting of a very tiny set of big files merging the
content of the "regular" library (please correct me if I am wrong.)
Creating those big files would be part of the distribution process, i.e.
done by scripts and perhaps helpers like improved unifdef(1); so it
would result indeed in a "deep clone" of the library; given the already
existing process (Python-generated documentation, autostuff) I do not
see it as particularly bulky.

Something I do see as bulky though is about bug reporting, more exactly
about patch contribution: any fix provided against such big files cannot
be applied to the "regular" source tree, it has first to be carved out
of the generated source and be re-applied in the correct place; this
should not be a real problem for 1-line patches, but it certainly would
prevent things more ambitious, for example a new platform, to be
successfully developed off those big files (I am sure this issue is
known to any amalgamationproject.)

> Unfortunately, creating an amalgamated version of FreeType would require some 
> changes to the way that source files are organized. Three issues come up:
>
> 1) Some FreeType #include statements use angle brackets instead of double 
> quotes. e.g. #include <ft2build.h> Why?
It is the way headers are intended to be used according to some readings
of the ANSI/ISO C standard: #include "xxx" is to be used for purely
local stuff, internal to the current module, while #include <yyy> is for
externally-visible stuff.
This model fits well for two-tiers developments, but less well for
three-tiers ones, like modularized libraries: it is missing one level.
Freetype chose to keep the first form for its own internal modules,
since it is the way most C compilers interpret the syntax, and also
since it is the most flexible way to distribute the library code. So the
<yyy> syntax serves for both the headers of the Freetype modules within
the library, and the users of the library itself.

Old timers will remember we were badly hurt by this confusion: up to the
2.1.x branch, several packagers and library users did use the internal
headers of the Freetype library to access directly its internals,
presumably because it held the <yyy> syntax (albeit it really read
<freetype/internal/zzz>); or perhaps it was not different enough from
the regular, to-be-used-externally headers like FT_FREETYPE_H.

Due to the way C compilers handle #include "" (they search the file
under the local directory, then in accordance to the standard, they
defer to the <> syntax), replacing <yyy> with "yyy" in "user"code is
unlikely to have adverse effect, or even noticeable effect; if it does,
it is probably a bug ;-)

> 2) Some FreeType #include statements use macros, like "#include FT_GLYPH_H". 
> This complicates automated tools.
Sure, it makes them slightly more complex: they have to take care of the
whole ANSI C syntax for headers. On the other hand, it simplifies a lot
(IMHO) the reading of the source, there is less need for convoluted
#ifdef HAVE_BETTY_BOOP or #define WANT_MARYLIN and vice-versa. It also
allows installation of the library which fits user needs, such as people
without hierarchical include tree, or without access or control to
either -I nor /usr/include (just edit config/ftheader.h to your taste!)

As Werner said, it is here for 11 years, it is reasonably debugged now,
users are (mostly) used to it, it even got some acceptance externally
;-) I think any new way to build the library will have to make the
effort to integrate with that syntax; so is life.

> 3) Depending on the platform, there might be a different ftconfig.h and/or 
> ftmodule.h needed (possibly other sources too)
This OTOH is about library customisation (or tailoring.)
In fact, a customisation of the library is represented by:
* a few Makefile or similar "project howtobuild" files; it reduces to
config.mk and modules.cfg with the provided GNU make infrastructure; or
even nothing with Jam ;-)
    and
* the ftconfig.h header, optionally expanded to the other config/*.h
headers, and perhaps ft2build.h tailored to the build system.
It appears to me a tiny set, particularly when compared with the
potential complexity of Freetype customisation, and variety of build
systems.

And yes, you want Freetype to be customisable, and not only depending on
the underlying platform: a clear issue was the use or not of the
patented algorithm; another issue is the use or not and the preferences
of auto-hinting, depending of the scripts you are rendering (for example
only Latin, or only Chinese); there is no such thing as one-fits-all here.

> 3) A new ftconfig.h should be created [...]
> These problems could be overcome a different way, by writing a custom script 
> that does most of this work.
Isn't already autoconf (well, really configure) task on Unix-like targets?
Autoconf and libtool are not Freetype specific (our local
bug-workarounds there are long gone, fortunately); and
builds/unix/configure.raw (autoconf source with a strange extension),
the Freetype-specific part of it, is 700 lines of documented
instructions. It handles silently many *nix-like configurations,
including cross-compilations, Apple or Msys.
And yes, it requires quite a bit of maintenance.

I understand this is the price to pay for fully automatic,
out-of-the-box use of the Freetype package, while providing as
alternative all the above-mentioned ways to customise and build the library.


I hope this long comment will help you in the design of your future
contribution!

Antoine