[libmicrohttpd] STF M1

[Christian Grothoff]

Dear all,

We're happy to announce reaching the first milestone for the
STF-funded MHD 2.0 project, which is completing the MHD HTTP header
and thus the design for the next generation API.  We have now
spent several months iterating between discussing, designing,
optimizing, editing, and testing details and are finally
happy with the result.

Key objectives for us were:

- simplify application code that uses MHD
- keep the API and the MHD code size small
- ensure the API is extensible
- enable use of different TLS backends, avoid backend-specific
  settings to use backends in the same way.
- make API work with HTTP 1.x, HTTP/2 and HTTP/3
- preserve or improve portability across platforms
- stay compatible to a wide range of C and C++ compilers

The main changes these objectives inspired are to:

- Split the MHD_AccessHandlerCallback functionality into
  various separate callbacks to keep the API simple for
  simple requests while allowing complex logic to be
  incrementally introduced via the new "struct MHD_Action".
  Main effects:
  * improved type safety
  * client code most likely to always need all arguments
    passed to callbacks, while keeping rarely used
    arguments available via the introspection API
  * harder or impossible to make calls at the wrong time
    or to forget to do key processing steps
  * clients more likely to avoid repeated URL dispatching
  * Easier to add commonly used features like a generic
    URL dispatcher
  * Improved modularity, easier to not compile in some
    features (actions) to minimize code size
- We changed how MHD is configured, providing a new
  strongly-typed but still extensible mechanism
  to set options, avoiding the use of 'varargs' for
  options while also not introducing new symbols for
  any kind of option. The new construction uses
  either inline functions or macros depending on the
  compiler to initialize a struct with a variant union;
  as a result, application developers get the experience
  of using well-typed functions, while no such functions
  actually exist in the library, keeping the code size
  minimal, especially if features are not even used :-).
- Unified the way how settings are used for daemon,
  connection and response objects.
- Removed the separation of options and flags and
  made it harder to pass inconsistent options
- improved terminology across the API, in particular by
  eliminating confusion between 'request' and 'connection',
  but also by introducing new 'nouns' such as 'session',
  'stream' and 'action', but also 'String' which returns a
  'String' that is both 0-terminated but ALSO has a
  known length (eliminating need for applications
  to call strlen() on all strings returned by MHD).
  Also changed the API to use more consistent
  prefixes for related functions by using
  "MHD_subject_verb_object" naming convention
- significantly simplified for application processing of
  client's upload. Removed the need for troublesome (for
  application) incremental processing (especially problematic
  for forms processing), while keeping automatic limits
  for memory allocations, preventing by design a wide range
  of remote attacks.
- Added unified and detailed introspection API for library,
  daemon, connection, stream and request objects.
  The API is designed in the same way for all object, simplifying
  use for the application. The new API is detailed and allow
  application to extract any required information in a simple
  way. Also separated "fixed" and "dynamic" properties of objects
  for letting compiler optimise application code better.
- Integrated HTTP status into the response object, as
  this is way more logical and we are aware of various
  implementations being forced to basically pass them
  around as a tuple.
- simplified API for common-case of one-shot responses by
  eliminating need for destroy response in most cases
- Improved portability by avoiding fixed types, like uint32_t,
  as they may not exist on some platforms. Instead use
  types like uint_fast32_t.  Avoided use of enums with very
  large bitmasks as 'int' could be just 16 bits on some platforms
  resulting in enum values higher than 65535 being silently dropped.
- Improved possibility of use of zero-copy style for parsing
  uploaded data, including of the PostProcessor parser while
  still allowing applications to do stream processing if data
  does not fit into main memory. This both simplifies usage
  in the common case where uploaded data is small, while also
  nicely supporting use-cases with large data streams.
- Made responses unmodifiable after first use. Modifiable responses
  cannot be thread-safe. However, MHD-generated headers (Date,
  Connection/Keep-Alive) are part of the *request* and do not count
  as part of the immutable "response" here.  Removed "footers" from
  responses.  With unmodifiable responses everything should be "headers".
  However, footers are supported as part of a *request* instead.
- Move response codes from MHD_HTTP_xxx namespace to MHD_HTTP_CODE_xxx
  namespace. This avoids potential clashes with other MHD constant names.
- Introduced various new "enums" especially for constants
  introduced in HTTP/2 where use of these constants can
  then avoid having to map between protocol numbers and
  strings (but applications may still also use the strings)
  This also includes better status codes returned from
  API calls to diagnose issues (no more just "YES/NO")
- Let application to use request methods as a enum, avoid repeated
  string comparison by sharing result of the already performed
  internal detection of the request method.  Keep ability to
  use non-standard HTTP requests methods if needed via use
  of introspection API.
- Introduced "MHD_APP_SOCKET_CNTX_TYPE" hack to allow
  applications to improve type-safety by overriding
  the type of "closure" arguments instead of using
  "void *" pointers.
- Significant re-design of the event loop APIs to simplify integrating
  MHD with external event loops while preserving O(1) processing cost.
- Added many annotations to help compiler determine invariants (if
  supported by the compiler), such as arguments not being NULL, etc.
- Removal of various legacy symbols only exported for API compatibility.


While these are lots of changes, we want to give you a first brief 
preview of
how this will impact client code using the library.  Here is an example 
of how
clients used to initialize the MHD daemon (from demo.c):

OLD>>
d = MHD_start_daemon (
    MHD_USE_AUTO | MHD_USE_INTERNAL_POLLING_THREAD
    | MHD_USE_ERROR_LOG,
    (uint16_t) port,
    NULL, NULL,
    &generate_page,
    NULL,
    MHD_OPTION_CONNECTION_MEMORY_LIMIT,
    (size_t) (256 * 1024),
#ifdef PRODUCTION
    MHD_OPTION_PER_IP_CONNECTION_LIMIT,
    (unsigned int) (64),
#endif
    MHD_OPTION_CONNECTION_TIMEOUT,
    (unsigned int) (120 /* seconds */),
    MHD_OPTION_THREAD_POOL_SIZE,
    (unsigned int) NUMBER_OF_THREADS,
    MHD_OPTION_NOTIFY_COMPLETED,
    &response_completed_callback,
    NULL,
    MHD_OPTION_END);
if (NULL == d)
  error();
<<

This was one big variadic mess, and getting any of the
types wrong for any of the options could result in
trouble, sometimes depending on the target platform.

With MHD 2.0, the same code will look like this:

NEW>>
d = MHD_daemon_create (&generate_page,
                       NULL);
if (MHD_SC_OK !=
    MHD_daemon_options_set (
      d,
      MHD_D_OPTION_BIND_PORT (MHD_AF_DUAL, port),
      MHD_D_OPTION_WM_WORKER_THREADS (NUMBER_OF_THREADS),
      MHD_D_OPTION_CONN_MEMORY_LIMIT (256*1024),
      MHD_D_OPTION_PER_IP_LIMIT (64),
      MHD_D_OPTION_DEFAULT_TIMEOUT (120),
      MHD_D_OPTION_NOTIFY_CONNECTION (&response_completed_callback,
                                      NULL)))
  error();
if (MHD_SC_OK !=
    MHD_daemon_start (d))
  error();
<<

Note that you can can call MHD_daemon_options_set() multiple times if 
you need to handle errors for individual options.  Thanks to extensive 
trickery on our part, the resulting type-safe code should *also* be 
almost as compact and fast as the previous version (mostly, this adds 
two additional library API calls, but in return you can get more precise 
status codes back).


While we thought hard about the new API and poured our experience into 
the re-design, we still might have overlooked something and thus value 
community feedback.

We thank Sovereign Tech Fund for funding this work.

Happy hacking!

Christian & Evgeny