Dear all,
We're happy to announce reaching the first milestone for the
STF-funded MHD 2.0 project, which is completing the MHD HTTP header
and thus the design for the next generation API. We have now
spent several months iterating between discussing, designing,
optimizing, editing, and testing details and are finally
happy with the result.
Key objectives for us were:
- simplify application code that uses MHD
- keep the API and the MHD code size small
- ensure the API is extensible
- enable use of different TLS backends, avoid backend-specific
settings to use backends in the same way.
- make API work with HTTP 1.x, HTTP/2 and HTTP/3
- preserve or improve portability across platforms
- stay compatible to a wide range of C and C++ compilers
The main changes these objectives inspired are to:
- Split the MHD_AccessHandlerCallback functionality into
various separate callbacks to keep the API simple for
simple requests while allowing complex logic to be
incrementally introduced via the new "struct MHD_Action".
Main effects:
* improved type safety
* client code most likely to always need all arguments
passed to callbacks, while keeping rarely used
arguments available via the introspection API
* harder or impossible to make calls at the wrong time
or to forget to do key processing steps
* clients more likely to avoid repeated URL dispatching
* Easier to add commonly used features like a generic
URL dispatcher
* Improved modularity, easier to not compile in some
features (actions) to minimize code size
- We changed how MHD is configured, providing a new
strongly-typed but still extensible mechanism
to set options, avoiding the use of 'varargs' for
options while also not introducing new symbols for
any kind of option. The new construction uses
either inline functions or macros depending on the
compiler to initialize a struct with a variant union;
as a result, application developers get the experience
of using well-typed functions, while no such functions
actually exist in the library, keeping the code size
minimal, especially if features are not even used :-).
- Unified the way how settings are used for daemon,
connection and response objects.
- Removed the separation of options and flags and
made it harder to pass inconsistent options
- improved terminology across the API, in particular by
eliminating confusion between 'request' and 'connection',
but also by introducing new 'nouns' such as 'session',
'stream' and 'action', but also 'String' which returns a
'String' that is both 0-terminated but ALSO has a
known length (eliminating need for applications
to call strlen() on all strings returned by MHD).
Also changed the API to use more consistent
prefixes for related functions by using
"MHD_subject_verb_object" naming convention
- significantly simplified for application processing of
client's upload. Removed the need for troublesome (for
application) incremental processing (especially problematic
for forms processing), while keeping automatic limits
for memory allocations, preventing by design a wide range
of remote attacks.
- Added unified and detailed introspection API for library,
daemon, connection, stream and request objects.
The API is designed in the same way for all object, simplifying
use for the application. The new API is detailed and allow
application to extract any required information in a simple
way. Also separated "fixed" and "dynamic" properties of objects
for letting compiler optimise application code better.
- Integrated HTTP status into the response object, as
this is way more logical and we are aware of various
implementations being forced to basically pass them
around as a tuple.
- simplified API for common-case of one-shot responses by
eliminating need for destroy response in most cases
- Improved portability by avoiding fixed types, like uint32_t,
as they may not exist on some platforms. Instead use
types like uint_fast32_t. Avoided use of enums with very
large bitmasks as 'int' could be just 16 bits on some platforms
resulting in enum values higher than 65535 being silently dropped.
- Improved possibility of use of zero-copy style for parsing
uploaded data, including of the PostProcessor parser while
still allowing applications to do stream processing if data
does not fit into main memory. This both simplifies usage
in the common case where uploaded data is small, while also
nicely supporting use-cases with large data streams.
- Made responses unmodifiable after first use. Modifiable responses
cannot be thread-safe. However, MHD-generated headers (Date,
Connection/Keep-Alive) are part of the *request* and do not count
as part of the immutable "response" here. Removed "footers" from
responses. With unmodifiable responses everything should be "headers".
However, footers are supported as part of a *request* instead.
- Move response codes from MHD_HTTP_xxx namespace to MHD_HTTP_CODE_xxx
namespace. This avoids potential clashes with other MHD constant names.
- Introduced various new "enums" especially for constants
introduced in HTTP/2 where use of these constants can
then avoid having to map between protocol numbers and
strings (but applications may still also use the strings)
This also includes better status codes returned from
API calls to diagnose issues (no more just "YES/NO")
- Let application to use request methods as a enum, avoid repeated
string comparison by sharing result of the already performed
internal detection of the request method. Keep ability to
use non-standard HTTP requests methods if needed via use
of introspection API.
- Introduced "MHD_APP_SOCKET_CNTX_TYPE" hack to allow
applications to improve type-safety by overriding
the type of "closure" arguments instead of using
"void *" pointers.
- Significant re-design of the event loop APIs to simplify integrating
MHD with external event loops while preserving O(1) processing cost.
- Added many annotations to help compiler determine invariants (if
supported by the compiler), such as arguments not being NULL, etc.
- Removal of various legacy symbols only exported for API compatibility.
While these are lots of changes, we want to give you a first brief
preview of
how this will impact client code using the library. Here is an example
of how
clients used to initialize the MHD daemon (from demo.c):
OLD>>
d = MHD_start_daemon (
MHD_USE_AUTO | MHD_USE_INTERNAL_POLLING_THREAD
| MHD_USE_ERROR_LOG,
(uint16_t) port,
NULL, NULL,
&generate_page,
NULL,
MHD_OPTION_CONNECTION_MEMORY_LIMIT,
(size_t) (256 * 1024),
#ifdef PRODUCTION
MHD_OPTION_PER_IP_CONNECTION_LIMIT,
(unsigned int) (64),
#endif
MHD_OPTION_CONNECTION_TIMEOUT,
(unsigned int) (120 /* seconds */),
MHD_OPTION_THREAD_POOL_SIZE,
(unsigned int) NUMBER_OF_THREADS,
MHD_OPTION_NOTIFY_COMPLETED,
&response_completed_callback,
NULL,
MHD_OPTION_END);
if (NULL == d)
error();
<<
This was one big variadic mess, and getting any of the
types wrong for any of the options could result in
trouble, sometimes depending on the target platform.
With MHD 2.0, the same code will look like this:
NEW>>
d = MHD_daemon_create (&generate_page,
NULL);
if (MHD_SC_OK !=
MHD_daemon_options_set (
d,
MHD_D_OPTION_BIND_PORT (MHD_AF_DUAL, port),
MHD_D_OPTION_WM_WORKER_THREADS (NUMBER_OF_THREADS),
MHD_D_OPTION_CONN_MEMORY_LIMIT (256*1024),
MHD_D_OPTION_PER_IP_LIMIT (64),
MHD_D_OPTION_DEFAULT_TIMEOUT (120),
MHD_D_OPTION_NOTIFY_CONNECTION (&response_completed_callback,
NULL)))
error();
if (MHD_SC_OK !=
MHD_daemon_start (d))
error();
<<
Note that you can can call MHD_daemon_options_set() multiple times if
you need to handle errors for individual options. Thanks to extensive
trickery on our part, the resulting type-safe code should *also* be
almost as compact and fast as the previous version (mostly, this adds
two additional library API calls, but in return you can get more precise
status codes back).
While we thought hard about the new API and poured our experience into
the re-design, we still might have overlooked something and thus value
community feedback.
We thank Sovereign Tech Fund for funding this work.
Happy hacking!
Christian & Evgeny
OpenPGP_signature.asc