Re: [PATCH 28/30] docs: update build system docs

[Marc-André Lureau]

Hi

On Fri, Dec 9, 2022 at 3:28 PM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> Now that configure is only doing compiler and host setup, adjust the
> relevant documentation.  It is also possible to build emulators with
> ninja directly if one is so inclined.
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

better if it didn't break the build :)
qemu/docs/devel/build-system.rst:446:Inline literal start-string
without end-string.

> ---
>  docs/devel/build-system.rst | 275 +++++++++++++++++++++---------------
>  1 file changed, 164 insertions(+), 111 deletions(-)
>
> diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst
> index 4a733fc0a747..a25070fcaac8 100644
> --- a/docs/devel/build-system.rst
> +++ b/docs/devel/build-system.rst
> @@ -4,30 +4,14 @@ The QEMU build system architecture
>
>  This document aims to help developers understand the architecture of the
>  QEMU build system. As with projects using GNU autotools, the QEMU build
> -system has two stages, first the developer runs the "configure" script
> +system has two stages; first the developer runs the "configure" script
>  to determine the local build environment characteristics, then they run
> -"make" to build the project. There is about where the similarities with
> +"make" to build the project.  This is about where the similarities with
>  GNU autotools end, so try to forget what you know about them.
>
> +The two general ways to perform a build are as follows:
>
> -Stage 1: configure
> -==================
> -
> -The QEMU configure script is written directly in shell, and should be
> -compatible with any POSIX shell, hence it uses #!/bin/sh. An important
> -implication of this is that it is important to avoid using bash-isms on
> -development platforms where bash is the primary host.
> -
> -In contrast to autoconf scripts, QEMU's configure is expected to be
> -silent while it is checking for features. It will only display output
> -when an error occurs, or to show the final feature enablement summary
> -on completion.
> -
> -Because QEMU uses the Meson build system under the hood, only VPATH
> -builds are supported.  There are two general ways to invoke configure &
> -perform a build:
> -
> - - VPATH, build artifacts outside of QEMU source tree entirely::
> + - build artifacts outside of QEMU source tree entirely::
>
>       cd ../
>       mkdir build
> @@ -35,80 +19,114 @@ perform a build:
>       ../qemu/configure
>       make
>
> - - VPATH, build artifacts in a subdir of QEMU source tree::
> + - build artifacts in a subdir of QEMU source tree::
>
>       mkdir build
>       cd build
>       ../configure
>       make
>
> -The configure script automatically recognizes
> -command line options for which a same-named Meson option exists;
> -dashes in the command line are replaced with underscores.
> +Most of the actual build process uses Meson under the hood, therefore
> +build artifacts cannot be placed in the source tree itself.
>
> -Many checks on the compilation environment are still found in configure
> -rather than ``meson.build``, but new checks should be added directly to
> -``meson.build``.
>
> -Patches are also welcome to move existing checks from the configure
> -phase to ``meson.build``.  When doing so, ensure that ``meson.build`` does
> -not use anymore the keys that you have removed from ``config-host.mak``.
> -Typically these will be replaced in ``meson.build`` by boolean variables,
> -``get_option('optname')`` invocations, or ``dep.found()`` expressions.
> -In general, the remaining checks have little or no interdependencies,
> -so they can be moved one by one.
> +Stage 1: configure
> +==================
>
> -Helper functions
> -----------------
> +The configure script has four tasks:
>
> -The configure script provides a variety of helper functions to assist
> -developers in checking for system features:
> + - detect the host architecture
>
> -``do_cc $ARGS...``
> -   Attempt to run the system C compiler passing it $ARGS...
> + - list the targets for which to build emulators; the list of
> +   targets also affects which firmware binaries and tests to build
>
> -``do_cxx $ARGS...``
> -   Attempt to run the system C++ compiler passing it $ARGS...
> + - find the compilers (native and cross) used to build executables,
> +   firmware and tests.  The results are written as either Makefile
> +   fragments (``config-host.mak``) or a Meson machine file
> +   (``config-meson.cross``)
>
> -``compile_object $CFLAGS``
> -   Attempt to compile a test program with the system C compiler using
> -   $CFLAGS. The test program must have been previously written to a file
> -   called $TMPC.  The replacement in Meson is the compiler object ``cc``,
> -   which has methods such as ``cc.compiles()``,
> -   ``cc.check_header()``, ``cc.has_function()``.
> + - invoke Meson to perform the actual configuration step for the
> +   emulator build
>
> -``compile_prog $CFLAGS $LDFLAGS``
> -   Attempt to compile a test program with the system C compiler using
> -   $CFLAGS and link it with the system linker using $LDFLAGS. The test
> -   program must have been previously written to a file called $TMPC.
> -   The replacement in Meson is ``cc.find_library()`` and ``cc.links()``.
> +The configure script automatically recognizes command line options for
> +which a same-named Meson option exists; dashes in the command line are
> +replaced with underscores.
> +
> +Almost all QEMU developers that need to modify the build system will
> +only be concerned with Meson, and therefore can skip the rest of this
> +section.
> +
> +
> +Modifying ``configure``
> +-----------------------
> +
> +``configure`` is a shell script; it uses ``#!/bin/sh`` and therefore
> +should be compatible with any POSIX shell. It is important to avoid
> +using bash-isms to avoid breaking development platforms where bash is
> +the primary host.
> +
> +The configure script provides a variety of functions to help writing
> +portable shell code and providing consistent behavior across architectures
> +and operating systems:
> +
> +``error_exit $MESSAGE $MORE...``
> +   Print $MESSAGE to stderr, followed by $MORE... and then exit from the
> +   configure script with non-zero status.
>
>  ``has $COMMAND``
>     Determine if $COMMAND exists in the current environment, either as a
>     shell builtin, or executable binary, returning 0 on success.  The
>     replacement in Meson is ``find_program()``.
>
> +``probe_target_compiler $TARGET``
> +  Detect a cross compiler and cross tools for the QEMU target $TARGET (e.g.,
> +  ``$CPU-softmmu``, ``$CPU-linux-user``, ``$CPU-bsd-user``).  If a working
> +  compiler is present, return success and set variables ``$target_cc``,
> +  ``$target_ar``, etc. to non-empty values.
> +
> +``write_target_makefile``
> +  Write a Makefile fragment to stdout, exposing the result of the most
> +  ``probe_target_compiler`` call as the usual Make variables (``CC``,
> +  ``AR``, ``LD``, etc.).
> +
> +
> +Configure does not generally perform tests for compiler options beyond
> +basic checks to detect the host platform and ensure the compiler is
> +functioning.  These are performed using a few more helper functions:
> +
> +``compile_object $CFLAGS``
> +   Attempt to compile a test program with the system C compiler using
> +   $CFLAGS. The test program must have been previously written to a file
> +   called $TMPC.
> +
> +``compile_prog $CFLAGS $LDFLAGS``
> +   Attempt to compile a test program with the system C compiler using
> +   $CFLAGS and link it with the system linker using $LDFLAGS. The test
> +   program must have been previously written to a file called $TMPC.
> +
>  ``check_define $NAME``
> -   Determine if the macro $NAME is defined by the system C compiler
> +   Determine if the macro $NAME is defined by the system C compiler.
> +
> +``do_compiler $CC $ARGS...``
> +   Attempt to run the C compiler $CC, passing it $ARGS...  This function
> +   does not use flags passed via options such as ``--extra-cflags``, and
> +   therefore can be used to check for cross compilers.  However, most
> +   such checks are done at ``make`` time instead (see for example the
> +   ``cc-option`` macro in ``pc-bios/option-rom/Makefile``).
>
>  ``write_c_skeleton``
>     Write a minimal C program main() function to the temporary file
> -   indicated by $TMPC
> -
> -``error_exit $MESSAGE $MORE...``
> -   Print $MESSAGE to stderr, followed by $MORE... and then exit from the
> -   configure script with non-zero status
> +   indicated by $TMPC.
>
>
>  Stage 2: Meson
>  ==============
>
> -The Meson build system is currently used to describe the build
> -process for:
> +The Meson build system describes the build and install process for:
>
>  1) executables, which include:
>
> -   - Tools - ``qemu-img``, ``qemu-nbd``, ``qga`` (guest agent), etc
> +   - Tools - ``qemu-img``, ``qemu-nbd``, ``qemu-ga`` (guest agent), etc
>
>     - System emulators - ``qemu-system-$ARCH``
>
> @@ -118,7 +136,8 @@ process for:
>
>  2) documentation
>
> -3) ROMs, which can be either installed as binary blobs or compiled
> +3) ROMs, whether provided as binary blobs in the QEMU distributions
> +   or cross compiled under the direction of the configure script
>
>  4) other data files, such as icons or desktop files
>
> @@ -273,8 +292,7 @@ system/userspace emulation target
>  Adding checks
>  -------------
>
> -New checks should be added to Meson.  Compiler checks can be as simple as
> -the following::
> +Compiler checks can be as simple as the following::
>
>    config_host_data.set('HAVE_BTRFS_H', cc.has_header('linux/btrfs.h'))
>
> @@ -351,22 +369,30 @@ script, which may point to something other than the 
> first python3
>  binary on the path.
>
>
> -Stage 3: makefiles
> -==================
> +Stage 3: Make
> +=============
>
> -The use of GNU make is required with the QEMU build system.
> +The next step in building QEMU is to invoke make.  GNU Make is required
> +to build QEMU, and may be installed as ``gmake`` on some hosts.
>
> -The output of Meson is a build.ninja file, which is used with the Ninja
> -build system.  QEMU uses a different approach, where Makefile rules are
> -synthesized from the build.ninja file.  The main Makefile includes these
> -rules and wraps them so that e.g. submodules are built before QEMU.
> -The resulting build system is largely non-recursive in nature, in
> -contrast to common practices seen with automake.
> +The output of Meson is a ``build.ninja`` file, which is used with the
> +Ninja build tool.  However, QEMU's build comprises other components than
> +just the emulators (namely firmware and the tests in ``tests/tcg``) which
> +need different cross compilers.  The QEMU Makefile wraps both Ninja and
> +the smaller build systems for firmware and tests; it also takes care of
> +running ``configure`` again when the script changes.  Apart from invoking
> +these sub-Makefiles, the resulting build is largely non-recursive.
>
> -Tests are also ran by the Makefile with the traditional ``make check``
> -phony target, while benchmarks are run with ``make bench``.  Meson test
> -suites such as ``unit`` can be ran with ``make check-unit`` too.  It is also
> -possible to run tests defined in meson.build with ``meson test``.
> +Tests, whether defined in ``meson.build`` or not, are also ran by the
> +Makefile with the traditional ``make check`` phony target, while benchmarks
> +are run with ``make bench``.  Meson test suites such as ``unit`` can be ran
> +with ``make check-unit``, and ``make check-tcg`` builds and runs "non-Meson"
> +tests for all targets.
> +
> +If desired, it is also possible to use ``ninja`` and ``meson test``,
> +respectively to build emulators and run tests defined in meson.build.
> +The main difference is that ``make`` needs the ``-jN`` flag in order to
> +enable parallel builds or tests.
>
>  Useful make targets
>  -------------------
> @@ -378,6 +404,7 @@ Useful make targets
>    Print the value of the variable VAR. Useful for debugging the build
>    system.
>
> +
>  Important files for the build system
>  ====================================
>
> @@ -391,8 +418,7 @@ number of dynamically created files listed later.
>  ``Makefile``
>    The main entry point used when invoking make to build all the components
>    of QEMU. The default 'all' target will naturally result in the build of
> -  every component. Makefile takes care of recursively building submodules
> -  directly via a non-recursive set of rules.
> +  every component.
>
>  ``*/meson.build``
>    The meson.build file in the root directory is the main entry point for the
> @@ -401,59 +427,86 @@ number of dynamically created files listed later.
>    other meson.build files spread throughout the QEMU source tree.
>
>  ``tests/Makefile.include``
> -  Rules for external test harnesses. These include the TCG tests,
> -  ``qemu-iotests`` and the Avocado-based integration tests.
> +  Rules for external test harnesses. These include the TCG tests
> +  and the Avocado-based integration tests.
>
>  ``tests/docker/Makefile.include``
> -  Rules for Docker tests. Like tests/Makefile, this file is included
> -  directly by the top level Makefile, anything defined in this file will
> -  influence the entire build system.
> +  Rules for Docker tests. Like ``tests/Makefile.include``, this file is
> +  included directly by the top level Makefile, anything defined in this
> +  file will influence the entire build system.
>
>  ``tests/vm/Makefile.include``
> -  Rules for VM-based tests. Like tests/Makefile, this file is included
> -  directly by the top level Makefile, anything defined in this file will
> -  influence the entire build system.
> +  Rules for VM-based tests. Like ``tests/Makefile.include``, this file is
> +  included directly by the top level Makefile, anything defined in this
> +  file will influence the entire build system.
>
>  Dynamically created files
>  -------------------------
>
> -The following files are generated dynamically by configure in order to
> -control the behaviour of the statically defined makefiles. This avoids
> -the need for QEMU makefiles to go through any pre-processing as seen
> -with autotools, where Makefile.am generates Makefile.in which generates
> -Makefile.
> +The following files are generated at run-time in order to control the
> +behaviour of the Makefiles. This avoids the need for QEMU makefiles to
> +go through any pre-processing as seen with autotools, where configure
> +generates ``Makefile`` from ``Makefile.in`.
>
>  Built by configure:
>
>  ``config-host.mak``
>    When configure has determined the characteristics of the build host it
> -  will write a long list of variables to config-host.mak file. This
> -  provides the various install directories, compiler / linker flags and a
> +  will write them to this file for use in ``Makefile`` and to a smaller
> +  extent ``meson.build``. These include the paths to various tools and a
>    variety of ``CONFIG_*`` variables related to optionally enabled features.
> -  This is imported by the top level Makefile and meson.build in order to
> -  tailor the build output.
>
> -  config-host.mak is also used as a dependency checking mechanism. If make
> +  ``config-host.mak`` is also used as a dependency checking mechanism. If 
> make
>    sees that the modification timestamp on configure is newer than that on
> -  config-host.mak, then configure will be re-run.
> +  ``config-host.mak``, then configure will be re-run.
>
> -  The variables defined here are those which are applicable to all QEMU
> -  build outputs. Variables which are potentially different for each
> -  emulator target are defined by the next file...
> +  The variables defined here apply to all QEMU
> +  build outputs.
> +
> +``config-meson.cross``
> +
> +  A Meson "cross file" (or native file) used to communicate the paths to
> +  the toolchain and other configuration options.
> +
> +``config.status``
> +
> +  A small shell script that will invoke configure again with the same
> +  environment variables that were set during the first run.  It's used to
> +  rerun configure after changes to the source code, but it can also be
> +  inspected manually to check the contents of the environment.
> +
> +``Makefile.prereqs``
> +
> +  A set of Makefile dependencies that order the build and execution of
> +  firmware and tests after the container images and emulators that they
> +  need.
> +
> +``pc-bios/*/config.mak``, ``tests/tcg/config-host.mak``, 
> ``tests/tcg/*/config-target.mak``
> +
> +  Configuration variables used to build the firmware and TCG tests,
> +  including paths to cross compilation toolchains.
>
>
>  Built by Meson:
>
> -``${TARGET-NAME}-config-devices.mak``
> -  TARGET-NAME is again the name of a system or userspace emulator. The
> -  config-devices.mak file is automatically generated by make using the
> -  scripts/make_device_config.sh program, feeding it the
> -  default-configs/$TARGET-NAME file as input.
> +``config-host.h``
> +  Used by C code to determine the properties of the build environment
> +  and the set of enabled features for the entire build.
>
> -``config-host.h``, ``$TARGET_NAME-config-target.h``, 
> ``$TARGET_NAME-config-devices.h``
> -  These files are used by source code to determine what features are
> -  enabled.  They are generated from the contents of the corresponding
> -  ``*.mak`` files using Meson's ``configure_file()`` function.
> +``${TARGET-NAME}-config-devices.mak``
> +  TARGET-NAME is the name of a system emulator. The file is
> +  generated by Meson using files under ``configs/devices`` as input.
> +
> +``${TARGET-NAME}-config-target.mak``
> +  TARGET-NAME is the name of a system or usermode emulator. The file is
> +  generated by Meson using files under ``configs/targets`` as input.
> +
> +``$TARGET_NAME-config-target.h``, ``$TARGET_NAME-config-devices.h``
> +  Used by C code to determine the properties and enabled
> +  features for each target.  enabled.  They are generated from
> +  the contents of the corresponding ``*.mak`` files using Meson's
> +  ``configure_file()`` function; each target can include them using
> +  the ``CONFIG_TARGET`` and ``CONFIG_DEVICES`` macro respectively.
>
>  ``build.ninja``
>    The build rules.
> --
> 2.38.1
>
>


-- 
Marc-André Lureau