[PATCH v1 09/14] scripts: misc: guix.sh: prevent OOM.

[Denis 'GNUtoo' Carikli]

TODO:
- Decide what to do with Trisquel 10: No Guix package.
- Add tests if possible. The tests would install PureOS 10, install
  the guix package and update it.
- What to do with substitutes.
--------------------------------------------------------------------------------
We want to use Guix as a non-optional dependency in GNU Boot.

And we also want to make it as easy as possible for new users to
contribute to GNU Boot, so if we depend on Guix, the build needs to
work out of the box the first time without needing to learn Guix
first, as starting with something that doesn't work can be very
frustrating and make people go away as well as increase the amount of
time assisting new contributors.

To achieve that we currently install the Guix package provided by the
host distributions and provide an update script
(resources/dependencies/guix) that can be be run manually and that
could even be integrated to run automatically later on.

Since we currently support PureOS 10 (byzantium) and Trisquel 10
(nabia) and that the later lacks a guix package, we would probably
need to handle that somehow by various ways (that are not mutually
exclusive) such as:

- Making it easy to install PureOS 10 in various setups. For instance
  by enabling to install PureOS with debootstrap in most FSDG
  compliant distributions and relying on the docker hub image for
  other distributions.

- Making it easier to support more recent distributions by providing
  ways of skipping builds for the KCMA-D8, KFSN4, KGPE-D16 mainboards
  that are stuck in Coreboot 4.11 branch and that requires
  distributions with older software to compile.

- By somehow building or providing the older software (compilers?)
  that are necessary to bootstrap the build of Coreboot 4.11.

- By moving more of the build inside Guix, and removing the need to
  depend on a host distribution completely.

- By adding other ways to install Guix, for instance by running the
  guix installer or a modified version of it.

Here we need to update guix in any case because we are already working
with upstream Guix, and so in the future we will most likely want to
use specific Guix revision(s) (with guix time-machine) inside GNU Boot
to use our contributions in upstream Guix once they are merged.

In addition using fixed revisions of Guix with guix time-machine also
makes sure that the command line argument we use do work and also
reduces the amount of work required to use Guix as tracking command
line argument changes across different guix revisions can be tedious.

PureOS 10 (byzantium) has Guix 1.2.0 and so if we want to use a fixed
revision that is more recent than that, Guix needs to be updated
somehow as for now it doesn't automatically update itself to find new
revisions.

The update script contains more logic than just running the 'guix
pull' command. Beside minor details like setting up the right
environment variables, it mainly tries to make sure that running 'guix
pull' doesn't result in an out of memory situation that kills some
Guix process because if for some reasons that happens, it would defeat
the point of this script as it would make it harder for users to
understand what is going on than if the script was not there in the
first place.

Having the system go out of memory can happen if the total RAM size
divided by the number of cores used by Guix is lower than a certain
value.

To workaround that issue we adjust the number of cores used by Guix
depending on the amount of RAM that the computer has.

How to test this patch:
-----------------------

We need help for testing this patch as we're unsure of how much RAM
per core Guix really requires. For x86_64 it's probably somewhere
between 2GiB and 3GiB. It might be less for i686 as the 32bit programs
occupy less space in RAM.

To test this patch users need to not have Guix already installed and
install it though the package manager and run the
resources/dependencies/guix script and see if it succeeds.

This should ideally be done on top of PureOS 10 (byzantium) with a
graphical desktop that uses a lot of RAM during typically desktop
usage that also use lot of RAM (like browsing with many tabs open).

Where the PureOS distributions are run is not very important. For
instance they could run in virtual machines, in LXC containers,
etc. What is more important is that commonly used RAM hungry programs
are being used in parallel and to also write down the amount of RAM
the test setup has.

Tests with other distributions are welcome as well as at some point we
might succeed in adding support for newer distributions, especially
because GNU Boot also has long term plans to upstream support for the
KGPE D16 back into upstream Coreboot, to fix this problem and many
other along the way (like issues in RAM initialization that requires
very specific RAM modules), so if this plan works, the requirement for
older distributions will be able to be dropped.

Signed-off-by: Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
---
 resources/scripts/misc/errno-base.sh | 41 ++++++++++++++++
 resources/scripts/misc/guix.sh       | 65 +++++++++++++++++++++++--
 resources/scripts/misc/resources.sh  | 71 ++++++++++++++++++++++++++++
 tests/lint                           |  2 +
 4 files changed, 176 insertions(+), 3 deletions(-)
 create mode 100755 resources/scripts/misc/errno-base.sh
 create mode 100755 resources/scripts/misc/resources.sh

diff --git a/resources/scripts/misc/errno-base.sh 
b/resources/scripts/misc/errno-base.sh
new file mode 100755
index 0000000..042445e
--- /dev/null
+++ b/resources/scripts/misc/errno-base.sh
@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note
+# (GNUtoo) 22 January 2024: Adapted from errno-base.h (that comes from
+# the Linux kernel)
+
+# shellcheck disable=SC2034
+
+EPERM=1    # Operation not permitted
+ENOENT=2   # No such file or directory
+ESRCH=3    # No such process
+EINTR=4    # Interrupted system call
+EIO=5      # I/O error
+ENXIO=6    # No such device or address
+E2BIG=7    # Argument list too long
+ENOEXEC=8  # Exec format error
+EBADF=9    # Bad file number
+ECHILD=10  # No child processes
+EAGAIN=11  # Try again
+ENOMEM=12  # Out of memory
+EACCES=13  # Permission denied
+EFAULT=14  # Bad address
+ENOTBLK=15 # Block device required
+EBUSY=16   # Device or resource busy
+EEXIST=17  # File exists
+EXDEV=18   # Cross-device link
+ENODEV=19  # No such device
+ENOTDIR=20 # Not a directory
+EISDIR=21  # Is a directory
+EINVAL=22  # Invalid argument
+ENFILE=23  # File table overflow
+EMFILE=24  # Too many open files
+ENOTTY=25  # Not a typewriter
+ETXTBSY=26 # Text file busy
+EFBIG=27   # File too large
+ENOSPC=28  # No space left on device
+ESPIPE=29  # Illegal seek
+EROFS=30   # Read-only file system
+EMLINK=31  # Too many links
+EPIPE=32   # Broken pipe
+EDOM=33    # Math argument out of domain of func
+ERANGE=34  # Math result not representable
diff --git a/resources/scripts/misc/guix.sh b/resources/scripts/misc/guix.sh
index 10daab7..17337c0 100755
--- a/resources/scripts/misc/guix.sh
+++ b/resources/scripts/misc/guix.sh
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
 #
 # Copyright (C) 2023 Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
 #
@@ -18,6 +18,10 @@
 true=0
 false=1
 
+# shellcheck source=resources/scripts/misc/resources.sh
+topdir="$(dirname "${BASH_SOURCE[0]}")"/../../../
+. "${topdir}"/resources/scripts/misc/resources.sh
+
 # shellcheck disable=SC2034
 guix_v0_0="6365068393254e1131ab80eb0d68a759e7fd2256"
 # shellcheck disable=SC2034
@@ -123,6 +127,52 @@ source_guix_profile()
     fi
 }
 
+# Commands like Guix pull use a lot of memory, it was measured to be
+# roughly about 2GiB per core. So if we want to avoid going out of
+# memory and avoid having some process killed, we need to set the
+# number of cores.
+guix_min_ram_per_core()
+{
+    nr_cpu_bits > /dev/null ; ret=$?
+    if [ "${ret}" -ne 0 ] ; then
+        return "${ret}"
+    fi
+
+    if [ "$(nr_cpu_bits)" = "32" ] ; then
+       core_min_ram=$(( 2 * gib))
+    else
+       # Guix pull between Guix 1.4.0 on Trisquel 11 in console mode
+       # requires more than 2GiB of RAM. I'm not completely sure how
+       # much though.
+       core_min_ram=$((3 * gib))
+    fi
+
+    echo "${core_min_ram}"
+
+    return 0
+}
+
+guix_nr_jobs()
+{
+    ram_bytes > /dev/null ; ret=$?
+    if [ "${ret}" -ne 0 ] ; then
+        return "${ret}"
+    fi
+    nr_cpus   > /dev/null ; ret=$?
+    if [ "${ret}" -ne 0 ] ; then
+        return "${ret}"
+    fi
+
+    core_min_ram=$((2 * gib))
+    if [ "$(( "$(ram_bytes)" / "${core_min_ram}"))" -lt "$(nr_cpus)" ] ; then
+        echo "$(( "$(ram_bytes)" / "${core_min_ram}"))"
+    else
+        nr_cpus
+    fi
+
+    return 0
+}
+
 guix_version()
 {
     guix --version | grep '^guix (GNU Guix)' | awk '{print $4}'
@@ -132,7 +182,7 @@ guix_version_commit()
 {
     version="$1"
 
-    eval echo "$(echo \$guix_"${version}" | sed 's#\.#_#g')"
+    eval echo "\$guix_""${version//./_}"
 }
 
 is_guix_system()
@@ -250,6 +300,9 @@ guix_next_version()
     esac
 
     if is_known_version "${version}" && is_rc_version "${version}"; then
+       # According to https://www.shellcheck.net/wiki/SC2001 bash
+       # substitution doesn't support regexes.
+       # shellcheck disable=SC2001
         echo "${version}" | sed 's/rc[0-9]\+//'
         return ${true}
     else
@@ -320,10 +373,16 @@ update_guix_to_latest_release()
         return ${true}
     fi
 
+    guix_nr_jobs > /dev/null ; ret=$?
+    if [ "${ret}" -ne 0 ] ; then
+        return "${ret}"
+    fi
+    nr_guix_jobs="$(guix_nr_jobs)"
+
     # We use a released version already
     if [ -n "${major}" ] ; then
         commit="$(guix_version_commit "v$(guix_next_version 
"${current_version}")")"
-        guix pull --commit="${commit}"
+        guix pull  -M "${nr_guix_jobs}" -c "${nr_guix_jobs}" 
--commit="${commit}"
         source_guix_profile
         update_guix_to_latest_release
     fi
diff --git a/resources/scripts/misc/resources.sh 
b/resources/scripts/misc/resources.sh
new file mode 100755
index 0000000..4fdcb85
--- /dev/null
+++ b/resources/scripts/misc/resources.sh
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+#
+# Copyright (C) 2023 Denis 'GNUtoo' Carikli <GNUtoo@cyberdimension.org>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+# shellcheck source=resources/scripts/misc/errno-base.sh
+topdir="$(dirname "${BASH_SOURCE[0]}")"/../../../
+. "${topdir}"/resources/scripts/misc/errno-base.sh
+
+kib=1024
+# shellcheck disable=SC2034
+mib=$((1024 * kib))
+# shellcheck disable=SC2034
+gib=$((1024 * mib))
+
+ram_bytes()
+{
+    if [ "$(grep '^MemTotal:' /proc/meminfo | \
+       awk '{print $3}')" != "kB" ] ; then
+        return "${EINVAL}"
+    fi
+
+    memory_kb="$(grep '^MemTotal:' /proc/meminfo | awk '{print $2}')"
+
+    echo $(("${memory_kb}" * "${kib}"))
+
+    return 0
+}
+
+nr_cpu_bits()
+{
+    # getconf is part of GCC toolchain packages. It returns 32 with
+    # 32bit rootfs so even if people run 64bit kernels on 32bit
+    # rootfs.
+    if [ -z "$(which getconf)" ] ; then
+        return "${EINVAL}"
+    fi
+
+    getconf LONG_BIT
+}
+
+nr_cpus()
+{
+    # We can't use /proc/cpuinfo as offline CPU don't show up in
+    # /proc/cpuinfo. This was tested with the following command on a
+    # Thinkpad X200:
+    # echo 0 > /sys/devices/system/cpu/cpu1/online
+
+    if [ -d /sys/devices/system/cpu/ ] ; then
+       find "/sys/devices/system/cpu/" \
+            -maxdepth 1 \
+            -name "cpu[0-9\+]" \
+            -type d | wc -l
+    else
+        return "${EINVAL}"
+    fi
+
+    return 0
+}
diff --git a/tests/lint b/tests/lint
index 9218cb4..ae5748d 100755
--- a/tests/lint
+++ b/tests/lint
@@ -58,7 +58,9 @@ run_shellcheck \
     resources/packages/src/distclean \
     resources/packages/u-boot-libre/distclean \
     resources/packages/website/distclean \
+    resources/scripts/misc/errno-base.sh \
     resources/scripts/misc/guix.sh \
+    resources/scripts/misc/resources.sh \
     resources/scripts/tasks/distclean.sh \
     tests/distclean \
     tests/lint
-- 
2.41.0