Re: Performance on NFS

[Ludovic Courtès]

Hi!

Roel Janssen <address@hidden> skribis:

> From these timings, I don't think it has a big impact.

Wait, wait.  If we take the best timings of each series of runs, we get:

  hpc-submit1: 26.4s -> 18.0s (-30%)
  hpc-guix:    26.2s -> 22.7s (-13%)

This is arguably insufficient but still non-negligible.  I’ve committed
it as b46712159c15f72fc28b71d17d5a7c74fcb64ed0.

With commit 015f17e8b9eff97f656852180ac51c75438d7f9d, the number of
open(2) calls for that same command drops from 991 to 795 (including 122
errors).  I suspect we can’t reduce it further:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env strace -o ,,s guix build coreutils -n
$ grep '^open.*' < ,,s |wc -l
795
$ grep '^open.*\.go"' < ,,s |wc -l
563
$ grep '^open.*\.patch"' < ,,s |wc -l
29
$ grep '^open.*\.scm"' < ,,s |wc -l
6
--8<---------------cut here---------------end--------------->8---

Could you check how this affects performance on your NFS system?

There’s possibly another low-hanging fruit, which is to disable file
name canonicalization (via ‘%file-port-name-canonicalization’.)  It
special care though, so I’ll try that later.

> This makes me wonder, can't we replace the disk-intensive stuff with a
> database?  If we only have to read the files on disk once, after which
> we extracted the information (the hashes?) needed to compute which
> links have to be created to make an environment, then actually
> creating the environment can be as fast as only creating those links.

Essentially we need to compute derivations as a function of local files
(sent to the daemon with ‘add-to-store’) and other derivations.  We
cannot avoid that.

In the case of a remote server, communications with the daemon play an
important role too.  Have you tried setting ‘GUIX_DAEMON_SOCKET’ as
suggested before instead of using the “socat hack”?  I think this should
be faster (see commit 950d51c9d9a5107c5dac279da7d2e431134b5f43.)

HTH,
Ludo’.