bug#26201: Heads-up: hydra.gnu.org uses ‘guix publish --cache’

[Ludovic Courtès]

address@hidden (Ludovic Courtès) skribis:

> address@hidden (Ludovic Courtès) skribis:
>
>>   2. Produce a narinfo and corresponding nar the first time they are
>>      requested.  So, the first time we receive “GET foo.narinfo”, return
>>      404 and spawn a thread to compute foo.narinfo and foo.nar.  Return
>>      200 only when both are ready.
>>
>>      The precomputed nar{,info}s would be kept in a cache and we could
>>      make sure a narinfo and its nar have the same lifetime, which
>>      addresses one of the problems we have.
>>
>>      pros: better HTTP latency and bandwidth
>>      pros: allows us to add a Content-Length for nars
>>      pros: helps keep narinfo/nar lifetime in sync
>>      cons: doesn’t reduce load on hydra.gnu.org
>>      cons: exposes inconsistency between the store contents and the HTTP
>>            response (you may get 404 even if the thing is actually in
>>            store), but maybe that’s not a problem
>
> Implemented in commit 00753f7038234a0f5a79be3ec9ab949840a18743.
>
> I’ll set up a test instance shortly.

I ended up deploying it on hydra.gnu.org directly.  :-)

Progressively the cached nar/narinfo at {,mirror.}hydra.gnu.org will be
replaced with the new ones.  Now, the /guix/nar URLs have a
‘Content-Length’ header you should see a progress bar when downloading
one of these:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build vim
The following file will be downloaded:
   /gnu/store/ax5cm9gr1741pcq17w7bhgss5nvq5470-vim-8.0.0566
@ substituter-started /gnu/store/ax5cm9gr1741pcq17w7bhgss5nvq5470-vim-8.0.0566 
/gnu/store/rnpz1svz4aw75kibb5qb02hhccy2m4y0-guix-0.12.0-7.aabe/libexec/guix/substitute
Downloading 
https://mirror.hydra.gnu.org/guix/nar/gzip/ax5cm9gr1741pcq17w7bhgss5nvq5470-vim-8.0.0566
 (23.4MiB installed)...
 vim-8.0.0566  7.8MiB                                      385KiB/s 00:21 
[####################] 100.0%

@ substituter-succeeded /gnu/store/ax5cm9gr1741pcq17w7bhgss5nvq5470-vim-8.0.0566
/gnu/store/ax5cm9gr1741pcq17w7bhgss5nvq5470-vim-8.0.0566
--8<---------------cut here---------------end--------------->8---

This new caching scheme should put an end to caching of truncated nars
in nginx, which has been too frequent lately.

It should also mostly avoid the problem where we have a narinfo for
something but not the corresponding nar, which leads to user frustration
(‘guix’ reports that the thing will be downloaded and eventually fails
with 410 “Gone” while trying to download it), because ‘guix publish’
caches narinfo/nar pairs together.  I say “mostly” because nginx caching
in front of ‘guix publish’ makes things more complicated.

The bandwidth issue reported at the beginning of this thread should be
mostly fixed: serving a narinfo or nar URL is now just sendfile(2),
which is the best we can do; 404s on narinfo should be immediate.

Of course, when the machine is overloaded, we’ll still experience
increased latency and lower bandwidth, but that should be less acute
than with the previous setting.

Please report any problems you may have!

Ludo’.