Cache eviction policy of ‘guix publish’

[Ludovic Courtès]

Hello,

(Continued from a thread on guix-sysadmin on how to make sure ‘guix
publish’ does not remove from its own cache things that correspond to
store items that are still valid.)

Mark H Weaver <address@hidden> skribis:

> address@hidden (Ludovic Courtès) writes:
>
>> Mark H Weaver <address@hidden> skribis:

[...]

>>> especially to ensure that all substitutes from the most recent release
>>> are always kept.  Furthermore, I'd like to see all NARs and NARINFOs
>>> kept for as long as the corresponding items are present in the store, so
>>> that users of less popular packages (or less popular systems) are not
>>> penalized for being in the minority.
>>
>> Agreed.  I’m not sure how to do this; perhaps some sort of a “GC root”
>> mechanism?
>
> How about simply adding an option to 'guix publish' that prevents any
> NAR/NARINFO from being expired from the cache as long as the
> corresponding item is present in the store?  It could be called
> something like --never-expire-valid-store-items.
>
> More concretely, at the point in the code where a cache item is expired
> in the current code, insert an additional check: if the
> --never-expire-valid-store-items option is enabled and the corresponding
> store path is still valid, then do not expire it.
>
> What do you think?

Good idea.  The patch below implements this policy.  I’m tempted to just
make that policy the default and not even add a command-line option to
switch to the current policy.  WDYT?

Thank you!

Ludo’.

diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index a7e3e6d62..775ba62b0 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -385,6 +385,24 @@ at a time."
                     (string-suffix? ".narinfo" file)))
       '()))
 
+(define (nar-expiration-time ttl)
+  "Return the narinfo expiration time (in seconds since the Epoch).  The
+expiration time is +inf.0 when passed an item that is still in the store; in
+other cases, it is the last-access time of the item plus TTL.
+
+This policy allows us to keep cached nars that correspond to valid store
+items.  Failing that, we could eventually have to recompute them and return
+404 in the meantime."
+  (let ((expiration-time (file-expiration-time ttl)))
+    (lambda (file)
+      (let ((item (string-append (%store-prefix) "/"
+                                 (basename file ".narinfo"))))
+        ;; Note: We don't need to use 'valid-path?' here because FILE would
+        ;; not exist if ITEM were not valid in the first place.
+        (if (file-exists? item)
+            +inf.0
+            (expiration-time file))))))
+
 (define* (render-narinfo/cached store request hash
                                 #:key ttl (compression %no-compression)
                                 (nar-path "nar")
@@ -435,7 +453,7 @@ requested using POOL."
                  (maybe-remove-expired-cache-entries cache
                                                      narinfo-files
                                                      #:entry-expiration
-                                                     (file-expiration-time ttl)
+                                                     (nar-expiration-time ttl)
                                                      #:delete-entry 
delete-entry
                                                      #:cleanup-period ttl))))
            (not-found request