--- Begin Message ---
|
Subject: |
[PATCH 0/4] 'github' importer gracefully handles rate limiting |
|
Date: |
Thu, 3 Mar 2022 22:13:26 +0100 |
Hi Guix!
These patches address a famous complaint about “the GitHub problem”
that affects ‘guix refresh’¹, shown here in its naked awfulness:
--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix refresh
gnu/packages/zig.scm:32:13: zig would be upgraded from 0.9.0 to 0.9.1
[...]
In guix/scripts/refresh.scm:
578:14 5 (_ _)
In srfi/srfi-1.scm:
634:9 4 (for-each #<procedure 7fe85c9a8e00 at
guix/scripts/refresh.scm:578:24 (t-916fdc98f4be2f1-1d48)> _)
In guix/scripts/refresh.scm:
378:2 3 (check-for-package-update #<package
redshift-wayland@1.12-1.7da875d gnu/packages/xdisorg.scm:1425 7fe85879e790>
(#<<upstream…>) …)
In guix/import/github.scm:
232:12 2 (latest-release _)
In ice-9/boot-9.scm:
1685:16 1 (raise-exception _ #:continuable? _)
1685:16 0 (raise-exception _ #:continuable? _)
ice-9/boot-9.scm:1685:16: In procedure raise-exception:
Error downloading release information through the GitHub
API. This may be fixed by using an access token and setting the environment
variable GUIX_GITHUB_TOKEN, for instance one procured from
https://github.com/settings/tokens
--8<---------------cut here---------------end--------------->8---
With this change, ‘guix refresh’ warns you when the GitHub rate limit
is reached, but it keeps going, falling back to the ‘generic-git’
updater if it’s among the applicable updaters:
--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix refresh -t github,generic-git
[...]
guix refresh: warning: GitHub rate limit exceeded; disallowing requests for
1477 seconds
hint: You can waive the rate limit by setting the `GUIX_GITHUB_TOKEN'
environment variable to a
token obtained from `https://github.com/settings/tokens' with your GitHub
account.
Alternatively, you can wait until your rate limit is reset, or use the
`generic-git' updater
instead.
gnu/packages/zile.scm:113:14: warning: no tags were found for zile-on-guile
gnu/packages/zig.scm:32:13: zig would be upgraded from 0.9.0 to 0.9.1
gnu/packages/xorg.scm:2830:7: warning: no valid tags found for
xf86-video-freedreno
gnu/packages/xml.scm:2132:13: java-kxml2 would be upgraded from 2.4.2 to 2.5.0
--8<---------------cut here---------------end--------------->8---
The GitHub updater becomes functional again once the rate limit has
been reset.
The code to deal with rate limiting is similar to that in (guix swh).
Thoughts?
Thanks,
Ludo’.
¹ https://issues.guix.gnu.org/53818#50
Ludovic Courtès (4):
http-client: Add response headers to '&http-get-error'.
import: github: Gracefully handle rate limit exhaustion.
http-client: Correctly handle redirects when #:keep-alive? #t.
import: github: Reuse HTTP connection for the /tags URL fallback.
.dir-locals.el | 1 +
guix/http-client.scm | 39 +++++++++-----
guix/import/github.scm | 119 +++++++++++++++++++++++++++++------------
3 files changed, 112 insertions(+), 47 deletions(-)
base-commit: be84fb701bf7a36a0eb50147ccbb988aa3f41209
--
2.34.0
--- End Message ---
--- Begin Message ---
|
Subject: |
Re: bug#54241: [PATCH 0/4] 'github' importer gracefully handles rate limiting |
|
Date: |
Sun, 06 Mar 2022 22:55:53 +0100 |
|
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) |
Hello,
I committed an updated version of these patches, taking some of your
suggestions into account:
a8d3033da6 import: github: Reuse HTTP connection for the /tags URL fallback.
8786c2e8d7 http-client: Correctly handle redirects when #:keep-alive? #t.
55e8e283ae import: github: Gracefully handle rate limit exhaustion.
ecad9b2213 http-client: Add response headers to '&http-get-error'.
049aefddb2 tests: Add (guix http-client) tests.
The first patch adds tests for ‘http-fetch’, as you suggested, which
allowed me to find a thinko in the keep-alive patch. The test doesn’t
test keep-alive support though, because the server in (guix tests http)
doesn’t support it currently (I tried to retrofit it based on (web
server http) but that’s not really possible because we’d need to either
access internals of the <http> record type, specifically it’s poll set,
or re-implement it entirely.)
Thanks a lot for your help!
Ludo’.
--- End Message ---