Re: [PATCH] draft addition of github updater

[Ben Woodcroft]

Thanks for the encouraging words. Here's the next revision.

On 16/11/15 19:15, Ludovic Courtès wrote:
> Hi!
>
> Ben Woodcroft <address@hidden> skribis:
>
> > Importing from GitHub seems very non-trivial, but can we update?
> > There's a number of issues with the attached patch but so far out of
> > the 171 github package in guix, it recognizes 101, and 17 are detected
> > as out of date (see below).
It seems I miscounted before, but now it is 129 of 146 github "release" 
packages recognised with 28 suggesting an update - see the end of email 
for details. There is one false positive:

gnu/packages/ocaml.scm:202:13: camlp4 would be upgraded from 4.02+6 to 
4.02.0+1

This happens because the newer versions were not made as official 
releases just tags, so the newer versions are omitted from the API 
response, plus there's the odd version numbering scheme. Guix is up to 
date.
> > I have two questions:
> >
> > 1. Some guess-work is required to get between the version as it is
> > defined in guix, and that presented in the github json, where only the
> > "tag_name" is available. Is it OK to be a little speculative in this
> > conversion e.g. "v1.0" => "1.0"?
> I guess so.  What I would do is do that conversion when the tag matches
> “^v[0-9]” and leave the tag as-is in other cases.  WDYT?
>
> We can always add more heuristics later if we find that there’s another
> widely-used convention for tag names.
Most seem to follow those few conventions, but there's still repos that 
decided to be different e.g.

https://github.com/vapoursynth/vapoursynth/archive/R28.tar.gz
https://github.com/synergy/synergy/archive/v1.7.4-stable.tar.gz

Having gotten this far, I wonder if I've gone about it backwards. 
Currently the updater works by asserting it is a refreshable package by 
interrogating the source URI only. But it might be easier to determine 
this with an API response on hand, by matching the current release 
version number to a tag. Then if we assume the same transformation of 
tag to version holds in the newest release, the reverse transformation 
can be used on the newest tag to convert it back into a version number. 
By transformation I mean addition of [a-z\.\-] characters before and 
after the version number. This is easier because guesswork is only 
needed to convert between the tag and version number, without reference 
to a URI.

This means more work for me, is it a good idea? As I understand it would 
involve returning #t more often from "github-package?". If #f is 
returned by an updater, do the updaters further down the chain get a 
bite at the cherry too? It doesn't matter for now since the github 
updater is last, but it might in the future.
> > 2. For mass-updates, it fails when it hits the abuse limit on github
> > (60 api requests per hour). This can be overcome by authenticating
> > with an access token, but I don't think that token should go in the
> > git repository. So I'm after some guidance on the best way of the user
> > providing a token to the updater (or some other workaround).
> Argh, that’s annoying.  How does it fail exactly?  What’s the impact on
> the behavior of ‘guix refresh’?
I didn't investigate thoroughly, but I believe it either gives a 403 or 
a more descriptive json string, dependent on the user-agent.

I added some words and errored out when json-fetch* returns #f. This was 
potentially a little lazy on my part as it might be better to detect the 
403 error as distinct from errors of other kinds, but it wasn't 
immediately obvious to me how to do this without going too deep into the 
fetching functions and/or duplicating code. WDYT?
> I guess (guix import github) could contain something like:
>
>    (define %github-token
>      ;; Token to be passed to Github.com to avoid the 60-request per hour
>      ;; limit, or #f.
>      (make-parameter (getenv "GUIX_GITHUB_TOKEN")))
>
> and we’d need to document that, or maybe write a message hinting at it
> when we know the limit has been reached.
>
> WDYT?
Seems we were all thinking the same thing - I've integrated this. Should 
we check that the token matches ^[0-9a-f]+$ for security and UI?
> I was thinking we could have a generic Git updater that would look for 
> available tags upstream. I wonder how efficient that would be compared 
> to using the GitHub-specific API, and if there would be other 
> differences. What are your thoughts on this?
This sounds like an excellent idea, but I was unable to find any way to 
fetch tags without a clone first. A clone could take a long time and a 
lot of bandwidth I would imagine. Also there's no way to discern regular 
releases from pre-releases I don't think. It is a bit unclear to me how 
conservative these updaters should be, are tags sufficiently synonymous 
with releases so as to be reported by refresh?

There's a number of github repos packaged that refer to git commits 
directly too, these are ignored by the current updater but might benefit 
from this approach (as well as non-github git repos of course).

Thanks,
ben



gnu/packages/xml.scm:380:13: pugixml would be upgraded from 1.6 to 1.7
gnu/packages/web.scm:353:13: libpsl would be upgraded from 0.7.1 to 0.11.0
gnu/packages/web.scm:685:6: sassc would be upgraded from 3.2.5 to 3.3.2
gnu/packages/version-control.scm:934:13: findnewest would be upgraded 
from 0.2 to 0.3
gnu/packages/telephony.scm:192:13: libsrtp would be upgraded from 1.5.2 
to 1.5.3
gnu/packages/ruby.scm:2373:13: ruby-sanitize would be upgraded from 
4.0.0 to 4.0.1
gnu/packages/ocaml.scm:202:13: camlp4 would be upgraded from 4.02+6 to 
4.02.0+1
gnu/packages/ninja.scm:31:13: ninja would be upgraded from 1.5.3 to 1.6.0
gnu/packages/maths.scm:1855:13: dealii would be upgraded from 8.2.1 to 8.3.0
gnu/packages/jrnl.scm:30:13: jrnl would be upgraded from 1.8.4 to 1.9.7
gnu/packages/gl.scm:453:13: libepoxy would be upgraded from 1.2 to 1.3.1
gnu/packages/game-development.scm:125:13: tiled would be upgraded from 
0.13.1 to 0.14.2
gnu/packages/fontutils.scm:285:13: libuninameslist would be upgraded 
from 0.4.20140731 to 0.5.20150701
gnu/packages/engineering.scm:58:13: librecad would be upgraded from 
2.0.6-rc to 2.0.8
gnu/packages/emacs.scm:436:13: haskell-mode would be upgraded from 
13.14.2 to 13.16
gnu/packages/conky.scm:35:13: conky would be upgraded from 1.10.0 to 1.10.1
gnu/packages/bioinformatics.scm:974:13: deeptools would be upgraded from 
1.5.11 to 1.5.12
gnu/packages/bioinformatics.scm:1532:13: htsjdk would be upgraded from 
1.129 to 2.0.1
gnu/packages/bioinformatics.scm:207:13: bedtools would be upgraded from 
2.24.0 to 2.25.0
gnu/packages/bioinformatics.scm:1880:13: orfm would be upgraded from 
0.4.1 to 0.5.2
gnu/packages/bioinformatics.scm:758:13: clipper would be upgraded from 
0.3.0 to 1.0
gnu/packages/bioinformatics.scm:1612:13: idr would be upgraded from 
2.0.0 to 2.0.2
gnu/packages/bioinformatics.scm:2592:13: preseq would be upgraded from 
2.0 to 2.0.2
gnu/packages/bioinformatics.scm:2978:13: vsearch would be upgraded from 
1.4.1 to 1.9.5
gnu/packages/bioinformatics.scm:1360:13: grit would be upgraded from 
2.0.2 to 2.0.4
gnu/packages/bioinformatics.scm:1577:13: htslib would be upgraded from 
1.2.1 to 1.3
gnu/packages/bioinformatics.scm:1013:13: diamond would be upgraded from 
0.7.9 to 0.7.10
gnu/packages/bioinformatics.scm:613:13: bowtie would be upgraded from 
2.2.4 to 2.2.6

0001-import-Add-github-updater.patch
Description: Text Data