Re: Let's fix core-updates!

[Marius Bakke]

Chris Marusich <address@hidden> writes:

> Hi everyone!
>
> Currently, 13% of builds on core-updates fail:
>
>   https://hydra.gnu.org/jobset/gnu/core-updates
>
> We need to fix this to help Ricardo prepare for the next release.
> Questions:
>
>   1) When is core-updates "done"?  Do we merge once we're below a
>      specific failure rate, once specific bugs have been fixed, or a
>      combination of the two?

Typically a combination.  Packages with many dependents usually get
fixed first, then once we're down to mostly "leaf" packages, if they are
not hugely popular (say GNOME, or SLiM), we can merge and deal with the
remainder on 'master'.

Right now, the biggest blockers are:

* Shepherd on armhf: 
<https://hydra.gnu.org/job/gnu/core-updates/shepherd-0.3.2.armhf-linux>

It's an indeterministic failure, but happens nearly every build.  This
is fixed in Shepherd git, but not yet in Guix.  Once the Hydra build
succeeds, we need to restart the builds for all of Shepherds dependent
packages in order to get substitutes for them.

At this point, we might consider applying the upstream fix directly for
armhf only.  Then we don't have to manually restart the dependent jobs,
since that would create a new derivation and Hydra will make new jobs.

* ruby-sqlite3: 
<https://hydra.gnu.org/job/gnu/core-updates/ruby-sqlite3-1.3.13.x86_64-linux>

I filed an issue upstream here:
<https://github.com/sparklemotion/sqlite3-ruby/issues/226>.  It seems
harmless, so I guess we can s/integer/INTEGER/ and move on.

* SBCL: <https://hydra.gnu.org/job/gnu/core-updates/sbcl-1.3.7.x86_64-linux>

I don't know anything about this package, so not sure what to do about
it.  It has 45 dependents, including MATE, so merging without it is not
great.  I tried updating to the latest version, which made the build
succeed, but then many of the dependent packages failed.

* python-django: 
<https://hydra.gnu.org/job/gnu/core-updates/python-django-1.10.8.x86_64-linux>

This version of Django is fairly old and riddled with security holes, so
I'm willing to "punt" on this and let Django users fix it when it hits
'master'.  Here I also tried updating to the latest LTS, but got some
odd and very difficult test failures.

In addition, there are many "leaf" Python packages that are failing,
which is typical when we go from 3.5->3.6.  Updating usually solves it.

Note that we have a bug with python2-flake8: if you see a python2
variant failing due to missing "enum34", it will have to be added as a
propagated-input until we can fix it in flake8 proper.  See these
commits:

<https://git.savannah.gnu.org/cgit/guix.git/commit/?h=core-updates&id=01af1e11a67466d5f6338bdb207ff99ef5cf094e>
<https://git.savannah.gnu.org/cgit/guix.git/commit/?h=core-updates&id=b7049b2e23a514a1d9c131af484e808ee0ee4226>
<https://git.savannah.gnu.org/cgit/guix.git/commit/?h=core-updates&id=53f826cd0f429864d46fc3bf6305c14356d0b2ad>

Most of the currently failing Python packages can be fixed after the
merge IMO, as they don't have a lot of dependents.  But don't let that
stop anyone from fixing them now :-)

>   2) How shall we prioritize and divvy up work for fixing the failures?
>      I'm guessing people just need to volunteer and start debugging!

I encourage everyone to check the Hydra core-updates page every now and
then, press "compare" and choose "master" and go on the "currently
failing" tab.  If you recognize any of the failing packages, please try
to fix them :-)

Here is the comparison for the latest evaluation (this might not load
since Hydra is currently very busy):
<https://hydra.gnu.org/eval/109908?compare=109907&full=1#tabs-now-fail>.

>   3) Are there any tools to help us understand what the failures might
>      have in common?  E.g., if half the failures occur because a package
>      deep in the dependency graph fails to build, clearly that package
>      should be prioritized for fixing.  I suppose we'll learn about
>      commonalities as we go, but it'd be nice if there were a tool that
>      might help us understand what to focus on first.

On the "currently failing" tab, only those with a bright red icon are
failing.  Most of the jobs have an orange icon which means a dependency
failed.  Clicking on the job will show you which dependency that is.

>   4) What other bugs/features need to be addressed to un-block release?

I believe most of the "hard" problems this round have been addressed, so
now we just need the fixes mentioned above.  That said, please try to
pull core-updates on your systems to make sure :-)

> I know that we want to update the default JDK used by Java packages from
> 7 to 8, but there are probably more important tasks to finish up, also.

We don't have a lot of Java dependents still, so IMO this can be done on
'master' (if everything builds) or maybe 'staging' for testing all arches.

> Let's get started!

I'm going on a road trip for a few days, so thanks for picking up this
thread.  I hope this branch is merged when I get back!

signature.asc
Description: PGP signature