groff maintainership, release, and blockers (was: groff 1.23.0.rc2 readiness)

[G. Branden Robinson]

Hi Ingo,

At 2022-08-26T13:51:25+0200, Ingo Schwarze wrote:
> > Bertrand and I are in private correspondence about this.
> > One or both of us will follow up to the list as things progress.
> 
> Very good news, this is nice.
> Also, my best wishes for Bertrands further and speedy recovery!

He has mine as well!

Bertrand is going to manage this release as it will apparently take
literally months to get me through FSF processes.  In the meantime, he's
hit a smaller process obstacle himself, so at present _he_ can't do a
release candidate either.  We expect this to be resolved "soon".

Bertrand is also delegating to me as many maintainer tasks as possible
while preserving process integrity (so, no, we're not sharing our GPG
secret keys with each other).  He feels there isn't all that much for me
to learn.  Fingers crossed!

> > There's a lot of FSF procedure to deal with.
> 
> Wouldn't it be better to simply abandon the the GNU roff project
> (i.e. leaving the FSF with no developer whatsoever), fork groff under
> a new name (say, "GPL roff"), and continue that new project outside
> the FSF?  This is not the first time FSF red tape has proven onerous
> to the point of creating major distractions from and significant
> obstacles to real development work.

The FSF provides useful infrastructure.  Consider the risks of
relocating to a site like GitHub, where a firm like Microsoft can come
along, acquire it, and all of a sudden arrogate extra rights to itself
for everything, and if you don't like it you have 24 hours to take your
project down.

The FSF has its problems but selling out to a copyright rentier firm
seems like a pretty low risk.

> Of course, the FSF would retain copyright of the existing GPLv3+ code.
> But that's not a problem because a free software license, once
> granted, cannot be retracted.  Future contributors could simply
> contribute in their own name, without transfering Copyright, under
> GPLv3+.
> 
> Of course, if most developers love the FSF, i'm not pressing for that
> change as i'm only mildly affected by the red tape.  I believe
> some others suffer more from it than i do (even though it did
> cause me some waste of time on a few occasions).

Consider also that relocating the project will have frictions of its
own, and any project anywhere requires administration.  The requirements
of administration are frequently the cause of "bureaucratic" delays.

The grass isn't so green.  In my experience nearly everything in
software management that looks "agile" and low-friction is that way
because there is some serious infrastructure beneath that people have
worked hard to make unobtrusive.

I am confident that groff decamping to _any_ other hosting site or
umbrella project, or building one for itself, is _not_ going to be
faster than waiting on the FSF's new maintainer procedure, which at
least has the virtue of being familiar to those administering it, as
well as to Bertrand (and Werner)--and numerous people in other GNU
projects who might be friendly and willing to explain things, or help.

> > But for the sake of transparency, in the meantime, he asked if the
> > current HEAD was good enough to tag as "rc2" and I said "yes".
> 
> Sorry, i fail to understand that.  The acronym "RC" stands for "release
> candidate".  I would define a "release candidate" as "a version that
> is believed to be ready for release".

Apparently we have a terminological and/or philosophical disagreement.

My objective since Bertrand added the first automatic tests before the
groff 1.22.4 release has been to _never_ have Git HEAD in a state where
_any_ tests fail, with one important exception: the introduction of a
new regression test should verify its own validity by failing.  (You
then apply the fix for the regression in the very next commit.)

Therefore, by that standard, any commit not marked "Test fails at this
commit."..._is a release candidate_.

On the other hand, that statement is unrealistic.  We don't have a
regression test for every defect in groff because, like any
non-formally-verified codebase of non-trivial complexity, groff has bugs
we don't know about and therefore cannot test for.  It can also have
bugs that we know about but don't understand well enough to a write a
test for, and bugs that manifest only on platforms or configurations
that its regular developers don't test.

None of these are novel observations; it's why people have "continuous
integration" infrastructures.

> The purpose of an RC is to have it tested on as many platforms and for
> as many different purposes as possible, to confirm that indeed no
> undiscovered regressions exist

I wouldn't say "no" undiscovered regressions, I'd say "minimal".  There
will be bugs we don't turn up, and bugs that even exotic platforms don't
expose.

> and that in particular the last few commits made before the RC did not
> cause regressions.

groff is an assembly of modular components.  We can be reasonably
confident that a change to a macro package cannot cause a bug in the
formatter.  (It might _expose_ one.)  We can be reasonably confident
that a change to xtotroff(1) is not going to cause a bug in grog(1).

More importantly and practically, everyone who looks at groff's commit
history knows that I make lots of changes to documentation.  Such a
change can be ill-advised, incorrect, or even stupid, but it's not going
to cause a SEGV in the formatter.

So, while I think your narrowly drawn concept of an RC works well for an
individual software module, it applies less comfortably to a more
complex assembly with loose coupling among components.

> These tests cause non-trivial work for significant numbers of people,
> most of whom are *not* groff developers, so an RC should only be made
> when the software is really believed to be ready - both out of respect
> for testers' time and because releasing multiple RCs will weary out
> testers and increase the likelihood of serious bugs slipping into the
> release: some testers will not have the time to test over and over
> again, so the more RCs you ship, the less test coverage you get.

I agree with this, but I reiterate that, in a sense, we've had literally
thousands of RCs since groff 1.22.4.  Stability, on the platforms that
are readily available to be tested by the developers, has not been a
problem.  Even when I've made what I consider to be a boneheaded mistake
and moved quickly to fix it, the stakes have typically not been high
(even if one considers typesetting software mission-critical).

Take the most recent example.

https://git.savannah.gnu.org/cgit/groff.git/commit/?id=87efb8ff373d5cf3b92be9d21445a80b264fa961

What happened here?  When an invalid document (one that selected an
impossible or unavailable font family) is rendered, groff 1.22.4 would
issue a diagnostic, albeit a slightly confusing one, but for nearly the
past year, groff Git would not.

A _valid_ input document was not affected at all.

> In particular, i'm firmly convinced that issuing an RC while even one
> single blocker issue is unresolved is a blatant contradiction.  Before
> an RC, all blockers must either be resolved or explicitly
> re-classified as "not release critical" and re-scheduled for the
> subsequent release.

Well, then, in a sense we don't have _any_ blockers, because "the tree
isn't red".

But according to Savannah, we have 7 blocker issues right now.

https://savannah.gnu.org/bugs/index.php?go_report=Apply&group=groff&func=&set=custom&msort=0&report_id=101&advsrch=0&status_id=1&resolution_id=0&submitted_by=0&assigned_to=0&category_id=0&bug_group_id=0&severity=8&summary=&details=&sumORdet=0&history_search=0&history_field=0&history_event=modified&history_date_dayfd=26&history_date_monthfd=8&history_date_yearfd=2022&chunksz=200&spamscore=5&boxoptionwanted=1#options

For those who don't want to click, here they are.

    #62933 [man] produce hyperlinks in PDF output
      Macro man 5 - Blocker gbranden 2022-08-21

      This is a feature enhancement.  Deri's done the bulk of the work
      on it; my task is to integrate it.  Obviously we _can_ ship
      without it; every previous version of groff did.

    #62926 [mdoc] align styling of titles and man page cross references with 
man(7)
      Macro mdoc 5 - Blocker gbranden 2022-08-20

      This is a complaint of Ingo's, and a goal of mine.  The idea is to
      have a more consistent presentation of man(7) and mdoc(7) pages,
      making it harder for the _reader_ of a man page to deduce which
      macros were used.  This makes a better user experience.  (Why
      should they care which package a document uses?  Man pages should
      look like man pages.)  groff man(7) and mdoc(7) have differed in
      minor rendering details for as long as they have existed.

    #62918 Wrong GhostScript version reported during build
      Font - others/general 5 - Blocker Need Info alhadis 2022-08-19

      This needs research.  I think it might be the same as #62860
      below.

    #62860 [build] audit for inappropriate use of system groff resources
      General 5 - Blocker gbranden 2022-08-04

      This is the most serious from a strict engineering perspective.
      Apparently sometimes the build finds itself with recourse to
      installed groff and will use it instead of the groff artifacts
      that are generated _by_ the build.  Since groff is pretty stable,
      no known _practical_ problems arise from this, but it still feels
      to me a bit like driving around town with your car doors removed.
      A strictly disciplined build environment (i.e., build in a chroot
      with no installed groff) won't have a problem with this, but users
      of the groff distribution archive may well play fast and loose.
      We should keep that from influencing our build artifacts.

      Still, if nothing is known to be _broken_, can it be a blocker?

      It is also the item on the list requiring the most effort, so I
      have to admit I've been procrastinating it a bit.  grepping
      straces is not my idea of a party.  Someone want to volunteer?  :)

    #62774 [mdoc] warn if any of `Dd`, `Dt`, `Os` not called
      Macro mdoc 5 - Blocker gbranden 2022-07-16

      This is Ingo and me wanting to be more fastidious about handling
      of invalid, degenerate mdoc(7) input.  I don't think handling
      invalid input in an ugly way can be a blocker unless you _fail_ to
      handle it, which in C all too often means an infinite loop, SEGV,
      or undefined behavior.  But here, the handler is a macro package.
      The consequence appears to be a choice between different ugly ways
      of rendering invalid pages.

    #61423 [libgroff] allow paths in "name" directive of font description file, 
restoring historical groff behavior
      Font devps 5 - Blocker Need Info gbranden barx 2021-11-04

      This ticket is, at this point, a documentation issue.  I need to
      decide whether a behavior change (to stop accepting dubious input)
      is worth documenting more prominently.

    #58930 take baby steps toward Unicode
      Core 5 - Blocker Need Info gbranden barx 2020-08-10

      Like the previous, this is a reminder to myself to decide how I
      want to document something.  Once that is addressed, the ticket
      itself will remain open, with a lower severity.  Sorry to say,
      "full Unicode support" is not pending and not expected for the
      1.23 release.  (But I'm getting better ideas about how to perturb
      the code base toward it afterward.)

There have been recent discussions about gropdf's handling of "download"
files, including multiple people expressing a desire for grops to work
the same way.  That's not a blocker yet because, last I checked, the
gropdf work wasn't done yet.  (Ralph had some good suggestions.)  I'd
really like to get these changes and alignment into 1.23 but it's
obviously at risk because the grops changes haven't been written yet.
But, under your definition, it _can't_ be a blocker because the programs
_work_.  As far as I know they work at least as well as they did in
groff 1.22.4.  gropdf assuredly works better; Deri has fixed several
Savannah tickets regarding it over the past 4 years.

_None_ of these are really release critical.  They are simply things
that I feel would be much better to address before release than not.

On the other hand, applying your definition strictly, we'd almost never
have any blocker bugs at all.  I can't remember one ever having arisen
in the five-plus years I've been contributing.  (Maybe there was, it was
my fault, I fixed it really quickly, and my sense of shame has effaced
it from my memory.)

You could indeed say that I am abusing the Savannah ticket tracker and
its "Blocker" severity to serve as a sort of to-do list, since we don't
have anything legitimately blocking the release.

A lot of software projects would call this a nice problem to have.

It has occurred to me that, once a Blocker item is resolved, I should
knock its severity back down where it really belongs; I'm not sure that
I have consistently done so in the past, and as the mantle of
"maintainer" slowly settles over my shoulders it will be more obviously
my (self-imposed) duty to do that.

So, we could indeed stop using the Savannah Blocker severity for the
purpose I'm employing it.

But I ask you: what _good_ would that do, apart from satisfying your
personal esthetic of release management?  groff needs release processes
that work for us.  I haven't caused anyone to leap onto any Blocker
tickets and spend late nights on them.  (If I have, TELL ME!)

[resequencing two paragraphs here]
> Not only do we have a significant numbers of open blockers, but i
> also reported that the mandoc test suite found thirty-seven changes
> of behaviour between the last groff release and groff-current that
> i did not find the time to analyze just jet (in addition to changes
> that i already investigated and that turned out to be in part groff
> regressions, in part mandoc bugs, and in part intentional and useful
> changes in groff behaviour).

It seems likely that the RC2 process is going to take long enough that
you will have time to triage the remainder of these 37 issues.  I regret
any bugs I've introduced to groff; at the same time it sounds like
mandoc's test suite is doing its job admirably--it is creating work,
yes, but also exposing bugs in multiple formatters and identifying
places where increasing behavioral parity between groff and mandoc will
redound to all users' benefit.

> After the RC, it is the critical to not commit anything except fixes
> for critical regressions that people reported from RC testing.
> In particular, after an RC, no bugs must be fixed that were already
> known before the RC was sent out.

Bertrand and I have agreed upon a less strict approach.  We, or
possibly I, will identify a recent commit in groff Git from which he
will upload a distribution archive of groff 1.23.0.rc2.  I've
volunteered to write (most of) a release announcement email to try to
drum excitement and get people who didn't test rc1 to test this one.

We'll get feedback on that, and either address or postpone work on
issues currently marked as Blockers, and address any (likely build
system-related) problems reported, probably from relatively exotic host
environments.  (I find it quite sad that any ISA that isn't x86 or ARM
is "exotic" these days.  I remember when Debian shipped its stable
release for 11 platforms, including PowerPC, HP PA-RISC, MIPS, SPARC,
DEC Alpha, Itanium, and m68k.)

That done, RC3 will be tagged and released.  At that point, no further
"code" changes will be done on the master branch until after final
release.  Even a build/install failure on a platform might not gate the
release at that point, unless it's a regression from RC2, because we
asked for testers _for_ RC2.  If an "exotic" host environment regresses,
it might have to wait for a 1.23.1 release (which I personally would be
anxious to do under such circumstances).

If, somehow, we break Linux x86-64 with the RC3 tag (I can't imagine
this would happen--Bertrand I would both refuse to attach such a tag to
a commit that misbehaved so badly), then we'd need to have an RC4.

But not otherwise.

Documentation updates will proceed throughout the RC process, all the
way up to final release.  If there are none after RC3, then the "1.23.0"
tag will be applied to the same commit as "1.23.0.rc3".  Then, an
official distribution archive gets created and announced to info-gnu
(and here, of course).

That's the plan.  This is, as far as I can recall, exactly what we did
for groff 1.22.4.

> So it is totally obvious to me that the code base is *not* in a good
> shape and quite far from being ready for an RC.

I don't agree.  I think it is very close, though of course I reserve the
right to change my mind if one of the 37 mandoc regression tests points
out something horrific.  But if it does, I will wonder how _I_ didn't
have a regression or unit test for the same misbehavior already, since
I've spent the last 4+ years studying groff's own man page renderings in
text, HTML, and PDF closely on a nearly daily basis.

Discussions like this are why I _didn't_ want the maintainer job.  It
takes a lot of time to explain and justify what are ultimately
irreducibly subjective criteria.  On the other hand, we all benefit from
the transparency.

[snipping your projected timeline because while I'd like to dismiss your
time estimates as too pessimistic, I can't]

Regards,
Branden

signature.asc
Description: PGP signature