[bug #40225] Deterministic output ordering

[Frank Heckenbach]

Follow-up Comment #1, bug #40225 (project make):

First, I wonder how deterministic the order in a non-parallel build actually
is. I guess, formally it's not, but for practical purposes it is, i.e. given
the same make version and the same makefiles, it's unlikely to change.

When I had a need like yours (rarely), what I did was either:

- Preferredly: Build with "-s" and fix warnings early, so my build logs are
quite small (ideally empty), even on large builds, so I'll see new messages
with my bare eyes. :)

- Diff sorted logs. This way, I can at least see new or eliminated messages,
but without context. (Here, output-sync will not help at all, though you still
may want it for other reasons.)

Apart from these workarounds, I do think a proper solution could be
implemented in make itself. Since output-sync keeps the output from each job
in a temp file, it's already half way there.

I think such a new feature only makes sense with "recursive" output-sync. The
purpose of line/target sync is to get messages earlier, which would conflict
with (or be negated by) ordering. So effectively you want a new sync mode,
let's call it "ordered" which implies "recursive", so we'd have a sequence of
5 modes (none, line, target, recursive, ordered) with successively stronger
synchronization at the cost of more deferred messages. This will simplify the
implementation since we'll only have to deal with one sync mode, and we won't
have to care about make's own messages which have to be treated specially with
line/target, but not recursive sync.

With output-sync, make dumps those temp files to the real stdout/stderr when
it reaps the children (job.c:935: reap_children() -> output_dump()), so one
idea would be to reap the children in the order they were started (by waiting
for a specific process rather than any one). But this has a big drawback
because it may drastically reduce parallelism (a long-running job would
prevent waiting for other children and starting new ones, even if job server
slots are free), so I don't think this will work well.

Instead, I think we'll have to separate reaping and syncing, i.e.
reap_children() won't call output_dump() in "ordered" mode, and when make
terminates (succesfully or not), it will clean up all children and dump their
output (free_child() -> output_close() -> output_dump()).

The active children are kept in a list (global variable "children" and "next"
pointer) until reaped. We cannot just move the children to a new list in
reap_children() because we'd lose the order at that point. But if we simply
keep the reaped children in the original list after reaping, this might also
lead to problems: All code which traverses this list (e.g. job.c:819) needs to
check for active children (not sure if child->file->command_state will do or a
new flag would be needed), and for huge builds with maybe thousands of total
jobs, this will basically add an O(n) factor there (not sure though if that's
really significant, compared to actually running the child jobs).

A way to avoid these problems might be to put the children in a dual list,
i.e. "children" and "next" continue to contain the active children, and a
second list (say, global variable "children_to_sync" and a new "next_to_sync"
pointer in struct child) contains all children (at least in "ordered" mode).
new_job() would then link the new entry in both lists, reap_children() ->
free_child() would only remove it from the former and the final syncing at the
end would traverse the latter, whereas other code that traverses the children
list would need no changes.

All of this assumes that the order that the children are started is
deterministic -- at least as deterministic as in the non-parallel case (see
above), which I haven't verified.

I might be missing something, but this could be a way to implement this
feature with moderate effort. I'm afraid I don't have the time to write it
myself (and the feature is not that important to me), and I don't know if Paul
does, but if you, Josh, would like to try it, I could offer some advice when
needed and review your changes, since I'm quite familiar with the output-sync
internals. But first, I'd like to know what Paul thinks about adding even more
complexity to output-sync ...

PS: While checking the above, I noticed vmsjobs.c calls free_child() which is
static in job.c. Of course, I can't test it, since I don't have VMS, but it
might be a bug.


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?40225>

_______________________________________________
  Nachricht gesendet von/durch Savannah
  http://savannah.gnu.org/