[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Using shell wrapper for descrambling parallel make output
From: |
Atte Peltomäki |
Subject: |
Re: Using shell wrapper for descrambling parallel make output |
Date: |
Mon, 14 Nov 2011 11:29:04 +0200 |
User-agent: |
Mutt/1.5.20 (2009-06-14) |
On Fri, Nov 11, 2011 at 10:24:25AM -0500, David Boyce wrote:
> On Fri, Nov 11, 2011 at 9:26 AM, Atte Peltomäki <address@hidden> wrote:
> > Hi,
> >
> > As you know, running parallel builds with '-jX' makes the shell output
> > difficult to read, since output from parallel jobs are mixed. To remedy
> > this, the use of a buffering shell wrapper has been suggested:
> >
> > http://cmcrossroads.com/cm-basics/12838-descrambling-parallel-build-logs
> >
> > I liked the idea, but implementation was lacking so I rewrote the
> > wrapper.
>
> That's a good thing to do but the work may be duplicative. See
> https://github.com/boyski/SYNCSH. Take a look and tell me if you think
> it's worth comparing notes in order to get the best of both.
>
> > If I have time and interest, I will look into implementing the wrapper
> > functionality into make itself. Don't hold your breath, though.
>
> Similarly, almost the same thing has already been done and a patch
> exists against 3.82 which I hope will make it into 3.83. See
> http://savannah.gnu.org/bugs/?33138. There's also a long thread in the
> bug-make archives from around the same time (4/2011) discussing it.
Nice work. Your implementation seems much more refined than mine. Only
one thing catches my attention; your version doesn't seem to properly
preserve the original line ordering between stdout and stderr. I suggest
solving this as I did:
- Don't write the output to files, just use a pipe, change vfork() to
fork() and read child output from the parent using the pipe.
- Use a simple struct as a buffer for the data: one member of the struct
is an int that is 0 for stdout and 1 for stderr, other member is a one
line of output (char *).
- Use a temporary buffer to read() from pipe, then dynamically allocate
only needed amount of memory per line to the the struct char* member.
See my code source for details.
--
Atte Peltomäki
address@hidden <> http://kameli.org
"Your effort to remain what you are is what limits you"