Re: O_DIRECT "packet mode" pipes on Linux

[Vito Caputo]

On Wed, Sep 23, 2020 at 11:53:10PM -0400, Lawrence Velázquez wrote:
> > On Sep 23, 2020, at 11:41 PM, Vito Caputo <vcaputo@pengaru.com> wrote:
> > 
> > On Wed, Sep 23, 2020 at 09:12:40AM -0400, Chet Ramey wrote:
> >> On 9/22/20 11:23 PM, Vito Caputo wrote:
> >>> Hello list,
> >>> 
> >>> Is there any chance we could get a | modifier for enabling O_DIRECT on the
> >>> created pipe?  "Packet" style pipes have some interesting and potentially
> >>> useful properties, it would be nice if bash made them more accessible.
> >> 
> >> Is there a general need, especially since they're Linux-specific?
> >> 
> > 
> > I'm not sure, but as far as GNU/Linux distros go bash, is kind of the
> > canonical shell, and this functionality is kind of inaccessible
> > without the shell wiring it up.
> 
> What functionality? I (and I'm sure some others) am not familiar
> with packet-style pipes and their benefits. You haven't actually
> described *how* exposing them would be useful, and why that would
> justify introducing new syntax that only matters/works on Linux.
> 

Packetized pipes establish well-defined boundaries between writes
reproduced at the read side.  If the write sizes are kept within
PIPE_BUF bounds, then you can be certain what's read is an atomic
record including nothing from a subsequent or previous write, with no
possibility for partial records.

It's useful if you're doing something like say, aggregating data from
multiple piped sources into a single bytestream.  With the default
pipe behavior, you'd have the output interleaved at random boundaries.
With packetized pipes, if your sources write say, newline-delimited
text records, kept under PIPE_BUF length, the aggregated output would
always interleave between the lines, never in the middle of them.

If we added this to the shell, I suppose the next thing to explore
would be how to get all the existing core shell utilities to detect a
packetized pipe on stdout and switch to a line-buffered mode instead
of block-buffered, assuming they're using stdio.  That should turn
their lines into packets on the pipe, and it all becomes generally
relevant across the existing shell utils landscape.  This heuristic
echoes of the terminal output detection for stdout line-buffering
already performed according to setvbuf(3).

Thanks,
Vito Caputo