Re: O_DIRECT "packet mode" pipes on Linux

[Robert Elz]

    Date:        Wed, 23 Sep 2020 23:18:36 -0700
    From:        Vito Caputo <vcaputo@pengaru.com>
    Message-ID:  <20200924061836.4cu5ttjzdn7rv4xd@shells.gnugeneration.com>

  | In my example, the aggregator can trivially read the separate records
  | at the write boundaries from each of the connected packetized pipes.
  | The reads return at the write boundaries.  Without packetized pipes
  | you'd need to parse the contents to search for record boundaries.

So all of this is to save this mythical process from needing to examine
its input?

  | Presumably this is *why* we don't already have an input
  | version of `tee`.

No, that's because there is no good syntax that works for creating it
(and certainly wasn't when the basic utilities were being constructed)
and because there isn't much in the way of demand for this functionality.

But I can assure you that if there was a real need, and a reasonable way
to use such a program, the need to examine the input to process it
correctly wouldn't be any kind of barrier to implementation.


  | This is probably true, though if a packetized pipe were introspectable
  | we could request the behavior via the ||| construction, while
  | simultaneously enabling record boundaries regardless of how the
  | contents are delimited.

If ... I assume that the O_xxx for a pipe could be examined (fcntl)
though whether stdio would want to bother (a slight slowdown for
everything for a rare case) is another question.

  | If consumers knew about packetized pipes,
  | they could treat the separately returned reads as records
  | independently of what's inside.

And do what with them, independently of what's inside?   Such a
process either knows what it is dealing with (text lines, binary
records, ...) in which case it can both do something with it, and
parse it, or it doesn't, in which case there is little useful that
it can do, is there?

kre