Re: [Qemu-devel] [Nbd] [PATCH] Improve documentation of FUA and FLUSH

[Wouter Verhelst]

On Fri, Apr 01, 2016 at 11:17:29AM +0100, Alex Bligh wrote:
> On 1 Apr 2016, at 11:10, Wouter Verhelst <address@hidden> wrote:
> > On Fri, Apr 01, 2016 at 10:28:03AM +0100, Alex Bligh wrote:
> >> On 1 Apr 2016, at 09:35, Wouter Verhelst <address@hidden> wrote:
> >>>> +* All write commands (that includes both `NBD_CMD_WRITE` and
> >>>> +  `NBD_CMD_TRIM`) that the server completes (i.e. replies to)
> >>>> +  prior to processing to a `NBD_CMD_FLUSH` MUST be written to 
> >>>> non-volatile
> >>>> +  storage prior to replying to that `NBD_CMD_FLUSH`. The server SHOULD 
> >>>> ensure
> >>>> +  that all write command received prior to processing the 
> >>>> `NBD_CMD_FLUSH`
> >>>> +  (whether they are replied to or not) are written to non-volatile
> >>>> +  storage prior to processing an `NBD_CMD_FLUSH`; note this is a
> >>>> +  stronger condition than the previous 'MUST' condition. This
> >>> 
> >>> This seems to make little sense. Are you saying that suddenly now
> >>> sending a reply for FLUSH with outstanding writes is wrong? If not, the
> >>> above should be clarified.
> >> 
> >> The MUST sentence does not cover that situation as it only refers
> >> to completed writes.
> >> 
> >> The SHOULD sentence says that's a 'SHOULD NOT' situation in respect
> >> of writes that have PROCESSED (i.e actioned) whether or not they
> >> have been replied to. Of course the client has no way of knowing
> >> whether they have been PROCESSED without a reply.
> >> 
> >> Personally I think the SHOULD clause is pretty pointless and is
> >> unnecessary, but that's where the conversation got to n years
> >> ago I believe.
> > 
> > I'm still not sure what it's supposed to mean, though. Clearly, you
> > should at the very least reword it, if not...
> > 
> >> Happy to delete the last sentence if that's wrong.
> > 
> > ... remove it instead.
> 
> If I can't even explain it, it doesn't bode well!

:-)

> I think there are three types of writes that are relevant at the point
> of replying to a FLUSH:
> 
> Type A: writes that are truly 'in flight', i.e. have been sent, but have
> not been 'processed' (i.e. write(2) has not been called).

Right.

> Type B: writes that are have been processed (i.e. write(2) has not been

ITYM s/not// here?

> called) but the reply has not yet been sent to the client - either the
> reply hasn't been made yet or (more likely) it's sitting in some queue.
>
> Type C: writes that have been processed and a reply sent.
> 
> 
> The first sentence (the 'MUST') refers to Type C writes and says these
> MUST be persisted to permanent storage prior to replying to the FLUSH.
> 
> Type A writes may be in a buffer server side or even client side, so
> no one expects anything to happen with respect to those.
> 
> The 'SHOULD' bit concerns Type B writes. It's saying that if you've
> actually done the write call (but not yet replied) you SHOULD persist
> these to disk prior to replying to the FLUSH. This type of situation
> doesn't (last time I looked) happen in nbd-server.c because it does
> not process requests in parallel.

Actually, it does, since a few months or so :-)

> However it may happen elsewhere.  The question is what to do with them.

Right.

As far as the client (and as a direct result, the protocol) is concerned,
however, there is no substantial difference between type A and type B.
The request has been sent, and the reply hasn't been received yet. This
is the same for A as well as for B.

I don't think it is at all useful to tell the server what it should do
in that situation. If the server implements FLUSH with fsync(), then
type B replies are handled exactly like type C replies. If the server
implements FLUSH with something else, however, then flushing type B
replies could potentially be much more expensive then flushing type C
replies.

I think this is one of those cases where the protocol spec simply shouldn't
say anything (at most, the protocol spec could explicitly say that it
doesn't prescribe any particular behaviour). Requests that have been
sent to the server MAY be flushed by a FLUSH request, but MAY equally
well not be. There is no reason to use SHOULD regarding anything about
that.

(I believe that is what the spec currently says, fwiw, but I didn't
check and may be wrong)

-- 
< ron> I mean, the main *practical* problem with C++, is there's like a dozen
       people in the world who think they really understand all of its rules,
       and pretty much all of them are just lying to themselves too.
 -- #debian-devel, OFTC, 2016-02-12