[libmicrohttpd] Re: libmicrohttpd question

[Christian Grothoff]

Dear Jared,

I've thought about your arguments and found that it would be indeed very 
difficult to implement such signaling by other means.  As a result, I've now 
added "MHD_CONTENT_READER_END_WITH_ERROR" (aka -2 as size_t) to the API -- and 
introduced "MHD_CONTENT_READER_END_OF_STREAM" as a new name for what used to 
be "-1" (for successful end-of-stream signalling).

These changes are in SVN 13718/13719.  Please let me know if this works for 
you (most of the diff is the documentation, the actual new code is like 4 
lines).

Happy hacking,

Christian

On Sunday 14 November 2010 22:09:35 Jared Cantwell wrote:
> I agree that there should be a cleaner way to handle this.  Our goal is
> simply to provide a URL through which content can be downloaded, and we
> would like to not have to specify the client used (nor write our own).
> 
> I can see many reasons why a chunked response can fail.  For example, data
> corruption or failed disks, both of which may only be detected once the
> response has started.  My understanding of the purpose behind chunked
> encoding in the first place is to not have to completely prepare the
> response ahead of time.  However, what if after partially preparing the
> response, an unrecoverable error occurs?  Based on my readings of the RFC,
> there doesn't seem to be a standard way of handling this situation.
> 
> Given the reasonable possibility of this situation, does it make sense for
> the API to provide a way to "abort" the response?  I think that closing the
> connection is a nice way to signal the other end that a fatal error has
> occurred, don't you?  To handle it otherwise, the developer would need to
> control both the client and the server, which seems unnecessary to me.
> 
> I can understand if you don't wish to incorporate this into your API, but
> would you mind giving a few pointers on where the code would need modified
> to make this happen (as a test run)?
> 
> Just a thought
> 
> Jared
> 
> On Sun, Nov 14, 2010 at 3:47 PM, Christian Grothoff
> 
> <address@hidden>wrote:
> > Well, this is not something a well-behaved HTTP server should ever do, so
> > I think the solution with the footers is cleaner (and we should probably
> > limit
> > the API to "legal" HTTP behavior --- after all, if the response
> > construction
> > could realistically fail, the developer should take care of this case in
> > a nice way).
> > 
> > My 2 cents
> > 
> > Christian
> > 
> > On Sunday, November 14, 2010 09:07:08 pm Jared Cantwell wrote:
> > > Thanks Christian!
> > > 
> > > I think this will help a lot.  Would it make sense to have the ability
> > > to prematurely close the connection with some error code (i.e. by
> > > returning -2)?  This is similar to what the try_ready_normal_body()
> > > does.  I would imagine that any normal client would interpret this as
> > > a failed request since they would have never received the trailing
> > > '0\n\r'.
> > > 
> > > ~Jared
> > > 
> > > On Sun, Nov 14, 2010 at 2:49 PM, Christian Grothoff
> > > 
> > > <address@hidden>wrote:
> > > > Dear Jared,
> > > > 
> > > > I was just about to fire back a response saying: "just add a footer",
> > 
> > but
> > 
> > > > then
> > > > I decided to look for the API for that and I found that -- while 99%
> > > > of the support for this is there -- a tiny function to do so was
> > > > missing from the API.
> > > > 
> > > > So I'm happy to introduce: "MHD_add_response_footer"; the only
> > 
> > difference
> > 
> > > > to
> > > > 'MHD_add_response_header' being that the 'kind' is set to
> > > > MHD_FOOTER_KIND. The
> > > > new API is in SVN 13683.
> > > > 
> > > > This helps you in that you simply add a footer that signals your
> > > > client software that some kind of error happened; doing so inside of
> > > > the response callback while also returning "-1" should do the trick.
> > > > 
> > > > In fact, the recommended approach would actually be to always add a
> > > > cryptographic checksum (MD5, SHA-1) as a footer.  That way, your
> > 
> > clients
> > 
> > > > will
> > > > also be better at detecting other transmission errors in your
> > > > gigabytes of data.  This is the best approach, and certainly also
> > > > how HTTP intended footers
> > > > to be used.
> > > > 
> > > > Naturally, this only works if you can teach your HTTP client to
> > > > respect footers.  If you cannot do that, I don't have a good answer
> > > > for you.
> > > > 
> > > > Happy hacking,
> > > > 
> > > > Christian
> > > > 
> > > > On Sunday, November 14, 2010 06:00:37 pm Jared Cantwell wrote:
> > > > > Hi Christian,
> > > > > 
> > > > > Thanks for a great piece of software-- it has helped me
> > > > > tremendously
> > 
> > to
> > 
> > > > > quickly implement a key feature for us.  I have a question about
> > > > > chunked uploads that I was hoping you could help me with. 
> > > > > According to the documentation (and my reading of the code), when
> > > > > using MHD_ContentReaderCallback to return chunks of data (i.e. in
> > > > > response
> > 
> > to
> > 
> > > > > a GET request), there doesn't seem to be a way to indicate that the
> > > > > chunked response failed.  Returning 0 will try again, -1 will
> > > > > assume the transfer completed successfully.  Is there a way to
> > > > > indicate to MHD that a chunked response failed?  The reason I ask
> > > > > is that we are transferring files on
> > > > 
> > > > the
> > > > 
> > > > > order of 10s of GBs, and we're getting this data from throughout
> > > > > our cluster.  Midway through the transfer, anything could happen,
> > > > > and it
> > > > 
> > > > would
> > > > 
> > > > > be nice to just prematurely close the connection to signal failure
> > > > 
> > > > (instead
> > > > 
> > > > > of assuming the end of the chunked data was reached).  Am I missing
> > > > > something in the interface, or is this just not currently possible?
> > > > > 
> > > > > Thanks for any help!!
> > > > > ~Jared