Re: Chunked Encoding

[Andy Wingo]

On Thu 29 Sep 2011 16:09, Ian Price <address@hidden> writes:

> If you've used the (web ...) modules, you may have noticed that guile
> does not currently support chunked-encoding. This is expected in a
> HTTP/1.1 world, so I wrote an implementation to cover my immediate
> need, but I'm not particularly convinced of it, so I wanted to discuss
> this before sending a patch.

Thanks for the note!  We totally need to do this.

> What I did was introduce two new exported procedures for reading (all I
> needed at the moment), namely 'read-chunk' and 'read-chunked-response-body'.
>
>   (read-chunk port)
>   reads one chunk from a port.
>
>   (read-chunked-response-body response)
>   read the full body for the response and returns it as a bytevector. It
>   was written to be similar to 'read-response-body'.

Is it possible to use soft ports?.  That would be nice, and it would
allow client code to be able to read from the port until it gets EOF,
without being concerned about the transfer-encoding.  Same thing goes
for gzip/deflate/compress encoding.

> For writing chunks then, the obvious thing is to write two procedures
> 'write-chunk' and 'write-chunked-response-body' which perform the
> inverse. However, it seems to me that 'write-chunked-response-body' is a
> practically useless procedure, because if you ever had a full body, you
> can just use 'write-response-body'.

Again, soft ports?

> Another option I've been thinking over would be to go for a sort of
> chunking version of R6RS' 'transcoded-port' which would handle it
> transparently for users of the returned port.

Is this equivalent?  I don't have much experience with these.  It seems
that custom binary input/output ports are more appropriate.

> I'd also suggest extending 'http-get' from (web client) to handle
> chunked encoding (and trailers too, I guess) for the user.

For trailers, the RFC2616 says:

   A server using chunked transfer-coding in a response MUST NOT use the
   trailer for any header fields unless at least one of the following is
   true:

   a)the request included a TE header field that indicates "trailers" is
     acceptable in the transfer-coding of the  response, as described in
     section 14.39; or,

   b)the server is the origin server for the response, the trailer
     fields consist entirely of optional metadata, and the recipient
     could use the message (in a manner acceptable to the origin server)
     without receiving this metadata.  In other words, the origin server
     is willing to accept the possibility that the trailer fields might
     be silently discarded along the path to the client.

   This requirement prevents an interoperability failure when the
   message is being received by an HTTP/1.1 (or later) proxy and
   forwarded to an HTTP/1.0 recipient. It avoids a situation where
   compliance with the protocol would have necessitated a possibly
   infinite buffer on the proxy.

So, in the default case in which we do not mention "trailers" in the
request, trailer fields should be strictly not required.  So the naive
case should just work.

However, I agree that it would be nice to get these fields, so I would
suggest that we store the parsed trailers in a weak map port ->
header-alist, and provide an accessor procedure.  That way if the client
knows that they should parse trailers, then they can get at them.  I
guess you're right that http-get should just DTRT.

> Comments kindly requested,

Again, very sorry for the delay!  I hope to be more on top of things
this month.

Cheers,

Andy
-- 
http://wingolog.org/