Squid Patches for Transfer-Encoding/TE (against 2.3devel2)

Folks,

As some of you might know, I'm very interested in the hop-to-hop
semantics of HTTP/1.1 as the basis for deploying new features into the
network.. the deployed basis of clients and servers makes it
challenging to try new techniques end-to-end; but as HTTP is defined
hop to hop we can implement these changes over a large proportion of a
transfer just by going from proxy to proxy.

I'm in the process of conducting a study to try out the effects of a
number of different transfer-encodings for this purpose and am using
squid as the base proxy. Of course squid (2.3devel2) has no support
for transfer-encoding, or TE (or making http/1.1 requests for that
matter)..

so, I created support for transfer-encoding, and TE.. I also
implemented a fairly generic mechanism for including your own
transfer-encodings, and implemented a few. (I implemented chunked,
gzip, deflate, and a couple called jones and biff for illustrative
purposes.. the last two are just byte swappers but they show how the
framework operates.. the compression formats are unfortunately done
with an AppliedTheory library that I can't share with you, but I left
the code in there as I think the zlib stuff is an almost drop in
replacement and can be adapted easily)..

I'm making these patches available to anybody who's interested at:

  http://proximate.appliedtheory.com/transfer-encoding/

here's the basic api..

/* Transfer Encoding Filters: They work like this..

  filter (a,b,c,d,e)
    a - is a character pointer to the data to be filtered
    b - is length of a
    c - is address of a character pointer where the resulting buffer
        should be stored
    d - pointer to integer to store length of c in
    e - pointer to a general data pointer that the filter may use to
        maintain state, it begins life as NULL.

the return value is a 4 bit mask
   0x01 - c is a newly allocated buffer and should be freed when the
          calling function has completed the filter
   0x02 - reserved internally for chunking
   0x04 - reserved internally for chunking
   0x08 - set this bit if your function was called with input, but did
          not produce any out (for instance if you're buffering it in the
          context provided by e.) prevents squid from thinking EOF
          has been reached.

 call sequence
     first call: *e is NULL but a is not and b>0
     body calls : a is not null, b>0 and value of *e determined by
                  previous calls
     last call: b==0.. good time to clean up *e if you've stored stuff
                   there.. you may produce output if necessary, but
                   last call will be repeated.
*/

and they are installed using the new_xlat() function in
client_side.c which there are several examples of..

Also, the patches are against 2.3devel2 which had a couple bugs for me
out of the box.. they were trivial and I fixed them, but those fixes
are in the patch too, so I should enumerate them:

  * dns_internal.c: parsing problem with extra \n at the end of my
    resolv.conf..
  * strListisMember() ignored the length argument it parsed, and
    indeed would only return true if the the argument it was searching
    for was the last one in a list (i.e. list="x,y"
    strListisMember(list,"x") would be false but
    strListisMember(list,"y") would be true.. fixed.. also added a
    strListIsMember_q that works the same way but allows the list
    members to have q values and returns the qvalue * 1000.

There are a few caveats however, both wrt specification and quality of
implementation..

specification: rfc 2616 section 3.6 prohibits the sending of
transfer-encodings to 1.0 clients which the squid<->squid hop is.. In
my implementation (which is only running privately for the purposes of
study and doesn't claim to be compliant to anything) I will send a
transfer-encoding to a client if that encoding is explicitly mentioned
on the TE line with a positive q value.. (which should be safe as both
the te todo and chunked need to be explicit on that line so the client
is capable)

 because 2616 also requires that any response with a non-identity
tr-enc be chunked and no content-length be present I *did* implement
those requirements.. which means squid can now both send and receive
chunked encodings while preserving persistent connections.. though it
will only send chunked either to a 1.1 or higher client or a 1.0
client with it on the TE header. (and because chunked is required to do
more interesting things, it always puts it in it's requests.. though
if it made 1.1 requests this would be superfluous.)

related thought: given that chunked is in there now, what is lacking
from the MUST list to make HTTP/1.1 requests?

quality of implementation: There are a few things that aren't suitable
for production code, but the basis of what needs to be there is in, so
I thought I'd pass it on and y'all could consider if it should be
dressed up.

  * selection of what objects to apply te's to is hardcoded to be
    text/* and *postscript* in the C.

  * new te's are just whacked into client_side.c, it lacks a good
    configure mechanism

  * squid.conf should be used to decide what TE's to announce on a
    request.. hardcoded in http.c for the moment.

  * no thought given to how an object should be stored.. right now it
    is stored with any t-e's it is received with..

  * when sending out new responses a very naive "strip to identity,
    rebuild to desired output" set of filters is constructed.. this
    can no doubt be optimized..

all that not withstanding, it's a very handy framework for what I'm
trying to do, so under normal opensource tenets I'll make the patches
available.

I'm very interested in dialogue and feedback, though I'm not on
squid-dev@.. I could request that be changed if convenient for folks.

-Patrick

--
Patrick R. McManus - AppliedTheory Corportation   -	Software Engineering
http://pat.appliedtheory.com/~mcmanus			Lead Developer
mcmanus@AppliedTheory.com	'Prince of Pollywood'	Standards, today!
*** - You Kill Nostalgia, Xenophobic Fears. It's Now or Neverland. - ***