Re: [squid-users] ecap adapted body limited to 64k

NP: squid-dev mailing list is over thataway --->

On 24/11/2012 10:27 a.m., carteriii wrote:
> It appears that squid limits the adapted body size of any ecap adapter to
> 65535, perhaps due to BodyPipe::MaxCapacity = 64*1024. I am certain of the
> limitation, but not 100% certain BodyPipe::MaxCapacity is the cause.
>
> My adapter properly receives all chunks of data via noteVbContentAvailable
> and assembles one larger internal buffer. The first time that abContent()
> is called in my adapter, it is called with abContent(offset=0,
> size=4294967295). My test case (www.google.com) has a body size of 100043
> bytes, so I pass it all back at once.
>
> My adapter next sees a call to abContentShift(size=65535) and then no
> additional calls to any ab* function in my adapter. I have set
> debug_options to ALL,9 and been looking for relevant lines. Below are a few
> lines that seem appropriate. Note that the lines starting with
> "Adapter::Xaction(RESPMOD,...)" are from my adapter, showing the calls to
> abContent
>
> 2012/11/23 15:01:54.986|
> Adapter::Xaction(RESPMOD,0xa4ff090)::abContent(offset=0, size=4294967295)
> 2012/11/23 15:01:54.986| XactionRep.cc(648) moveAbContent: up to 100043
> bytes
> 2012/11/23 15:01:54.987|
> Adapter::Xaction(RESPMOD,0xa4ff090)::abContentShift(size=65535)
>
> Note that XactionRep.cc shows that it has received the full 100043 bytes of
> the adapted body from my adapter.

Nope. It received your variable value saying you had 100K to send.
The move notice is informing how much it received / copied out of your
adaptor buffer.

> In looking at adaptation/ecap/XactionRep.cc, I see it in turn calls
> BodyPipe::putMoreData(). In looking at the definition of BodyPipe, I see
> enum { MaxCapacity = 64*1024 }. That would seem to potentially be the
> source of the size limitation; ***HOWEVER*** I believe that the proper
> behavior is that abContent should be called repeatedly until all of the body
> has been transferred. So even with the MaxCapacity limitation, I believe
> squid should be repeatedly or recursively calling abContent and
> abContentShift in my adapter until it has retrieved the entire adapted body.

Possibly. You will have to talk to the eCAP developers about that. It
may also be that the abContentShift() was your notification of how mauch
was handled and your adaptors task to track what action is needed for
followup.

> In XactionRep.cc, Adaptation::Ecap::XactionRep::moveAbContent(), I see that
> libecap::nsize is being passed as the size parameter to abContent().
> Perhaps it should instead pass a maximum value of the
> BodyPipe::MaxCapacity??? At a minimum, the code should continue to call
> abContent until it has acquired all of the adapted body. Perhaps the
> function needs to check to see if "const size_t used" is less than c.size
> and then "putMoreData()" ???
>
> As far as I can tell, this seems like an bug that limits all adapters to a
> maximum adapted body size of 64k.

It is a limitation on adapters which try to hoard the entire size of
large objects before delivery.
eCAP is designed to work at high speed in moderately sized chunks on
inifinitely long objects.

Amos
Received on Sat Nov 24 2012 - 03:27:45 MST