Re: Partial Content...

On Sun, 2003-04-27 at 15:25, Leeann BENT wrote:
> Hi All - I posted this question a few days ago on squid-users with no
> response, so I thought I'd try again here. Basically, I'm looking into how
> to start working on implementing "Partial Content" responses, so I had a
> few general questions. I've also spoken to Rob Collins about this a
> little, but he suggested I bring the discussion over here...

Yep - the best forum for it :}.

> So - here are my questions:
>
> I wanted to inquire about any other extensions that might support "Partial
> Content" (or something similar in nature) as part of their implementation
> (for example, caching streaming data in Squid).

I'm not aware of any finished patches, but there might be something
around :}.

> Also, I wanted to ask if anyone knew whether any of the other caches out
> there support partial content. I looked into Apache, but couldn't find it
> in the documentation. I would assume it must, since the filesystem support
> must already be there for the server. I was wondering if anyone on here
> knew anything about it (or any other cache whose source code I can take a
> gander at) offhand.

Partial content support is not trivial - I would be very very surprised
if Apache's proxy function supports cached partial objects.

> Finally, I spoke with Rob Collins and he mentioned that objects in memory
> can be sparse in Squid 3, but I was a little unclear what this meant. Can
> anyone clear this up for me and explain why it's needed for Partial
> Response caching?

In squid 2.5, objects that pass through memory are represented as
follows:

memory object:
Offset of lowest data
Head of data chunk list

chunk:
pointer to next chunk
used count of bytes (length)
buffer array

Now, as there is no offest in each chunk, squid can *only* represent a
single contiguous region of a given entity.

In squid 3.0 this becomes:

memory object:
head of data chunk list

chunk:
pointer to next chunk
offset of this chunk in the entity
length of this chunk
buffer array

Which means that squid can represent multiple regions of a given entity,
and that there can be gaps in an entity's available data.

For example, squid 3.0 can represent this:
bytes 0-400
bytes 500-1000

whereas squid 2.5 cannot. The best squid 2.5 can do is:
bytes 0-1000

What does all this mean? Well, to combine ranges, you need to be able to
feed the data you have to the client, which may or may not be asking for
a contiguous set of regions. Secondly, when serving partial content, we
don't want to have to read all the object into ram, we can simply read
in what we need, into this sparse structure.

It's called sparse, because there are gaps in it :}.

Hope that helps, apologies if it's unclear - it's 3am here ... {

Rob

-- 
GPG key available at: <http://users.bigpond.net.au/robertc/keys.txt>.