Re: Storing partial responses

----- Original Message -----
From: "Joe Cooper" <joe@swelltech.com>
To: "Robert Collins" <robert.collins@itdomain.com.au>; "Squid-Dev" <squid-dev@squid-cache.org>
Sent: Tuesday, December 12, 2000 7:45 PM
Subject: Re: Storing partial responses

> Hi Robert,
>
> I won't volunteer to 'help' per se, but I will volunteer to peer
> interestedly over your shoulder the whole time. I might even point at
> stuff, and say "Hmmmm" on occasion.

Hmmmmm :] That sounds good. At this point with my knowledge of the squid code, I am wary of making design or code choices that may
impact squid' performance. Squid is not all that simple :]

> Ok, that's my way of saying this sounds like an interesting area of
> work, and I was just looking at that part of the code this past weekend
> (which hasn't been touched since 2.2STABLE5+patch, as far as I could
> tell) for a client who had a curiosity about byte range requests and
> Squid. And what I'll actually do in the way of helping is try to keep
> up with your development and help you where I can with testing and idea
> bouncing. I might even attempt to write a line or two of code.
>
> Let me know if there's anything I can do to help get things rolling.

Exactly what's happen since I opened my mouth: discussions from interested parties on the architecture: just keep on talking.

> First thoughts:
>
> Requires a disk format or swap.state change, probably. (Whether we
> break it up or put the pieces all in one place.)

Almost certainly. I alluded to that possibility in the mail I sent out. What I am wondering is whether the on-disk metadata for a
store obejct should have a version tag, and a little bit of logic like thus:

* on reading: if no version (how to test this?) or an old version tag, call that versions unpack code. Each change to the ondisk
metadata then just needs the creation of a new unpack code version, and update the old ones to provide sensible defaults to variable
they don't know. (Or even just have a StoreSetSensibleDefaults for the old versions to call before they unpack).
* on writing, the metadata is always written with the current metadata version code. Thus over time the cached objects in the store
will be updated with newer metadata versions until none have the old version.

The reason to bother doing that is that in-memory changes to swap.state information are easy to code and take effect on the next
squid restart, but on-disk changes are hard at the moment, as admins (quite rightfully) don't want to throw away 18Gb of cache just
to upgrade squid.

>
> Breaks the storetree stuff that is going to be merged in from the
> squidng project. storetree does a hash of the URL for storage
> indexing...and keeps no record of what was just stored. If the object
> is not complete, reiser_raw won't know that and will serve it
> incomplete. So a new field would also be needed in reiser_raw. Which I
> guess applies to the standard object store as well...so chalk that one
> up to "necessary change" for version DEVEL2.5.

Yes. the fields available for the object store will have to change, and that may hit memory utilisation hard on big caches. This
needs a careful approach.

> Possibilities:
>
> The idea of rewriting files and expiring the 'pieces' as new pieces come
> in, until we have a complete object. Adds significant write overhead,
> but keeps the index simple and objects end up being as large as they can
> be...reducing read and seek overhead. I think we should avoid
> fragmenting of the object store, if possible, for performance reasons.
> But that's just a hunch. I could be wrong, since range requests are
> being used on mostly big objects anyway, I guess? But this plan doesn't
> account for 'holes' in the ranges being requested. How likely is that?
> And would it be wise to just accept having to fetch the whole object
> or the data in between two points in such a case in order to avoid the
> complexity of having several separate parts of one object at different
> locations in the store?

rewriting the files and expiring the pieces? As an example do you mean I a client grabs bytes 0-999 of a file, and then 1000-1999
you would expire the 0-999 range? or do you mean release the 0-999 object and store the new one with 0-1999? I'll assume you mean b)
and yes that might well work. Certainly I think the overhead should be less than retrieving the object from origin for every single
user.

The pet peeve that got me looking at this project is adobe acrobat reader. I haven't dug into the request sequence yet, but the way
it grabs individual pages seems almost certain to be range requests. So we end up getting the whole file, and not caching it :[

regarding fragmenting the object store - yes I agree. Sending out a hit of a 1 MB file from 1 different locations on the squid disks
isn't ideal.

I think that some tuning parameters will be needed - for your example of getting the whole object. Well if the obejct is "small" -
say 1 MB and cachable, and a user's agent grabs the first 100Kb (header info) and then the last 200Kb you only incur 700kb
overhead - AND DELAY - in getting the whole object. The users agent is allowed to send a series of requests like
0-999
10000-10999
1000-1050
1000-2000

and we should not delay the last three just because a near-end-of-file request was made on a big file - say 10Mb. But a heuristic
like (if more than 3 partial responses are stored or in progess, get all the missing bits of the file on the next request) may make
sense.

Rob
Received on Tue Dec 12 2000 - 16:43:49 MST