Re: [squid-users] Caching files from Amazon S3

On Mon, Mar 17, 2008, Robert Collins wrote:

> > I've looked at the -3 stuff, and its missing about as much as the -2
> > stuff is missing. The memory store is only a small part of the overall
> > problem handling sparse objects.
> >
> > (Unless there's some code I've missed that handles other range-request relevant
> > stuff.)
>
> In -3 the client side was overhauled to talk in object offsets, so a
> range request would ask the store for the relevant bytes, and rather
> than getting an opaque stream and range parsing it, it gets the bytes
> back; likewise the store insertion by the server side writes offset,
> length data into the store, not opaque data.

Yes, but I think more thought needed to go into teaching the store about
sparse objects (which the backend store hooks don't cover); teaching the
store about re-populating the memory cache (which is needed for range request
processing -and- for general better performance); is there any code to properly
process range replies and spit stuff into the store.

As I said, a lot of it hasn't been done (I don't think 60% is a credible
answer) and I think "handle range requests" is something that needs to be
thought of as part of a squid internal planning/discussion rather than
something seperate.

(The handling of the range request thing was one of my reasons for being
initially upset with the -3 direction and becoming less interested in
contributing; the whole point why I ripped out the range request handling
in early -3 days was so we could have this discussion and figure out the
best way of doing it without being saddled with the partially-implemented
range request stuff, far before we implemented anything..)

Adrian

-- 
- Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support -
- $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -