Re: Storing partial responses

On Tue, Dec 12, 2000, Joe Cooper wrote:

> Requires a disk format or swap.state change, probably. (Whether we
> break it up or put the pieces all in one place.)
>
> Breaks the storetree stuff that is going to be merged in from the
> squidng project. storetree does a hash of the URL for storage
> indexing...and keeps no record of what was just stored. If the object
> is not complete, reiser_raw won't know that and will serve it
> incomplete. So a new field would also be needed in reiser_raw. Which I
> guess applies to the standard object store as well...so chalk that one
> up to "necessary change" for version DEVEL2.5.

Well, I still want to do some more work on my new store fs code.
The storetree code is a good idea, but it needs to be completed
(read: turned into *the* FS layer instead of being tacked on like
it is) and I just don't have the time.

> Possibilities:
>
> The idea of rewriting files and expiring the 'pieces' as new pieces come
> in, until we have a complete object. Adds significant write overhead,
> but keeps the index simple and objects end up being as large as they can
> be...reducing read and seek overhead. I think we should avoid
> fragmenting of the object store, if possible, for performance reasons.
> But that's just a hunch. I could be wrong, since range requests are
> being used on mostly big objects anyway, I guess? But this plan doesn't
> account for 'holes' in the ranges being requested. How likely is that?
> And would it be wise to just accept having to fetch the whole object
> or the data in between two points in such a case in order to avoid the
> complexity of having several separate parts of one object at different
> locations in the store?
>
> I'll stop talking for now until I actually understand what I'm talking
> about.

If you want to support partial objects in the store, you're better off
having it implemented as part of the FS interface and allow different
FSes to support partial objects or not.

-- 
Adrian Chadd			"Here's five for the cake, and
<adrian@creative.net.au>	  five to buy a clue."
				    - Ryan, Whatever it Takes