Re: memory-mapped files in Squid

Kevin Littlejohn wrote:

> I think this is going to be a big part of the tuning for the cyclic
> fs - how do you deal with, for instance, the next MS service pack -
> 50Mb of heavily-hit object. Personally, re-writing objects on hits
> doesn't give me warm fuzzy feelings...

I would probably use a rewrite rate limit of 30%. It the object are in
the newest 30% of the store, then don't rewrite it to the head.

> Except if you've already incurred the overhead of writing to disk, why
> incur it again?

As I see it there are three options in a cyclic file system:

1. Rewrite reused objects when reused, with a limit on by which rate the
object is rewritten to the head.
Penalties:
  * extra writes
  * changing store index numbers.

2. When rewriting a store chunk, preserve those objects that we are
interested in keeping, but pack them together.
Penalties:
  * extra reads
  * extra writes
  * no obvious way to determine what to keep and what to throw away
  * changing store index numbers.

3. When rewriting a store chunk, preserve those objects that we are
interested in keeping where they are in the chunk.
Penalties:
  * fragmentation
  * no obvious way to determine what to keep and what to throw away.

Currently I think 1 is the way to go, but 2 or 3 allows for a more
flexible or customisable replacement policy.

> I know that there's many other things affecting disk access
> speeds - but that media is still the slow part of the chain (well, after
> network), so it makes sense to me to keep disk use low.

There will always be some kinds of tradeoffs. Goal is to find a design
that allows for some balancing between the different kinds.

> In fact, squid already handles a drive 'going away'.

In theory yes, in practice no. Squid may continue to function, but
complains loudly.

> I'd be curious to see what the 'best' lower limit for size of objects is before
> you start using this 'staging area'. I'd also be curious to see what impact
> it has on performance if the object sizes drift - if you're heavily hitting
> that area, you may start to loose some of the cyclic gains :(

I see this FIFO + standard fs as a possible future extension if deemed
neccesary. Not as part of the basic cyclic fs design.

Also the standard FS staging area should only be used if the object is
larger than we want to queue in memory. Smaller objects should go
directly to the cyclic store.

> There's definately some nifty ideas there, but I'm not convinced that the
> gains from a cyclic fs over a more traditional style fs are enough to warrant
> the extra management complexity - shuffling objects around on disk, etc.

I wouldn't say that there is much extra management complexity, but I
admit that some of Squids interfaces does not fit well.

> Guess the only real way to tell is to implement and see ;)

Unfortunately I don't have the time needed to implement anything right
now, and I am not sure if I will be able to later.

/Henrik
Received on Tue Jul 29 2003 - 13:15:56 MDT