Re: Frequency-based replacement and store_avg_object_size

John Dilley wrote:

> Duane and other Squid Masters,
>
> I ran into an ussue when using GDSF the other day. I was using the
> default value of store_avg_object_size (13 KB) and ran into a problem
> where filemap utilization was too high, eventually leading to a crash.
> The reason is that GDSF leads Squid to store more, but smaller objects,
> thereby improving hit rate. (By contrast LFUDA leads to storing fewer,
> larger objects, reducing overall bandwidth demand. Both favor popular
> objects, rather than recently accessed objects.)
>
> In looking at the source I've come up with a couple suggestions on a
> fix but want to run them by you. First, it can be addressed in the
> config file with a comment to reduce store_avg_object_size. But who
> will see this? These parameters are related, so I don't think this is
> ideal.

Not enough people would see it. We'd be answering questions about it till
the cows came home.

> Second it can be addressed whereever avgObjectSize is used, for example
> to multiply it by a constant depending upon replacement policy (e.g.,
> 1.0 for LRU, 4.0 for GDSF, 0.4 for LFUDA). But this is not scalable and
> leads to problems when someone forgets to include the appropriate
> translation. Places that seem to matter are storeInitHashValues,
> storeDigestCalcCap, maybe storeDirConfigure.
>
> Finally the modification could be made where the value is read from the
> config file (cf_parser.c). The disadvantage here is that the value used
> at runtime is different from what is specified in the config file,
> which may lead to confusion (but you'd know this better than I).

Lies to users? Probably not good. we _might_ be able to get away with it
if we spat a cache.log message out during operation to tell them what we
were doing.

> Instead of this modification, which seems like a hack to me, the config
> parameter could be eliminated completely and Squid could determine the
> average object size strictly based upon replacement policy without any
> input from the user. I tend to favor making things as automatic as
> possible, limiting the things users can screw up. Then the parameter
> would be set as LRU: 10 KB; GDSF: 2 KB; LFUDA: 15 KB. What do you
> think?

The numbers tend to vary too much, methinks. I have a deployment of squids
where the average object size (using LRU) is 4KB. A most unusual result,
and I was rather disturbed by it. All I can attribute it to is network and
proxy architectures downstream of the deployment. There are more proxies
down there than we have at other deployments, and only ten or fifteen
percent of them are squid (other deployments run to as much as 60% squid,
downstream).

> One final note - my personal home page has gone away on the HPL web
> server when I "defected" to Akamai. There is an HPL technical report
> that describes the policies, though, that can be referenced instead:
> http://www.hpl.hp.com/techreports/1999/HPL-1999-69.html. Please replace
> the URL in store_heap_replacement.c with that one... Thanks,

I don't have a hugely good answer. The best _I_ can think of (in my
not-yet-finished-the-first-cup-of-coffee haze) is to, yes, have some sort
of internal scaling factor, remove the average object size config item,
and either supply some sensible default (12K, say), then scale it...or, if
we have cache-objects, work out the actual average size for the current
setup, and use that for the rest of the run. Either way, spit a message to
cache.log.

There's startup overhead to this, of course. Can't build the filemap
without knowing the average object size. Got to scan the cache to find
that. Unless we can do a skip-sampling, and not check _every_ object, but
just a representative number.

Half-conscious thoughts,
D
Received on Wed Nov 03 1999 - 14:24:52 MST