squid-3 update

Hi,

I've been doing some profiling of squid-3 over the last few days and
I thought I'd share my experiences with the list.

My current aim with the squid-3 branch is trying to hunt down and
squish any bugs that I find before we start running trigger-happy with
features. I think there's been enough changes to the codebase in -3
that its worth trying to fix stuff.

I've profiled squid with both poll and kqueue. With poll, the obvious
kernel and userland bottlenecks are poll related - building the
arrays, processing them, spitting them out again.. not very pretty.

With kqueue that goes away, and the request rate is free to climb
higher - but not by much. I think that may be a tuning and polygraph-related
thing however, but I'm not terribly interested in delving into that
just right now.

The interesting point is the call graphs and the per-function CPU
utilisation - there's .. well, lots of them. In what could be said as
the understatement of the year on this list - I believe that the current
squid codebase (squid-2 _and_ squid-3) does, well, far too much.

The commercial cache vendors out there will snigger. :)

Perhaps we're doing the same thing multiple times on a request
(like parsing). Perhaps we could look at removing some of the
steps (eg, using an async version of writev() to write out headers
instead of packing them into a buffer first).

I have the call graphs available from gprof if anyone is interested.

So, I'm going to continue bug hunting and I'd like to put forward
the suggestion that we try very hard to only bugfix and not, well,
change the codebase too much until the codebase is stable.
I personally have _plenty_ of stuff to throw in to squid-3 to rework
some more of the comm-stuff, but I'm trying to hold off. :)

I'd then like to propose that, instead of throwing in any features,
we take one good, hard, intense look at the codebase and decide
what we can do to improve efficiency. I think the two suggestions
above are a good start. We'll also stop thrashing the poor CPU
bus if we try to limit the copies that we do. (On this note,
does anyone know of any way to extract how far we're pushing
the CPU FSB under FreeBSD/Linux?)

Finally, just to formalise things, what bugs are people seeing in squid-3?
I'd like to put them somewhere (even under a squid3 category under bugzilla!)
so we can squish things before they become a long-standing problem.
Robert, Reuben - stand up and be heard with your buglist, please! :)

Adrian, squidding out.
Received on Mon Oct 28 2002 - 02:20:24 MST