Re: subscribe squid-dev interest.

On Mon, 2002-11-18 at 19:40, l. walsh wrote:
> Hi, I'm Linda. I've been known to do development. I've been known to
> have
> opinions (informed, intuitive or My birth sign is
> in Leo and rising in Sag. How's that for an intro? Ok, that wasn't
> the intro whoever sits in judgement was looking for?

Good enough, but I don't sit in judgement, you'll have to wait for that
to occur...

> I'm interested in performance, extensibility and ease of use.

Neato.

> With squid, I'm wondering if anyone has done broken down where delays
> are. Like when a user request comes in, where does the time go before
> the user gets back an answer. How much is spent on waiting for DNS
> lookups, how much time before a remote system accepts the connection,
> how much time did the request take to send (likely close to zero in most
> cases), how much time was spent waiting for the user request to be
> returned and how much time to feed it back to the user? How much time
> is spent writing log files? For example with 'diskd', cache writes
> are moved into another process because disk writes are deemed to take
> too long, but are log files written synchronously? I assume reads from
> hosts and writes to clients are non-blocking.

There have been many analyses done. There are some generally known areas
for improvement, and many areas where squid is simply inefficient
internally, and could *really* use some maintenance. Also, it's
flexability allows folk to misconfigure it (see for instance the recent
emails regarding ACL types on squid-users.)

> I'd like to see such performance stats to be builtin, user configurable
> since many of the answers are going to vary depending on the the
> particular
> workload and hardware. But inclusion of such information allows a user
> to find hot spots and tune the server for their particular load.

This would be very cool. Some aspects of this already exist, but a clear
cachemgr page would be most excellent. I presume you are thinking of
something like:

Mean elapsed time to:
permit or deny request: 0.02 ms
redirect request : 0.1 ms
establish a connection to a server: 45.03 ms
....

?

> I'm also more immediately in having integrated pre and post filter
> plugins.
> Redirect provides essentially user output filtering before it gets to
> the squid process, but to my knowledge, nothing provides prefiltering
> on the content coming back from the host (or net) before it goes into
> the squid cache and there also doesn't seem to be a way to plug in a
> post filter that filters content coming out of squid but before it is
> returned to the client. Forgive me if these plugins are already
> provided
> but when I mentioned the concept on the user list, I got a bunch of
> blank
> looks.

Check out the iCAP patches for squid, they are a 'standard' to do that.

Also there are various projects to do specific things in squid - and
some generic projects like the clientStreams code in 3.0 that enable
such alterations to be more structured.

> I'd also like to see easier keyword implementation to log all traffic
> to/from specific clients in a 'recorder' type fashion. I have some
> 'smart devices' that want to talk to the web -- including some programs
> on my winPC's that want to talk to various sites on a regular basis or
> they
> get 'cranky'. I don't want to prevent the communication, but I do want
> to
> monitor what is going on. In the case of the devices, it would help
> with
> writing compatible home servers, for example.

Another good idea. There are several log alteration projects around, we
are looking for a sponsor to make the most generic one 'happen', which
is very similar to what you suggest (in fact, it's a superset in all but
one aspect - the recording of all data).

> I'm not sure how inter-client communication is currently done, but
> in some cases, a client library that uses shared memory might be more
> efficient (or then again may not be...heck if I really know anything
> until
> I've tried it).

Yep. We use that with diskd. I'm currently rewriting the IPC support
routines to be more modular, but that shouldn't interact badly with what
you mention.

Rob