[squid-users] Increased memory usage (memory leak?)

Hello list (sorry for the long message),

we are using eight servers (running FreeBSD 7.3 amd64 with 2G ram each)
as transparent proxy servers (Squid 2.7.9 along with squidGuard 1.4) for
the content filtering service of the Greek School Network.

Our current squid configuration includes:
   http_port 80 transparent
   http_port 8080
   cache_mem 192 MB
   cache_swap_low 80
   cache_swap_high 85
   maximum_object_size 4096 KB
   ipcache_size 94208
   ipcache_low 90
   ipcache_high 95
   cache_store_log none
   cache_dir null /tmp
   redirect_program /usr/local/bin/squidGuard
   redirect_children 48
   redirector_bypass on
   uri_whitespace encode
   request_header_max_size 25 KB

Although it is not peak season (schools have exams and are about to
close for summer), we are experiencing high memory usage in some cases
(see cache01 at http://noc.ntua.gr/~christia/squid-vsize.png). All boxes
are identical. The memory usage seems to be related to the type of the
requests (not the volume, http://noc.ntua.gr/~christia/squid-reqs.png).
In order to rule out any possibility of server specific problems we
moved the client requests from one cache box to another and the memory
allocation problem moved along (at first it was cache04 and now it is
cache01).

We compared the cache.log files and found no special messages in cache01
or in any other server that showed increased memory usage. After looking
into the access.log we tend to believe that the memory leaps occur
during periods that squid receives too many concurrent requests for
sites that are responding too slowly or not responding at all
(firewalled, not sending any packets back). In these cases, concurrent
requests vary from 40 reqs/sec to 700 reqs/sec, the average service
varies from 570 msec to 178502 msec and the HTTP return codes are
usually TCP_MISS/417, TCP_MISS/504 and TCP_MISS/000.

Cache manager reports for cache01 692882 allocated cbdata
clientHttpRequest objects occupying 742381 KB and 100% of them in use
(ten times more than the other cache boxes).

Full report is available here:
   http://noc.ntua.gr/~christia/cache01.mem.20100609-21:35.txt
   http://noc.ntua.gr/~christia/cache01.info.20100609-21:35.txt

At the same 'top' in cache01 reports about the squid process:
   1575 MB virtual size
    303 MB resident size

Finally, pmap (FreeBSD sysutils/pmap port) report is available here (in
case it is useful):
   http://noc.ntua.gr/~christia/cache01.pmap.20100609-21:35.txt

Anyway, is this behaviour normal or is it a bug or memory leak? I am
willing to try any suggestion, provide any other information and help
debugging.

For the moment, I am manually excluding from the transparent cache
schema the sites that seem to cause problems. I am also considering
adding a maxconn acl and lowering some connection timeouts. Waiting for
180000 msec for a firewalled connection is *too* much.

Regards,
Panagiotis

-- 
Panagiotis J. Christias    Network Management Center
p.christias_at_noc.ntua.gr    National Technical Univ. of Athens, GREECE