Re: [squid-users] Increased memory usage (memory leak?)

On Thu, Jun 10, 2010 at 12:33:18AM +0000, Amos Jeffries wrote:
> On Wed, 09 Jun 2010 22:29:50 +0300, Panagiotis Christias
> <p.christias_at_noc.ntua.gr> wrote:
> > Hello list (sorry for the long message),
> >
> > we are using eight servers (running FreeBSD 7.3 amd64 with 2G ram each)
> > as transparent proxy servers (Squid 2.7.9 along with squidGuard 1.4) for
>
> > the content filtering service of the Greek School Network.
> >
> > Our current squid configuration includes:
> > http_port 80 transparent
> > http_port 8080
> > cache_mem 192 MB
> > cache_swap_low 80
> > cache_swap_high 85
> > maximum_object_size 4096 KB
> > ipcache_size 94208
> > ipcache_low 90
> > ipcache_high 95
> > cache_store_log none
> > cache_dir null /tmp
> > redirect_program /usr/local/bin/squidGuard
> > redirect_children 48
> > redirector_bypass on
> > uri_whitespace encode
> > request_header_max_size 25 KB
> >
> > Although it is not peak season (schools have exams and are about to
> > close for summer), we are experiencing high memory usage in some cases
> > (see cache01 at http://noc.ntua.gr/~christia/squid-vsize.png). All boxes
>
> > are identical. The memory usage seems to be related to the type of the
> > requests (not the volume, http://noc.ntua.gr/~christia/squid-reqs.png).
> > In order to rule out any possibility of server specific problems we
> > moved the client requests from one cache box to another and the memory
> > allocation problem moved along (at first it was cache04 and now it is
> > cache01).
> >
> > We compared the cache.log files and found no special messages in cache01
>
> > or in any other server that showed increased memory usage. After looking
>
> > into the access.log we tend to believe that the memory leaps occur
> > during periods that squid receives too many concurrent requests for
> > sites that are responding too slowly or not responding at all
> > (firewalled, not sending any packets back). In these cases, concurrent
> > requests vary from 40 reqs/sec to 700 reqs/sec, the average service
> > varies from 570 msec to 178502 msec and the HTTP return codes are
> > usually TCP_MISS/417, TCP_MISS/504 and TCP_MISS/000.
> >
> > Cache manager reports for cache01 692882 allocated cbdata
> > clientHttpRequest objects occupying 742381 KB and 100% of them in use
> > (ten times more than the other cache boxes).
> >
> > Full report is available here:
> > http://noc.ntua.gr/~christia/cache01.mem.20100609-21:35.txt
> > http://noc.ntua.gr/~christia/cache01.info.20100609-21:35.txt
> >
> > At the same 'top' in cache01 reports about the squid process:
> > 1575 MB virtual size
> > 303 MB resident size
> >
> > Finally, pmap (FreeBSD sysutils/pmap port) report is available here (in
> > case it is useful):
> > http://noc.ntua.gr/~christia/cache01.pmap.20100609-21:35.txt
> >
> > Anyway, is this behaviour normal or is it a bug or memory leak? I am
> > willing to try any suggestion, provide any other information and help
> > debugging.
> >
> > For the moment, I am manually excluding from the transparent cache
> > schema the sites that seem to cause problems. I am also considering
> > adding a maxconn acl and lowering some connection timeouts. Waiting for
> > 180000 msec for a firewalled connection is *too* much.
> >
> > Regards,
> > Panagiotis
>
> Looks like broken almost-HTTP/1.1 clients to me. The behaviour when a
> client sends "Expect: 100-continue" without properly supporting it
> themselves. Squid does not fully support that feature of HTTP/1.1, just
> enough to send back the right 417 error and negotiate its non-existence
> with the client.
>
> What I think is happening is that many requests come in using the Expect:
> header for HTTP/1.1 (which Squid does not yet support) the clients then
> wait for the 100 status code to return. Which will never happen.
> Standards-compliant Squid will reply with 417 errors and hope the client
> retries without Expect:.
>
> Check that your cache01 is configured with "ignore_expect_100 off". This
> will send back the 417 error immediately for any clients which are actually
> working to negotiate a faster response back. Broken clients will die
> immediately with the error instead of dying after holding your resources
> for that timeout.

ignore_expect_100 parameter is not set in any of our cache boxes's
squid.conf. I assume that this means that the default value ("off")
is used.

Panagiotis

-- 
Panagiotis J. Christias    Network Management Center
P.Christias_at_noc.ntua.gr    National Technical Univ. of Athens, GREECE