Re: [squid-users] Slow memory leak

Thanks. The thing is, I don't even think an iptables ACL will fix this leak, because I have other squid processes running without a lengthy ACL (using an auth program instead) and they too exhibit the leak.

Squid on most of my servers is from the standard CentOS 6.3 package. Compilation options as follows:

Squid Cache: Version 3.1.10
configure options: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr' '--libexecdir=/usr/lib64/squid' '--localstatedir=/var' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-logdir=$(localstatedir)/log/squid' '--with-pidfile=$(localstatedir)/run/squid.pid' '--disable-dependency-tracking' '--enable-arp-acl' '--enable-follow-x-forwarded-for' '--enable-auth=basic,digest,ntlm,negotiate' '--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SMB,YP,getpwnam,multi-domain-NTLM,SASL,DB,POP3,squid_radius_auth' '--enable-ntlm-auth-helpers=smb_lm,no_check,fakeauth' '--enable-digest-auth-helpers=password,ldap,eDirectory' '--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-icap-client' '--enable-ident-lookups' '--enable-linux-netfilter' '--enable-referer-log' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl' '--enable-storeio=aufs,diskd,ufs' '--enable-useragent-log' '--enable-wccpv2' '--enable-esi' '--with-aio' '--with-default-user=squid' '--with-filedescriptors=16384' '--with-dl' '--with-openssl' '--with-pthreads' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fpie' 'LDFLAGS=-pie' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fpie' --with-squid=/builddir/build/BUILD/squid-3.1.10

Regarding the config directives, I have tried changing the values of those in the past, but really the defaults are very conservative so they shouldn't be responsible for all this memory usage, unless the default is being ignored and treated as unbounded instead.

I would in fact love to set up a lot of in-mem caching, but I can't because this leak is killing me.

My hunch is that I've got some combination of config params which creates a very slow leak, but it's not well-known because it only shows up on very heavy-utilization servers. My 50-100Mbps throughput comes in the form of normal web traffic, i.e. lots and lots of page hits as opposed to a few large file downloads, so the requests per second are quite high. I just took a quick sample off one server and it was doing around 200 requests per second.

Further, I expect the leak is some little bit of mem that's getting allocated in the same piece of code over and over, and with gigabytes worth of leaked memory sitting in my processes, it should be easy to find once I get the proper debugging environment in place.

I am ready to help with debugging this leak. Would be great to get it patched.

Thanks
-Ty
Received on Wed Sep 26 2012 - 21:28:59 MDT