Re: [squid-users] Slow memory leak

On 27.09.2012 07:09, tcr_at_raynersw.com wrote:
> Hi Eliezer,
>
> Thanks for the feedback. This memory leak causes real-world problems
> for me, as squid starts to do a lot of swapping when it exhausts
> physical RAM, and things slow down. Also, the sheer magnitude of the
> memory numbers is just ridiculous... squid easily grows to in excess
> of 10GB of resident memory as shown in top. I've got the servers
> restarting their squids periodically, but that's a pretty bad
> solution.
>
> Here is my squid.conf. Note two include files...
> squid_ns5_allowed_ips.conf and squid_blacklist_ips.conf . These are
> lists of IPs in an ACL. The allowed IPs one has lots of entries
> (almost 20,000) and that's the only thing I think is really unusual
> about my setup, so I'm wondering if that is exposing a leak
> somehwere.
>
>
> ##### BEGIN squid.conf #######
> http_port 5000
> http_port 5001
> http_port 5002
> http_port 5003
> http_port 5020
> http_port 5021
> http_port 5022
> http_port 5023
>
> cache_mgr [omitted]
> visible_hostname [omitted]

No need to omit hostname. This is the *public* FQDN which your squid
uses in error pages to load icon and sub-object URLs.

> max_filedesc 32768
>
> hierarchy_stoplist cgi-bin ?
> acl QUERY urlpath_regex cgi-bin \?
> cache deny QUERY

Dropping "QUERY" is a good idea. Most of the web is now dynamic pages
which this will block and a lot of that actually provides caching
information Squid-2.7+ can use. I've found the 10-20% HIT barrier to be
mostly caused by these rules.

> access_log /var/log/squid/access.log squid
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440

Add:
  refresh_pattern -i (/cgi-bin/|\?) 0 0% 0

NP: this replaces the "QUERY" ACLs and makes squid act properly when
handed dynamic content without proper caching controls.

> refresh_pattern . 0 20% 4320

NP: the following refresh_pattern are duplicates, and would never be
used due to the '.' pattern above.

> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern . 0 20% 4320
> acl demo_sites dstdomain .raynersoftware.com
> acl banned_sites dstdomain client.versiontracker.com
> acl banned_sites dstdomain .rapidshare.com
>
> # allow yuki2
> acl a_ips src 64.62.244.50/32
>
> include /etc/squid/squid_ns5_allowed_ips.conf
> include /etc/squid/squid_blacklist_ips.conf
>
> # acl all src 0.0.0.0/0.0.0.0
> acl manager proto cache_object
> acl localhost src 127.0.0.1/32 ::1
> acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1

NP: when you move to 3.2 the above are also built-in ACLs.

> acl SSL_ports port 443 563
> acl Safe_ports port 80
> acl Safe_ports port 21
> acl Safe_ports port 443 563
> acl Safe_ports port 70
> acl Safe_ports port 210
> acl Safe_ports port 1025-65535
> acl Safe_ports port 280
> acl Safe_ports port 488
> acl Safe_ports port 591
> acl Safe_ports port 777
> acl CONNECT method CONNECT
> http_access allow manager localhost
> http_access deny manager
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access deny to_localhost
> http_access deny banned_sites
> http_access deny banned_ips
> http_access allow demo_sites
> # we shorten allowed_ips to a_ips to save space in our include file
> http_access allow a_ips

Optimization:
   If you pre-filter your a_ips details to remove banned_ips, you shoudl
be able to convert to:

http_access allow demo_sites !banned_ips
http_access allow a_ips

> http_access deny all
> http_reply_access allow all
> #icp_access allow all
>
> pid_filename /var/run/squid.pid
>
> coredump_dir /var/spool/squid
>
> via off
> forwarded_for off
> ##### END squid.conf #######

> When I do a mgr:info on one particular box, I get this:
>
> Memory usage for squid via mallinfo():
> Total space in arena: 1101288 KB
> Ordinary blocks: 1098718 KB 287 blks
> Small blocks: 0 KB 0 blks
> Holding blocks: 17636 KB 9 blks
> Free Small blocks: 0 KB
> Free Ordinary blocks: 2569 KB
> Total in use: 1116354 KB 100%
> Total free: 2569 KB 0%
> Total size: 1118924 KB
> Memory accounted for:
> Total accounted: 284679 KB 25%
> memPool accounted: 284679 KB 25%
> memPool unaccounted: 834245 KB 75%
> memPoolAlloc calls: 620095300
> memPoolFree calls: 640694955
>
> Yet in ps, I see this:
>
> squid 2576 1.1 32.5 5388508 5314184 ? S Sep05 92:51
> (squid) -f /etc/squid/squid.conf
>
> So, squid is using 5GB of RAM yet it only knows about 1 of those GB.
>

FYI: mallinfo() function used by Squid to display some of those memory
statistics is known to contain a 32-bit wrap problem on 64-bit OS.

Since you said your Squid is at 5GB memory usage that is 1x 4GB wraps
and the remainder (+1 GB) is what we can expect to see mallinfo()
reporting as total. It may just be that which you are seeing.

NP: "Memory accounted for" is the Squid internal accounting. 284MB is
on record as currently in-use, with "unaccounted" being the mallinfo()
value minus the Squid "total accounted" (when 32-bit wrap happens this
shows large negative numbers, which is a dead giveaway. You are in the
fuzzy area of +N remainders on 32-bit wrap where it is unclear).

To be absolutely certain of memory leaks, build with valgrind support.
The cache mgr memory reports get appended with valgrind 'snapshot'
reports of usage/leaks and IIRC something for global leaks reports is
logged on shutdown.
  NP: its about time we ran 3.2 through another valgrind session, your
traffic looks high enough to provide useful details. If you are happy to
do this the valgrind report will be welcomed on the squid-dev mailing
list.

Amos
Received on Wed Sep 26 2012 - 22:32:43 MDT