Re: [squid-users] Squid uses way too much RAM and starts swapping ...

From: guest01 <guest01_at_gmail.com>
Date: Mon, 30 May 2011 10:07:13 +0200

Hi,

Any news on this topic? Unfortunately, RAM is running full within days
and at the moment, our workaround is to do a reboot ... We would
appreciate any other solution!

thanks,
peter

On Wed, May 11, 2011 at 3:47 PM, guest01 <guest01_at_gmail.com> wrote:
> On Wed, May 11, 2011 at 10:47 AM, Amos Jeffries <squid3_at_treenet.co.nz> wrote:
>> On 11/05/11 19:19, guest01 wrote:
>>>
>>> Hi,
>>>
>>> I am currently using squid 3.1.12 as forward-proxy without
>>> harddisk-caching (only RAM is used for caching). Each server is
>>> running on RHEL5.5 and is pretty strong (16 CPUs, 28GB RAM), but each
>>> servers starts swapping a few days after start. The workaround at the
>>> moment is to reboot the server once a week, which I don't really like.
>>> But swapping leads to serious side effects, e.g. performance troubles,
>>> ...
>>>
>>> way too much swapping:
>>> http://imageshack.us/m/52/6149/memoryday.png
>>>
>>> I already read a lot of posts and mails for similar problems, but
>>> unfortunately, I was not able to solve this problem. I added following
>>> infos to my squid.conf-file:
>>> # cache specific settings
>>> cache_replacement_policy heap LFUDA
>>> cache_mem 1600 MB
>>> memory_replacement_policy heap LFUDA
>>> maximum_object_size_in_memory 2048 KB
>>> memory_pools off
>>> cache_swap_low 85
>>> cache_swap_high 90
>>>
>>> (There are four squid instances per server, which means that 1600*4 =
>>> 6400MB RAM used for caching, which is not even 1/4 of the total
>>> available amount of RAM. Plenty enough, don't you think?)
>>
>> Not that is "for HTTP object caching", emphasis on *caching* and "HTTP
>> object". In-transit objects and non-HTTP caches (Ip cache, domain name
>> cache, persistent connections cache, client database, via/fwd database,
>> network performance cache, auth caches, external ACL caches) and the indexes
>> for all those caches use other memory.
>>
>> Then again they should all be using no more than a few GB combined. So you
>> may have hit a new leak (all the known ones are resolved before 3.1.12).
>
> Ok, very strange. But at least it is reproducible, it takes about a
> week until squid is starting to swap ...
> http://img191.imageshack.us/img191/9615/memorymonth.png
>
>>>
>>> Very strange are the negative values (Memory usage for squid via
>>> mallinfo():) from the output below. Maybe that is a reason for running
>>> out of RAM?
>>
>> mallinfo() sucks badly when going above 2GB of RAM. It can be ignored.
>>
>> The section underneath it "Memory accounted for:" is Squids own accounting
>> and more of a worry. It should not have had negatives since before 3.1.10.
>>
>>>
>>> HTTP/1.0 200 OK
>>> Server: squid/3.1.12
>>> Mime-Version: 1.0
>>> Date: Wed, 11 May 2011 07:06:10 GMT
>>> Content-Type: text/plain
>>> Expires: Wed, 11 May 2011 07:06:10 GMT
>>> Last-Modified: Wed, 11 May 2011 07:06:10 GMT
>>> X-Cache: MISS from xlsqip03_1
>>> Via: 1.0 xlsqip03_1 (squid/3.1.12)
>>> Connection: close
>>>
>>> Squid Object Cache: Version 3.1.12
>>> Start Time:     Wed, 27 Apr 2011 11:01:13 GMT
>>> Current Time:   Wed, 11 May 2011 07:06:10 GMT
>>> Connection information for squid:
>>>         Number of clients accessing cache:      1671
>>>         Number of HTTP requests received:       16144359
>>>         Number of ICP messages received:        0
>>>         Number of ICP messages sent:    0
>>>         Number of queued ICP replies:   0
>>>         Number of HTCP messages received:       0
>>>         Number of HTCP messages sent:   0
>>>         Request failure ratio:   0.00
>>>         Average HTTP requests per minute since start:   810.3
>>>         Average ICP messages per minute since start:    0.0
>>>         Select loop called: 656944758 times, 1.820 ms avg
>>> Cache information for squid:
>>>         Hits as % of all requests:      5min: 17.4%, 60min: 18.2%
>>>         Hits as % of bytes sent:        5min: 45.6%, 60min: 39.9%
>>>         Memory hits as % of hit requests:       5min: 86.1%, 60min: 88.9%
>>>         Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
>>>         Storage Swap size:      0 KB
>>>         Storage Swap capacity:   0.0% used,  0.0% free
>>>         Storage Mem size:       1622584 KB
>>>         Storage Mem capacity:   100.0% used,  0.0% free
>>
>> Okay 1.6 GB of RAM used for caching HTTP objects. Fully used.
>>
>>>         Mean Object Size:       0.00 KB
>>
>> Problem #1. It *may* be Squid not accounting for the memory objects in the
>> mean.
>>
>>>         Requests given to unlinkd:      0
>>> Median Service Times (seconds)  5 min    60 min:
>>>         HTTP Requests (All):   0.01648  0.01235
>>>         Cache Misses:          0.05046  0.04277
>>>         Cache Hits:            0.00091  0.00091
>>>         Near Hits:             0.01469  0.01745
>>>         Not-Modified Replies:  0.00000  0.00091
>>>         DNS Lookups:           0.00190  0.00190
>>>         ICP Queries:           0.00000  0.00000
>>> Resource usage for squid:
>>>         UP Time:        1195497.286 seconds
>>>         CPU Time:       22472.507 seconds
>>>         CPU Usage:      1.88%
>>>         CPU Usage, 5 minute avg:        5.38%
>>>         CPU Usage, 60 minute avg:       5.44%
>>>         Process Data Segment Size via sbrk(): 3145032 KB
>>>         Maximum Resident Size: 0 KB
>>>         Page faults with physical i/o: 8634
>>> Memory usage for squid via mallinfo():
>>>         Total space in arena:  -1049140 KB
>>>         Ordinary blocks:       -1277813 KB  87831 blks
>>>         Small blocks:               0 KB      0 blks
>>>         Holding blocks:          2240 KB      5 blks
>>>         Free Small blocks:          0 KB
>>>         Free Ordinary blocks:  228673 KB
>>>         Total in use:          -1275574 KB 122%
>>>         Total free:            228674 KB -22%
>>>         Total size:            -1046900 KB
>>> Memory accounted for:
>>>         Total accounted:       -1375357 KB 131%
>>>         memPool accounted:     2818947 KB -269%
>>>         memPool unaccounted:   -3865847 KB 0%
>>>         memPoolAlloc calls:       111
>>>         memPoolFree calls:  8322084644
>>> File descriptor usage for squid:
>>>         Maximum number of file descriptors:   1024
>>>         Largest file desc currently in use:    563
>>>         Number of file desc currently in use:  472
>>>         Files queued for open:                   0
>>>         Available number of file descriptors:  552
>>>         Reserved number of file descriptors:   100
>>>         Store Disk files open:                   0
>>> Internal Data Structures:
>>>          96996 StoreEntries
>>>          96996 StoreEntries with MemObjects
>>>          96980 Hot Object Cache Items
>>>              0 on-disk objects
>>>
>>> Has anyone experienced similar things or does even know a solution?
>>
>> Not since we fixed Squid's capacity to calculate storage used by >2GB
>> objects.
>>
>> Can you find the actual maximum and average object size for the cached
>> objects in that Squid? (the mgr:vm_objects report should have all their
>> details, "inmem_hi:" being the object size)
>
> Unfortunately, we had to reboot this server. At the moment there is no
> swapping. But the mean object size is still 0.0kb:
> xlsqip03:/root>/opt/squid/bin/squidclient -h 10.122.125.4
> cache_object://10.122.125.4/info
> HTTP/1.0 200 OK
> Server: squid/3.1.12
> Mime-Version: 1.0
> Date: Wed, 11 May 2011 13:33:58 GMT
> Content-Type: text/plain
> Expires: Wed, 11 May 2011 13:33:58 GMT
> Last-Modified: Wed, 11 May 2011 13:33:58 GMT
> X-Cache: MISS from xlsqip03_1
> Via: 1.0 xlsqip03_1 (squid/3.1.12)
> Connection: close
>
> Squid Object Cache: Version 3.1.12
> Start Time:     Wed, 11 May 2011 09:35:49 GMT
> Current Time:   Wed, 11 May 2011 13:33:58 GMT
> Connection information for squid:
>        Number of clients accessing cache:      814
>        Number of HTTP requests received:       504290
>        Number of ICP messages received:        0
>        Number of ICP messages sent:    0
>        Number of queued ICP replies:   0
>        Number of HTCP messages received:       0
>        Number of HTCP messages sent:   0
>        Request failure ratio:   0.00
>        Average HTTP requests per minute since start:   2117.6
>        Average ICP messages per minute since start:    0.0
>        Select loop called: 15248273 times, 0.937 ms avg
> Cache information for squid:
>        Hits as % of all requests:      5min: 17.6%, 60min: 15.8%
>        Hits as % of bytes sent:        5min: 38.0%, 60min: 34.3%
>        Memory hits as % of hit requests:       5min: 88.2%, 60min: 87.0%
>        Disk hits as % of hit requests: 5min: 0.1%, 60min: 0.1%
>        Storage Swap size:      0 KB
>        Storage Swap capacity:   0.0% used,  0.0% free
>        Storage Mem size:       1622020 KB
>        Storage Mem capacity:   100.0% used,  0.0% free
>        Mean Object Size:       0.00 KB
>        Requests given to unlinkd:      0
> Median Service Times (seconds)  5 min    60 min:
>        HTTP Requests (All):   0.02190  0.01847
>        Cache Misses:          0.04277  0.04776
>        Cache Hits:            0.00091  0.00091
>        Near Hits:             0.02069  0.02069
>        Not-Modified Replies:  0.00091  0.00091
>        DNS Lookups:           0.00190  0.00094
>        ICP Queries:           0.00000  0.00000
> Resource usage for squid:
>        UP Time:        14288.578 seconds
>        CPU Time:       642.922 seconds
>        CPU Usage:      4.50%
>        CPU Usage, 5 minute avg:        5.70%
>        CPU Usage, 60 minute avg:       5.27%
>        Process Data Segment Size via sbrk(): 3035580 KB
>        Maximum Resident Size: 0 KB
>        Page faults with physical i/o: 0
> Memory usage for squid via mallinfo():
>        Total space in arena:  -1158592 KB
>        Ordinary blocks:       -1187512 KB   8338 blks
>        Small blocks:               0 KB      0 blks
>        Holding blocks:          2240 KB      5 blks
>        Free Small blocks:          0 KB
>        Free Ordinary blocks:   28920 KB
>        Total in use:          -1185273 KB 103%
>        Total free:             28921 KB -3%
>        Total size:            -1156352 KB
> Memory accounted for:
>        Total accounted:       -1290919 KB 112%
>        memPool accounted:     2903385 KB -251%
>        memPool unaccounted:   -4059737 KB 0%
>        memPoolAlloc calls:       111
>        memPoolFree calls:  263492993
> File descriptor usage for squid:
>        Maximum number of file descriptors:   1024
>        Largest file desc currently in use:    372
>        Number of file desc currently in use:  298
>        Files queued for open:                   0
>        Available number of file descriptors:  726
>        Reserved number of file descriptors:   100
>        Store Disk files open:                   0
> Internal Data Structures:
>        105790 StoreEntries
>        105790 StoreEntries with MemObjects
>        105780 Hot Object Cache Items
>             0 on-disk objects
>
> xlsqip03:/root>/opt/squid/bin/squidclient -h 10.122.125.4
> cache_object://10.122.125.4/vm_objects | head -n 50
> HTTP/1.0 200 OK
> Server: squid/3.1.12
> Mime-Version: 1.0
> Date: Wed, 11 May 2011 13:34:39 GMT
> Content-Type: text/plain
> Expires: Wed, 11 May 2011 13:34:39 GMT
> Last-Modified: Wed, 11 May 2011 13:34:39 GMT
> X-Cache: MISS from xlsqip03_1
> Via: 1.0 xlsqip03_1 (squid/3.1.12)
> Connection: close
>
> KEY 00C06887AD5B919595711834A5195511
>        STORE_OK      IN_MEMORY     SWAPOUT_NONE PING_DONE
>        CACHABLE,DISPATCHED,VALIDATED
>        LV:1305110370 LU:1305110371 LM:1157000708 EX:-1
>        0 locks, 0 clients, 1 refs
>        Swap Dir -1, File 0XFFFFFFFF
>        GET http://www.hardwaremuseum.at/mitarbeiter/keske.jpg
>        inmem_lo: 0
>        inmem_hi: 37138
>        swapout: 0 bytes queued
>
> KEY 00A02D3964813C2CC4A7D6553D2CF0A7
>        STORE_OK      IN_MEMORY     SWAPOUT_NONE PING_DONE
>        CACHABLE,DISPATCHED,VALIDATED
>        LV:1305110512 LU:1305111016 LM:1275291526 EX:-1
>        0 locks, 0 clients, 2 refs
>        Swap Dir -1, File 0XFFFFFFFF
>        GET http://nyx.at/bikeboard/images/design/icon_hotels.png
>        inmem_lo: 0
>        inmem_hi: 1507
>        swapout: 0 bytes queued
>
> KEY 0020B363AC66E4EAF4B812DCCB561B99
>        STORE_OK      IN_MEMORY     SWAPOUT_NONE PING_DONE
>        CACHABLE,DISPATCHED,VALIDATED
>        LV:1305110827 LU:1305110833 LM:966956701 EX:-1
>        0 locks, 0 clients, 1 refs
>        Swap Dir -1, File 0XFFFFFFFF
>        GET http://windfind.ivwbox.de/blank.gif
>        inmem_lo: 0
>        inmem_hi: 296
>        swapout: 0 bytes queued
>
> KEY 00C09B8188CC913834CF292CC2A4D72E
>        STORE_OK      IN_MEMORY     SWAPOUT_NONE PING_DONE
>        CACHABLE,DISPATCHED,VALIDATED
>        LV:1305110909 LU:1305110991 LM:1153932790 EX:-1
>        0 locks, 0 clients, 3 refs
>        Swap Dir -1, File 0XFFFFFFFF
>
> xlsqip03:/root>/opt/squid/bin/squidclient -h 10.122.125.4
> cache_object://10.122.125.4/vm_objects | grep inmem_hi
>        inmem_hi: 37138
>        inmem_hi: 1507
>        inmem_hi: 296
>        inmem_hi: 747
>        inmem_hi: 5149
>        inmem_hi: 1449
>        inmem_hi: 5231
>        inmem_hi: 3594
>        inmem_hi: 7911
>        inmem_hi: 699
>        inmem_hi: 141554
>        inmem_hi: 8997
>        inmem_hi: 977
>        inmem_hi: 37114
>        inmem_hi: 6229
>        inmem_hi: 79923
>        inmem_hi: 1016
>        inmem_hi: 11075
>        inmem_hi: 1475
>        inmem_hi: 6113
>        inmem_hi: 3118
>        inmem_hi: 26894
>        inmem_hi: 1433
>        inmem_hi: 2763
>        inmem_hi: 2388
>        inmem_hi: 287
>        inmem_hi: 2481
>        inmem_hi: 297
>        inmem_hi: 3399
>        inmem_hi: 217
>        inmem_hi: 1549
>        inmem_hi: 41865
>        inmem_hi: 4920
>        inmem_hi: 7857
>        inmem_hi: 1608
>        inmem_hi: 5666
>        inmem_hi: 9507
>        inmem_hi: 8653
>        inmem_hi: 11728
>        inmem_hi: 6106
>        inmem_hi: 452
>        inmem_hi: 2740
>        inmem_hi: 1122
>        inmem_hi: 2183
>        inmem_hi: 3861
>        inmem_hi: 14803
>        inmem_hi: 861
>        inmem_hi: 26080
>        inmem_hi: 28724
>        inmem_hi: 2094
>        inmem_hi: 428
>        inmem_hi: 1445
>        inmem_hi: 468
>        inmem_hi: 21754
>
> I also checked our Test-Squid with Squid 3.2.0.7:
> squid_at_xlsqit01 /etc/squid $ /opt/squid/bin/squidclient -h 10.122.125.2
> cache_object://10.122.125.2/info
> HTTP/1.1 200 OK
> Server: squid/3.2.0.7
> Mime-Version: 1.0
> Date: Wed, 11 May 2011 13:40:19 GMT
> Content-Type: text/plain
> Expires: Wed, 11 May 2011 13:40:19 GMT
> Last-Modified: Wed, 11 May 2011 13:40:19 GMT
> Connection: close
>
> Squid Object Cache: Version 3.2.0.7
> Start Time:     Fri, 29 Apr 2011 07:56:13 GMT
> Current Time:   Wed, 11 May 2011 13:40:19 GMT
> Connection information for squid:
>        Number of clients accessing cache:      28
>        Number of HTTP requests received:       44628
>        Number of ICP messages received:        0
>        Number of ICP messages sent:    0
>        Number of queued ICP replies:   0
>        Number of HTCP messages received:       0
>        Number of HTCP messages sent:   0
>        Request failure ratio:   0.00
>        Average HTTP requests per minute since start:   2.5
>        Average ICP messages per minute since start:    0.0
>        Select loop called: 1549800905 times, 1.365 ms avg
> Cache information for squid:
>        Hits as % of all requests:      5min: 0.0%, 60min: 0.0%
>        Hits as % of bytes sent:        5min: 100.0%, 60min: 57.0%
>        Memory hits as % of hit requests:       5min: 0.0%, 60min: 0.0%
>        Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
>        Storage Swap size:      0 KB
>        Storage Swap capacity:   0.0% used,  0.0% free
>        Storage Mem size:       30452 KB
>        Storage Mem capacity:    5.9% used, 94.1% free
>        Mean Object Size:       0.00 KB
>        Requests given to unlinkd:      0
> Median Service Times (seconds)  5 min    60 min:
>        HTTP Requests (All):   0.00189  0.00000
>        Cache Misses:          0.00000  0.12262
>        Cache Hits:            0.00000  0.00000
>        Near Hits:             0.00000  0.00000
>        Not-Modified Replies:  0.00000  0.00000
>        DNS Lookups:           0.00047  0.00047
>        ICP Queries:           0.00000  0.00000
> Resource usage for squid:
>        UP Time:        1057446.006 seconds
>        CPU Time:       366.647 seconds
>        CPU Usage:      0.03%
>        CPU Usage, 5 minute avg:        0.08%
>        CPU Usage, 60 minute avg:       0.04%
>        Process Data Segment Size via sbrk(): 63720 KB
>        Maximum Resident Size: 0 KB
>        Page faults with physical i/o: 0
> Memory usage for squid via mallinfo():
>        Total space in arena:   63984 KB
>        Ordinary blocks:        63645 KB     22 blks
>        Small blocks:               0 KB      0 blks
>        Holding blocks:         18480 KB     12 blks
>        Free Small blocks:          0 KB
>        Free Ordinary blocks:     339 KB
>        Total in use:             339 KB 0%
>        Total free:               339 KB 0%
>        Total size:             82464 KB
> Memory accounted for:
>        Total accounted:        54669 KB  66%
>        memPool accounted:      54669 KB  66%
>        memPool unaccounted:    27795 KB  34%
>        memPoolAlloc calls:   9115016
>        memPoolFree calls:   13310573
> File descriptor usage for squid:
>        Maximum number of file descriptors:   32768
>        Largest file desc currently in use:     60
>        Number of file desc currently in use:   41
>        Files queued for open:                   0
>        Available number of file descriptors: 32727
>        Reserved number of file descriptors:   200
>        Store Disk files open:                   0
> Internal Data Structures:
>          1461 StoreEntries
>          1461 StoreEntries with MemObjects
>          1459 Hot Object Cache Items
>             0 on-disk objects
> squid_at_xlsqit01 /etc/squid $ /opt/squid/bin/squidclient -h 10.122.125.2
> cache_object://10.122.125.2/info  | grep -i mean
>        Mean Object Size:       0.00 KB
>
> Seems to be the same problem:
> squid_at_xlsqit01 /etc/squid $ /opt/squid/bin/squidclient -h 10.122.125.2
> cache_object://10.122.125.2/vm_objects  | grep inmem_hi
>        inmem_hi: 410
>        inmem_hi: 4859
>        inmem_hi: 5927
>        inmem_hi: 1494
>        inmem_hi: 766
>        inmem_hi: 681
>        inmem_hi: 2215
>        inmem_hi: 423
>        inmem_hi: 236
>        inmem_hi: 386
>        inmem_hi: 3899
>        inmem_hi: 1060
>        inmem_hi: 1093
>        inmem_hi: 39037
>        inmem_hi: 2321
>        inmem_hi: 53163
>        inmem_hi: 822
>        inmem_hi: 272
>        inmem_hi: 420
>        inmem_hi: 421
>        inmem_hi: 381
>        inmem_hi: 2186
>        inmem_hi: 307
>        inmem_hi: 394
>
> Very strange indeed ... Maybe it is related to our RHEL5.5 environment
> or compile options? Otherwise my guess is that many people would have
> the same problem?
>
> xlsqip03:/root>/opt/squid/sbin/squid -v
> Squid Cache: Version 3.1.12
> configure options:  '--enable-ssl' '--enable-icap-client'
> '--sysconfdir=/etc/squid' '--enable-async-io' '--enable-snmp'
> '--enable-poll' '--with-maxfd=32768' '--enable-storeio=aufs'
> '--enable-removal-policies=heap,lru' '--enable-epoll'
> '--disable-ident-lookups' '--enable-truncate'
> '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid'
> '--with-default-user=squid' '--prefix=/opt/squid' '--enable-auth=basic
> digest ntlm negotiate'
> '-enable-negotiate-auth-helpers=squid_kerb_auth'
> --with-squid=/home/squid/squid-3.1.12 --enable-ltdl-convenience
>
> squid_at_xlsqit01 /etc/squid $ /opt/squid/sbin/squid -v
> Squid Cache: Version 3.2.0.7
> configure options:  '--enable-ssl' '--enable-icap-client'
> '--sysconfdir=/etc/squid' '--enable-async-io' '--enable-snmp'
> '--enable-poll' '--with-maxfd=32768' '--enable-storeio=aufs'
> '--enable-removal-policies=heap,lru' '--enable-epoll'
> '--disable-ident-lookups' '--enable-truncate'
> '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid'
> '--with-default-user=squid' '--prefix=/opt/squid'
> '-enable-negotiate-auth-helpers=squid_kerb_auth'
> --enable-ltdl-convenience
>
> We are basically only doing authentication (Kerberos, LDAP, NTLM) and
> ICAP authorization.
> Is there anything else I can do?
>
> thanks and regards!
> Peter
>
Received on Mon May 30 2011 - 08:07:23 MDT

This archive was generated by hypermail 2.2.0 : Mon May 30 2011 - 12:00:05 MDT