Re: [squid-users] Squid performance with high load

The above config for cache_dirs is not working probably.
I can see the aufs dir growing rapidly while the Rock directory has
been created but it is empty !

-------------------------------------------------------------------------------
Store Directory Statistics:
Store Entries : 1166040
Maximum Swap Size : 1740800000 KB
Current Store Swap Size: 85456552.00 KB
Current Capacity : 4.91% used, 95.09% free

Store Directory #0 (rock): /mnt/ssd/cache/
FS Block Size 1024 Bytes

Maximum Size: 307200000 KB
Current Size: 760592.00 KB 0.25%
Maximum entries: 2399999
Current entries: 5942 0.25%
Pending operations: 137 out of 0
Flags:

Store Directory #1 (aufs): /mnt/sas1/cache/store1
FS Block Size 4096 Bytes
First level subdirectories: 32
Second level subdirectories: 512
Maximum Size: 1433600000 KB
Current Size: 84695960.00 KB
Percent Used: 5.91%
Filemap bits in use: 1159378 of 2097152 (55%)
Filesystem Space in use: 121538556/-1957361748 KB (-5%)
Filesystem Inodes in use: 1176103/146243584 (1%)
Flags:
Removal policy: lru
LRU reference age: 0.17 days

On Sat, Mar 30, 2013 at 5:10 PM, Hasanen AL-Bana <hasanen_at_gmail.com> wrote:
> Thank you Amos for clarifying these issues.
> I will skip SMP and use single worker since Rock limit my max object
> size to 32kb when used in shared environments.
> My new cache_dir configuration looks like this now :
>
> cache_dir rock /mnt/ssd/cache/ 300000 max-size=131072
> cache_dir aufs /mnt/sas1/cache/store1 1400000 32 512
>
> I have enabled store.log to be used with some other software
> collecting data from it
>
> my disks are now mounted with
> noatime,barrier=0,journal_async_commit,noauto_da_alloc,nobh,data=writeback,commit=10
>
> I will keep the list posted with my results.
> Thanks.
>
> On Sat, Mar 30, 2013 at 12:57 PM, Amos Jeffries <squid3_at_treenet.co.nz> wrote:
>> On 30/03/2013 6:33 a.m., Hasanen AL-Bana wrote:
>>>
>>> Hi,
>>>
>>> I am running squid 3.2 with an average of 50k req/min. Total received
>>> bandwidth is around 200mbit/s.
>>> I have problem when my aufs cache_dirs reaches size above 600GB.
>>> Traffic starts dropping and going up again , happening every 20~30
>>> minutes.
>>> I have more that enough RAM in the system (125GB DDR3 !) , All disks
>>> are SAS 15k rpm, one of them is and SSD 450GB.
>>> So hardware should not cause any problem and I should easily spawn
>>> multiple squid workers any time.
>>> So what could cause such problems ?
>>
>>
>> #1 - AUFS is *not* SMP-aware component in Squid.
>>
>> Each of the two workers you are using will be altering the on-disk portion
>> of the cache without updating the in-memory index. When the other worker
>> encounters these over-written files it will erase them.
>>
>> For now you are required to use the macros hacks to split the cache_dir
>> lines between the workers.
>>
>>
>> #2 - you have multiple cache_dir per disk. Or so it seems from your
>> configuration. Is that correct?
>>
>> * squid load balances between cache_dir treating them as separate physcal
>> HDD when it comes to loading calculations.
>>
>> * the memory requirement for indexing these 1.6 TB of disk space is ~24GB
>> per worker plus the 20GB of shared emmory cache ==> 68GB of RAM.
>>
>>
>>
>> I advise to you inform Squid about the real disk topology it is working
>> with. No more than one AUFS cache_dir per physical disk and allocate the
>> cache_dir size to ~80-90% the full available disk space. With each worker
>> assigned to use a sub-set of the AUFS cache_dir and associated disks.
>>
>> For best performance and object de-duplication you should consider using
>> rock cache_dir for small objects. They *are* shared between workers, and can
>> also share disk space with an AUFS disk.
>>
>>
>>
>>> Thank you.
>>>
>>>
>>> ====================================================================================
>>> include /etc/squid3/refresh.conf
>>>
>>>
>>> cache_mem 20 GB
>>>
>>> acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
>>> acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
>>> acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
>>> acl localnet src fc00::/7 # RFC 4193 local private network range
>>> acl localnet src fe80::/10 # RFC 4291 link-local (directly
>>> plugged) machines
>>>
>>> acl SSL_ports port 443
>>> acl Safe_ports port 80 # http
>>> acl Safe_ports port 21 # ftp
>>> acl Safe_ports port 443 # https
>>> acl Safe_ports port 70 # gopher
>>> acl Safe_ports port 210 # wais
>>> acl Safe_ports port 1025-65535 # unregistered ports
>>> acl Safe_ports port 280 # http-mgmt
>>> acl Safe_ports port 488 # gss-http
>>> acl Safe_ports port 591 # filemaker
>>> acl Safe_ports port 777 # multiling http
>>> acl CONNECT method CONNECT
>>>
>>>
>>> http_access allow localhost manager
>>> http_access deny manager
>>> http_access deny !Safe_ports
>>> http_access deny CONNECT !SSL_ports
>>>
>>> http_access allow localnet
>>> http_access allow localhost
>>>
>>> http_access allow all
>>
>>
>> Sure, why bother having security at all?
>>
>>
>>> maximum_object_size_in_memory 512 KB
>>> memory_cache_mode always
>>> memory_replacement_policy heap GDSF
>>> cache_replacement_policy heap LFUDA
>>> store_dir_select_algorithm least-load
>>> max_open_disk_fds 0
>>> maximum_object_size 200 MB
>>>
>>> cache_swap_high 98
>>> cache_swap_low 97
>>>
>>>
>>> # access_log stdio:/var/log/squid3/access.log
>>> access_log none
>>> cache_log /var/log/squid3/cache.log
>>> cache_store_log stdio:/var/log/squid3/store.log
>>
>>
>> Usually the log you want to use is access_log, with the store debug log
>> (cache_store_log) disabled.
>>
>>
>>> workers 2
>>>
>>> cache_dir aufs /mnt/ssd/cache/store1 50000 32 512
>>> cache_dir aufs /mnt/ssd/cache/store2 50000 32 512
>>> cache_dir aufs /mnt/ssd/cache/store3 50000 32 512
>>> cache_dir aufs /mnt/ssd/cache/store4 50000 32 512
>>> cache_dir aufs /mnt/ssd/cache/store5 50000 32 512
>>> cache_dir aufs /mnt/ssd/cache/store6 50000 32 512
>>> cache_dir aufs /mnt/ssd/cache/store7 50000 32 512
>>>
>>>
>>> cache_dir aufs /mnt/sas1/cache/store1 50000 32 512
>>> cache_dir aufs /mnt/sas1/cache/store2 50000 32 512
>>> cache_dir aufs /mnt/sas1/cache/store3 50000 32 512
>>> cache_dir aufs /mnt/sas1/cache/store4 50000 32 512
>>> cache_dir aufs /mnt/sas1/cache/store5 50000 32 512
>>> cache_dir aufs /mnt/sas1/cache/store6 50000 32 512
>>> cache_dir aufs /mnt/sas1/cache/store7 50000 32 512
>>> cache_dir aufs /mnt/sas1/cache/store8 50000 32 512
>>> cache_dir aufs /mnt/sas1/cache/store9 50000 32 512
>>> cache_dir aufs /mnt/sas1/cache/store10 50000 32 512
>>> cache_dir aufs /mnt/sas1/cache/store11 50000 32 512
>>> cache_dir aufs /mnt/sas1/cache/store12 50000 32 512
>>>
>>>
>>> cache_dir aufs /mnt/sas2/cache/store1 50000 32 512
>>> cache_dir aufs /mnt/sas2/cache/store2 50000 32 512
>>> cache_dir aufs /mnt/sas2/cache/store3 50000 32 512
>>> cache_dir aufs /mnt/sas2/cache/store4 50000 32 512
>>> cache_dir aufs /mnt/sas2/cache/store5 50000 32 512
>>> cache_dir aufs /mnt/sas2/cache/store6 50000 32 512
>>> cache_dir aufs /mnt/sas2/cache/store7 50000 32 512
>>> cache_dir aufs /mnt/sas2/cache/store8 50000 32 512
>>> cache_dir aufs /mnt/sas2/cache/store9 50000 32 512
>>> cache_dir aufs /mnt/sas2/cache/store10 50000 32 512
>>> cache_dir aufs /mnt/sas2/cache/store11 50000 32 512
>>> cache_dir aufs /mnt/sas2/cache/store12 50000 32 512
>>>
>>> ===========================================================
>>>
>>> and my refresh patterns are :
>>>
>>> #general
>>> refresh_pattern \.(jp(e?g|e|2)|tiff?|bmp|gif|png)
>>> 12560 99% 30240 ignore-no-cache
>>> ignore-no-store override-expire override-lastmod ignore-private
>>> refresh_pattern
>>> \.(z(ip|[0-9]{2})|r(ar|[0-9]{2})|jar|bz2|gz|tar|rpm|vpu)
>>> 12560 99% 30240 ignore-no-cache ignore-no-store
>>> override-expire override-lastmod ignore-private
>>> refresh_pattern \.(mp3|wav|og(g|a)|flac|midi?|rm|aac|wma|mka|ape)
>>> 12560 99% 30240 ignore-no-cache
>>> ignore-no-store override-expire override-lastmod ignore-private
>>> refresh_pattern \.(exe|msi|dmg|bin|xpi|iso|mar|psf|cab|ipk|ipa|apk)
>>> 12560 99% 30240 ignore-no-cache
>>> ignore-no-store override-expire override-lastmod ignore-private
>>> refresh_pattern
>>>
>>> \.(mpeg|ra?m|avi|mp(g|e|4)|mov|divx|asf|wmv|m\dv|rv|vob|asx|ogm|flv|3gp|on2)
>>> 12560 99% 30240 ignore-no-cache ignore-no-store override-expire
>>> override-lastmod ignore-private
>>> refresh_pattern fbcdn 12560 99% 30240 ignore-no-cache
>>> ignore-no-store override-expire override-lastmod ignore-private
>>> refresh_pattern \.(php|jsp|cgi|asx|html|htm) 12560 100% 30240
>>> reload-into-ims ignore-no-cache
>>>
>>> refresh_pattern -i \.adobe\.com 161280 500% 525948 override-expire
>>> ignore-reload
>>>
>>> #specific sites
>>> refresh_pattern \.(ico|video-stats)
>>> 12560 100% 30240 override-expire ignore-reload
>>> ignore-no-cache ignore-private ignore-auth override-lastmod
>>> refresh_pattern ytimg\.com.*\.(jpg|png)
>>> 12560 100% 30240 override-expire ignore-reload
>>> refresh_pattern ((facebook.com)|(85.131.151.39)).*\.(png|gif)
>>> 12560 100% 30240 override-expire ignore-reload
>>> ignore-no-cache ignore-private
>>> refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)
>>> 12560 100% 30240 override-expire ignore-reload
>>> refresh_pattern .fbcdn.net.*\.(jpg|gif|png)
>>> 12560 100% 30240 ignore-no-cache override-expire
>>> ignore-reload
>>> refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\.
>>> 12560 100% 30240 ignore-no-cache ignore-reload
>>> override-expire
>>> refresh_pattern
>>> mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 12560
>>> 100% 30240 reload-into-ims override-expire ignore-private
>>>
>>> refresh_pattern . 1440 100% 2250 reload-into-ims
>>
>>
>> NP: if you have the latest 3.2 releases ignore-no-cache is useless.
>>
>> Amos
Received on Sat Mar 30 2013 - 20:08:13 MDT