Re: [squid-users] Squid performance with high load

On 30/03/2013 6:33 a.m., Hasanen AL-Bana wrote:
> Hi,
>
> I am running squid 3.2 with an average of 50k req/min. Total received
> bandwidth is around 200mbit/s.
> I have problem when my aufs cache_dirs reaches size above 600GB.
> Traffic starts dropping and going up again , happening every 20~30 minutes.
> I have more that enough RAM in the system (125GB DDR3 !) , All disks
> are SAS 15k rpm, one of them is and SSD 450GB.
> So hardware should not cause any problem and I should easily spawn
> multiple squid workers any time.
> So what could cause such problems ?

#1 - AUFS is *not* SMP-aware component in Squid.

Each of the two workers you are using will be altering the on-disk
portion of the cache without updating the in-memory index. When the
other worker encounters these over-written files it will erase them.

For now you are required to use the macros hacks to split the cache_dir
lines between the workers.

#2 - you have multiple cache_dir per disk. Or so it seems from your
configuration. Is that correct?

  * squid load balances between cache_dir treating them as separate
physcal HDD when it comes to loading calculations.

* the memory requirement for indexing these 1.6 TB of disk space is
~24GB per worker plus the 20GB of shared emmory cache ==> 68GB of RAM.

I advise to you inform Squid about the real disk topology it is working
with. No more than one AUFS cache_dir per physical disk and allocate the
cache_dir size to ~80-90% the full available disk space. With each
worker assigned to use a sub-set of the AUFS cache_dir and associated disks.

For best performance and object de-duplication you should consider using
rock cache_dir for small objects. They *are* shared between workers, and
can also share disk space with an AUFS disk.

> Thank you.
>
> ====================================================================================
> include /etc/squid3/refresh.conf
>
>
> cache_mem 20 GB
>
> acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
> acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
> acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
> acl localnet src fc00::/7 # RFC 4193 local private network range
> acl localnet src fe80::/10 # RFC 4291 link-local (directly
> plugged) machines
>
> acl SSL_ports port 443
> acl Safe_ports port 80 # http
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 # https
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl CONNECT method CONNECT
>
>
> http_access allow localhost manager
> http_access deny manager
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
>
> http_access allow localnet
> http_access allow localhost
>
> http_access allow all

Sure, why bother having security at all?

> maximum_object_size_in_memory 512 KB
> memory_cache_mode always
> memory_replacement_policy heap GDSF
> cache_replacement_policy heap LFUDA
> store_dir_select_algorithm least-load
> max_open_disk_fds 0
> maximum_object_size 200 MB
>
> cache_swap_high 98
> cache_swap_low 97
>
>
> # access_log stdio:/var/log/squid3/access.log
> access_log none
> cache_log /var/log/squid3/cache.log
> cache_store_log stdio:/var/log/squid3/store.log

Usually the log you want to use is access_log, with the store debug log
(cache_store_log) disabled.

> workers 2
>
> cache_dir aufs /mnt/ssd/cache/store1 50000 32 512
> cache_dir aufs /mnt/ssd/cache/store2 50000 32 512
> cache_dir aufs /mnt/ssd/cache/store3 50000 32 512
> cache_dir aufs /mnt/ssd/cache/store4 50000 32 512
> cache_dir aufs /mnt/ssd/cache/store5 50000 32 512
> cache_dir aufs /mnt/ssd/cache/store6 50000 32 512
> cache_dir aufs /mnt/ssd/cache/store7 50000 32 512
>
>
> cache_dir aufs /mnt/sas1/cache/store1 50000 32 512
> cache_dir aufs /mnt/sas1/cache/store2 50000 32 512
> cache_dir aufs /mnt/sas1/cache/store3 50000 32 512
> cache_dir aufs /mnt/sas1/cache/store4 50000 32 512
> cache_dir aufs /mnt/sas1/cache/store5 50000 32 512
> cache_dir aufs /mnt/sas1/cache/store6 50000 32 512
> cache_dir aufs /mnt/sas1/cache/store7 50000 32 512
> cache_dir aufs /mnt/sas1/cache/store8 50000 32 512
> cache_dir aufs /mnt/sas1/cache/store9 50000 32 512
> cache_dir aufs /mnt/sas1/cache/store10 50000 32 512
> cache_dir aufs /mnt/sas1/cache/store11 50000 32 512
> cache_dir aufs /mnt/sas1/cache/store12 50000 32 512
>
>
> cache_dir aufs /mnt/sas2/cache/store1 50000 32 512
> cache_dir aufs /mnt/sas2/cache/store2 50000 32 512
> cache_dir aufs /mnt/sas2/cache/store3 50000 32 512
> cache_dir aufs /mnt/sas2/cache/store4 50000 32 512
> cache_dir aufs /mnt/sas2/cache/store5 50000 32 512
> cache_dir aufs /mnt/sas2/cache/store6 50000 32 512
> cache_dir aufs /mnt/sas2/cache/store7 50000 32 512
> cache_dir aufs /mnt/sas2/cache/store8 50000 32 512
> cache_dir aufs /mnt/sas2/cache/store9 50000 32 512
> cache_dir aufs /mnt/sas2/cache/store10 50000 32 512
> cache_dir aufs /mnt/sas2/cache/store11 50000 32 512
> cache_dir aufs /mnt/sas2/cache/store12 50000 32 512
>
> ===========================================================
>
> and my refresh patterns are :
>
> #general
> refresh_pattern \.(jp(e?g|e|2)|tiff?|bmp|gif|png)
> 12560 99% 30240 ignore-no-cache
> ignore-no-store override-expire override-lastmod ignore-private
> refresh_pattern
> \.(z(ip|[0-9]{2})|r(ar|[0-9]{2})|jar|bz2|gz|tar|rpm|vpu)
> 12560 99% 30240 ignore-no-cache ignore-no-store
> override-expire override-lastmod ignore-private
> refresh_pattern \.(mp3|wav|og(g|a)|flac|midi?|rm|aac|wma|mka|ape)
> 12560 99% 30240 ignore-no-cache
> ignore-no-store override-expire override-lastmod ignore-private
> refresh_pattern \.(exe|msi|dmg|bin|xpi|iso|mar|psf|cab|ipk|ipa|apk)
> 12560 99% 30240 ignore-no-cache
> ignore-no-store override-expire override-lastmod ignore-private
> refresh_pattern
> \.(mpeg|ra?m|avi|mp(g|e|4)|mov|divx|asf|wmv|m\dv|rv|vob|asx|ogm|flv|3gp|on2)
> 12560 99% 30240 ignore-no-cache ignore-no-store override-expire
> override-lastmod ignore-private
> refresh_pattern fbcdn 12560 99% 30240 ignore-no-cache
> ignore-no-store override-expire override-lastmod ignore-private
> refresh_pattern \.(php|jsp|cgi|asx|html|htm) 12560 100% 30240
> reload-into-ims ignore-no-cache
>
> refresh_pattern -i \.adobe\.com 161280 500% 525948 override-expire
> ignore-reload
>
> #specific sites
> refresh_pattern \.(ico|video-stats)
> 12560 100% 30240 override-expire ignore-reload
> ignore-no-cache ignore-private ignore-auth override-lastmod
> refresh_pattern ytimg\.com.*\.(jpg|png)
> 12560 100% 30240 override-expire ignore-reload
> refresh_pattern ((facebook.com)|(85.131.151.39)).*\.(png|gif)
> 12560 100% 30240 override-expire ignore-reload
> ignore-no-cache ignore-private
> refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png)
> 12560 100% 30240 override-expire ignore-reload
> refresh_pattern .fbcdn.net.*\.(jpg|gif|png)
> 12560 100% 30240 ignore-no-cache override-expire
> ignore-reload
> refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\.
> 12560 100% 30240 ignore-no-cache ignore-reload
> override-expire
> refresh_pattern
> mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 12560
> 100% 30240 reload-into-ims override-expire ignore-private
>
> refresh_pattern . 1440 100% 2250 reload-into-ims

NP: if you have the latest 3.2 releases ignore-no-cache is useless.

Amos
Received on Sat Mar 30 2013 - 09:58:06 MDT