[squid-users] Heavy load squid with high CPU utilization...

Regards,

we have heavy load (over 6k requests per minute) intercepting squid
loading about 70-80Mbps traffic.
I have notices that CPU usage of squid process never goes down from 50%
and usually goes up to over 90%.

We plan to upgrade to 120Mbps link and this can be major problem if we
cannot solve it.
Every suggestions will be appreciated...

Best regards,
Zivanic Dejan

Server: FujitsuSiemens Primergy Econel 200 S2 (Quad Xeon E5310 on
1.6Ghz) with 6GB ECC ram.
Storage: 2xSata udma6 250GB
OS: Fedora14 x64

OS is installed on sda1, sda2 is only for cache_dir.

Squid Cache: Version 3.1.11
configure options: '--prefix=/usr' '--localstatedir=/var'
'--sysconfdir=/etc/squid' '--with-filedescriptors=16384'
'--enable-removal-policies=heap,lru' '--enable-delay-pools'
'--enable-epoll' '--enable-stopreio=ufs,aufs,diskd' '--enable-async-io'
'--with-pthreads' '--disable-dlmalloc' '--with-large-files'
'--enable-htcp' '--enable-large-cache-files' '--enable-wccpv2'
'--enable-esi' '--with-aio' '--with-dl' '--enable-ltdl-convenience'
'--enable-linux-netfilter' --with-squid=/root/install/squid-3.1.11

Linux 2.6.35.6-45.fc14.x86_64 #1 SMP Mon Oct 18 23:57:44 UTC 2010 x86_64
x86_64 x86_64 GNU/Linux
squid]# uptime
20:27:36 up 4 days, 22:07, 3 users, load average: 0.93, 0.84, 0.85
squid]# free
              total used free shared buffers cached
Mem: 6125216 4956620 1168596 0 377916 2568624
-/+ buffers/cache: 2010080 4115136
Swap: 4194296 9824 4184472

Squid Object Cache: Version 3.1.11
Start Time: Mon, 21 Mar 2011 12:42:30 GMT
Current Time: Tue, 22 Mar 2011 19:19:01 GMT
Connection information for squid:
     Number of clients accessing cache: 999
     Number of HTTP requests received: 11471323
     Number of ICP messages received: 0
     Number of ICP messages sent: 0
     Number of queued ICP replies: 0
     Number of HTCP messages received: 0
     Number of HTCP messages sent: 0
     Request failure ratio: 0.00
     Average HTTP requests per minute since start: 6246.2
     Average ICP messages per minute since start: 0.0
     Select loop called: 320350087 times, 0.344 ms avg
Cache information for squid:
     Hits as % of all requests: 5min: 10.0%, 60min: 9.4%
     Hits as % of bytes sent: 5min: 12.2%, 60min: 11.9%
     Memory hits as % of hit requests: 5min: 11.5%, 60min: 10.3%
     Disk hits as % of hit requests: 5min: 60.4%, 60min: 59.5%
     Storage Swap size: 72469364 KB
     Storage Swap capacity: 39.3% used, 60.7% free
     Storage Mem size: 1038552 KB
     Storage Mem capacity: 100.0% used, 0.0% free
     Mean Object Size: 53.58 KB
     Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
     HTTP Requests (All): 0.22004 0.23230
     Cache Misses: 0.27332 0.25890
     Cache Hits: 0.01648 0.01745
     Near Hits: 0.25890 0.18699
     Not-Modified Replies: 0.00179 0.00179
     DNS Lookups: 0.07618 0.08334
     ICP Queries: 0.00000 0.00000
Resource usage for squid:
     UP Time: 110191.361 seconds
     CPU Time: 41856.331 seconds
     CPU Usage: 60.99%
     CPU Usage, 5 minute avg: 75.64%
     CPU Usage, 60 minute avg: 81.66%
     Process Data Segment Size via sbrk(): 1474024 KB
     Maximum Resident Size: 5983984 KB
     Page faults with physical i/o: 13
Memory usage for squid via mallinfo():
     Total space in arena: 1474156 KB
     Ordinary blocks: 1467177 KB 10620 blks
     Small blocks: 0 KB 0 blks
     Holding blocks: 24216 KB 9 blks
     Free Small blocks: 0 KB
     Free Ordinary blocks: 6978 KB
     Total in use: 1491393 KB 100%
     Total free: 6978 KB 0%
     Total size: 1498372 KB
Memory accounted for:
     Total accounted: 1332348 KB 89%
     memPool accounted: 1332348 KB 89%
     memPool unaccounted: 166023 KB 11%
     memPoolAlloc calls: 2768926328
     memPoolFree calls: 2896226918
File descriptor usage for squid:
     Maximum number of file descriptors: 16384
     Largest file desc currently in use: 1069
     Number of file desc currently in use: 867
     Files queued for open: 0
     Available number of file descriptors: 15517
     Reserved number of file descriptors: 100
     Store Disk files open: 8
Internal Data Structures:
     1364435 StoreEntries
      65986 StoreEntries with MemObjects
      65640 Hot Object Cache Items
     1352638 on-disk objects

sysctl.conf:
//
fs.file-max = 360000
vm.drop_caches = 3
vm.swappiness = 3
net.ipv4.ip_local_port_range = 2048 65000
net.core.rmem_max = 16777216
net.core.wmem_max = 32777216
net.ipv4.tcp_low_latency = 1
net.core.netdev_max_backlog = 4000
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_mem = 786432 1048576 1572864
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 87380 16777216
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
vm.min_free_kbytes = 70000
net.core.somaxconn = 65536
net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.netfilter.nf_conntrack_max = 1048576
net.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_buckets = 1048576
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 3
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 100
net.netfilter.nf_conntrack_log_invalid = 0
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_icmp_timeout = 10
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.netfilter.nf_conntrack_udp_timeout = 10
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 5
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 10
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 10
net.netfilter.nf_conntrack_tcp_timeout_established = 21600
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 5
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 5
net.netfilter.nf_conntrack_checksum = 1
//

squid.conf:
//
http_port 5555 intercept
icp_port 0
maximum_icp_query_timeout 3000
mcast_icp_query_timeout 2000
dead_peer_timeout 10 seconds
hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY
cache_mem 1024 MB
cache_swap_low 98
cache_swap_high 99
maximum_object_size 250 MB
minimum_object_size 10 KB
maximum_object_size_in_memory 50 KB
ipcache_size 16384
ipcache_low 90
ipcache_high 95
cache_replacement_policy heap LFUDA
memory_replacement_policy heap GDSF
cache_dir aufs /cache00 180000 32 256
cache_access_log /var/log/squid/access.log common
cache_log /var/log/squid/cache.log
cache_store_log none
cache_swap_log /var/spool/squid/cache_swap_log
log_ip_on_direct off
pid_filename /var/run/squid.pid
debug_options ALL,1
dns_timeout 20 seconds
dns_nameservers 212.200.191.150 212.200.113.130
auth_param basic children 15
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours
auth_param basic casesensitive off
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320
forward_timeout 20 seconds
connect_timeout 10 seconds
peer_connect_timeout 10 seconds
read_timeout 30 seconds
request_timeout 10 seconds
persistent_request_timeout 10 seconds
client_lifetime 360 minutes
half_closed_clients off
pconn_timeout 10 seconds
shutdown_lifetime 10 seconds
acl manager proto cache_object
acl localhost src 127.0.0.1
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny opstina2 !vreme
http_access allow kladovonet
http_access allow kladovonet-public
http_access deny all
tcp_outgoing_address 172.29.203.200
cache_peer 172.29.202.252 parent 8080 7 name=dsl1 round-robin no-query
no-digest weight=2
cache_peer 172.29.202.253 parent 8080 7 name=dsl2 round-robin no-query
no-digest weight=2
cache_peer 172.29.202.251 parent 8080 7 name=dsl3 round-robin no-query
no-digest weight=2
acl kroz-adsl url_regex -i "/etc/adsl"
cache_peer_access adsl1 allow kroz-adsl
cache_peer_access adsl2 allow kroz-adsl
cache_peer_access adsl3 allow kroz-adsl
never_direct allow kroz-adsl
http_reply_access allow all
icp_access deny all
cache_mgr Zivanic_Dejan_[zivanicd_at_kladovonet.com]
cache_effective_user squid
logfile_rotate 10
memory_pools off
error_directory /usr/share/errors/sr-latn
ignore_unknown_nameservers off
client_persistent_connections off
server_persistent_connections off
//
Received on Tue Mar 22 2011 - 19:30:20 MDT