Re: [squid-users] tuning an overloaded server

Hi,

On Sun, 23 Nov 2008, Amos Jeffries wrote:
> Chris Robertson wrote:
>> Al - Image Hosting Services wrote:
>>> I hope that someone can help me. I have 3 servers running squid acting as
>>> a web accelerator for a single http server. I have been having a large
>>> amount of problems with the http server locking up and having drive
>>> failures and squid seems to be a great solution. I was pleasantly
>>> surprised at the performance of squid when I first installed it. Even on
>>> the slowest of my servers, it seemed faster than directly from the http
>>> server, but after I put the load back on the slowest of my servers it was
>>> 0% idle.
>>
>> Where is the CPU usage at, User, System or Wait?
>>
>>> Since this server had a second hard drive that was basically not being
>>> used, I moved the squid cache over to it. I then did some googling and I
>>> made some other changes and the server is now only 0% idle a few times a
>>> day. I am really suprised at just how tunable squid is and it was
>>> wonderful that a change in configuration could make such a difference. But
>>> since the busy season is coming, I still think that more can be done to
>>> tune it.
>>>
>>> Here are the changes that I made to the squid.conf:
>>> redirect_rewrites_host_header off
>>> acl all src 0.0.0.0/0.0.0.0
>>> acl manager proto cache_object
>>> acl localhost src 127.0.0.1/255.255.255.255
>>> acl to_localhost dst 127.0.0.0/8
>>> acl SSL_ports port 443
>>> acl CONNECT method CONNECT
>>> http_access allow manager localhost
>>> http_access deny manager
>>> http_access deny !Safe_ports
>>> http_access deny CONNECT !SSL_ports
>>> http_access allow all
>
> Not a good idea, limit the access to just the domains you are serving where
> at all possible.
>
>>> icp_access allow all
>>> http_port 80 accel vhost
>>> cache_peer 10.10.1.4 parent 80 0 no-query originserver
>>> hierarchy_stoplist cgi-bin ?
>>> cache_mem 16 MB
>
> cache_mem is worth expanding, it lets more objects stay in memory and skip
> the disk IO waiting times.
>
>>> maximum_object_size_in_memory 128 KB
>>> memory_replacement_policy heap GDSF
>>> cache_replacement_policy heap GDSF
>>> cache_dir ufs /var/squid/cache 200 16 256
>>> maximum_object_size 2048 KB
>>> cache_swap_low 90
>>> cache_swap_high 98
>>> access_log /var/log/squid/access.log squid
>>> cache_log /var/log/squid/cache.log
>>> cache_store_log /var/log/squid/store.log
>
> dump cache_store_log its a waste of disk IO unless you really need it for
> something weird.
>
>>> emulate_httpd_log on
>>> buffered_logs on
>>> acl QUERY urlpath_regex cgi-bin \?
>>> cache deny QUERY
>>> refresh_pattern ^ftp: 1440 20% 10080
>>> refresh_pattern ^gopher: 1440 0% 1440
>>> refresh_pattern . 0 20% 4320
>>> acl apache rep_header Server ^Apache
>>> broken_vary_encoding allow apache
>>> half_closed_clients off
>>> icp_port 0
>>> log_icp_queries off
>>> coredump_dir /var/squid/cache
>>>
>>> Are there any other changes that can be made to improve cpu usage?
>>>
>>> I also did an install from source, since I am using NetBSD systems I used
>>> pkgsrc to install it. There are several build options:
>>>
>>> arp-acl
>>> aufs
>>> carp
>>> coss
>>> diskd
>>> icmp
>>> ipf-transparent
>>> pam-helper
>>> pf-transparent
>>> snmp
>>> ssl
>>> unlinkd
>>>
>>> I looked up what some of these are, but I am not sure how they are used by
>>> squid or if I really need them. My thought is that if I compiled squid
>>> without these options it could improve performance. I wish there was more
>>> information on running squid on BSD systems (especially recent info) and
>>> then maybe I could be sure what effect compiling without an option would
>>> have. For example, I know what pam is, but do I need it?
>>
>> You are not using authentication, so could do without PAM.
>>
>>> I looked up what unlinkd does and I can see it running, but will it help
>>> cpu usage to use it?
>>
>> Using --enable-truncate "gives a little performance improvement, but may
>> cause problems when used with async I/O. Truncate uses more filesystem
>> inodes than unlink.." asynch I/O refers to diskd, aufs and coss. The only
>> way to see if that will lower your CPU usage is to try, but...
>>
>>> I don't see diskd running as a seperate process, so I don't know that I am
>>> even using it.
>>
>> Your cache_dir line specifies "ufs". To use diskd, you need to change that
>> line to...
>>
>> cache_dir diskd /var/squid/cache 200 16 256

I tried setting it to diskd and then had a problem with it shutting down
after a few minutes. Is there something else that I need to do to before I
put in this change?

Thanks,
Al
Received on Tue Dec 02 2008 - 00:23:17 MST