Re: [squid-users] tuning an overloaded server

> Hi,
>
>
> On Sun, 23 Nov 2008, Amos Jeffries wrote:
>> Chris Robertson wrote:
>>> Al - Image Hosting Services wrote:
>>>> I hope that someone can help me. I have 3 servers running squid acting
>>>> as
>>>> a web accelerator for a single http server. I have been having a large
>>>> amount of problems with the http server locking up and having drive
>>>> failures and squid seems to be a great solution. I was pleasantly
>>>> surprised at the performance of squid when I first installed it. Even
>>>> on
>>>> the slowest of my servers, it seemed faster than directly from the
>>>> http
>>>> server, but after I put the load back on the slowest of my servers it
>>>> was
>>>> 0% idle.
>>>
>>> Where is the CPU usage at, User, System or Wait?
>>>
>>>> Since this server had a second hard drive that was basically not being
>>>> used, I moved the squid cache over to it. I then did some googling and
>>>> I
>>>> made some other changes and the server is now only 0% idle a few times
>>>> a
>>>> day. I am really suprised at just how tunable squid is and it was
>>>> wonderful that a change in configuration could make such a difference.
>>>> But
>>>> since the busy season is coming, I still think that more can be done
>>>> to
>>>> tune it.
>>>>
>>>> Here are the changes that I made to the squid.conf:
>>>> redirect_rewrites_host_header off
>>>> acl all src 0.0.0.0/0.0.0.0
>>>> acl manager proto cache_object
>>>> acl localhost src 127.0.0.1/255.255.255.255
>>>> acl to_localhost dst 127.0.0.0/8
>>>> acl SSL_ports port 443
>>>> acl CONNECT method CONNECT
>>>> http_access allow manager localhost
>>>> http_access deny manager
>>>> http_access deny !Safe_ports
>>>> http_access deny CONNECT !SSL_ports
>>>> http_access allow all
>>
>> Not a good idea, limit the access to just the domains you are serving
>> where
>> at all possible.
>>
>>>> icp_access allow all
>>>> http_port 80 accel vhost
>>>> cache_peer 10.10.1.4 parent 80 0 no-query originserver
>>>> hierarchy_stoplist cgi-bin ?
>>>> cache_mem 16 MB
>>
>> cache_mem is worth expanding, it lets more objects stay in memory and
>> skip
>> the disk IO waiting times.
>>
>>>> maximum_object_size_in_memory 128 KB
>>>> memory_replacement_policy heap GDSF
>>>> cache_replacement_policy heap GDSF
>>>> cache_dir ufs /var/squid/cache 200 16 256
>>>> maximum_object_size 2048 KB
>>>> cache_swap_low 90
>>>> cache_swap_high 98
>>>> access_log /var/log/squid/access.log squid
>>>> cache_log /var/log/squid/cache.log
>>>> cache_store_log /var/log/squid/store.log
>>
>> dump cache_store_log its a waste of disk IO unless you really need it
>> for
>> something weird.
>>
>>>> emulate_httpd_log on
>>>> buffered_logs on
>>>> acl QUERY urlpath_regex cgi-bin \?
>>>> cache deny QUERY
>>>> refresh_pattern ^ftp: 1440 20% 10080
>>>> refresh_pattern ^gopher: 1440 0% 1440
>>>> refresh_pattern . 0 20% 4320
>>>> acl apache rep_header Server ^Apache
>>>> broken_vary_encoding allow apache
>>>> half_closed_clients off
>>>> icp_port 0
>>>> log_icp_queries off
>>>> coredump_dir /var/squid/cache
>>>>
>>>> Are there any other changes that can be made to improve cpu usage?
>>>>
>>>> I also did an install from source, since I am using NetBSD systems I
>>>> used
>>>> pkgsrc to install it. There are several build options:
>>>>
>>>> arp-acl
>>>> aufs
>>>> carp
>>>> coss
>>>> diskd
>>>> icmp
>>>> ipf-transparent
>>>> pam-helper
>>>> pf-transparent
>>>> snmp
>>>> ssl
>>>> unlinkd
>>>>
>>>> I looked up what some of these are, but I am not sure how they are
>>>> used by
>>>> squid or if I really need them. My thought is that if I compiled squid
>>>> without these options it could improve performance. I wish there was
>>>> more
>>>> information on running squid on BSD systems (especially recent info)
>>>> and
>>>> then maybe I could be sure what effect compiling without an option
>>>> would
>>>> have. For example, I know what pam is, but do I need it?
>>>
>>> You are not using authentication, so could do without PAM.
>>>
>>>> I looked up what unlinkd does and I can see it running, but will it
>>>> help
>>>> cpu usage to use it?
>>>
>>> Using --enable-truncate "gives a little performance improvement, but
>>> may
>>> cause problems when used with async I/O. Truncate uses more filesystem
>>> inodes than unlink.." asynch I/O refers to diskd, aufs and coss. The
>>> only
>>> way to see if that will lower your CPU usage is to try, but...
>>>
>>>> I don't see diskd running as a seperate process, so I don't know that
>>>> I am
>>>> even using it.
>>>
>>> Your cache_dir line specifies "ufs". To use diskd, you need to change
>>> that
>>> line to...
>>>
>>> cache_dir diskd /var/squid/cache 200 16 256
>
> I tried setting it to diskd and then had a problem with it shutting down
> after a few minutes. Is there something else that I need to do to before I
> put in this change?
>

It may be the mentioned problems wit the -truncate option, or it may be
something to do with diskd itself. Can you get a trace of why diskd
helpers are closing, they should stay active for the full lifetime of
Squid. closing only on reconfigure and shutdown.

Amos
Received on Tue Dec 02 2008 - 01:01:19 MST