Re: [squid-users] Squid and Squidguard using high disk IO

Just found this is Squid cache log:

2013/11/09 19:28:25 kid1| /var/squid/cache/04/7A: (24) Too many open files
2013/11/09 19:31:31 kid1| WARNING: All 20/20 redirector processes are busy.
2013/11/09 19:31:31 kid1| WARNING: 20 pending requests queued
2013/11/09 19:31:31 kid1| WARNING: Consider increasing the number of
redirector processes in your config file.

The cache size is 2GB.... though that shouldn't affect performance as
far as I understand.

On 11/09/2013 05:23 PM, Eliezer Croitoru wrote:
> Hey,
>
> Notes inside.
>
> On 11/09/2013 05:58 PM, Kaya Saman wrote:
>>
>> What can I do to improve performance with this?
>>
>>
>> Is this line too high: url_rewrite_children 500
> YES!!
>
>> or simply have a misconfigured something?
>
>
>> I additionally have 'c-icap' running with squidclamav coupled to clamd
>> in case that is of importance - not using the squidGuard line in the
>> squidclamav.conf file!!!
>>
>> Basically how can I get the IO usage down and get the system to work
>> again?
> For how many users exactly?
> Just a note that I am not in a favor of any OS by default but I would
> feel better Using Linux.
>
>>
>> - the logs don't indicate anything outside of 'starting squidGuard
>> process' many times.
> The basic assumption of using 500 child process is that you have
> atleast 100 CPUs.
> SquidGuard was design for performance which is lots of urls per sec.
> It can be tested just to clear the point out.
> for example in a rate of 1500k requests per second you should not have
> a need in more then 40-50 children.
> In practice it works a bit different speed since there is a speed
> limit on STDIN and STDOUT which slows down the speed of squid and
> squidguard communication blocking the whole squid instance(in a way).
>
> If you need basic url filtering you can use ICAP which has an option
> to run as a standalone service outside of squid settings and machine.
>
> I have written in the past a small ICAP service for the favor of
> requests manipulation and filtering.
> I have never finished it in a level I was happy with but the basic
> code can be seen here:
> https://github.com/elico/echelon
>
> I know for a fact that ICAP interface adds concurrency by the "nature"
> of it using TCP.
>
> This is not the place to ask about concurrency in squidguard which can
> allow the usage of square less processes(children) for more requests.
>
> In order to find the right number of children start with 40 and see if
> it fits you and then see what is the bottle neck in the whole setup.
>
> Eliezer
Received on Sat Nov 09 2013 - 19:39:46 MST