Re: [squid-users] Squid and Squidguard using high disk IO

On Sat, Nov 09, 2013 at 11:16:12PM +0100, Loïc BLOT wrote:
> Hello Kaya,
> first, don't forget to look at sysctl kern.maxfiles values.
> Also improve daemon FD values in login.conf for squid. Don't forget each
> connection is a FD (1 connection for the client, 1 for the transaction
> to remote site, somes for the caching).
>
> Also to improve performances of squidguard, i stored all blacklists DB
> to a memory fs (mfs) this improve massively squidguard performance

If the disk I/O is really the bottleneck, consider ufdbGuard.
ufdbGuard loads the URL database in memory and easily does
25,000 URL lookups/sec, much more than you will ever need.

Marcus

> I have wrote an article to improve squid perfs on OpenBSD:
> http://www.unix-experience.fr/2013/monter-un-proxy-cache-performant-avec-squid-et-openbsd/
>
>
>
> --
> Best regards,
> Loïc BLOT,
> UNIX systems, security and network engineer
> http://www.unix-experience.fr
>
>
>
> Le samedi 09 novembre 2013 à 19:39 +0000, Kaya Saman a écrit :
> > Just found this is Squid cache log:
> >
> > 2013/11/09 19:28:25 kid1| /var/squid/cache/04/7A: (24) Too many open files
> > 2013/11/09 19:31:31 kid1| WARNING: All 20/20 redirector processes are busy.
> > 2013/11/09 19:31:31 kid1| WARNING: 20 pending requests queued
> > 2013/11/09 19:31:31 kid1| WARNING: Consider increasing the number of
> > redirector processes in your config file.
> >
> >
> > The cache size is 2GB.... though that shouldn't affect performance as
> > far as I understand.
> >
> > On 11/09/2013 05:23 PM, Eliezer Croitoru wrote:
> > > Hey,
> > >
> > > Notes inside.
> > >
> > > On 11/09/2013 05:58 PM, Kaya Saman wrote:
> > >>
> > >> What can I do to improve performance with this?
> > >>
> > >>
> > >> Is this line too high: url_rewrite_children 500
> > > YES!!
> > >
> > >> or simply have a misconfigured something?
> > >
> > >
> > >> I additionally have 'c-icap' running with squidclamav coupled to clamd
> > >> in case that is of importance - not using the squidGuard line in the
> > >> squidclamav.conf file!!!
> > >>
> > >> Basically how can I get the IO usage down and get the system to work
> > >> again?
> > > For how many users exactly?
> > > Just a note that I am not in a favor of any OS by default but I would
> > > feel better Using Linux.
> > >
> > >>
> > >> - the logs don't indicate anything outside of 'starting squidGuard
> > >> process' many times.
> > > The basic assumption of using 500 child process is that you have
> > > atleast 100 CPUs.
> > > SquidGuard was design for performance which is lots of urls per sec.
> > > It can be tested just to clear the point out.
> > > for example in a rate of 1500k requests per second you should not have
> > > a need in more then 40-50 children.
> > > In practice it works a bit different speed since there is a speed
> > > limit on STDIN and STDOUT which slows down the speed of squid and
> > > squidguard communication blocking the whole squid instance(in a way).
> > >
> > > If you need basic url filtering you can use ICAP which has an option
> > > to run as a standalone service outside of squid settings and machine.
> > >
> > > I have written in the past a small ICAP service for the favor of
> > > requests manipulation and filtering.
> > > I have never finished it in a level I was happy with but the basic
> > > code can be seen here:
> > > https://github.com/elico/echelon
> > >
> > > I know for a fact that ICAP interface adds concurrency by the "nature"
> > > of it using TCP.
> > >
> > > This is not the place to ask about concurrency in squidguard which can
> > > allow the usage of square less processes(children) for more requests.
> > >
> > > In order to find the right number of children start with 40 and see if
> > > it fits you and then see what is the bottle neck in the whole setup.
> > >
> > > Eliezer
> >
Received on Sat Nov 09 2013 - 22:37:10 MST