Re: [squid-users] Large ACL's list, the ways to distribute squid caches, asking pro-users for advice.

Сергей А. Андросов wrote:
> Hi there,
>
> There is some problems and I asking for advice for spread squid load and
> increase perfomance.
>
> Now we have near 4000 users in our university, and one squid proxy
> server running on paravirtualized virtual machine (2 x 1.6Ghz Xeon E5310)
>
> We use Squid Accounting system calling SAcc for traffic accounting. It's
> generates squid configuration file and reloads squid when user is
> banned. Caching is off and squid uses only for proxying and accounting.
>
> There is NCSA authentication with passwd file for 4000 users, and 4000
> ACL's generated in squid configuration file like this:
>
> acl popovasi proxy_auth popovasi
> http_access allow popovasi
> http_access allow popovasi CONNECT
>
> acl halenko proxy_auth halenko
> http_access allow halenko
> http_access allow halenko CONNECT
>
> acl mamatovaa proxy_auth mamatovaa
> http_access allow mamatovaa
> http_access allow mamatovaa CONNECT
>
> etc..

Wow. wow. This takes the prize for the year I think.
Do you have any hair or sanity left after working with that?

>
> Internet connection is not so wide, only 4Mbps and, as I think, most
> hard on server is working with huge ACL lists.
>
> ----------
> What we need and what we can:
>
> We need to spread squid load for better latency and responce to queries
> and turn on caching because of tiny internet channel.
>
> And so we have a couple of servers for new squid proxies.
>
> We can use mysql_auth helper for authentication because of all user
> database stored in mysql (but in clear-text, we working on it).
>
> We need some expirience and advices for better ACL management. How we
> can use one acl for all authorized users in passwd file? Can we get
> perfomance increase using CARP and parent proxy? What the perfomance hit
> will be if we will use gnu-regexp instead of built-in?

*Any* regexp, give huge performance downgrade. There are better
alternatives in most situations and some juggling to reduce the hit in
others where its needed.

> Maybe we should change all user and traffic accounting management. Can
> you advice on dynamic distributed accounting and authentication
> solutions based on squid (I called it cache-grid :))?
>
> Now, ask you for your expirience. Thanks for reply. Feel free to give
> man and URL's for reading on this thread.
>
> Sorry for my English.
>
> Serg Androsov.

Okay. I think a few small additions may help out a lot here. Lets start
at the top.

If you want to help out the community could you please record your
current speed/hit statistics (squidclient mgr:info) before starting any
of this. And again after. We would be very interested in real-life
benefit examples.

Now the changes:

First,
   Are you running a squid 2.6 stable 18? That release has the fastest
squid code out so far.

Second,
   You will need a helper script to do the authentication better than
reloading squid. The helper script needs to accept "<id> <username>
<password>" on input and give out "<id> OK" or "<id> ERR" for accept/deny.

If you have a standard system such as LDAP , ActiveDirectory,
eDirectory, etc there are helpers already floating around for those.

You have two choices in setting it up here:

Option #1, full auth_* helper such as this:

auth_param basic program /etc/squid/helper/users.sh
auth_param basic children 10
auth_param basic concurrency 50
auth_param basic realm Your School Name or Realm
auth_param basic credentialsttl 2 seconds
authenticate_ip_ttl 10 minutes
authenticate_ttl 3 minutes
acl users proxy_auth REQUIRED

(with maybe different TTL to suit your setup)
References:
http://www.squid-cache.org/Versions/v2/2.6/cfgman/auth_param.html
http://www.squid-cache.org/Versions/v2/2.6/cfgman/authentical_ttl.html
http://www.squid-cache.org/Versions/v2/2.6/cfgman/authentical_ip_ttl.html

Or option #2, a custom external acl.

external_acl_type user_test
              ttl=5 negative_ttl=0 concurrency=50 children=5
              %LOGIN /etc/squid/helper/users.sh
(above all on one line)

acl users external user_test

References:
http://www.squid-cache.org/Versions/v2/2.6/cfgman/extrnal_acl_type.html

Then just this once:

   http_access allow users
   http_access deny all

When thats going, kill the system reloading squid. Your accounting
system may still be good for making the database which the helper uses.
But it no longer needed to touch squid.

That should cut down your performance issues with reloading, and many
ACL all in one cut.

Third,
   You say caching turned off at present. You would do well (up to 30%
we think bandwidth savings and faster internet) just by turning it on.
Before any actual tuning.

References:
http://www.squid-cache.org/Versions/v2/2.6/cfgman/cache_dir.html

If you post your squid.conf we shall see what else is possible to be done.

Amos

-- 
Please use Squid 2.6STABLE17+ or 3.0STABLE1+
There are serious security advisories out on all earlier releases.