Re: [squid-users] Slow Internet with squid

On 22/07/11 16:00, Wilson Hernandez wrote:
> On 7/20/2011 9:12 PM, Amos Jeffries wrote:
>> On Wed, 20 Jul 2011 11:12:04 -0400, Wilson Hernandez wrote:
>>> Hello.
>>>
>>> I am puzzled to see how my bandwidth is used when running squid. I have
>>> a total of 25M/3M of bandwidth, lately I've noticed with iptraf that my
>>> external interface traffic/bandwidth is almost maxed out at 24.8M and my
>>> internal interface (squid) is only at 2.9M as a result most clients have
>>> been calling saying "their internet is slow".
>>>
>>> I'm wondering why that big of a difference on the interfaces' traffic.
>>>
>>> This is what cachemgr shows:
>>>
>>> Squid Object Cache: Version 3.1.14
>>>
>>> Start Time: Fri, 15 Jul 2011 08:01:48 GMT
>>> Current Time: Wed, 20 Jul 2011 14:39:02 GMT
>>>
>>>
>>> Connection information for squid:
>>> Number of clients accessing cache: 113
>>> Number of HTTP requests received: 5198204
>>> Number of ICP messages received: 0
>>> Number of ICP messages sent: 0
>>> Number of queued ICP replies: 0
>>> Number of HTCP messages received: 0
>>> Number of HTCP messages sent: 0
>>> Request failure ratio: 0.00
>>> Average HTTP requests per minute since start: 684.2
>>> Average ICP messages per minute since start: 0.0
>>> Select loop called: 479758718 times, 0.950 ms avg
>>> Cache information for squid:
>>> Hits as % of all requests: 5min: 23.2%, 60min: 19.4%
>>> Hits as % of bytes sent: 5min: -219.3%, 60min: -314.7%
>>> Memory hits as % of hit requests: 5min: 13.2%, 60min: 9.5%
>>> Disk hits as % of hit requests: 5min: 64.6%, 60min: 62.5%
>>> Storage Swap size: 66028580 KB
>>> Storage Swap capacity: 64.5% used, 35.5% free
>>> Storage Mem size: 1042556 KB
>>> Storage Mem capacity: 100.0% used, 0.0% free
>>> Mean Object Size: 23.52 KB
>>> Requests given to unlinkd: 0
>>> Median Service Times (seconds) 5 min 60 min:
>>> HTTP Requests (All): 0.12106 0.02069
>>> Cache Misses: 0.24524 0.30459
>>> Cache Hits: 0.05046 0.02899
>>> Near Hits: 0.17711 0.22004
>>> Not-Modified Replies: 0.00307 0.00091
>>> DNS Lookups: 0.31806 0.17048
>>
>> DNS is very slow as well. Probably due to remote queries over this
>> full link.
>>
>>>
>>> Please help me understand why this is happening and if there is a
>>> solution to make squid perform better.
>>
>> Squid "optimizes web delivery" as the slogan goes. So when the server
>> side is acting very inefficiently it can consume more than the client
>> side. Could be any of these or a few other things I'm not aware of:
>>
>> 1) client requests an object. Squid has it cached, but server is
>> requiring 'must-revalidate'. While revalidating the server forces an
>> entire new object back at squid, along with a timestamp stating it has
>> not changed. Squid only sends the small no-change reply to the client.
>>
>> 2a) client requests a small range of an object. Squid passes this on.
>> Server replies with again, forcing an entire new object back at squid.
>> Squid only sends the small range asked for to the client.
>>
>> 2b) client requests a small range of an object. Squid passes this on
>> but requests the full object (refresh_offset_limit). Server replies
>> with the whole object. Squid stores it and only sends the small range
>> asked for to the client.

Sorry, mistake. 'refresh_offset_limit' should have been 'range_offset_limit.

>>
>> 3) client requests info about an object (HEAD). Squid relays this
>> request on. Server replies, forcing an entire new object back at
>> squid. Squid only sends the small header asked for to the client.
>>
>> 4) client requests an object, then abandons it before receiving the
>> reply. Squid continues to wait and receive it, in hopes that it can be
>> stored. If not storable it may be discarded and the cycle repeat. Or
>> it could be stored but never again requested. This behaviour is
>> modified by the quick_abort_* directives.
>>
>>
>> Or it could be you configured an open proxy. Configuration problems
>> can allow external access to external sites. When discovered attackers
>> can use this and consume all your external bandwidth. Usually its
>> caused by mistakenly removing or bypassing the controls on CONNECT
>> tunnels. Though it can also happen on other requests.
>>
>> Amos
> Amos.
>
> Thanks for replying. Now, you have left me confused and in doubt. I
> don't know if my actual configuration is ok or not. I will post it here
> so, you can take a look and let me know where or what I'm doing wrong.
> Thanks again.

Sorry. "slow" is a vague problem :(

>
> squid.conf:
> # Port Squid listens on
> http_port 172.16.0.1:3128 intercept disable-pmtu-discovery=off

Port 3128 is very dangerous for interception. It is best to pick a
completely random one.
  Then to firewall any access to it. These ports are _only_ used between
squid and its box NAT system. Some hints on locking it down (iptabes
mangle table) are on the wiki intercept pages
http://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxDnat

>
> error_default_language es-do
>
> # Access-lists (ACLs) will permit or deny hosts to access the proxy
> acl lan-access src 172.16.0.0/16
> acl localhost src 127.0.0.1
> acl localnet src 172.16.0.0/16

lan-access and localnet look like the same thing. You can simplify be
removing one.

> acl proxy src 172.16.0.1
> acl clientes_registrados src "/etc/msd/ipAllowed"
>
> acl adstoblock dstdomain "/etc/squid/blockAds"
>
> acl CONNECT method CONNECT
>
>
<snip comments>
>
> #-----------------------
>
> http_access allow proxy
> http_access allow localhost

What is "172.16.0.1" and localhost doing having completely unrestricted
access?

>
> #---- Block some sites
>
> acl blockanalysis01 dstdomain .scorecardresearch.com clkads.com
> acl blockads01 dstdomain .rad.msn.com ads1.msn.com ads2.msn.com
> ads3.msn.com ads4.msn.com
> acl blockads02 dstdomain .adserver.yahoo.com
> pagead2.googlesyndication.com ad.yieldmanager.com
> acl blockads03 dstdomain .doubleclick.net .fastclick.net
> .googleadservices.com
> acl blockads04 dstdomain .ero-advertising.com .adsomega.com
> acl blockads05 dstdomain .adyieldmanager.com .yieldmanager.com
> .adyieldmanager.net .yieldmanager.net
> acl blockads06 dstdomain .e-planning.net .super-publicidad.com
> .super-publicidad.net
> acl blockads07 dstdomain .adbrite.com .contextweb.com .adbasket.net
> .clicktale.net
> acl blockads08 dstdomain .adserver.com .adv-adserver.com
> .zerobypass.info .zerobypass.com
> acl blockads09 dstdomain ads.ak.facebook.com .pubmatic.com .baynote.net
> .publicbt.com
>
> http_access deny blockanalysis01
> http_access deny blockads01
> http_access deny blockads02
> http_access deny blockads03
> http_access deny blockads04
> http_access deny blockads05
> http_access deny blockads06
> http_access deny blockads07
> http_access deny blockads08
> http_access deny blockads09
>
> http_access deny adstoblock

All of these block lists can be combined into one ACL name. ie
"blockads". Which will be much faster to check and block with.

>
> acl bank dstdomain .popularenlinea.com .bpd.com.do .google.com
> .google.com.do

google.com is your _bank_? wow.

> acl ourPublicServer dst 190.80.159.7
>
> http_access allow bank
> http_access allow ourPublicServer
>

Attempt at a reverse-proxy using squid-2.5 method? you may want to take
a look at how this is done with modern Squid.
   http://wiki.squid-cache.org/ConfigExamples/Reverse/BasicAccelerator
   http://wiki.squid-cache.org/ConfigExamples/Reverse/VirtualHosting

This is one possible cause of "slow". Almost every single request has to
do DNS lookups to figure out if its going to 190.80.159.7 or not.
  Surely you have one/some domain names used to access that server?
Adding a dstdomain list of those will speed things up quite a bit.

Moving to the reverse-proxy config for that same list of domains will
speed these particular requests even more.

<snip comments>
>
> acl manager proto cache_object
> # replace 10.0.0.1 with your webserver IP
> acl webserver src 172.16.0.1
> http_access allow manager webserver
> http_access deny manager
>

You called 172.16.0.1 "proxy" earlier and gave it unlimited access.

This management access security is one of the things which should be at
the top of your http_access sequence.
  http://wiki.squid-cache.org/SquidFaq/SecurityPitfalls#The_manager_ACLs

<snip comments>
> #access_log /var2/squid/access.log
> access_log none

It would be a good idea to enable logging for a while and see what type
of requests are going through the proxy. They are likely to be a big
clue about the imbalance.

<snip comments>
>
> # Also videos are LARGE; make sure you aren't killing them as 'too big
> to save'
> # - squid defaults to 4MB, which is too small for videos and even some
> sound files
> maximum_object_size 32500 KB
>
> maximum_object_size_in_memory 15625 KB

Perhapse it would be easier to write this:
   maximum_object_size 32 MB
   maximum_object_size_in_memory 16 MB

<snip>
>
> acl windowsupdate dstdomain windowsupdate.microsoft.com
> acl windowsupdate dstdomain .update.microsoft.com
> acl windowsupdate dstdomain download.windowsupdate.com
> acl windowsupdate dstdomain redir.metaservices.microsoft.com
> acl windowsupdate dstdomain images.metaservices.microsoft.com
> acl windowsupdate dstdomain c.microsoft.com
> acl windowsupdate dstdomain www.download.windowsupdate.com
> acl windowsupdate dstdomain wustat.windows.com
> acl windowsupdate dstdomain crl.microsoft.com
> acl windowsupdate dstdomain sls.microsoft.com
> acl windowsupdate dstdomain productactivation.one.microsoft.com
> acl windowsupdate dstdomain ntservicepack.microsoft.com
> acl windowsupdate dstdomain .go.microsoft.com
> acl windowsupdate dstdomain
> .update.microsoft.com/windowsupdate/v7/default.aspx
> acl windowsupdate dstdomain .download.microsoft.com
> acl windowsupdate dstdomain activex.microsoft.com
> acl windowsupdate dstdomain codecs.microsoft.com
> acl windowsupdate dstdomain urs.microsoft.com
>
> acl wuCONNECT dstdomain www.update.microsoft.com
> acl wuCONNECT dstdomain sls.microsoft.com
>
> http_access allow CONNECT wuCONNECT localnet
> http_access allow windowsupdate localnet
>
> # --- Windows update ends -----------------------------
>
> # ------ Test AntiVirus Caching --------------
> acl avast_allow_url url_regex -i \.vpu
> acl avast_allow_url url_regex -i \.vpx
>
> url_rewrite_access allow avast_allow_url
>
> acl avast dstdomain avast.com
> http_access allow CONNECT localnet

I notice this is the last mention of CONNECT. Nowhere to be seen is the
safety control preventing internal viral infections from using the proxy.
Please read:
 
http://wiki.squid-cache.org/SquidFaq/SecurityPitfalls#The_Safe_Ports_and_SSL_Ports_ACL

> http_access allow avast localnet
>
>
<snip comments and default options>
>
> range_offset_limit 0 KB
>
> quick_abort_min 0 KB

Okay. That prevents scenario (4) from happening. The side effect is
larer total bandwidth consued if clients about, but not an imbalance
like you are seeing.

>
> #quick_abort_pct 95
>
> range_offset_limit -1

This line replaces the earlier range_offset_limit of 0.

This one may cause scenario (2b) I mentioned above.

<snip>
>
> balance_on_multiple_ip on
>

This is not such a good idea. It breaks persistent connections,
IPv4/Ipv6 gateway and any websites which assume the client session will
only go to one of their servers.

> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
> refresh_pattern . 0 20% 4320
>

The "." pattern matches absolutely every URL that can possible exist.
Squid will never even attempt to evaluate the patterns which follow that
rule.

If you find that things have been working okay, then you can erase every
refresh_pattern below this line.

>
> #Suggested default:
> refresh_pattern -i \.jpg$ 0 50% 21600 reload-into-ims
> refresh_pattern -i \.gif$ 0 50% 21600 reload-into-ims
> refresh_pattern -i \.png$ 0 50% 21600 reload-into-ims
> refresh_pattern -i \.jpeg$ 0 50% 21600 reload-into-ims
> refresh_pattern -i \.bmp$ 0 50% 21600 reload-into-ims
> refresh_pattern -i \.tif$ 0 50% 21600 reload-into-ims
> refresh_pattern -i \.tiff$ 0 50% 21600 reload-into-ims
> refresh_pattern -i \.swf$ 0 50% 21600 reload-into-ims
> refresh_pattern -i \.html$ 0 20% 1440 reload-into-ims
> refresh_pattern -i \.htm$ 0 20% 1440 reload-into-ims
> refresh_pattern -i \.shtml$ 0 20% 1440 reload-into-ims
> refresh_pattern -i \.shtm$ 0 20% 1440 reload-into-ims
> refresh_pattern -i \.nub$ 2880 80% 21600 reload-into-ims
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern . 0 20% 8640
> refresh_pattern -i exe$ 0 50% 525600
> refresh_pattern -i zip$ 0 50% 525600
> refresh_pattern -i \.flv$ 10080 90% 525600 ignore-no-cache
> override-expire ignore-private
> refresh_pattern -i \.swf$ 10080 90% 525600 ignore-no-cache
> override-expire ignore-private
>
>
> read_ahead_gap 32 KB
>
> visible_hostname Optimum-Wireless-Services

Funny looking domain name.

http://Optimum-Wireless-Services/ doesn't go anywhere useful for me.

<snip comments>
> #Default:
> client_persistent_connections on
> server_persistent_connections on
>

But, but, you also have "balance_on_mutiple_ip on" Which means
round-robin the destination IPs. This will raise the number of
connections you have, eroding benefits from the persistence.

<snip>
>
> http_access allow clientes_registradosr
>
> shutdown_lifetime 45 seconds
>
> http_access deny all
>

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.14
   Beta testers wanted for 3.2.0.9