Re: [squid-users] limiting connections

On 27/03/2012 17:27, Carlos Manuel Trepeu Pupo wrote:
> On Mon, Mar 26, 2012 at 5:45 PM, Amos Jeffries<squid3_at_treenet.co.nz> wrote:
>> On 27.03.2012 10:13, Carlos Manuel Trepeu Pupo wrote:
>>>
>>> On Sat, Mar 24, 2012 at 6:31 PM, Amos Jeffries<squid3_at_treenet.co.nz>
>>> wrote:
>>>>
>>>> On 25/03/2012 7:23 a.m., Carlos Manuel Trepeu Pupo wrote:
>>>>
>>>>> On Thu, Mar 22, 2012 at 10:00 PM, Amos Jeffries wrote:
>>>>>>
>>>>>>
>>>>>> On 23/03/2012 5:42 a.m., Carlos Manuel Trepeu Pupo wrote:
>>>>>>>
>>>>>>>
>>>>>>> I need to block each user to make just one connection to download
>>>>>>> specific extension files, but I dont know how to tell that can make
>>>>>>> just one connection to each file and not just one connection to every
>>>>>>> file with this extension.
>>>>>>>
>>>>>>> i.e:
>>>>>>> www.google.com #All connection that required
>>>>>>> www.any.domain.com/my_file.rar #just one connection to that file
>>>>>>> www.other.domain.net/other_file.iso #just connection to this file
>>>>>>> www.other_domain1.com/other_file1.rar #just one connection to that
>>>>>>> file
>>>>>>>
>>>>>>> I hope you understand me and can help me, I have my boss hurrying me
>>>>>>> !!!
>>>>>>
>>>>>>
>>>>>>
>>>>>> There is no easy way to test this in Squid.
>>>>>>
>>>>>> You need an external_acl_type helper which gets given the URI and
>>>>>> decides
>>>>>> whether it is permitted or not. That decision can be made by querying
>>>>>> Squid
>>>>>> cache manager for the list of active_requests and seeing if the URL
>>>>>> appears
>>>>>> more than once.
>>>>>
>>>>>
>>>>> Hello Amos, following your instructions I make this external_acl_type
>>>>> helper:
>>>>>
>>>>> #!/bin/bash
>>>>> result=`squidclient -h 192.168.19.19 mgr:active_requests | grep -c "$1"`
>>>>> if [ $result -eq 0 ]
>>>>> then
>>>>> echo 'OK'
>>>>> else
>>>>> echo 'ERR'
>>>>> fi
>>>>>
>>>>> # If I have the same URI then I denied. I make a few test and it work
>>>>> for me. The problem is when I add the rule to the squid. I make this:
>>>>>
>>>>> acl extensions url_regex "/etc/squid3/extensions"
>>>>> external_acl_type one_conn %URI /home/carlos/script
>>>>> acl limit external one_conn
>>>>>
>>>>> # where extensions have:
>>>>>
>>>>>
>>>>>
>>>>> \.(iso|avi|wav|mp3|mp4|mpeg|swf|flv|mpg|wma|ogg|wmv|asx|asf|deb|rpm|exe|zip|tar|tgz|rar|ppt|doc|tiff|pdf)$
>>>>>
>>>>> http_access deny extensions limit
>>>>>
>>>>>
>>>>> So when I make squid3 -k reconfigure the squid stop working
>>>>>
>>>>> What can be happening ???
>>>>
>>>>
>>>>
>>>> * The helper needs to be running in a constant loop.
>>>> You can find an example
>>>>
>>>>
>>>> http://bazaar.launchpad.net/~squid/squid/3.2/view/head:/helpers/url_rewrite/fake/url_fake_rewrite.sh
>>>> although that is re-writer and you do need to keep the OK/ERR for
>>>> external
>>>> ACL.
>>>
>>>
>>> Sorry, this is my first helper, I do not understand the meaning of
>>> running in a constant loop, in the example I see something like I do.
>>> Making some test I found that without this line :
>>> result=`squidclient -h 192.168.19.19 mgr:active_requests | grep -c "$1"`
>>> the helper not crash, dont work event too, but do not crash, so i
>>> consider this is in some way the problem.
>>
>>
>>
>> Squid starts helpers then uses the STDIN channel to pass it a series of
>> requests, reading STDOUt channel for the results. The helper once started is
>> expected to continue until a EOL/close/terminate signal is received on its
>> STDIN.
>>
>> Your helper is exiting without being asked to be Squid after only one
>> request. That is logged by Squid as a "crash".
>>
>>
>>>
>>>>
>>>> * "eq 0" - there should always be 1 request matching the URL. Which is
>>>> the
>>>> request you are testing to see if its>1 or not. You are wanting to deny
>>>> for
>>>> the case where there are *2* requests in existence.
>>>
>>>
>>> This is true, but the way I saw was: "If the URL do not exist, so
>>> can't be duplicate", I think isn't wrong !!
>>
>>
>> It can't not exist. Squid is already servicing the request you are testing
>> about.
>>
>> Like this:
>>
>> receive HTTP request -> (count=1)
>> - test ACL (count=1 -> OK)
>> - done (count=0)
>>
>> receive a HTTP request (count-=1)
>> - test ACL (count=1 -> OK)
>> receive b HTTP request (count=2)
>> - test ACL (count=2 -> ERR)
>> - reject b (count=1)
>> done a (count=0)
>
> With your explanation and code from Eliezer Croitoru I made this:
>
> #!/bin/bash
>
> while read line; do
> result=`squidclient -h 192.168.19.19 mgr:active_requests | grep
> -c "$line"`
>
> echo "$line">> /home/carlos/guarda # -> Add this line to
> see in a file the $URI I passed to the helper
>
> if [ $result -eq 1 ] # ->
> With your great explain you made me, I change to "1"
> then
> echo 'OK'
> else
> echo 'ERR'
> fi
> done
>
> It's look like it's gonna work, but, here another miss.
> 1- The "echo "$line">> /home/carlos/guarda" do not save anything to the file.
> 2- When I return 'OK' then in my .conf I can't make a rule like I
> wrote before, I have to make something like this: "http_access deny
> extensions !limit", in the many helps you bring me guys, I learn that
> the name "limit" here its not functional. The deny of "limit" its
> because when there are just one connection I cant block the page.
> 3- With the script just like Eliezer tape it the page with the URL to
> download stay loading infinitely.
>
> So, I have less work, can you help me ??
>

1. the first is that "squidclient -h 192.168.19.19 mgr:active_requests"
can take awhile in some cases.
the first time i tried to run the command it took couple of minutes for
squid to send the list (1 connections).
so your hanging stuff is probably because of this issue.

2. why you are writing to a file? if it's for debugging ok.
and what you need to do is ti use the echo $? to get from the grep the
lookup answer first.
so psudo:
-----------
read the uri line. (just notice that there is a possibility for 2 uris
on two different hosts just to notice it._
request from squid the list of active downloads and see if any of the
downloads in the output has a match to the uri in the line before.
in case the uri exists the outpot of "echo $?" (exit code) will produce 0

case it will find 1 echo OK
case it will find 0 echo ERR

end
goto read uri...

-----------
the reason you cant add info to the file is because the file is owned by
other user then the one that is executing the script for squid.
so change the file permissions to 666 or change the group and user i
thing to squid unprivileged user.
the whole thing is a simple while loop with a nice if (echo $? == 1)

#!/bin/bash

while read line; do
#i'm throwing the echo to background in case of slow disc access(dont
really know how much it will improve)
          echo $line>> /home/carlos/guarda &
# -> Add this line to see in a file the $URI I passed to the helper
        result=`squidclient -h 192.168.19.19 mgr:active_requests | grep -c
"$line"|echo $?`

    if [ $result == 1 ]
          then
          echo 'OK'
          echo 'OK'>>/home/carlos/guarda &
    else
          echo 'ERR'
         echo 'ERR'>>/home/carlos/guarda &
    fi
done

about the acls.
you can use the follow
http_access deny external_helper_acl

deny_info http://block_url_site_page/block.html external_helper_acl

http_access allow loalhost manager
http_access allow loalhost
...

this will do the trick for you.
unless... squidclient is stuck with the output.
and also the echo statements that writes to the file gives error output
that can cause trouble for that.

by the way this external acl can limit number of current connections to
more then just 1 with some wc -l stuff.

Regards,
ELiezer

>
>
>>
>>
>>>
>>>>
>>>> * ensure you have manager requests form localhost not going through the
>>>> ACL
>>>> test.
>>>
>>>
>>> I was making this wrong, the localhost was going through the ACL, but
>>> I just changed !!! The problem persist, What can I do ???
>>
>>
>> which problem?
>>
>>
>> Amos

-- 
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il