[Fwd: Re: [squid-users] help with squid redirectors]

-------- Forwarded Message --------
From: Osmany <osmany_at_oc.quimefa.cu>
Reply-to: osmany_at_oc.quimefa.cu
To: squid-users <squid-users_at_squid-cache.org>
Subject: Re: [squid-users] help with squid redirectors
Date: Tue, 08 Mar 2011 07:41:47 -0500

On Wed, 2011-03-09 at 01:33 +1300, Amos Jeffries wrote:
> On 09/03/11 01:20, Osmany wrote:
> > On Tue, 2011-03-08 at 12:21 +1300, Amos Jeffries wrote:
> >> On Tue, 08 Mar 2011 11:58:57 +1300, Amos Jeffries wrote:
> >>> On Mon, 07 Mar 2011 16:59:07 -0500, Osmany wrote:
> >>>> Greetings everyone,
> >>>>
> >>>> So I'm having trouble with my squid proxy-cache server. I recently
> >>>> added
> >>>> a redirect program because I had to make users go to my kaspersky
> >>>> admin
> >>>> kit and my WSUS services to get their updates and it works fine but
> >>>> I
> >>>> get constantly a warning and squid just collapses after a few
> >>>> minutes of
> >>>> run time. This is what I get in my cache.log:
> >>>>
> >>>> 2011/03/07 15:54:17| WARNING: All url_rewriter processes are busy.
> >>>> 2011/03/07 15:54:17| WARNING: up to 465 pending requests queued
> >>>> 2011/03/07 15:54:17| storeDirWriteCleanLogs: Starting...
> >>>> 2011/03/07 15:54:17| WARNING: Closing open FD 1455
> >>>> 2011/03/07 15:54:17| commSetEvents: epoll_ctl(EPOLL_CTL_DEL): failed
> >>>> on
> >>>> fd=1455: (1) Operation not permitted
> >>>> 2011/03/07 15:54:17| 65536 entries written so far.
> >>>> 2011/03/07 15:54:17| 131072 entries written so far.
> >>>> 2011/03/07 15:54:17| WARNING: Closing open FD 1456
> >>>> 2011/03/07 15:54:17| commSetEvents: epoll_ctl(EPOLL_CTL_DEL): failed
> >>>> on
> >>>> fd=1456: (1) Operation not permitted
> >>>> 2011/03/07 15:54:17| Finished. Wrote 139965 entries.
> >>>> 2011/03/07 15:54:17| Took 0.1 seconds (1288729.1 entries/sec).
> >>>> FATAL: Too many queued url_rewriter requests (465 on 228)
> >>>> Squid Cache (Version 2.7.STABLE7): Terminated abnormally.
> >>>>
> >>>> This is what I have in the squid.conf
> >>>>
> >>>> # TAG: url_rewrite_program
> >>>> url_rewrite_program /etc/squid/redirect
> >>>>
> >>>> # TAG: url_rewrite_children
> >>>> url_rewrite_children 100
> >>>>
> >>>> # TAG: url_rewrite_concurrency
> >>>> url_rewrite_concurrency 50
> >>>>
> >>>> # TAG: url_rewrite_access
> >>>> url_rewrite_access allow redirect
> >>>>
> >>>> And this is what I have in my redirector script
> >>>>
> >>>> #!/usr/bin/perl
> >>>> BEGIN {$|=1}
> >>>> while (<>) {
> >>>> @X = split;
> >>>> $url = $X[0];
> >>>> if ($url =~ /^http:\/\/dnl(.*)kaspersky(.*)com(.*)/) {
> >>>> print
> >>>> "301:ftp:\/\/dnl-kaspersky\.quimefa\.cu\:2122\/Updates";
> >>>> }
> >>>> elsif ($url =~ /^http:\/\/(.*)windowsupdate(.*)/) {
> >>>> print "301:http:\/\/windowsupdate\.quimefa\.cu\:8530";
> >>>> }
> >>>> }
> >>>>
> >>>> Can you please help me to solve this?
> >>>
> >>> Your script does not support concurrency. When that is configured in
> >>> squid there will be 2 space-delimited fields to handle.
> >>> First one being the ID of the request channel, not the URL.
> >>>
> >>
> >> Oops, I missed a few other things too:
> >> * 'else' case is needed to print the no-change result back to Squid
> >> * newlines need to be printed in perl
> >>
> >>
> >> $url = $X[1];
> >> if ($url =~ /^http:\/\/dnl(.*)kaspersky(.*)com(.*)/) {
> >> print $X[0]."
> >> 301:ftp:\/\/dnl-kaspersky\.quimefa\.cu\:2122\/Updates\n";
> >> }
> >> elsif ($url =~ /^http:\/\/(.*)windowsupdate(.*)/) {
> >> print $X[0]." 301:http:\/\/windowsupdate\.quimefa\.cu\:8530\n";
> >> }
> >> else {
> >> print $X[0]."\n";
> >> }
> >>
> >> Amos
> >
> > So this is what I have now but it doesn't work. I've tried it manually:
> >
> > #!/usr/bin/perl
> > BEGIN {$|=1}
> > while (<>) {
> > @X = split;
> > $url = $X[1];
> > if ($url =~ /^http:\/\/dnl(.*)kaspersky(.*)com(.*)/) {
> > print $X[0]."
> > 301:ftp:\/\/dnl-kaspersky\.quimefa\.cu\:2122\/Updates\n";
> > }
> > elsif ($url =~ /^http:\/\/(.*)windowsupdate(.*)/) {
> > print $X[0]." 301:http:\/\/windowsupdate\.quimefa\.cu\:8530\n";
> > }
> > else {
> > print $X[0]."\n";
> > }
> > }
> >
> > it just keeps on returning the same url that I enter. help please?
> >
>
> Did you add the concurrency channel ID before the URL on each manually
> entered line?
> eg $id $url $garbage
>
> Amos

Yes you are right. I was missing the concurrency channel ID on my manual
test in fact when I supervise my access.log I see it is actually working
and when I look at the cache.log I see that it working concurrently with
all the available channels.

Now actually I found out that my script only works for one of the two
conditions I have. the one that works: windowsupdate, I can see that it
is actually redirecting because I see it in the access.log file like
this:

TCP_MISS/301 330 HEAD
http://download.microsoft.com/windowsupdate/v5/redir/wuredir.cab?

But the other condition, the kaspersky one I am pretty sure it is not
working because in the access.log this is what I see:
TCP_MISS/200 946 GET
http://dnl-17.geo.kaspersky.com/diffs/bases/av/kdb/i386/base048c.kdc.hax

Not only that. I can also see in the cache.log that it is giving me a
warning that I really don't understand.

helperHandleRead: unexpected reply on channel 3301 from url_rewriter #26
'3301:ftp://dnl-kaspersky.quimefa.cu:2122/Updates'

could you please help me with this now?

thanks in advance
Received on Tue Mar 08 2011 - 12:57:14 MST