Re: [squid-users] Squid redirectors children doesn't shutdown/die with squid

2010/9/7 Jorge Iván Burgos Aguilar <jorgeburgos_at_inaipyucatan.org.mx>:
> Hi,
>
> I have been building for the past week a special redirector for squid
> in python 2.6.x; the redirectors works good they do their job but when
> i hit shutdown or reload (squid -k shutdown|reload) they just won't
> simple die and in fact they start to became huge resource monster and
> lag the cpu until obviously i kill them manually (which is in fact the
> workaround i have just made for when squid hit reload or shutdown on
> the init.d scripts).
> I don't know if its a bug or configuration problem so here are some
> relevant logs...
>
> ==cache.log==
> 2010/09/06 13:55:04.217| helperOpenServers: Starting 2/2 'python26' processes
> .....
> 2010/09/06 13:55:41.808| helperShutdown: redirector #1 shutting down.
> 2010/09/06 13:55:41.808| comm_close: start closing FD 9
> 2010/09/06 13:55:41.808| The AsyncCall comm_close_start constructed,
> this=0x2b7fce5b3590 [call66]
> 2010/09/06 13:55:41.808| comm.cc(1611) will call comm_close_start(FD 9) [call66]
> 2010/09/06 13:55:41.808| comm.cc(1195) commSetTimeout: FD 9 timeout -1
> 2010/09/06 13:55:41.808| comm.cc(1206) commSetTimeout: FD 9 timeout -1
> 2010/09/06 13:55:41.808| commio_finish_callback: called for FD 9 (-10, 11)
> 2010/09/06 13:55:41.808| comm.cc(165) will call SomeCommReadHandler(FD
> 9, errno=11, flag=-10, data=0x2b7fce58b528, size=0,
> buf=0x2b7fce58b5f0) [call4]
> 2010/09/06 13:55:41.808| commCallCloseHandlers: FD 9
> 2010/09/06 13:55:41.808| commCallCloseHandlers: ch->handler=0x2b7fce568a80*1
> 2010/09/06 13:55:41.808| comm.cc(1460) will call SomeCloseHandler(FD
> 9, data=0x2b7fce58b528) [call3]
> 2010/09/06 13:55:41.809| The AsyncCall comm_close_complete
> constructed, this=0x2b7fce5b3600 [call67]
> 2010/09/06 13:55:41.809| comm.cc(1643) will call
> comm_close_complete(FD 9) [call67]
> 2010/09/06 13:55:41.809| helperShutdown: redirector #2 shutting down.
> 2010/09/06 13:55:41.809| comm_close: start closing FD 11
> 2010/09/06 13:55:41.809| The AsyncCall comm_close_start constructed,
> this=0x2b7fce5b3670 [call68]
> 2010/09/06 13:55:41.809| comm.cc(1611) will call comm_close_start(FD
> 11) [call68]
> 2010/09/06 13:55:41.809| comm.cc(1195) commSetTimeout: FD 11 timeout -1
> 2010/09/06 13:55:41.809| comm.cc(1206) commSetTimeout: FD 11 timeout -1
> 2010/09/06 13:55:41.809| commio_finish_callback: called for FD 11 (-10, 11)
> 2010/09/06 13:55:41.809| comm.cc(165) will call SomeCommReadHandler(FD
> 11, errno=11, flag=-10, data=0x2b7fce58d7e8, size=0,
> buf=0x2b7fce58d8b0) [call8]
> 2010/09/06 13:55:41.809| commCallCloseHandlers: FD 11
> 2010/09/06 13:55:41.809| commCallCloseHandlers: ch->handler=0x2b7fce58f8c0*1
> 2010/09/06 13:55:41.809| comm.cc(1460) will call SomeCloseHandler(FD
> 11, data=0x2b7fce58d7e8) [call7]
> 2010/09/06 13:55:41.809| The AsyncCall comm_close_complete
> constructed, this=0x2b7fce58ad70 [call69]
> 2010/09/06 13:55:41.809| comm.cc(1643) will call
> comm_close_complete(FD 11) [call69]
> .....
> 2010/09/06 13:55:41.810| Open FD UNSTARTED     9 python26 #1
> 2010/09/06 13:55:41.810| Open FD UNSTARTED    11 python26 #2
> ==cache.log==
>
> And a sample/stub of the redirector python script if it helps...
> ==python redirector script==
> #!/usr/bin/python26
> import sys
>
> # Variables
> EOF = False
> RecentBorn = True
>
> # Life Cycle
> try:
>    while ( not EOF ):
>          if ( RecentBorn ):
>             RecentBorn = False
>          else:
>             sys.stdout.flush()
>
>          line = sys.stdin.readline()
>          sys.stdout.write("\n")
>
> except KeyboardInterrupt:
>    EOF = True
> ==python redirector script==
>
> And my running box of course:
> # uname -msr
> Linux 2.6.18-194.11.3.el5.centos.plus x86_64
> # squid -v
> Squid Cache: Version 3.1.8
> configure options:  '--build=x86_64-redhat-linux-gnu'
> '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu'
> '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr'
> '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
> '--datadir=/usr/share' '--includedir=/usr/include'
> '--libdir=/usr/lib64' '--libexecdir=/usr/libexec'
> '--sharedstatedir=/usr/com' '--mandir=/usr/share/man'
> '--infodir=/usr/share/info' '--exec_prefix=/usr'
> '--libexecdir=/usr/lib64/squid' '--localstatedir=/var'
> '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid'
> '--with-logdir=$(localstatedir)/log/squid'
> '--with-pidfile=$(localstatedir)/run/squid.pid'
> '--disable-dependency-tracking' '--enable-arp-acl'
> '--enable-follow-x-forwarded-for'
> '--enable-auth=basic,digest,ntlm,negotiate'
> '--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SMB,YP,getpwnam,multi-domain-NTLM,SASL,DB,POP3,squid_radius_auth'
> '--enable-ntlm-auth-helpers=smb_lm,no_check,fakeauth'
> '--enable-digest-auth-helpers=password,ldap,eDirectory'
> '--enable-negotiate-auth-helpers=squid_kerb_auth'
> '--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group'
> '--enable-cache-digests' '--enable-cachemgr-hostname=localhost'
> '--enable-delay-pools' '--enable-epoll' '--enable-icap-client'
> '--enable-ident-lookups' '--enable-linux-netfilter'
> '--enable-referer-log' '--enable-removal-policies=heap,lru'
> '--enable-snmp' '--enable-ssl' '--enable-storeio=aufs,diskd,ufs'
> '--enable-useragent-log' '--enable-wccpv2' '--enable-esi' '--with-aio'
> '--with-default-user=squid' '--with-filedescriptors=16384' '--with-dl'
> '--with-openssl' '--with-pthreads' '--disable-loadable-modules'
> 'build_alias=x86_64-redhat-linux-gnu'
> 'host_alias=x86_64-redhat-linux-gnu'
> 'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-fPIE -Os -g -pipe
> -fsigned-char -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
> -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'
> 'LDFLAGS=-pie' 'CXXFLAGS=-fPIE -O2 -g -pipe -Wall
> -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
> --param=ssp-buffer-size=4 -m64 -mtune=generic'
> --with-squid=/home/constructor/rpmbuild/BUILD/squid-3.1.8
> --enable-ltdl-convenience
>
> Best Regards
>

The redirectors and external_acls must finish when they receive an
empty line. The skeleton of your redirector should be something like
this:

#!/usr/bin/python -u <--- -u is very important, it disables
buffering on stdin, stdout and stderr

import sys

while True:
     line = sys.stdin.readline()
     if not line:
        break

   do_something(line)

Regards,
  DIego

-- 
Diego Woitasen
XTECH