Re: [squid-users] Squid 2.7STABLE7 randomly crashes

Mark Nottingham wrote:
> Can you paste in your acl-related configuration? That's where the crash is...
>

... and a couple of lines logged above the FATAL which say what Squid
was trying to do at the time.

Amos

>
> On 27/01/2010, at 10:50 AM, myocella wrote:
>
>> Anyone?
>>
>> On Wed, Jan 20, 2010 at 4:17 PM, myocella <myocella_at_gmail.com> wrote:
>>> Greetings,
>>>
>>> A Squid server (RHEL-5.2 64-bit) is serving around 1000 users with
>>> NTLM (winbindd),
>>> Basic, wbinfo, and URL rewriter (Websense). Squid has randomly crashed with
>>> the following messages:
>>>
>>> FATAL: Received Segment Violation...dying.
>>> 2010/01/20 15:53:29| storeDirWriteCleanLogs: Starting...
>>> 2010/01/20 15:53:29| WARNING: Closing open FD 183
>>> 2010/01/20 15:53:29| commSetEvents: epoll_ctl(EPOLL_CTL_DEL): failed
>>> on fd=183: (1) Operation not permitted
>>> 2010/01/20 15:53:29| 65536 entries written so far.
>>> 2010/01/20 15:53:29| 131072 entries written so far.
>>> ...
>>> ...
>>> 2010/01/20 15:53:35| Finished. Wrote 10511684 entries.
>>> 2010/01/20 15:53:35| Took 6.2 seconds (1702503.8 entries/sec).
>>> CPU Usage: 3406.086 seconds = 2431.073 user + 975.013 sys
>>> Maximum Resident Size: 0 KB
>>> Page faults with physical i/o: 3
>>> Memory usage for squid via mallinfo():
>>> total space in arena: -1092812 KB
>>> Ordinary blocks: -1096748 KB 3986 blks
>>> Small blocks: 0 KB 24 blks
>>> Holding blocks: 24108 KB 8 blks
>>> Free Small blocks: 1 KB
>>> Free Ordinary blocks: 3935 KB
>>> Total in use: -1072640 KB 100%
>>> Total free: 3936 KB 0%
>>> 2010/01/20 15:54:48| Starting Squid Cache version 2.7.STABLE7 for
>>> x86_64-unknown-linux-gnu...
>>>
>>>
>>>
>>> Ran gdb over the core dump file and got this result:
>>>
>>> # gdb /usr/local/squid/sbin/squid core.30068
>>> GNU gdb Red Hat Linux (6.5-37.el5rh)
>>> Copyright (C) 2006 Free Software Foundation, Inc.
>>> GDB is free software, covered by the GNU General Public License, and you are
>>> welcome to change it and/or distribute copies of it under certain conditions.
>>> Type "show copying" to see the conditions.
>>> There is absolutely no warranty for GDB. Type "show warranty" for details.
>>> This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db
>>> library "/lib64/libthread_db.so.1".
>>>
>>> Reading symbols from /lib64/libcrypt.so.1...done.
>>> Loaded symbols for /lib64/libcrypt.so.1
>>> Reading symbols from /lib64/libpthread.so.0...done.
>>> Loaded symbols for /lib64/libpthread.so.0
>>> Reading symbols from /lib64/libm.so.6...done.
>>> Loaded symbols for /lib64/libm.so.6
>>> Reading symbols from /lib64/libnsl.so.1...done.
>>> Loaded symbols for /lib64/libnsl.so.1
>>> Reading symbols from /lib64/libc.so.6...done.
>>> Loaded symbols for /lib64/libc.so.6
>>> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
>>> Loaded symbols for /lib64/ld-linux-x86-64.so.2
>>> Reading symbols from /lib64/libnss_files.so.2...done.
>>> Loaded symbols for /lib64/libnss_files.so.2
>>> Reading symbols from /lib64/libnss_dns.so.2...done.
>>> Loaded symbols for /lib64/libnss_dns.so.2
>>> Reading symbols from /lib64/libresolv.so.2...done.
>>> Loaded symbols for /lib64/libresolv.so.2
>>> Core was generated by `(squid)'.
>>> Program terminated with signal 6, Aborted.
>>> #0 0x000000376b630215 in raise () from /lib64/libc.so.6
>>> (gdb) bt
>>> #0 0x000000376b630215 in raise () from /lib64/libc.so.6
>>> #1 0x000000376b631cc0 in abort () from /lib64/libc.so.6
>>> #2 0x0000000000474515 in death (sig=<value optimized out>) at tools.c:327
>>> #3 <signal handler called>
>>> #4 0x000000000044271a in httpHeaderGetEntry (hdr=0x141868f8,
>>> pos=0x7fff2c609774) at HttpHeader.c:595
>>> #5 0x00000000004427fe in httpHeaderFindEntry (hdr=0x141868f8,
>>> id=HDR_CONTENT_TYPE) at HttpHeader.c:619
>>> #6 0x0000000000443037 in httpHeaderGetStr (hdr=0x141868f8,
>>> id=HDR_CONTENT_TYPE) at HttpHeader.c:1086
>>> #7 0x00000000004075b9 in aclMatchAclList (list=0x2aaac246fa40,
>>> checklist=0x2aaad3524738) at acl.c:2010
>>> #8 0x0000000000407d82 in aclCheck (checklist=0x2aaad3524738) at acl.c:2178
>>> #9 0x000000000042daae in externalAclHandleReply (data=0x2aaab26015d0,
>>> reply=0x2aaac6951e38 "") at external_acl.c:985
>>> #10 0x000000000043cd65 in helperHandleRead (fd=<value optimized out>,
>>> data=<value optimized out>) at helper.c:769
>>> #11 0x0000000000427511 in comm_select (msec=<value optimized out>)
>>> at comm_generic.c:264
>>> #12 0x000000000044f7a6 in main (argc=<value optimized out>,
>>> argv=0x7fff2c609a58) at main.c:863
>>>
>>> The configure options read
>>>
>>> ./configure --prefix=/usr/local/squid --enable-async-io --enable-snmp --enab
>>> le-epoll --enable-storeio=aufs null --enable-removal-policies=heap lru
>>> --enable-
>>> large-cache-files --with-large-files --disable-poll --disable-select
>>> --disable-c
>>> arp --disable-wccp --disable-ident-lookups --with-maxfd=32768
>>> --enable-ntlm-auth
>>> -helpers=SMB --enable-external-acl-helpers=wbinfo_group ldap_group
>>> --enable-auth
>>> =basic ntlm --enable-basic-auth-helpers=LDAP --enable-err-languages=English
>>>
>>>
>>> Snippet of squid.conf
>>>
>>> emulate_httpd_log on
>>> buffered_logs on
>>> debug_options ALL,1
>>>
>>> logfile_daemon /usr/local/squid/libexec/logfile-daemon
>>> access_log daemon:/var/log/squid/access.log
>>>
>>> shutdown_lifetime 10 seconds
>>>
>>> cache_dir aufs /var/spool/squid/cache01 62500 32 256
>>> cache_dir aufs /var/spool/squid/cache02 62500 32 256
>>> cache_dir aufs /var/spool/squid/cache03 62500 32 256
>>> cache_dir aufs /var/spool/squid/cache04 62500 32 256
>>>
>>> cache_replacement_policy heap GDSF
>>> memory_replacement_policy heap GDSF
>>>
>>> cache_log /var/log/squid/cache.log
>>> cache_store_log none
>>> pid_filename /var/run/squid.pid
>>> coredump_dir /var/spool/squid/crash
>>> log_icp_queries off
>>> client_db on
>>> half_closed_clients off
>>>
>>> cache_mem 512 MB
>>> maximum_object_size 768000 KB
>>> maximum_object_size_in_memory 96 KB
>>> memory_pools off
>>>
>>> forwarded_for off
>>>
>>> snmp_port 1601
>>> snmp_incoming_address 0.0.0.0
>>> snmp_outgoing_address 255.255.255.255
>>>
>>> auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
>>> auth_param ntlm children 30
>>> auth_param ntlm keep_alive on
>>>
>>> auth_param basic program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-basic
>>> auth_param basic children 5
>>> auth_param basic realm User Authentication
>>>
>>> external_acl_type ads-group children=20 %LOGIN
>>> /usr/local/squid/libexec/wbinfo_group.pl
>>>
>>> url_rewrite_children 50
>>> redirector_bypass off
>>> url_rewrite_program /opt/Websense/bin/WsRedtor
>>>
>>>
>>> Does anyone have any idea how to fix this problem?
>>>
>>> Many Thanks
>>>
>>> myOcella
>>>
>
> --
> Mark Nottingham mnot_at_yahoo-inc.com
>
>

-- 
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE21
   Current Beta Squid 3.1.0.15