Re: [squid-users] Finally, the first real coredump, was Re: [squid-users] Squid Error

To continue this neverending story:
===================================

Yesterday, squid crashed several times:

-rw------- 1 root nogroup 192921780 Apr 23 18:36 core-18:36:13
-rw------- 1 root nogroup 87204020 Apr 23 19:14 core-19:14:31
-rw------- 1 root nogroup 75980916 Apr 23 19:16 core-19:16:09
-rw------- 1 root nogroup 37683316 Apr 23 19:16 core-19:16:40
-rw------- 1 root nogroup 10961012 Apr 23 19:16 core-19:16:46
-rw------- 1 root nogroup 13835476 Apr 23 19:16 core-19:16:54
-rw------- 1 root nogroup 7920852 Apr 23 19:16 core-19:16:57
-rw------- 1 root nogroup 9026772 Apr 23 19:17 core-19:17:01
-rw------- 1 root nogroup 8232148 Apr 23 19:17 core-19:17:04
-rw------- 1 root nogroup 8715476 Apr 23 19:17 core-19:17:08
-rw------- 1 root nogroup 150905012 Apr 24 09:21 core-09:21:10

Let's start with 18:36:13:
==========================

Core was generated by /usr/local/squid/sbin/squid -NsYC'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /usr/lib/libcrypt_i.so.1...done.
Loaded symbols for /usr/lib/libcrypt_i.so.1
Reading symbols from /usr/lib/libpthread.so.1...done.
Loaded symbols for /usr/lib/libpthread.so.1
Reading symbols from /usr/lib/libm.so.1...done.
Loaded symbols for /usr/lib/libm.so.1
Reading symbols from /usr/lib/libresolv.so.2...done.
Loaded symbols for /usr/lib/libresolv.so.2
Reading symbols from /usr/lib/libsocket.so.1...done.
Loaded symbols for /usr/lib/libsocket.so.1
Reading symbols from /usr/lib/libnsl.so.1...done.
Loaded symbols for /usr/lib/libnsl.so.1
Reading symbols from /usr/lib/libc.so.1...done.
Loaded symbols for /usr/lib/libc.so.1
Reading symbols from /usr/lib/libgen.so.1...done.
Loaded symbols for /usr/lib/libgen.so.1
Reading symbols from /usr/lib/libdl.so.1...done.
Loaded symbols for /usr/lib/libdl.so.1
Reading symbols from /usr/lib/libmp.so.2...done.
Loaded symbols for /usr/lib/libmp.so.2
Reading symbols from /usr/platform/SUNW,Ultra-60/lib/libc_psr.so.1...done.
Loaded symbols for /usr/platform/SUNW,Ultra-60/lib/libc_psr.so.1
Reading symbols from /usr/lib/libthread.so.1...done.
Loaded symbols for /usr/lib/libthread.so.1
Reading symbols from /usr/lib/nss_files.so.1...done.
Loaded symbols for /usr/lib/nss_files.so.1
Reading symbols from /usr/lib/nss_dns.so.1...done.
Loaded symbols for /usr/lib/nss_dns.so.1
#0 0x000ce9ac in free (mem=0xb088778) at dlmalloc.c:2356
2356 if (p->fd == last_remainder) /* keep as last_remainder */

(gdb) where
#0 0x000ce9ac in free (mem=0xb088778) at dlmalloc.c:2356
#1 0x000d8a74 in rfc1035RRDestroy (rr=0xb088930, n=0) at rfc1035.c:487
#2 0x00046638 in idnsGrokReply (buf=0x110400 "UO\201\200", sz=107) at dns_internal.c:485
#3 0x00046ad0 in idnsRead (fd=5, data=0x0) at dns_internal.c:556
#4 0x000412b4 in comm_check_incoming_poll_handlers (nfds=1, fds=0xffbe7c38) at comm_select.c:238
#5 0x00042bf0 in comm_poll_dns_incoming () at comm_select.c:927
#6 0x00042a70 in comm_poll (msec=570) at comm_select.c:497
#7 0x0007a1f0 in main (argc=2, argv=0xffbefe94) at main.c:735

Ah, much better than the old dumps.

Cache.log for that incident:

...
2003/04/23 18:09:01| CACHEMGR: <unknown>@192.168.220.203 requesting 'info'
2003/04/23 18:10:01| CACHEMGR: <unknown>@192.168.220.203 requ2003/04/23 18:36:15| Starting Squid Cache version 2.5.STABLE2-20030422 for ...
2003/04/23 18:36:15| Process ID 4187
2003/04/23 18:36:15| With 4096 file descriptors available
2003/04/23 18:36:15| Performing DNS Tests...
2003/04/23 18:36:15| Successful DNS name lookup tests...
2003/04/23 18:36:15| DNS Socket created at 0.0.0.0, port 23368, FD 5
2003/04/23 18:36:15| Adding nameserver 193.175.66.33 from squid.conf
2003/04/23 18:36:15| Adding nameserver 193.175.70.246 from squid.conf
...

19:16:09:
=========

...
Loaded symbols for /usr/lib/nss_dns.so.1
#0 0x000ce9f4 in free (mem=0x455f7d0) at dlmalloc.c:2359
2359 unlink(p, bck, fwd);
(gdb) bt
#0 0x000ce9f4 in free (mem=0x455f7d0) at dlmalloc.c:2359
#1 0x000d8a74 in rfc1035RRDestroy (rr=0x455f738, n=0) at rfc1035.c:487
#2 0x00046638 in idnsGrokReply (buf=0x110400 "", sz=107) at dns_internal.c:485
#3 0x00046ad0 in idnsRead (fd=5, data=0x0) at dns_internal.c:556
#4 0x000412b4 in comm_check_incoming_poll_handlers (nfds=1, fds=0xffbe7c38) at comm_select.c:238
#5 0x00042bf0 in comm_poll_dns_incoming () at comm_select.c:927
#6 0x00042a70 in comm_poll (msec=496) at comm_select.c:497
#7 0x0007a1f0 in main (argc=2, argv=0xffbefe94) at main.c:735

cache.log:
...
2003/04/23 19:14:33| User-Agent logging is disabled.
2003/04/23 19:14:33| Referer logging is disabled.
2003/04/23 19:16:10| Starting Squid Cache version 2.5.STABLE2-20030422 for ...
2003/04/23 19:16:10| Process ID 4727
2003/04/23 19:16:10| With 4096 file descriptors available
2003/04/23 19:16:10| Performing DNS Tests...
2003/04/23 19:16:10| Successful DNS name lookup tests...
...

19:16:40:
=========
...
Loaded symbols for /usr/lib/nss_dns.so.1
#0 0x000ce9f4 in free (mem=0x1fe3a48) at dlmalloc.c:2359
2359 unlink(p, bck, fwd);
(gdb) bt
#0 0x000ce9f4 in free (mem=0x1fe3a48) at dlmalloc.c:2359
#1 0x000d8a74 in rfc1035RRDestroy (rr=0x1fe39b0, n=0) at rfc1035.c:487
#2 0x00046638 in idnsGrokReply (buf=0x110400 "", sz=107) at dns_internal.c:485
#3 0x00046ad0 in idnsRead (fd=5, data=0x0) at dns_internal.c:556
#4 0x000412b4 in comm_check_incoming_poll_handlers (nfds=1, fds=0xffbe7c38) at comm_select.c:238
#5 0x00042bf0 in comm_poll_dns_incoming () at comm_select.c:927
#6 0x00042a70 in comm_poll (msec=0) at comm_select.c:497
#7 0x0007a1f0 in main (argc=2, argv=0xffbefe94) at main.c:735

cache.log:

...
2003/04/23 19:16:10| User-Agent logging is disabled.
2003/04/23 19:16:10| Referer logging is disabled.
2003/04/23 19:16:41| Starting Squid Cache version 2.5.STABLE2-20030422 for ...
2003/04/23 19:16:41| Process ID 4739
2003/04/23 19:16:41| With 4096 file descriptors available
2003/04/23 19:16:41| Performing DNS Tests...
2003/04/23 19:16:41| Successful DNS name lookup tests...
2003/04/23 19:16:41| DNS Socket created at 0.0.0.0, port 23426, FD 5
2003/04/23 19:16:41| Adding nameserver 193.175.66.33 from squid.conf
2003/04/23 19:16:41| Adding nameserver 193.175.70.246 from squid.conf
2003/04/23 19:16:41| User-Agent logging is disabled.
...

core-19:16:46:
==============

...
Reading symbols from /usr/lib/nss_dns.so.1...done.
Loaded symbols for /usr/lib/nss_dns.so.1
#0 0x000ce9f4 in free (mem=0x666590) at dlmalloc.c:2359
2359 unlink(p, bck, fwd);
(gdb) bt
#0 0x000ce9f4 in free (mem=0x666590) at dlmalloc.c:2359
#1 0x000d8a74 in rfc1035RRDestroy (rr=0x6664f8, n=0) at rfc1035.c:487
#2 0x00046638 in idnsGrokReply (buf=0x110400 "", sz=107) at dns_internal.c:485
#3 0x00046ad0 in idnsRead (fd=5, data=0x0) at dns_internal.c:556
#4 0x000412b4 in comm_check_incoming_poll_handlers (nfds=1, fds=0xffbe7c38) at comm_select.c:238
#5 0x00042bf0 in comm_poll_dns_incoming () at comm_select.c:927
#6 0x00042a70 in comm_poll (msec=0) at comm_select.c:497
#7 0x0007a1f0 in main (argc=2, argv=0xffbefe94) at main.c:735

cache.log:
==========
...
2003/04/23 19:16:41| Process ID 4739
2003/04/23 19:16:41| With 4096 file descriptors available
2003/04/23 19:16:41| Performing DNS Tests...
2003/04/23 19:16:41| Successful DNS name lookup tests...
2003/04/23 19:16:41| DNS Socket created at 0.0.0.0, port 23426, FD 5
2003/04/23 19:16:41| Adding nameserver 193.175.66.33 from squid.conf
2003/04/23 19:16:41| Adding nameserver 193.175.70.246 from squid.conf
2003/04/23 19:16:41| User-Agent logging is disabled.
2003/04/23 19:16:41| Referer logging is disabled.
2003/04/23 19:16:47| Starting Squid Cache version 2.5.STABLE2-20030422 for ...
2003/04/23 19:16:47| Process ID 4746
2003/04/23 19:16:47| With 4096 file descriptors available
2003/04/23 19:16:47| Performing DNS Tests...
2003/04/23 19:16:47| Successful DNS name lookup tests...
2003/04/23 19:16:47| DNS Socket created at 0.0.0.0, port 23430, FD 5
2003/04/23 19:16:47| Adding nameserver 193.175.66.33 from squid.conf
2003/04/23 19:16:47| Adding nameserver 193.175.70.246 from squid.conf
2003/04/23 19:16:47| User-Agent logging is disabled.
2003/04/23 19:16:47| Referer logging is disabled.
2003/04/23 19:16:55| Starting Squid Cache version 2.5.STABLE2-20030422 for ...
2003/04/23 19:16:55| Process ID 4754
2003/04/23 19:16:55| With 4096 file descriptors available
...

core-19:16:54:
==============

Reading symbols from /usr/lib/nss_dns.so.1...done.
Loaded symbols for /usr/lib/nss_dns.so.1
#0 0x000ce9f4 in free (mem=0x9bc358) at dlmalloc.c:2359
2359 unlink(p, bck, fwd);
(gdb) bt
#0 0x000ce9f4 in free (mem=0x9bc358) at dlmalloc.c:2359
#1 0x000d8a74 in rfc1035RRDestroy (rr=0x9bc2c0, n=0) at rfc1035.c:487
#2 0x00046638 in idnsGrokReply (buf=0x110400 "", sz=107) at dns_internal.c:485
#3 0x00046ad0 in idnsRead (fd=5, data=0x0) at dns_internal.c:556
#4 0x000412b4 in comm_check_incoming_poll_handlers (nfds=1, fds=0xffbe7c38) at comm_select.c:238
#5 0x00042bf0 in comm_poll_dns_incoming () at comm_select.c:927
#6 0x00041980 in comm_poll (msec=0) at comm_select.c:340
#7 0x0007a1f0 in main (argc=2, argv=0xffbefe94) at main.c:735

core-19:16:57:
==============

...
Reading symbols from /usr/lib/nss_dns.so.1...done.
Loaded symbols for /usr/lib/nss_dns.so.1
#0 0x000ce9f4 in free (mem=0x418cb0) at dlmalloc.c:2359
2359 unlink(p, bck, fwd);
(gdb) bt
#0 0x000ce9f4 in free (mem=0x418cb0) at dlmalloc.c:2359
#1 0x000d8a74 in rfc1035RRDestroy (rr=0x418c18, n=0) at rfc1035.c:487
#2 0x00046638 in idnsGrokReply (buf=0x110400 "", sz=107) at dns_internal.c:485
#3 0x00046ad0 in idnsRead (fd=5, data=0x0) at dns_internal.c:556
#4 0x000412b4 in comm_check_incoming_poll_handlers (nfds=1, fds=0xffbe7c38) at comm_select.c:238
#5 0x00042bf0 in comm_poll_dns_incoming () at comm_select.c:927
#6 0x00042a70 in comm_poll (msec=0) at comm_select.c:497
#7 0x0007a1f0 in main (argc=2, argv=0xffbefe94) at main.c:735

core-19:17:01:
==============

...
Reading symbols from /usr/lib/nss_dns.so.1...done.
Loaded symbols for /usr/lib/nss_dns.so.1
#0 0x000ce9f4 in free (mem=0x526630) at dlmalloc.c:2359
2359 unlink(p, bck, fwd);
(gdb) bt
#0 0x000ce9f4 in free (mem=0x526630) at dlmalloc.c:2359
#1 0x000d8a74 in rfc1035RRDestroy (rr=0x526598, n=0) at rfc1035.c:487
#2 0x00046638 in idnsGrokReply (buf=0x110400 "", sz=107) at dns_internal.c:485
#3 0x00046ad0 in idnsRead (fd=5, data=0x0) at dns_internal.c:556
#4 0x000412b4 in comm_check_incoming_poll_handlers (nfds=1, fds=0xffbe7c38) at comm_select.c:238
#5 0x00042bf0 in comm_poll_dns_incoming () at comm_select.c:927
#6 0x00042a70 in comm_poll (msec=0) at comm_select.c:497
#7 0x0007a1f0 in main (argc=2, argv=0xffbefe94) at main.c:735

-- 
Ralf Hildebrandt (Im Auftrag des Referat V a)   Ralf.Hildebrandt@charite.de
Charite Campus Mitte                            Tel.  +49 (0)30-450 570-155
Referat V a - Kommunikationsnetze -             Fax.  +49 (0)30-450 570-916
AIM: ralfpostfix