[squid-users] Squid crashes, core dumps and restarts automatically about once a day

Squid-users,

We have plenty of servers running all kinds of versions of Squid but one of the most stable version we run on a fast server, keeps crashing for an unknown reason. We hope someone on this mailing list can shed some light on the issue.
About once a day (not every day) Squid crashes, makes a core dump and restarts. Everything else on the server experiences no problems, nothing odd in the systems log files and the memory came out just fine in a memtest. We didn't even notice it until we saw the diskspace usage grow with a few spikes. The core dumps were about 5 gigabytes each, the latest just 500 megabytes but they all show the same backtrace.

Server information:

Squid version: 2.7.STABLE6-2
Kernel: 2.6.24-24-server
Issue: Ubuntu 8.04.3 LTS
Memory: 12G
CPU: single QC Intel(R) Xeon(R) CPU L5506 @ 2.13GHz

Regular messages in the cache log (garbage has been edited out in all of the following pieces):

"""
httpReadReply: Excess data from "GET http://...."
"""
"""
storeAddVaryReadOld: New index aborted at 8188 (836)
"""
"""
storeAufsOpenDone: (2) No such file or directory
"""
"""
VaryData: accept-encoding="gzip,%20deflate", user-agent="Mozilla%2F4.0%20(compatible . %20CLR%203.0.30618)"
"""

Cache log right before the crash/restart:

"""
storeLocateVaryRead: Unexpected data ' . ' . Starting Squid Cache version 2.7.STABLE6 for amd64-debian-linux-gnu...
"""

gdb backtrace on the core dump:

"""
(gdb) backtrace
#0  0x00007fdec9826095 in raise () from /lib/libc.so.6
#1  0x00007fdec9827af0 in abort () from /lib/libc.so.6
#2  0x00007fdec981f2df in __assert_fail () from /lib/libc.so.6
#3  0x00000000004ac124 in xstrndup (
    s=0x16377744 "\nÙ\206A/\210\216p4< . \003\026ô", n=0) at util.c:616
#4  0x0000000000477ae4 in storeLocateVaryRead (data=0x1a08fc48, buf=0x1e6ebe00 "гj\036", size=4050) at store.c:878
#5  0x0000000000478274 in storeClientCallback (sc=0x12f7d428, sz=4050) at store_client.c:146
#6  0x000000000048fd6b in storeAufsReadDone (fd=<value optimized out>, my_data=0x140c8e58,
    buf=0x12f15ef0 "\215F\004\"\t\234G\b . 203ôõ\023 2º", len=<value optimized out>, errflag=0) at aufs/store_io_aufs.c:400
#7  0x0000000000491f4a in aioCheckCallbacks (SD=<value optimized out>) at aufs/async_io.c:319
#8  0x000000000047a7d8 in storeDirCallback () at store_dir.c:511
#9  0x000000000042a7ac in comm_select (msec=307) at comm_generic.c:377
#10 0x00000000004573e6 in main (argc=<value optimized out>, argv=0x7fffd28695d8) at main.c:884
"""

Thijs Stuurman
System Administrator

Nxs Internet BV
Kabelweg 37
1014 BA  AMSTERDAM

Tel.:  +31 (0) 20 46 88 071
Fax:  +31 (0) 20 46 88 236
Mail: beheer.linux_at_nxs.nl
Received on Wed Aug 19 2009 - 08:54:42 MDT