Re: [squid-users] Squid crashes, core dumps and restarts automatically about once a day

2009/8/19 Thijs Stuurman <Thijs.Stuurman_at_nxs.nl>:
> Squid-users,
>
> We have plenty of servers running all kinds of versions of Squid but one of the most stable version we run on a fast server, keeps crashing for an unknown reason. We hope someone on this mailing list can shed some light on the issue.
> About once a day (not every day) Squid crashes, makes a core dump and restarts. Everything else on the server experiences no problems, nothing odd in the systems log files and the memory came out just fine in a memtest. We didn't even notice it until we saw the diskspace usage grow with a few spikes. The core dumps were about 5 gigabytes each, the latest just 500 megabytes but they all show the same backtrace.
>
> Server information:
>
> Squid version: 2.7.STABLE6-2
> Kernel: 2.6.24-24-server
> Issue: Ubuntu 8.04.3 LTS
> Memory: 12G
> CPU: single QC Intel(R) Xeon(R) CPU L5506 @ 2.13GHz
>
>
> Regular messages in the cache log (garbage has been edited out in all of the following pieces):
>
> """
> httpReadReply: Excess data from "GET http://...."
> """
> """
> storeAddVaryReadOld: New index aborted at 8188 (836)
> """
> """
> storeAufsOpenDone: (2) No such file or directory
> """
> """
> VaryData: accept-encoding="gzip,%20deflate", user-agent="Mozilla%2F4.0%20(compatible . %20CLR%203.0.30618)"
> """
>
> Cache log right before the crash/restart:
>
> """
> storeLocateVaryRead: Unexpected data ' . ' . Starting Squid Cache version 2.7.STABLE6 for amd64-debian-linux-gnu...
> """
>
>
> gdb backtrace on the core dump:
>
> """
> (gdb) backtrace
> #0  0x00007fdec9826095 in raise () from /lib/libc.so.6
> #1  0x00007fdec9827af0 in abort () from /lib/libc.so.6
> #2  0x00007fdec981f2df in __assert_fail () from /lib/libc.so.6
> #3  0x00000000004ac124 in xstrndup (
>     s=0x16377744 "\nÙ\206A/\210\216p4< . \003\026ô", n=0) at util.c:616
> #4  0x0000000000477ae4 in storeLocateVaryRead (data=0x1a08fc48, buf=0x1e6ebe00 "гj\036", size=4050) at store.c:878
> #5  0x0000000000478274 in storeClientCallback (sc=0x12f7d428, sz=4050) at store_client.c:146
> #6  0x000000000048fd6b in storeAufsReadDone (fd=<value optimized out>, my_data=0x140c8e58,
>     buf=0x12f15ef0 "\215F\004\"\t\234G\b . 203ôõ\023 2º", len=<value optimized out>, errflag=0) at aufs/store_io_aufs.c:400
> #7  0x0000000000491f4a in aioCheckCallbacks (SD=<value optimized out>) at aufs/async_io.c:319
> #8  0x000000000047a7d8 in storeDirCallback () at store_dir.c:511
> #9  0x000000000042a7ac in comm_select (msec=307) at comm_generic.c:377
> #10 0x00000000004573e6 in main (argc=<value optimized out>, argv=0x7fffd28695d8) at main.c:884
> """
>
>
> Thijs Stuurman
> System Administrator
>
> Nxs Internet BV
> Kabelweg 37
> 1014 BA  AMSTERDAM
>
> Tel.:  +31 (0) 20 46 88 071
> Fax:  +31 (0) 20 46 88 236
> Mail: beheer.linux_at_nxs.nl
>
>
>

Squid usually prints a FATAL error after a crash. Look for it in
cache.log and copy that lines if you don't undertand them.

-- 
Diego Woitasen
XTECH