Re: [squid-users] squid consuming near all (95+ %) CPU, it is normal?

Amos Jeffries wrote:
> On 20.11.2012 09:33, Frantisek Hanzlik wrote:
>> Amos Jeffries wrote:
>>> On 12.11.2012 03:25, Frantisek Hanzlik wrote:
>>>> With this squid configuration:
...<snip>
>> And then I found in /var/log/messages these lines
>> (/var/log/squid/cache.log contained nothing suspicious(*) ):
>>
>> Nov 19 16:53:40 172 squid[2834]: Squid Parent: (squid-1) process 32702
>> exited due to signal 6 with status 0
>
> Ouch. That needs to be found and fixed.
> Are you able to assist with that by running Squid under a debugger and
> getting a backtrace?
> (http://wiki.squid-cache.org/SquidFaq/BugReporting#Using_gdb_debugger_on_a_live_proxy_.28with_minimal_downtime.29)
>
>
> <snip repeats>
... <snip repeats>

Within several hours I have six core dumps:
# LANG=C ls -l
-rw------- 1 squid squid 31424512 Nov 21 06:03 core.15363
-rw------- 1 squid squid 75161600 Nov 21 06:54 core.17776
-rw------- 1 squid squid 19992576 Nov 21 06:54 core.18266
-rw------- 1 squid squid 91512832 Nov 21 07:27 core.18272
-rw------- 1 squid squid 144785408 Nov 21 07:46 core.18579
-rw------- 1 squid squid 361881600 Nov 21 09:14 core.18754
-rw------- 1 squid squid 307638272 Nov 21 11:46 core.19564

Backraces (without core.15363, it was in previous mail) are:

# for i in core.17776 core.18266 core.18272 core.18579 core.18754 core.19564; do echo -e "\n#gdb ./$i :"; gdb -batch -x gdb.commands ./$i; done

#gdb ./core.17776 :
[New Thread 17776]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/c4/1c574f31a203492b9389c783adad6ff1989915
[Thread debugging using libthread_db enabled]
Core was generated by `(squid-1) -f /etc/squid/squid.conf'.
Program terminated with signal 6, Aborted.
#0 0x00881424 in __kernel_vsyscall ()
#0 0x00881424 in __kernel_vsyscall ()
#1 0x0055f2f1 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0x00560d5e in abort () at abort.c:92
#3 0x00557c08 in __assert_fail (assertion=0x83cf443 "false", file=0x83ff936 "Address.cc", line=689, function=0x83ffca0 "void Ip::Address::GetAddrInfo(addrinfo*&, int) const") at assert.c:81
#4 0x082e1b60 in Ip::Address::GetAddrInfo (this=0xadea304, dst=@0xbfd5372c, force=0) at Address.cc:689
#5 0x082da72b in comm_openex (sock_type=1, proto=6, addr=..., flags=1, tos=0 '\000', nfmark=0, note=0xcbc5fd0 "[2a02:598:1::9999]") at comm.cc:525
#6 0x0832fd65 in Comm::ConnOpener::start (this=0xae04840) at ConnOpener.cc:164
#7 0x082d13be in NullaryMemFunT<AsyncJob>::doDial (this=0xcb01e04) at ../../src/base/AsyncJobCalls.h:101
#8 0x082d15d6 in JobDialer<AsyncJob>::dial (this=0xcb01e04, call=...) at ../../src/base/AsyncJobCalls.h:175
#9 0x082d1775 in AsyncCallT<NullaryMemFunT<AsyncJob> >::fire (this=0xcb01de8) at ../../src/base/AsyncCall.h:142
#10 0x082ce34b in AsyncCall::make (this=0xcb01de8) at AsyncCall.cc:36
#11 0x082d196b in AsyncCallQueue::fireNext (this=0x92086f0) at AsyncCallQueue.cc:54
#12 0x082d1ae0 in AsyncCallQueue::fire (this=0x92086f0) at AsyncCallQueue.cc:40
#13 0x0818edc0 in EventLoop::dispatchCalls (this=0xbfd53b24) at EventLoop.cc:154
#14 0x0818eff7 in EventLoop::runOnce (this=0xbfd53b24) at EventLoop.cc:131
#15 0x0818f0e0 in EventLoop::run (this=0xbfd53b24) at EventLoop.cc:95
#16 0x081ff650 in SquidMain (argc=3, argv=0xbfd53c94) at main.cc:1500
#17 0x0820076f in SquidMainSafe (argc=3, argv=0xbfd53c94) at main.cc:1215
#18 main (argc=3, argv=0xbfd53c94) at main.cc:1207

#gdb ./core.18266 :
[New Thread 18266]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/c4/1c574f31a203492b9389c783adad6ff1989915
[Thread debugging using libthread_db enabled]
Core was generated by `(squid-1) -f /etc/squid/squid.conf'.
Program terminated with signal 6, Aborted.
#0 0x00665424 in __kernel_vsyscall ()
#0 0x00665424 in __kernel_vsyscall ()
#1 0x0092c2f1 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0x0092dd5e in abort () at abort.c:92
#3 0x00924c08 in __assert_fail (assertion=0x83cf443 "false", file=0x83ff936 "Address.cc", line=689, function=0x83ffca0 "void Ip::Address::GetAddrInfo(addrinfo*&, int) const") at assert.c:81
#4 0x082e1b60 in Ip::Address::GetAddrInfo (this=0xa62fdcc, dst=@0xbfbd4fcc, force=0) at Address.cc:689
#5 0x082da72b in comm_openex (sock_type=1, proto=6, addr=..., flags=1, tos=0 '\000', nfmark=0, note=0xa6316a8 "[2a02:598:1::9999]") at comm.cc:525
#6 0x0832fd65 in Comm::ConnOpener::start (this=0xa6348f0) at ConnOpener.cc:164
#7 0x082d13be in NullaryMemFunT<AsyncJob>::doDial (this=0xa634874) at ../../src/base/AsyncJobCalls.h:101
#8 0x082d15d6 in JobDialer<AsyncJob>::dial (this=0xa634874, call=...) at ../../src/base/AsyncJobCalls.h:175
#9 0x082d1775 in AsyncCallT<NullaryMemFunT<AsyncJob> >::fire (this=0xa634858) at ../../src/base/AsyncCall.h:142
#10 0x082ce34b in AsyncCall::make (this=0xa634858) at AsyncCall.cc:36
#11 0x082d196b in AsyncCallQueue::fireNext (this=0x9cd16f0) at AsyncCallQueue.cc:54
#12 0x082d1ae0 in AsyncCallQueue::fire (this=0x9cd16f0) at AsyncCallQueue.cc:40
#13 0x0818edc0 in EventLoop::dispatchCalls (this=0xbfbd53c4) at EventLoop.cc:154
#14 0x0818eff7 in EventLoop::runOnce (this=0xbfbd53c4) at EventLoop.cc:131
#15 0x0818f0e0 in EventLoop::run (this=0xbfbd53c4) at EventLoop.cc:95
#16 0x081ff650 in SquidMain (argc=3, argv=0xbfbd5534) at main.cc:1500
#17 0x0820076f in SquidMainSafe (argc=3, argv=0xbfbd5534) at main.cc:1215
#18 main (argc=3, argv=0xbfbd5534) at main.cc:1207

#gdb ./core.18272 :
[New Thread 18272]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/c4/1c574f31a203492b9389c783adad6ff1989915
[Thread debugging using libthread_db enabled]
Core was generated by `(squid-1) -f /etc/squid/squid.conf'.
Program terminated with signal 6, Aborted.
#0 0x00c01424 in __kernel_vsyscall ()
#0 0x00c01424 in __kernel_vsyscall ()
#1 0x0055f2f1 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0x00560d5e in abort () at abort.c:92
#3 0x00557c08 in __assert_fail (assertion=0x83cf443 "false", file=0x83ff936 "Address.cc", line=689, function=0x83ffca0 "void Ip::Address::GetAddrInfo(addrinfo*&, int) const") at assert.c:81
#4 0x082e1b60 in Ip::Address::GetAddrInfo (this=0xdb3db4c, dst=@0xbfe206fc, force=0) at Address.cc:689
#5 0x082da72b in comm_openex (sock_type=1, proto=6, addr=..., flags=1, tos=0 '\000', nfmark=0, note=0xec8e250 "[2a02:598:1::9999]") at comm.cc:525
#6 0x0832fd65 in Comm::ConnOpener::start (this=0xaba9620) at ConnOpener.cc:164
#7 0x082d13be in NullaryMemFunT<AsyncJob>::doDial (this=0xdeb5adc) at ../../src/base/AsyncJobCalls.h:101
#8 0x082d15d6 in JobDialer<AsyncJob>::dial (this=0xdeb5adc, call=...) at ../../src/base/AsyncJobCalls.h:175
#9 0x082d1775 in AsyncCallT<NullaryMemFunT<AsyncJob> >::fire (this=0xdeb5ac0) at ../../src/base/AsyncCall.h:142
#10 0x082ce34b in AsyncCall::make (this=0xdeb5ac0) at AsyncCall.cc:36
#11 0x082d196b in AsyncCallQueue::fireNext (this=0xa0d06f0) at AsyncCallQueue.cc:54
#12 0x082d1ae0 in AsyncCallQueue::fire (this=0xa0d06f0) at AsyncCallQueue.cc:40
#13 0x0818edc0 in EventLoop::dispatchCalls (this=0xbfe20af4) at EventLoop.cc:154
#14 0x0818eff7 in EventLoop::runOnce (this=0xbfe20af4) at EventLoop.cc:131
#15 0x0818f0e0 in EventLoop::run (this=0xbfe20af4) at EventLoop.cc:95
#16 0x081ff650 in SquidMain (argc=3, argv=0xbfe20c64) at main.cc:1500
#17 0x0820076f in SquidMainSafe (argc=3, argv=0xbfe20c64) at main.cc:1215
#18 main (argc=3, argv=0xbfe20c64) at main.cc:1207

#gdb ./core.18579 :
[New Thread 18579]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/c4/1c574f31a203492b9389c783adad6ff1989915
[Thread debugging using libthread_db enabled]
Core was generated by `(squid-1) -f /etc/squid/squid.conf'.
Program terminated with signal 6, Aborted.
#0 0x00d09424 in __kernel_vsyscall ()
#0 0x00d09424 in __kernel_vsyscall ()
#1 0x0055f2f1 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0x00560d5e in abort () at abort.c:92
#3 0x00557c08 in __assert_fail (assertion=0x83cf443 "false", file=0x83ff936 "Address.cc", line=689, function=0x83ffca0 "void Ip::Address::GetAddrInfo(addrinfo*&, int) const") at assert.c:81
#4 0x082e1b60 in Ip::Address::GetAddrInfo (this=0xcf9d544, dst=@0xbfa0d25c, force=0) at Address.cc:689
#5 0x082da72b in comm_openex (sock_type=1, proto=6, addr=..., flags=1, tos=0 '\000', nfmark=0, note=0x11f8b9f8 "[2a02:598:1::9999]") at comm.cc:525
#6 0x0832fd65 in Comm::ConnOpener::start (this=0xb04fb30) at ConnOpener.cc:164
#7 0x082d13be in NullaryMemFunT<AsyncJob>::doDial (this=0x11f8ae74) at ../../src/base/AsyncJobCalls.h:101
#8 0x082d15d6 in JobDialer<AsyncJob>::dial (this=0x11f8ae74, call=...) at ../../src/base/AsyncJobCalls.h:175
#9 0x082d1775 in AsyncCallT<NullaryMemFunT<AsyncJob> >::fire (this=0x11f8ae58) at ../../src/base/AsyncCall.h:142
#10 0x082ce34b in AsyncCall::make (this=0x11f8ae58) at AsyncCall.cc:36
#11 0x082d196b in AsyncCallQueue::fireNext (this=0xa28a6f0) at AsyncCallQueue.cc:54
#12 0x082d1ae0 in AsyncCallQueue::fire (this=0xa28a6f0) at AsyncCallQueue.cc:40
#13 0x0818edc0 in EventLoop::dispatchCalls (this=0xbfa0d654) at EventLoop.cc:154
#14 0x0818eff7 in EventLoop::runOnce (this=0xbfa0d654) at EventLoop.cc:131
#15 0x0818f0e0 in EventLoop::run (this=0xbfa0d654) at EventLoop.cc:95
#16 0x081ff650 in SquidMain (argc=3, argv=0xbfa0d7c4) at main.cc:1500
#17 0x0820076f in SquidMainSafe (argc=3, argv=0xbfa0d7c4) at main.cc:1215
#18 main (argc=3, argv=0xbfa0d7c4) at main.cc:1207

#gdb ./core.18754 :
[New Thread 18754]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/c4/1c574f31a203492b9389c783adad6ff1989915
[Thread debugging using libthread_db enabled]
Core was generated by `(squid-1) -f /etc/squid/squid.conf'.
Program terminated with signal 6, Aborted.
#0 0x00e91424 in __kernel_vsyscall ()
#0 0x00e91424 in __kernel_vsyscall ()
#1 0x0055f2f1 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0x00560d5e in abort () at abort.c:92
#3 0x00557c08 in __assert_fail (assertion=0x83cf443 "false", file=0x83ff936 "Address.cc", line=689, function=0x83ffca0 "void Ip::Address::GetAddrInfo(addrinfo*&, int) const") at assert.c:81
#4 0x082e1b60 in Ip::Address::GetAddrInfo (this=0x1e63c0ac, dst=@0xbfb7435c, force=0) at Address.cc:689
#5 0x082da72b in comm_openex (sock_type=1, proto=6, addr=..., flags=1, tos=0 '\000', nfmark=0, note=0x1cf28b28 "[2a02:598:1::9999]") at comm.cc:525
#6 0x0832fd65 in Comm::ConnOpener::start (this=0xaf93c88) at ConnOpener.cc:164
#7 0x082d13be in NullaryMemFunT<AsyncJob>::doDial (this=0x1e4ee0a4) at ../../src/base/AsyncJobCalls.h:101
#8 0x082d15d6 in JobDialer<AsyncJob>::dial (this=0x1e4ee0a4, call=...) at ../../src/base/AsyncJobCalls.h:175
#9 0x082d1775 in AsyncCallT<NullaryMemFunT<AsyncJob> >::fire (this=0x1e4ee088) at ../../src/base/AsyncCall.h:142
#10 0x082ce34b in AsyncCall::make (this=0x1e4ee088) at AsyncCall.cc:36
#11 0x082d196b in AsyncCallQueue::fireNext (this=0x9d866f0) at AsyncCallQueue.cc:54
#12 0x082d1ae0 in AsyncCallQueue::fire (this=0x9d866f0) at AsyncCallQueue.cc:40
#13 0x0818edc0 in EventLoop::dispatchCalls (this=0xbfb74754) at EventLoop.cc:154
#14 0x0818eff7 in EventLoop::runOnce (this=0xbfb74754) at EventLoop.cc:131
#15 0x0818f0e0 in EventLoop::run (this=0xbfb74754) at EventLoop.cc:95
#16 0x081ff650 in SquidMain (argc=3, argv=0xbfb748c4) at main.cc:1500
#17 0x0820076f in SquidMainSafe (argc=3, argv=0xbfb748c4) at main.cc:1215
#18 main (argc=3, argv=0xbfb748c4) at main.cc:1207

#gdb ./core.19564 :
[New Thread 19564]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/c4/1c574f31a203492b9389c783adad6ff1989915
[Thread debugging using libthread_db enabled]
Core was generated by `(squid-1) -f /etc/squid/squid.conf'.
Program terminated with signal 6, Aborted.
#0 0x0032b424 in __kernel_vsyscall ()
#0 0x0032b424 in __kernel_vsyscall ()
#1 0x0055f2f1 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0x00560d5e in abort () at abort.c:92
#3 0x00557c08 in __assert_fail (assertion=0x83cf443 "false", file=0x83ff936 "Address.cc", line=689, function=0x83ffca0 "void Ip::Address::GetAddrInfo(addrinfo*&, int) const") at assert.c:81
#4 0x082e1b60 in Ip::Address::GetAddrInfo (this=0x177e1f34, dst=@0xbf9db7dc, force=0) at Address.cc:689
#5 0x082da72b in comm_openex (sock_type=1, proto=6, addr=..., flags=1, tos=0 '\000', nfmark=0, note=0x1b023308 "[2a02:598:1::9999]") at comm.cc:525
#6 0x0832fd65 in Comm::ConnOpener::start (this=0xe7fdf08) at ConnOpener.cc:164
#7 0x082d13be in NullaryMemFunT<AsyncJob>::doDial (this=0x1b01e594) at ../../src/base/AsyncJobCalls.h:101
#8 0x082d15d6 in JobDialer<AsyncJob>::dial (this=0x1b01e594, call=...) at ../../src/base/AsyncJobCalls.h:175
#9 0x082d1775 in AsyncCallT<NullaryMemFunT<AsyncJob> >::fire (this=0x1b01e578) at ../../src/base/AsyncCall.h:142
#10 0x082ce34b in AsyncCall::make (this=0x1b01e578) at AsyncCall.cc:36
#11 0x082d196b in AsyncCallQueue::fireNext (this=0x95966f0) at AsyncCallQueue.cc:54
#12 0x082d1ae0 in AsyncCallQueue::fire (this=0x95966f0) at AsyncCallQueue.cc:40
#13 0x0818edc0 in EventLoop::dispatchCalls (this=0xbf9dbbd4) at EventLoop.cc:154
#14 0x0818eff7 in EventLoop::runOnce (this=0xbf9dbbd4) at EventLoop.cc:131
#15 0x0818f0e0 in EventLoop::run (this=0xbf9dbbd4) at EventLoop.cc:95
#16 0x081ff650 in SquidMain (argc=3, argv=0xbf9dbd44) at main.cc:1500
#17 0x0820076f in SquidMainSafe (argc=3, argv=0xbf9dbd44) at main.cc:1215
#18 main (argc=3, argv=0xbf9dbd44) at main.cc:1207

Although core dump image are from 20MB to 362MB size, all backtraces are similar.
Can I, please, do anything further?
Received on Wed Nov 21 2012 - 11:33:10 MST