Re: [squid-users] problems with ssl_crtd

On Sep 19, 2012, at 9:03 AM, Linos <info_at_linos.es> wrote:

> On 19/09/12 15:30, Guy Helmer wrote:
>> On Sep 19, 2012, at 5:44 AM, Linos <info_at_linos.es> wrote:
>>
>>> Hi,
>>> i have been using Squid squid-3.2.0.17-20120527-r11561 in an Ubuntu Server
>>> 12.04 some time with ssl-bump without problems for a year, the ca cert expired
>>> some days ago and with the new ca cert i installed squid 3.2.1 stable.
>>>
>>> Now the proxy exists every time 10 or more users use https at the same time,
>>> it's pretty strange, i have tried to downgrade to the old squid version but i
>>> can't get the proxy to be stable no matter if using new or old version, i have
>>> tried to recreate other cert just in case, same problem, i recreated too
>>> squid_ssl_db and cache_dir, no matter what i do it keeps crashing, the cache log
>>> read as this:
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> 2012/09/19 11:58:00| Starting Squid Cache version 3.2.1 for x86_64-pc-linux-gnu...
>>> 2012/09/19 11:58:00| Process ID 30077
>>> 2012/09/19 11:58:00| Process Roles: master worker
>>> 2012/09/19 11:58:00| With 65535 file descriptors available
>>> 2012/09/19 11:58:00| Initializing IP Cache...
>>> 2012/09/19 11:58:00| DNS Socket created at [::], FD 4
>>> 2012/09/19 11:58:00| DNS Socket created at 0.0.0.0, FD 5
>>> 2012/09/19 11:58:00| Adding nameserver 80.58.61.250 from squid.conf
>>> 2012/09/19 11:58:00| Adding nameserver 8.8.8.8 from squid.conf
>>> 2012/09/19 11:58:00| helperOpenServers: Starting 5/10 'ssl_crtd' processes
>>> 2012/09/19 11:58:00| helperOpenServers: Starting 5/20 'request_body_max_size.sh'
>>> processes
>>> 2012/09/19 11:58:00| Logfile: opening log daemon:/var/log/squid3/access.log
>>> 2012/09/19 11:58:00| Logfile Daemon: opening log /var/log/squid3/access.log
>>> 2012/09/19 11:58:00| Unlinkd pipe opened on FD 31
>>> 2012/09/19 11:58:00| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
>>> 2012/09/19 11:58:00| Store logging disabled
>>> 2012/09/19 11:58:00| Swap maxSize 15360000 + 262144 KB, estimated 312442 objects
>>> 2012/09/19 11:58:00| Target number of buckets: 15622
>>> 2012/09/19 11:58:00| Using 16384 Store buckets
>>> 2012/09/19 11:58:00| Max Mem size: 262144 KB
>>> 2012/09/19 11:58:00| Max Swap size: 15360000 KB
>>> 2012/09/19 11:58:00| Rebuilding storage in /mnt/squid/squid3 (clean log)
>>> 2012/09/19 11:58:00| Using Least Load store dir selection
>>> 2012/09/19 11:58:00| Set Current Directory to /mnt/squid/squid3
>>> 2012/09/19 11:58:00| Loaded Icons.
>>> 2012/09/19 11:58:00| HTCP Disabled.
>>> 2012/09/19 11:58:00| Squid plugin modules loaded: 0
>>> 2012/09/19 11:58:00| Adaptation support is off.
>>> 2012/09/19 11:58:00| Accepting NAT intercepted HTTP Socket connections at
>>> local=0.0.0.0:3128 remote=[::] FD 36 flags=41
>>> 2012/09/19 11:58:00| Accepting SSL bumped HTTP Socket connections at
>>> local=[::]:3150 remote=[::] FD 37 flags=9
>>> 2012/09/19 11:58:00| Store rebuilding is 16.55% complete
>>> 2012/09/19 11:58:00| Done reading /mnt/squid/squid3 swaplog (24167 entries)
>>> 2012/09/19 11:58:00| Finished rebuilding storage from disk.
>>> 2012/09/19 11:58:00| 24167 Entries scanned
>>> 2012/09/19 11:58:00| 0 Invalid entries.
>>> 2012/09/19 11:58:00| 0 With invalid flags.
>>> 2012/09/19 11:58:00| 24167 Objects loaded.
>>> 2012/09/19 11:58:00| 0 Objects expired.
>>> 2012/09/19 11:58:00| 0 Objects cancelled.
>>> 2012/09/19 11:58:00| 0 Duplicate URLs purged.
>>> 2012/09/19 11:58:00| 0 Swapfile clashes avoided.
>>> 2012/09/19 11:58:00| Took 0.12 seconds (204025.29 objects/sec).
>>> 2012/09/19 11:58:00| Beginning Validation Procedure
>>> 2012/09/19 11:58:00| Completed Validation Procedure
>>> 2012/09/19 11:58:00| Validated 24167 Entries
>>> 2012/09/19 11:58:00| store_swap_size = 732468.00 KB
>>> 2012/09/19 11:58:01| storeLateRelease: released 0 objects
>>> (ssl_crtd): Cannot create ssl certificate or private key.
>>> 2012/09/19 12:03:20| WARNING: ssl_crtd #1 exited
>>> 2012/09/19 12:03:20| Too few ssl_crtd processes are running (need 1/10)
>>> 2012/09/19 12:03:20| Starting new helpers
>>> 2012/09/19 12:03:20| helperOpenServers: Starting 1/10 'ssl_crtd' processes
>>> 2012/09/19 12:03:20| client_side.cc(3477) sslCrtdHandleReply: "ssl_crtd" helper
>>> return <NULL> reply
>>> (ssl_crtd): Cannot create ssl certificate or private key.
>>> 2012/09/19 12:03:20| WARNING: ssl_crtd #2 exited
>>> 2012/09/19 12:03:20| Too few ssl_crtd processes are running (need 1/10)
>>> 2012/09/19 12:03:20| Closing HTTP port 0.0.0.0:3128
>>> 2012/09/19 12:03:20| Closing HTTP port [::]:3150
>>> 2012/09/19 12:03:20| storeDirWriteCleanLogs: Starting...
>>> 2012/09/19 12:03:20| Finished. Wrote 24195 entries.
>>> 2012/09/19 12:03:20| Took 0.02 seconds (1321120.45 entries/sec).
>>> FATAL: The ssl_crtd helpers are crashing too rapidly, need help!
>>>
>>> Squid Cache (Version 3.2.1): Terminated abnormally.
>>> CPU Usage: 1.896 seconds = 0.740 user + 1.156 sys
>>> Maximum Resident Size: 144640 KB
>>> Page faults with physical i/o: 0
>>> Memory usage for squid via mallinfo():
>>> total space in arena: 18900 KB
>>> Ordinary blocks: 18674 KB 54 blks
>>> Small blocks: 0 KB 1 blks
>>> Holding blocks: 37552 KB 9 blks
>>> Free Small blocks: 0 KB
>>> Free Ordinary blocks: 225 KB
>>> Total in use: 56226 KB 297%
>>> Total free: 225 KB 1%
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> configure and kernel versions:
>>>
>>> kernel: 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012 x86_64
>>> x86_64 x86_64 GNU/Linux
>>>
>>> Squid Cache: Version 3.2.1
>>> configure options: '--build=x86_64-linux-gnu' '--prefix=/usr'
>>> '--includedir=${prefix}/include' '--mandir=${prefix}/share/man'
>>> '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var'
>>> '--libexecdir=${prefix}/lib/squid3' '--srcdir=.' '--disable-maintainer-mode'
>>> '--disable-dependency-tracking' '--disable-silent-rules'
>>> '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3'
>>> '--mandir=/usr/share/man' '--with-cppunit-basedir=/usr' '--enable-inline'
>>> '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd'
>>> '--enable-removal-policies=lru,heap' '--enable-delay-pools'
>>> '--enable-cache-digests' '--enable-underscores' '--enable-icap-client'
>>> '--enable-follow-x-forwarded-for' '--enable-auth-basic' '--enable-auth-digest'
>>> '--enable-auth-ntlm' '--enable-auth-negotiate'
>>> '--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SASL,SMB,YP,DB,POP3,getpwnam,squid_radius_auth,multi-domain-NTLM'
>>> '--enable-ntlm-auth-helpers=smb_lm,'
>>> '--enable-digest-auth-helpers=ldap,password'
>>> '--enable-negotiate-auth-helpers=squid_kerb_auth'
>>> '--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group'
>>> '--enable-arp-acl' '--enable-esi' '--enable-zph-qos' '--enable-wccpv2'
>>> '--disable-translation' '--with-logdir=/var/log/squid3'
>>> '--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536'
>>> '--with-large-files' '--with-default-user=proxy' '--enable-ssl'
>>> '--enable-ssl-crtd' '--disable-epoll' '--enable-linux-netfilter'
>>> 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector
>>> --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security'
>>> 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now'
>>> 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector
>>> --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security'
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> i get this in dmesg:
>>>
>>> [3433312.743391] init: squid3 main process (29801) terminated with status 1
>>> [3433417.181960] ssl_crtd[29848]: segfault at 0 ip 00007f2ee2494ac5 sp
>>> 00007fff51dbd260 error 4 in ssl_crtd[7f2ee248b000+10000]
>>> [3433417.328898] init: squid3 main process (29847) terminated with status 1
>>> [3433557.765878] init: squid3 main process (29877) killed by KILL signal
>>> [3433944.030777] init: squid3 main process (30077) terminated with status 1
>>>
>>>
>>> ------------------------------------------------------------------------------
>>>
>>
>> OK, ssl_crtd is dying with a segfault when it is handling a request.
>>
>> What is in the squid_ssl_db/size file? If it is empty (0 bytes), that would cause a crash in older (pre-3.2.1) squid.
>>
>> Otherwise, is there an ssl_crtd core file in the squid log directory? A stack trace from it might help diagnose the problem.
>>
>> Guy
>>
>
> Thanks for reply.
>
> i checked the squid_ssl_db/size because i found the empty file problem searching
> for my own problem in the mailing list, it's ok in my host, the file have the
> content "139264" right now.
>
> I can't found the core file, do i need to do something for it to generate? maybe
> a configure script option or squid.conf change to activate it?
>
> Regards,
> Miguel Angel.

I have

coredump_dir /var/log/squid

to get coredumps in my /var/log/squid directory. Now that I think about it, I don't remember if this works for ssl_crtd though -- seems like I have had to start "gdb ssl_crtd" and then attach to one of the ssl_crtd processes, then generate HTTPS traffic to trigger the request to ssl_crtd and get a backtrace when ssl_crtd gets the segfault signal…

Guy
Received on Wed Sep 19 2012 - 14:46:31 MDT