Re: [squid-users] xmalloc failure, yet FAQ items check out ok

On Mon, 9 Jun 2003 adam-s@pacbell.net wrote:

> Hello,
>
> I continue to have a problem where the squid process crashes but
> recovers on it's own. The error is FAQ 8.7's "xmalloc unable to
> allocate..." error. It started back when system only had 300MB of
> swap. So I recreated a new filesystem swap of 1.5GB. At that time I
> also added an extra cache_dir (on separate disk). The system is a Sun
> Ultra 60 running Solaris 8 with 1GB RAM. Squid is 2.5STABLE2.
>
> The effect of the changes I made (larger swap and more cache_dirs) was
> that the squid process is staying less than 300MB when before (with
> one cache_dir and a small swap) the squid process got as large as
> 600+MB. I am assuming that this is because now it can write that
> extra memory stuff out to swap, but if that is not the case please
> enlighten me.
>
> Anyhow, once swap is eliminated as a problem, then FAQ 8.7 says it
> must be a data segment size problem. However "ulimit -aHd" shows
> unlimited and I checked with Sun and they said there is no per process
> limit in the kernel, process limits are only a shell thing. It's not
> like filedescriptors which can be bumped up in the kernel. For the
> kernel, the data segment size is (essentially) unlimited and is a
> shell thing, not a kernel thing. Furthermore, prior to my changes,
> squid's process size was regularly larger than 600MB and I haven't
> made any kernel type changes.
>
> So have I mistweaked something in the squid.conf file? I haven't
> changed the system (e.g. in the /etc/system file), just added more
> filesystem swap. Below are my squid compile options, the cache.log
> error, an extract of my squid.conf file, and before/after's of top and
> ps.
>
> Side question: for Sun Solaris 8, is dlmalloc the recommended malloc
> to compile against? I've read that I should avoid Sun's own malloc
> (though that may only have applied to x86) but I am wondering if the
> problem is the dlmalloc library interacting with Solaris...

I had trouble with my squid process growing out of control until I
recompiled with --emable-dlmalloc. Since doing so the squid process
behaves memory wise.

However, there is a bug in Squid 2.5S2 (and earlier) where some sites will
cause squid running on Solaris to crash with either a segmentation, bus
error or 'xmalloc: Unable to allocate xxxx bytes' messages. At least that
is what I was getting. Be sure to either upgrade to 2.5S3 or apply this
patch:

http://www.squid-cache.org/Versions/v2/2.5/bugs/#squid-2.5.STABLE2-dns_root_label

If you read through the patch description there is a link that will cause
Squid to crash on Solaris (for testing).

-Mike

>
> thanks in advance for any corrections or advice.
>
> thanks,
>
> Adam
>
> Squid Cache: Version 2.5.STABLE2
> configure options: --enable-dlmalloc --enable-async-io
> --enable-storeio=aufs,diskd,ufs --enable-removal-policies=heap,lru
> --enable-delay-pools --disable-icmp
> --enable-cachemgr-hostname=aocproxy --enable-snmp
> --disable-ident-lookups --with-pthreads
>
> From the cache.log file:
> [ usual lines like comm_accept and http_accept deleted ]
> 2003/06/05 10:47:25| httpAccept: FD 16: accept failure: (130) Software
> caused connection abort
> FATAL: xmalloc: Unable to allocate 24576 bytes!
>
> Squid Cache (Version 2.5.STABLE2): Terminated abnormally.
> CPU Usage: 2627.620 seconds = 1421.140 user + 1206.480 sys
> Maximum Resident Size: 0 KB
> Page faults with physical i/o: 57391
> Memory usage for squid via mallinfo():
> total space in arena: 218032 KB
> Ordinary blocks: 214550 KB 17911 blks
> Small blocks: 0 KB 0 blks
> Holding blocks: 7128 KB 7 blks
> Free Small blocks: 0 KB
> Free Ordinary blocks: 3482 KB
> Total in use: 221678 KB 102%
> Total free: 3482 KB 2%
> 2003/06/05 10:47:39| storeDirWriteCleanLogs: Starting...
> [ snip ]
>
> Main items from squid.conf:
> cache_mem 64 MB
> cache_swap_low 95
> cache_swap_high 95
> maximum_object_size_in_memory 16 KB
> cache_replacement_policy heap GDSF
> memory_replacement_policy heap GDSF
> cache_dir aufs /cache 7256 16 256
> cache_dir aufs /cache3 6800 16 256
>
> TOP and PS before crash:
> top -b:
> Memory: 1024M real, 541M free, 237M swap in use, 1994M swap free
> PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
> 29604 root 36 43 0 216M 213M sleep 43:06 10.35% squid
> 29600 root 3 18 0 4216K 1464K sleep 0:00 0.00% squid
> 29605 squid 4 28 0 1256K 912K sleep 0:00 0.00% unlinkd
>
> /usr/ucb/ps -auxwww:
> USER PID %CPU %MEM SZ RSS TT S START TIME COMMAND
> squid 29604 10.5 21.8221040218360 ? O 21:15:25 43:06 (squid)
> -F -f /usr/local/squid/etc/squid.conf -s
> root 29600 0.0 0.2 4216 1464 ? S 21:15:25 0:00
> /usr/local/squid/sbin/squid -F -f /usr/local/squid/etc/squid.conf -s
> squid 29605 0.0 0.1 1256 912 ? S 21:15:25 0:00 (unlinkd)
>
> TOP and PS after crash:
> Memory: 1024M real, 640M free, 156M swap in use, 2076M swap free
> PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
> 3268 root 36 32 0 135M 132M sleep 1:13 15.43% squid
> 29600 root 4 38 0 4224K 1472K sleep 0:00 0.00% squid
> 3269 squid 4 38 0 1256K 912K sleep 0:00 0.00% unlinkd
>
> /usr/ucb/ps -auxwww:
> USER PID %CPU %MEM SZ RSS TT S START TIME COMMAND
> squid 3268 15.4 13.5138616135616 ? R 10:47:42 1:13 (squid)
> -F -f /usr/local/squid/etc/squid.conf -s
> squid 3269 0.0 0.1 1256 912 ? S 10:47:43 0:00 (unlinkd)
> root 29600 0.0 0.2 4224 1472 ? S 21:15:25 0:00
> /usr/local/squid/sbin/squid -F -f /usr/local/squid/etc/squid.conf -s
>
> from sysdef -i:
> * Tunable Parameters
> *
> 20815872 maximum memory allowed in buffer cache (bufhwm)
> 15882 maximum number of processes (v.v_proc)
> 99 maximum global priority in sys class (MAXCLSYSPRI)
> 15877 maximum processes per user id (v.v_maxup)
> 30 auto update time limit in seconds (NAUTOUP)
> 25 page stealing low water mark (GPGSLO)
> 5 fsflush run rate (FSFLUSHR)
> 25 minimum resident memory for avoiding deadlock
> (MINARMEM)
> 25 minimum swapable memory for avoiding deadlock
> (MINASMEM)
> *
> * Process Resource Limit Tunables (Current:Maximum)
> *
> Infinity:Infinity cpu time
> Infinity:Infinity file size
> Infinity:Infinity heap size
> 0x0000000000800000:Infinity stack size
> Infinity:Infinity core file size
> 0x0000000000000800:0x0000000000002000 file descriptors
> Infinity:Infinity mapped memory
>
Received on Tue Jun 10 2003 - 07:27:03 MDT