RE: Squid-2.5

On 2 Oct 2001, at 20:19, Chemolli Francesco (USI) <ChemolliF@GruppoCredit.it> wrote:

> > > Memory leaks, but I think nothing functionality-wise.
> > ..
> >
> > This issue is very confusing. I'm here almost 100% confident, that
> > there are no leaks in squid. Of course, it also depends on how squid
> > is compiled and abit on how its used.
>
> Sure :)
> However, I'm leaking about 40-60 bytes/hit.
> Swapin/swapout rate is low, thus I expect that squid's leaking
> at least PAGE_SIZE objects (or it'd be swapping like crazy).
>
> Current-running state (excerpts from cachemgr), cache_mem is low (4mb,
> maybe):

> Memory usage for squid via mallinfo():
> Total space in arena: 389927 KB
> Ordinary blocks: 371749 KB 52507 blks
> Holding blocks: 696 KB 2 blks
> Total in use: 372445 KB 96%
> Total free: 18178 KB 5%
> Internal Data Structures:
> 203088 StoreEntries
> Cumulative allocated volume: 300.26 GB
> Idle pool limit: 10.00 MB

 that looks pretty awful indeed.

> This would show that the leaked memory is not from mempool-related
> allocs, but xallocs or mallocs or whatnot.

 what is mempool Idle high KB, Alloced, inUse?
 Afaik most stuff has been turned to mempools by now. Only stuff in
 lib/ is left. heap-repl is using mem from there.

> The fact that you show no leaks leads me to think it might be
> NTLM-auth-related (ouch!), Robert has audited the code found
> nothing though.

 if you suspect NTLM, can you try a run with NTLM not compiled in?

> I'm willing to do some debugging and profiling, if anybody could
> help me in this I'd be obviously very happy.

> This amount of leaking (when squid starts it's 20mb altoghether) is a
> tad too much to be traced to heap fragmentation IMO.

 Sure. I browsed few leak reports on bugzilla. Thats where I got
 my impression.

> > - stupid OS memory allocator. Solaris is notorious for that. It
> > I'm not sure how is default Linux malloc tuned, but I'd not be
> > surprised if it has similar behaviour.
>
> For allocs smaller than PAGE_SIZE, it only returns memory to the system
> LIFO. So a small unfreed alloc could "keep around" lots of memory.

 Thats expected. But depends on malloc in use. keeping ram without
 returning is actually optimisation ;). I'm not familiar with what
 Linux has, so I bet you are right.

> For allocs bigger than PAGE_SIZE, it should use anonymous mmapping,
> and thus return it to the system when freed.

 Thats not sure. Most probably all mallocs your squid has done are
 from heap. Allocs from mmap are accounted to "Holding blocks", at
 least on my Linux box, they are at 2 in your case.
 most mmapping mallocs have quite strict limits on how much to use
 mmap. typical is no more that 64 mmaps of size at least 128KB.
 I had to deal with that issue in chunked pools. They benefit most
 from being mmapped, and I had to convince malloc to do more than
 64 mmaps.

 but thats not an issue in your case anyway. If the memory was in
 fragmented space, it should be seen as "Total free" in mallinfo().

> > - increased memory footprint of Squid. I've seen this when I compiled
> > in heap-replacement. Not sure how its now, but when I tried it, it
> > consumed about 2-4 pointers per storeentry in addition. Together
> > with malloc roundups this was quite notable. One of my 2.3 squids
> > is experiencing this. heap memory overhead is not accounted for
> > as it doesn't use mempools.
>
> 10x the initial footprint? (i've in time noticed squid be as big as 600 Mb,
> which is more than the available core).

 Is heap maintained by someone? It was added very long ago.
 I wouldn't be too surprised if it leaks after so many changes
 around other places.
 try without heap. it doesn't cost anything.

> > As I understand, noone really knows if and where they are.
>
> I'm not sure I agree on the "if", unfortunately I do on the "where".

 right. Whats different in in our configure options?

 ./configure --enable-dlmalloc --enable-xmalloc-statistics
     --enable-storeio=ufs,diskd
     --enable-delay-pools --enable-snmp --enable-cache-digests
     --enable-underscores --enable-cpu-profiling --disable-wccp
     --enable-htcp --enable-forw-via-db --enable-gnuregex

------------------------------------
 Andres Kroonmaa <andre@online.ee>
 CTO, Microlink Online
 Tel: 6501 731, Fax: 6501 725
 Pärnu mnt. 158, Tallinn,
 11317 Estonia
Received on Tue Oct 02 2001 - 14:05:22 MDT