RE: Squid-2.5

> > Memory leaks, but I think nothing functionality-wise.
> ..
> > Well, I have stretches up to 10M hits until I have to
> restart for the
> > aforementioned leaks (and unfortunately the occasional
> leaked helper process
> > :().
>
> This issue is very confusing. I'm here almost 100% confident, that
> there are no leaks in squid. Of course, it also depends on how squid
> is compiled and abit on how its used.

Sure :)
However, I'm leaking about 40-60 bytes/hit.
Swapin/swapout rate is low, thus I expect that squid's leaking
at least PAGE_SIZE objects (or it'd be swapping like crazy).

Current-running state (excerpts from cachemgr), cache_mem is low (4mb,
maybe):
Connection information for squid:
        Number of clients accessing cache: 4577
        Number of HTTP requests received: 6048917
        Number of ICP messages received: 0
        HTTP requests per minute: 2878.1
Memory usage for squid via mallinfo():
        Total space in arena: 389927 KB
        Ordinary blocks: 371749 KB 52507 blks
        Small blocks: 0 KB 0 blks
        Holding blocks: 696 KB 2 blks
        Free Small blocks: 0 KB
        Free Ordinary blocks: 18178 KB
        Total in use: 372445 KB 96%
        Total free: 18178 KB 5%
Memory accounted for:
        Total accounted: -1 KB
        memPoolAlloc calls: 723118842
        memPoolFree calls: 722414709
Internal Data Structures:
        203088 StoreEntries
           880 StoreEntries with MemObjects
           867 Hot Object Cache Items
        203027 on-disk objects

Cumulative allocated volume: 300.26 GB
Current overhead: 672862 bytes (2.264%)
Idle pool limit: 10.00 MB
memPoolAlloc calls: 723381704
memPoolFree calls: 722677776

This would show that the leaked memory is not from mempool-related
allocs, but xallocs or mallocs or whatnot.
The fact that you show no leaks leads me to think it might be
NTLM-auth-related (ouch!), Robert has audited the code found
nothing though.
I'm willing to do some debugging and profiling, if anybody could
help me in this I'd be obviously very happy.

> I have 2 squid's on HEAD, neither has shown any signs of leaks of
> any kind. Both have done (as of now) over 500GB cumulative mempool
> allocs and both are well beyond 10M reqs serviced.
> There is a slight difference to HEAD - I use chunked_mempools here.
> but this hasn't changed anything in regards to memory leaks.
>
> I think that many so-called memory leaks are misnamed. I may be
> wrong and then let me be corrected. Why not go through this.

This amount of leaking (when squid starts it's 20mb altoghether) is a
tad too much to be traced to heap fragmentation IMO.

> From my experience, there are several things that can cause false
> sense of memory leaks:
> - stupid OS memory allocator. Solaris is notorious for that. It
> allocs memory, but never frees. Process at any time is as large
> as its peak memory usage has ever been.
> Fix: --enable-dlmalloc
> I'm not sure how is default Linux malloc tuned, but I'd not be
> surprised if it has similar behaviour.

For allocs smaller than PAGE_SIZE, it only returns memory to the system
LIFO. So a small unfreed alloc could "keep around" lots of memory.
For allocs bigger than PAGE_SIZE, it should use anonymous mmapping, and thus
return it to the system when freed.

> - idle pools limit. Default is "none", which means 2G. mempools do
> not return memory to OS. Heavy fragmentation occurs over time.
> Fix: memory_pools_limit=8M. Going much up doesn't really help.

10MB a shown.

> - increased memory footprint of Squid. I've seen this when I compiled
> in heap-replacement. Not sure how its now, but when I tried it, it
> consumed about 2-4 pointers per storeentry in addition. Together
> with malloc roundups this was quite notable. One of my 2.3 squids
> is experiencing this. heap memory overhead is not accounted for
> as it doesn't use mempools.

10x the initial footprint? (i've in time noticed squid be as big as 600 Mb,
which is more than the available core).

> I can't say that there isn't any leaks. It just doesn't show up
> for me. If we know where leaks are, then we can address them.
> As I understand, noone really knows if and where they are.

I'm not sure I agree on the "if", unfortunately I do on the "where".

> We should really first try determine conditions when leaks occur.
> In this group and together with bug reporters.

-- 
	/kinkie