Re: dlmalloc

On 17 Dec 2003, at 9:57, Henrik Nordstrom <hno@squid-cache.org> wrote:

> > But most interesting to me is in what ways has it bitten us?
>
> For a start there would need to be a official standard in how one is to
> replace the malloc implementation without risk for conflicts with the OS
> provided C/C++ libraries or other libraries used by Squid, and this
>
> The most common error and risk is that dlmalloc does not implement all
> entry points to the OS malloc implementation, and this causes a big mess
> of things both in Squid and in any library used by Squid if any piece of
> code ever tries to use one of the malloc functions/symbols not
> implemented by dlmalloc. When this happens parts of both malloc
> implementations gets linked into the program in uncompatible manners.
>
> Another risk is dynamic linking of the OS libc if any of the symbols
> exported by the OS provided malloc package is not a weak symbol. In such
> case functions in the OS libc using such sybmol will continue to use the
> OS malloc code even if dlmalloc provides another implementation of the
> same.

 Aha, okay, thanks for clarifying. That could be quite serious.
 Have we actually been hit by such mess? Isn't this issue of a past,
 or question of some unix compliance?

> > I have understood that we bundle dlmalloc because it fits well enough
> > for Squid and can be used as alternate malloc on OS'es that would suffer
> > if using native libs, so that instead of forcing users into an OS hunt
> > we can switch over to dlmalloc transparently. Having one source bundled
> > allows us better integrate expectations of both.
>
> True, but it is also true that mempools is supposed to take most of the
> heat away from the malloc implementation.

 Yes, but there remains issue with memory fragmentation, to solve which
 there is sense in coordinating assumptions of both malloc and pools.
 Chunks are longlived objects, and they can cause nasty holes in memspace,
 that can be relieved by use of mmaped allocs. We could rely on known
 malloc lib to do that, or opt to do that ourselves, repeating all the
 portability stuff.

> > dlmalloc is very tunable, at both compiletime and runtime. This has been
> > ignored so far.
>
> None of the developers has needed to care about dlmalloc tuning afaik.

 There are actually several things worthwhile to tune. mmap threshold for
 eg, which defaults to 256K in dlmalloc, and probably much higher with
 some others, can have good effect of fragmentation with mempools.
 Also max number of mmaps malloc allows could use some tuneup. Default
 is limited to 64, meaning no more than 64 chunks would get mmapped.
 That seems abit small.
 Finally, dlmalloc latest version has thread_safety options.
 Rest, internal tunables are probably indeed not needed.

------------------------------------
 Andres Kroonmaa <andre@online.ee>
 CTO, Microlink Data AS
 Tel: 6501 731, Fax: 6501 725
 Pärnu mnt. 158, Tallinn
 11317 Estonia
Received on Wed Dec 17 2003 - 04:03:37 MST