Re: [squid-users] ipcCreate: fork: (12) Cannot allocate memory

On 19/03/11 12:23, david_at_lang.hm wrote:
> On Sat, 19 Mar 2011, Amos Jeffries wrote:
>
>> On 19/03/11 02:43, Winfield Henry wrote:
>>>
>>> Hi,
>>> Squid has started to NOT come back up after log rotate. Here is
>>> snippett from cache.log.
>>> Machine has 1G ram and cache_mem is set to 500MB,
>>
>> Squid uses fork() instead of vfork() to spawn helpers, on some OS the
>> fork() implementation prevents extremely huge amounts of virtual
>> memory being "allocated" (even though it is neither allocated nor used).
>
> I think you mean that on some OS the form implementation 'results in'
> rather than 'prevents'
>
> on linux this is the 'overcommit' option, on by default in the kernel,
> but many people think it makes their systems more reliable to disable it.

I kind of meant both. All the systems seen during the investigation of
this problem both working and failing resulted in huge VMZ sizes being
allocated. Not one left it small. The error only hits when the limiter
prevents success though.
   (If you know of an OS which allows hundreds of GB of virtual memory
please say, the crowd trying to use huge RAM-caches need to know.)

>
> you can work around the problem by makeing sure the system has enough
> virtual memory available, usually by increasing the amount of swap space
> avialable.

That size has to be N * S.
  N being the total count of all helpers Squid uses.
  S being the maximum memory Squid allocates for itself during operation.

Thus to reduce helpers:
>
>> The helper multiplexer has been created to get around this problem:
>> ftp://ftp.squid-cache.org/pub/squid/contrib/helper-mux/
>> Details on how to use it are in the Squid-3.2 release notes:
>> http://www.squid-cache.org/Versions/v3/3.2/RELEASENOTES.html
>> (published as part of 3.2, but works on all squid-2.6 or later).
>>
>> PS.
>> A few of use have looked at making Squid use vfork() but been defeated
>> by the parent doing followup logics. If anyone is keen to attack the
>> problem patches that work will be VERY welcome.
>>

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.11
   Beta testers wanted for 3.2.0.5