[PATCH] SMP Caching: Core, IPC, Shared memory cache, and Rock Store

Hello,

    Attached compressed patch contains changes related to SMP shared
memory cache and Rock Store support. Most of these changes have been
posted earlier for preview. The patch accumulates more than two years
worth of work and, at 20K lines, is probably too huge and too difficult
to audit correctly, for which I do apologize. I should have found a way
to merge parts of this earlier.

The patch preamble is several pages long and attempts to document all
major changes.

Squid automatically disables shared memory caching and Rock Store
support if the required IPC primitives are not supported, making the
code quite portable overall. The code has passed Jenkins tests; thanks
to Kinkie and Dmitry Kurochkin for making that possible!

The code has been working in several lab environments for many months,
but saw only limited short-term production deployment. We are working on
a couple of optimizations that should allow us to push this back into
limited production, but I am sure more changes will be needed for wide
deployment.

FWIW, earlier non-SMP Rock Store code (based on Squid v3.1) has been
successfully deployed for a long time, but only in a limited set of
high-performance environments.

Overall, I think this code deserves to be in trunk, even though there
are some rough edges, especially in the code that was written before
various coding rules went into effect. I expect it to be disruptive
short-term, but very useful long-term.

My earlier plan was to submit these changes in three parts: Core
changes, IPC primitives plus Shared memory cache, and Rock Store. During
the last Preview, I did split the patch into three files, but even then
the exercise was painful and created many mistakes (stuff that should
have been in partA went into partB, was duplicated, or lost).

Today, there are even more links or dependencies between the parts so I
gave up on that idea as I was getting nervous I will miss yet another
chance to submit these changes. However, if somebody insists on a split,
we will work on that.

Besides extra work, my primary worry is that a 270-line commit message
is already difficult to handle, and most new changes would have to be
added to the patch virtually undocumented, even though they sometimes
reflect significant new knowledge.

I even considered suggesting doing "bzr merge" in trunk to get all the
commit messages, but I am guessing that would import too many entries, a
few of which are not interesting (e.g., "merged from trunk") or of low
quality ("atomic queue implementation #4"). On the other hand, doing
true branch merges could be the way how we are supposed to use bzr :-)

You can see the current tree on Launchpad. If the changes are approved,
and you think it is better to "bzr merge" them instead of patching them
in as usual, please say so.

    https://code.launchpad.net/~measurement-factory/squid/3p2-rock

This patch should address Amos comments raised during the last preview
dated 2011/05/15. I will respond to that old email separately.

Thank you,

Alex.