Re: Large Rock Store

On 17.10.2012 11:02, Alex Rousskov wrote:
> Hello,
>
> We have started working on caching "large" (i.e., multi-slot)
> responses in Rock Store. Initial design overview is available at the
> following wiki page, along with several design alternatives we have
> considered. Feedback is welcomed.
>
> http://wiki.squid-cache.org/Features/LargeRockStore
>
>
> As a part of Large Rock work, I will also try to fix at least one of
> Store API quality problems that complicate any Store-related
> improvements:
>
> 1) Store class is a parent of too many, barely related classes:
> StoreController, StoreHashIndex, SwapDir, MemStore (and TestStore).
> This
> "top heavy" design forces us to add pure virtual methods to Store and
> then supply dummy implementations in many Store kids. And, more
> importantly, it makes it difficult to understand which part of the
> storage API each Store kid is responsible for, leading to boundaries
> violation and other problems.
>
> 2) There is no class dedicated to a non-shared memory cache.
> StoreController currently implements most of non-shared memory cache
> logic while MemStore implements shared memory cache. StoreController
> non-shared memory sharing code should be moved to a dedicated class
> instead.
>
> 3) StoreHashIndex should probably become responsible for all disk
> caches
> (SwapDirs) as a whole, while StoreController will coordinate disk and
> memory caching (using StoreHashIndex and MemStore objects).
> Currently,
> some disk-related manipulations reside in StoreHashIndex and some in
> StoreController.
>
> 4) Get rid of the global store_table. Make it local to non-shared
> caching code. As we have discussed previously, the existence of this
> global makes shared caching life very difficult because we cannot
> share
> such a complex table and yet some older code expects to find all
> entries
> there. It also leads to problems with entry locking where the code
> assumes that an idle entry will remain in the global table at least
> for
> a while (I believe there are a few bugzilla reports about associated
> core dumps). I doubt I can solve all the StoreEntry locking problems
> (it
> may require significant client side rewrite), but removing
> store_table
> global is a step in the right direction.
>
>
> If you have suggestions on priorities or want to add other
> large-scale
> Store annoyances to the above list, please let me know.

Thank you, architecture clarification for store would be very welcome.

  IMHO;
  - "Store" should be the namespace - or the name of a base class for
shared API accessors to a *single* store regardless of type.
  - Low-level virtual methods should be in another class specific to the
type of store (disk, memory, shared, ...). Which the particular store
controller inherit from alongside "Store" interface class.
  - Index hash object/s should be independent of the store backend and
the class re-used by all (separate instances though).
  - store controller - managing the set of available stores for
startup/reload/clean/dirty/up/down/search/object-move handling?

Since you mention a difficulty identifying what each of the store API
classes does, can you suggest anything better than by arch brainstorm
above?

For example; its not clear to me in the slightest why StoreHashIndex is
relevant to disks but not in-memory caches. Surely the in-memory have a
hash index of their content as well? (or should after global store_table
is gone) - or is the class name irrelevant to what it actually does
nowdays?

Amos
Received on Tue Oct 16 2012 - 22:45:27 MDT