Re: Pinning objects in Squid Cache

On 03/13/2014 11:46 PM, aditya agarwal wrote:

> We have a set of new requirements in which we need to provide pinning
> of objects in squid cache such that they are not evicted by squid's
> LRU policy. I needed help in doing this and therefore had posted in
> squid-users group regarding the same. I got responses from Alex and
> Amos on some of the approaches that I can take to achieve this.

> Alex also mentioned that I should post the same on squid-dev group
> and discuss the best possible option. As per Alex following are the
> approaches:

> 1. Adding an interface to cache manager to pin-unpin specific cached objects.
> 2. An extension of HTTP request method
> 3. Adding an ICAP/eCAP interfaces to mark misses for pinning
> 4. Mark a pinning set configurable via squid.conf

> I understand Options 1 and 4 but I am not very clear by what he means
> in options 2 & 3. Also we might only have a few 10s or 100-200 of
> videos to pin.

Option 2 means supporting a new HTTP method like X-PIN that will pin the
referenced object to the cache. To use that option, the admin would have
to send an HTTP X-PIN request to Squid.

Option 3 uses an ICAP or eCAP response meta header that tells Squid that
the requested object should be pinned. To use that option, the admin
would have to use an ICAP service or an eCAP adapter.

> Can you please provide your opinions on which option would be best and why?

I think you should go with option #4. Add a new pin_cached directive
similar to send_hit and store_miss in Squid trunk:

  acl ToPin ...
  pin_cached allow ToPin
  pin_cached deny !ToPin

* Implementation sketch 4.1 (simpler and more flexible):

Check pin_cached whenever Squid replacement policy wants to delete a
cached entry. Move rejected candidates to the top of the replacement
policy index (as if they were "used" now) to avoid rechecking them too
frequently. Eventually, you can add specialized ACLs that would match on
the specific deletion reason/location and check pin_cached in other
deletion cases/places.

* Implementation sketch 4.2 (faster):

Check pin_cached whenever Squid starts caching an entry or loads cached
entry metadata during a cache index rebuild. Based on the results of
that check, raise or clear a new STORE_PINNED entry flag. Do not let
replacement policy to delete pinned entries, moving them to the top of
the replacement policy index (as in 4.1). A Squid restart would be
required to unpin pinned entries.

If pinning protection goes beyond replacement policy decisions, it
violates HTTP proxy caching rules and the feature should be marked as such.

> Note: Our squid version is 2.7.

Could have been worse!

HTH,

Alex.

> On Thursday, 13 March 2014 11:13 PM, Alex Rousskov <rousskov_at_measurement-factory.com> wrote:
> On 03/13/2014 09:52 AM, aditya agarwal wrote:
>
>> We had already thought of the second option to fetch the objects at
>> regular intervals so that they are always at the head of the queue in
>> cache, but it doesn't seem to be very scalable as we can have 100s of
>> videos which the client might want to pin to cache.
>
> Please also keep in mind that not all cache_dir types support LRU. For
> example, frequently requesting URLs in Rock storage would not help much.
>
>
>> I wanted to know if there is any modification that can be done in
>> squid to support pinning of objects.
>
> Yes, it would be possible to add such support. I can think of several
> options:
>
> * A cache manager interface to pin and unpin individual cached objects.
> It will not be simple if you want pinning to last across Squid restarts
> or if you want to pin using regular expressions and such.
>
> * An extension HTTP request method for the same purpose, but cache
> manager may be an overall better approach, especially from access
> control point of view.
>
> * It is also possible to add an eCAP/ICAP (or even a new helper)
> interface to mark misses for pinning. Adaptation makes pinning using
> regular expressions easy, but it will add performance overheads unless
> you are already using an adaptation service.
>
> * Finally, one could make a pinning set configurable via squid.conf
> ACLs. For mostly static sets that can be stored in a few MB or RAM
> (thousands of URLs, not millions), this is probably the most efficient
> and simple option.
>
> If you decide to work on any of this, please consider discussing
> specifics on squid-dev first. There are caveats related to each option
> and the choice of the best option is not obvious IMO.
>
>
> Cheers,
>
> Alex.
>
>
>
>> On Thursday, 13 March 2014 3:05 PM, Amos Jeffries <squid3_at_treenet.co.nz> wrote:
>> On 13/03/2014 9:22 p.m., aditya agarwal wrote:
>>
>>> Hi,
>>>
>>> I wanted to know if there is a way to PIN certain objects in Squid's
>>> cache, so that they are not removed or subjected to eviction because
>>> of the LRU policy running in squid.
>>>
>>> Thanks, Aditya
>>>
>>
>> That depends on what the objects are ... so "what exactly are you trying
>> to achieve?"
>>
>> Meanwhile ... objects locally served up by Squid using the
>> /squid-internal-static/ well-known URL path prefix have it. Such things
>> as icons for the error pages and FTP directory listings.
>> See the mime.conf file installed with your Squid on how to configure
>> those URL objects.
>>
>>
>> However, if you are wanting this for arbitrary objects served up elsewhere:
>>
>> * the best way is not to bother.
>> Cache is a _temporary_ storage area (a type of buffer) not a long term
>> archive. Correctly following HTTPP protocol ensures up to date reliable
>> content at all times.
>>
>> * the second-best way is to simply poll your proxy with a request for
>> it before the replacement policy removes it. This works on the same
>> principle as prefetching and has all the same problems with generating
>> correct client headers.
>>
>> Amos
>>
Received on Fri Mar 14 2014 - 15:08:55 MDT