Fw: [squid-users] Pinning objects in Squid Cache

Hi Guys, I am a software engineer working in a company in Bangalore. Our product primarily provides caching for video objects and we use squid (2.7) to do so. We have a set of new requirements in which we need to provide pinning of objects in squid cache such that they are not evicted by squid's LRU policy. I needed help in doing this and therefore had posted in squid-users group regarding the same. I got responses from Alex and Amos on some of the approaches that I can take to achieve this. Alex also mentioned that I should post the same on squid-dev group and discuss the best possible option. As per Alex following are the approaches: 1. Adding an interface to cache manager to pin-unpin specific cached objects. 2. An extension of HTTP request method 3. Adding an ICAP/eCAP interfaces to mark misses for pinning 4. Mark a pinning set configurable via squid.conf I understand Options 1 and 4 but I am not very clear by what he means in options 2 & 3. Also we might only have a few 10s or 100-200 of videos to pin. Can you please provide your opinions on which option would be best and why? Note: Our squid version is 2.7. Thanks, Aditya On Thursday, 13 March 2014 11:13 PM, Alex Rousskov <rousskov_at_measurement-factory.com> wrote: On 03/13/2014 09:52 AM, aditya agarwal wrote: > We had already thought of the second option to fetch the objects at > regular intervals so that they are always at the head of the queue in > cache, but it doesn't seem to be very scalable as we can have 100s of > videos which the client might want to pin to cache. Please also keep in mind that not all cache_dir types support LRU. For example, frequently requesting URLs in Rock storage would not help much. > I wanted to know if there is any modification that can be done in > squid to support pinning of objects. Yes, it would be possible to add such support. I can think of several options: * A cache manager interface to pin and unpin individual cached objects. It will not be simple if you want pinning to last across Squid restarts or if you want to pin using regular expressions and such. * An extension HTTP request method for the same purpose, but cache manager may be an overall better approach, especially from access control point of view. * It is also possible to add an eCAP/ICAP (or even a new helper) interface to mark misses for pinning. Adaptation makes pinning using regular expressions easy, but it will add performance overheads unless you are already using an adaptation service. * Finally, one could make a pinning set configurable via squid.conf ACLs. For mostly static sets that can be stored in a few MB or RAM (thousands of URLs, not millions), this is probably the most efficient and simple option. If you decide to work on any of this, please consider discussing specifics on squid-dev first. There are caveats related to each option and the choice of the best option is not obvious IMO. Cheers, Alex. > On Thursday, 13 March 2014 3:05 PM, Amos Jeffries <squid3_at_treenet.co.nz> wrote: > On 13/03/2014 9:22 p.m., aditya agarwal wrote: > >> Hi, >> >> I wanted to know if there is a way to PIN certain objects in Squid's >> cache, so that they are not removed or subjected to eviction because >> of the LRU policy running in squid. >> >> Thanks, Aditya >> > > That depends on what the objects are ... so "what exactly are you trying > to achieve?" > > Meanwhile ... objects locally served up by Squid using the > /squid-internal-static/ well-known URL path prefix have it. Such things > as icons for the error pages and FTP directory listings. > See the mime.conf file installed with your Squid on how to configure > those URL objects. > > > However, if you are wanting this for arbitrary objects served up elsewhere: > > * the best way is not to bother. > Cache is a _temporary_ storage area (a type of buffer) not a long term > archive. Correctly following HTTPP protocol ensures up to date reliable > content at all times. > > * the second-best way is to simply poll your proxy with a request for > it before the replacement policy removes it. This works on the same > principle as prefetching and has all the same problems with generating > correct client headers. > > Amos >
Received on Fri Mar 14 2014 - 05:46:55 MDT