Neighboring woes

Hi,

users have been bugging us about problems with our Squid proxy cache for
the last couple of days:

they will load a page through our proxy (e.g., http://www.tagesschau.de),
get a page that's out-of-date by weeks, press reload, get the up-to-date
page loaded into the cache; and the same sequence repeats an hour later!

The same thing happens for several popular pages.

After wading through the several log files, here's what I found was
happening:

 - the out-of-date version of the page had a last-modified date, the
   current one doesn't.

 - the local ttl for pages without expiry or last-modified date is only
   a couple of minutes---enough to handle frequent reloads, and to avoid
   problems with out-of-date pages. Thus, the current page is quickly
   purged from the cache again.

 - Loading the page again after it has expired from the cache results in
   a neighbor hit (UDP_HIT_OBJ) for a page that's definitely out-of-date
   (expired) by our local ttl_pattern standards, but it's delivered to
   our users and stored into the cache nevertheless.

Having located the neighbor that sends the out-dated pages, it was
necessary to force a reload on the offending page(s) in their cache.

I do not think that it's correct that a remote neighbor cache can override
local ttl patterns like that. And I do not like the fact that I need to
manually maintain a remote cache server to fix that problem.

As a maintainer of a local cache, I need *total* control over the age of
objects delivered to my users. Otherwise I will loose those users, to the
detriment of the whole cooperative system, and (what bugs me most;-)
through no fault of mine.

Conclusions:

 - Neighbor caches *MUST* send last-checked information along with any
   peered objects. ICP *MUST* support this, if it doesn't do so already.
   That's the only way to guarantee locality of age-control.

 - Objects sent by neighbors *MUST* be checked against local ttl_patterns
   to determine whether to send them on to the user or to retrieve them
   directly (neighbor) or force a refresh/IMS (parent).

 - Neighbors should be notified when objects received from them turn out
   to be out-of-date by checking the source (wasn't that supposed to be
   done using Mbone?). They should also be notified, if possible, if
   an object they delivered is known to have changed through a user
   reload (this is a lot harder to implement properly, I suspect).

The first two would be acceptable as a 'solution' of the problem I
outlined, but only the combination of all three would guarantee consistent
behaviour of a local cache embedded in a collaborative caching effort.

Users can deal with consistency, but they justly complain bitterly about
inconsistencies.

Best,

Andreas

-- 
Andreas Strotmann       / ~~~~~~ \________________A.Strotmann@Uni-Koeln.DE
Universitaet zu Koeln  /| University of Cologne   \
Regionales Rechenzentrum| Regional Computer Center \
Robert-Koch-Str. 10    /|    Tel: +49-221-478-5524 |\   Home: -221-4200663
D-50931  Koeln        __|__  FAX: +49-221-478-5590 |__________~~~~~~~~~~~~