RE: [squid-users] Predictive caching?

On Tue, 22 Jul 2003, Chris Wilcox wrote:

> > > -----Original Message-----
> > > From: Chris Wilcox [mailto:not_rich_yet@hotmail.com]
> >
> > > Just had a suggestion about a project I'm working on: can we provide
> > > predictive caching? I know it's possible to use cron and
> > > wget to schedule
> > > downloads of pages to keep them in the cache, but is there
> > > any way I can get
> > > squid to follow links on pages it downloads so they load even
> > > quicker when
> > > requested by users?
> >
>
> hehe, I wasn't thinking along the lines of writing my own proxy or software.
> Reading into it, I don't think it would actually have the overwhelming
> impact on speed which it may appear it has when it was first mentioned to
> me.

Aah, the old Oracular cache that does the thinking for you?

This is not a new concept, fellas. At least one vendor (and several whose
names I've forgotten - lost in the debris from the .com crash) springs to
mind who attempted to do this kinda stuff.

 Cacheflow - active prefetching, settable according to policies

 Inktomi - I'm sure these guys had prefetch strategies ...or one of their
    partners, who developed on their API

 Network Appliance - has switches to complete downloads even when client
    aborts; now they also have "confill", which allows you to prefetch
    objects into the cache, setting expiry policies or even pinning the
    objects in the cache indefinitely

 ( and about a dozen others ... )

The upshot of this was that the link (talking Cacheflow here) was kept
saturated most of the time, and an awful lot of wasted traffic was
generated on objects with relatively low TTLs.

These days with Gigabit pipes, the prefetch argument is almost a mute
point: once upon a time the notion to download sites during quiet times
for prepositioning for the next busy days' request profiles, but how can
you predict user behaviour? You can argue that the bulk of browsers will
hit www.microsoft.com when they first start up, but then you have to
consider whether the site(s) is/are cacheable and for how long ... blah,
blah.

It is of interest to note that the prefetching plots apparently gave birth
to a lot of the CDN technology (this is my perception) that is being
applied to Content Devilry today.

;-)
                     _______
+------------------ //////// -----+----------------------------+
| John V Cougar __ | Voice: +61 2 6208 1683 |
| Cache Manager / / /\ +----------------------------+
| Telstra Internet \/_/ \ Direct | E-Mail: cougar@telstra.net |
+--------------------____/ -------+----------------------------+
Received on Tue Jul 22 2003 - 18:11:42 MDT