Re: Squid enhancement thought

On Jun 2, 9:56am, Dave J Woolley (possibly) wrote:
> > From: Reuben Farrelly [SMTP:reuben-squid@mira.net]
> >
> > There are client programs which, certainly under Windows, accomplish the
> > same task as what is being suggested. I would suggest that they may be a
> > better approach to use in situations where this sort of behaviour is
> > required. That way, the client pays the cost of the extra speed, rather
> > than the squid-maintainer.
> >
> Many content providers object violently to these,
> especially when they don't honour robots.txt and don't
> provide a recognizable User-Agent string (so that they
> can be rejected!). IMDB is, I believe, one site which
> takes an active stance against them.

I don't really see any reason that content providers should get
identifiable User-Agent strings. If they don't want to serve up the
information, they shouldn't provide it. Regarding robots.txt, I'd
point out that much of the potentially prefetchable material is made
up of inline images, which are automatically requested by most graphic
browsers anyway. Do people contend that such automatic requests should
be blocked by robots.txt?

> The problem is that they waste bandwidth at the content
> provider by accessing, often dynamic, pages that are
> never going to be read.

Nowadays, is the real reason this, or that it disrupts the ability to
count banner clicks and other advertising? Dynamic content is
cache-unfriendly in any event; I wouldn't suggest prefetching if
something's from a non-cacheable page, but I don't really see any
reason to make an effort to make content providers using unnecessarily
uncacheable pages happy.

> It must also identify accesses made in this way in a manner
> that makes it easy for the content provider to reject them.

Umm... the motivation for a cache is to speed up access and reduce
bandwidth, not to make the content providers happy.

> Generally, given that there is not a very high penetration of
> cache usage by end users, the likely effect of adding this
> capability to an ISP cache is that sites will reject any proxied
> requests and force those users using caches to disable them.

The solution to this would appear to be to not let them know that
requests are proxied. I've done some patches that optionally eliminate
X-Forwarded-For, for instance.

        -Allen

-- 
Allen Smith				easmith@beatrice.rutgers.edu