Re: [squid-users] Prefetching HTTP GET requests, but under HTTPS

On 6/05/2014 5:53 p.m., Jianshi Huang wrote:
> Hi,
>
> I need to build a prefetching proxy to speedup page loading/clicks and
> I'm currently investigating Squid for prototype. Websites I want to
> speedup are all under HTTPS.
>
> I briefly scanned Squid's document and google the keywords, looks like
> I need to do the following setup:
>
> 1) Use SSL-Bump (and install Squid's cert in client's machine)

Yes this is the only way to get around the HTTPS "problem".

> 2) Two Squid setup, one runs prefetching script, another one does the caching.

Any reason for that design?
 Prefetching only needs three components: The cache (Squid), the logic
deciding what to fetch (script), and possibly a database of past info to
inform those decisions.

Check out the squidclient tool we provide for making arbitrary web
requests. It is the best tool around for scripting web requests, similar
levels of control to libcurl which is (probably) the best for use in
compiled code.

> Does that make sense?

Pre-fetching is a very old idea based around metrics from very old
protocols such as HTTP/1.0 where the traffic was static, predictable and
prefetching makes a fair bit of sense.

However there are several popular features built into HTTP/1.1 protocol
which greatly alter that balance. Dynamic content with variants makes it
far less static. Response negotiation makes the responses far more
unpredictable. Persistent connections greatly reduce the lag times.
Revalidation reduces the bandwidth costs. Together these all make
prefetching in HTTP/1.1 a much less beneficial operation than most of
the literature makes it seem.

Whether it makes sense depends entirly on where it is being installed,
what the traffic is like, how and why the prefetching decisions are
being made. Only you can really answer those and it may actually take
doing to figure out whether it was a bad choice to begin with.

> Is there a better solution?

At the current point of Internet development I believe throwing efforts
into assisting us with HTTP/2 development would be more beneficial. But
I am somewhat biased being a member of the HTTPbis WG and seeking to get
Squid HTTP/2 support off the ground.

> Or has anybody done similar things?

We get a fairly regular flow of questions from people wanting to do
pre-fetching. They all hit that above issues eventually and drop out of
sight.

This thread summarizes teh standard problems and answers:
http://arstechnica.com/civis/viewtopic.php?f=16&t=1204579
(see fandingo's answer near the bottom)

I am aware of this tool being used by more than a few dozen
installations, although its popularity does seem to be in decline:
https://packages.debian.org/unstable/web/squid-prefetch

>
> It would be great if someone could point out some
> configuration/scripting files to me. Code speaks :)
>

Everythig in this regard is situation dependent. The above are likely to
be the best you can find. People who actually get it going (apparently
anyway) keep the secrets of how to avoid the HTTP/1.1 issues pretty close.

HTH
Amos
Received on Tue May 06 2014 - 09:04:34 MDT