Re: HTTP delta-encoding in squid - please help...

Dmitry Fink (aka FINiK) wrote:

> I'm working on adding delta-encoding to squid (when the only thing that is
> sent from the server to the squid is the diff between the new document on
> server and the chached document in squid).

Interesting, but see the end of this message.

> I am having a very hard time understaning the squid sources, and therefore
> would be very happy if someone will answer my questions.

Will try to.

> First of all, it is clear that some new entry is needed in the StoreEntry
> (my guess), which is some sort of TAG tht idntifies the version of the
> cached document which will be sent to the server in the request.
> Therefore, in httpBuildRequestHeader I should have access to this
> information i.e have access to the data of the expired(old) version
> of the document. Is it possible? How do I do that?

Either you put only keep it in the object headers and bring these in
from disk prior to issuing the update request, or you add it to the
StoreEntry. Which approach to select depends on your priority between
performance (latency) and memory consumption.

> Then, when parsing the reply from the server, I need access to the
> whole data of the old expired version of the document in order to apply
> the patch on it. Again, the same question: do I have that kind of data on
> this stage?

If you want to yes.

> Or maybe when refresh is happening, the old version is deleted before
> the new version is brought?

For a start, objects are only deleted if they are not currently being
referenced (locked). Second, the revalidation process is very similar to
what you need to do except that it does not touch the actual content. It
gets a request for a "stale" object, issues a refresh request to the
origin server (or next hop) and then sends the result back to the client
based on both the new input and the old data.

The difference in what you are trying to do is that in this case you
need to create a new object, and play back the updated content into this
object. The client is then sent this new object.

Hmm.. have you done any calculations on what might be gained from this?
It can only be applied to one level (bad for scaling), and required new
protocol extensions to be defined (bad for portability).

It is probably a better payoff at looking at transfer encoding (i.e.
compression) of data. The needed protocol definitions are already
standardized, and there exists a useable skeleton implementation for
Squid (see "te" on http://squid.sourceforge.net/). What is missing is
the interactions with a available compression engine, and servers
capable/willing to compressing the data.

--
Henrik Nordstrom
Squid hacker