Re: [squid-users] Compute digest as content is written to cache

On 11/08/12 11:37 PM, Amos Jeffries wrote:
> On 11/08/2012 10:21 p.m., Jack Bates wrote:
>> On 11/08/12 12:30 AM, Amos Jeffries wrote:
>>> On 11/08/2012 7:22 p.m., Jack Bates wrote:
>>>> I am interested in intercepting content as it is written to the cache,
>>>> and computing a digest from the content. Do you know if this can be
>>>> done in some kind of add on, or would it require a change to the core?
>>>
>>> What type of digest and to what purpose?
>>
>> I was thinking of using OpenSSL
>> SHA256_Init()/SHA256_Update()/SHA256_Final(). The purpose I have in
>> mind is to detect identical content at different URLs
>>
>> Given a response with a "Location: ..." header and a "Digest:
>> SHA-256=..." header (such as from MirrorBrain), if the URL in the
>> "Location: ..." header is not already cached but the "Digest:
>> SHA-256=..." header matches the content at some other URL that is
>> already cached, then I want to update the "Location: ..." header with
>> the cached URL. I think this should redirect clients to mirrors that
>> are already cached
>
> Small problem there. The digest is not calculated/known until the object
> is finished arriving. By then it is too late to attach new headers. And
> way too late to decide whether to ask that source for it.

Thanks Amos, but I want to calculate the digest of different objects
than the ones that I want to rewrite the "Location: ..." header of

As objects are cached, I want to compute the digest and record it
(possibly in some external database). Then when a response with a
"Location: ..." header and a "Digest: SHA-256=..." header is received
(such as from MirrorBrain), if the URL in the "Location: ..." header is
not already cached, then I want to compare the digest in the "Digest:
SHA-256=..." header to the previously recorded digests. If a match is
found, than I want to rewrite the "Location: ..." header with the URL of
the match
Received on Sun Aug 12 2012 - 07:13:03 MDT