Re: [RFC] One helper to rewrite them all

On 12.09.2012 19:18, Eliezer Croitoru wrote:
> On 09/12/2012 02:50 AM, Alex Rousskov wrote:
>> Yes, of course. Both ICAP and eCAP have meta-headers that can go
>> both
>> ways. But some people do prefer the helper simplicity, and my
>> suggestion
>> was meant to give those folks a performance boost.
>>
>> If the consensus is that performance-sensitive deployments should
>> use
>> performance-centric APIs such as eCAP while keeping the URL
>> rewriting
>> helpers as simple as possible, I am happy to subscribe to that
>> approach.
> I asked before about this option to use Icap for url_rewriting and
> the answer that I got that we need the ICAP guy for that.
> I have specific ICAP server I wrote which I use for url_rewriting and
> his performance is more then 20,000 request per/s on a very little
> atom cpu.
> The main problem was that squid after about 900 request per/s would
> have a problem.
> Else then that the ICAP server fit's for the task even more then a
> helper (my opinion).
>
> But the heavy part in the process is not really one or another
> interface more then actually making the use of store_url url for
> mem_cache and cache_dir properly.
>
> I am having a bit trouble now because of the trillion loops and jumps
> in the code while using over 100 times the mem_obj->original_url and
> StoreEntry::origianlUrl (was ::url)
> most of them are debugs so it helps me a bit but I am far from one
> place that the hash is being calculated with the original url and
> neededs to be replaced with store_url.
>
> For the "no cachable" situation something should be done to lower
> performance issues that could come from using store_url, some kind of
> flag validation before sending it to the helper?
>
> I will do my best to make the heart of the feature possible leaving
> the rest for later to any other specialist who knows specifics of
> eCAP
> ICAP etc.
>
>
> Regards,
> Eliezer

I think long-term we want to have ICAP/eCAP generating ETag+Digest
headers for requests and responses and store using those headers as the
index key. Under that environment *CAP service can adapt the ETag+Digest
header on request/response in order to de-dup instead of playing with
store-URL.

That requires a fair bit of store alterations for Digest though, so
will have to come after this feature port.

Amos
Received on Wed Sep 12 2012 - 23:20:52 MDT