Re: issue with ICAP message for redirecting HTTPS/CONNECT

On 06/08/2014 10:02 AM, Nathan Hoad wrote:

> There's a bug in your ICAP server with how it's handling the
> Encapsulated header that it sends back to Squid.
...
> The Encapsulated header says that the HTTP object that has been sent
> back contains HTTP response headers, and no body. This leads Squid to
> believe it should be parsing a HTTP response

Hello Marcus,

    In addition to the Encapsulated header wrongly promising an HTTP
response, the ICAP response also contains an encapsulated HTTP body
chunk (of zero size) when the Encapsulated header promised no body at
all. That ICAP server bug is present in both GET and CONNECT adaptation
transactions (but the correct behavior would be different in each of
those two cases).

If you are writing a yet another ICAP server, please note that free and
commercial ICAP servers are available. Are you sure you want to go
through the pains of writing a yet another broken one? And that you
actually need ICAP?

Finally, please note that rewriting and even satisfying CONNECT requests
is difficult because the browser has certain expectations about the
origin server and the browser's security model prevent many CONNECT
request and response manipulations.

Cheers,

Alex.

> On 9 June 2014 00:22, Marcus Kool <marcus.kool_at_urlfilterdb.com> wrote:
>> I ran into an issue with the ICAP interface.
>> The issue is that a GET/HTTP-based URL can be successfully rewritten but a
>> CONNECT/HTTPS-based URL cannot. I used debug_options ALL,9 to find out what
>> is going wrong
>> but I fail to understand Squid.
>>
>> GET/HTTP to http://googleads.g.doubleclick.net works:
>>
>> Squid writes:
>> REQMOD icap://127.0.0.1:1344/reqmod_icapd_squid34 ICAP/1.0<0d>
>> Host: 127.0.0.1:1344<0d>
>> Date: Sun, 08 Jun 2014 13:54:09 GMT<0d>
>> Encapsulated: req-hdr=0, null-body=135<0d>
>> Preview: 0<0d>
>> Allow: 204<0d>
>> X-Client-IP: 127.0.0.1<0d>
>> <0d>
>> GET http://googleads.g.doubleclick.net/ HTTP/1.0<0d>
>> User-Agent: Wget/1.12 (linux-gnu)<0d>
>> Accept: */*<0d>
>> Host: googleads.g.doubleclick.net<0d>
>> <0d>
>>
>> ICAP daemon responds:
>> ICAP/1.0 200 OK<0d>
>> Server: ufdbICAPd/1.0<0d>
>> Date: Sun, 08 Jun 2014 13:54:09 GMT<0d>
>> ISTag: "5394572c-4567"<0d>
>> Connection: keep-alive<0d>
>> Encapsulated: res-hdr=0, null-body=233<0d>
>> X-Next-Services: <0d>
>> <0d>
>> HTTP/1.0 200 OK<0d>
>> Date: Sun, 08 Jun 2014 13:54:09 GMT<0d>
>> Server: ufdbICAPd/1.0<0d>
>> Last-Modified: Sun, 08 Jun 2014 13:54:09 GMT<0d>
>> ETag: "498a-00000001-5394572c-4567"<0d>
>> Cache-Control: max-age=10<0d>
>> Content-Length: 0<0d>
>> Content-Type: text/html<0d>
>> <0d>
>> 0<0d>
>> <0d>
>>
>>
>> CONNECT/HTTPS does not work:
>>
>> Squid writes:
>> REQMOD icap://127.0.0.1:1344/reqmod_icapd_squid34 ICAP/1.0<0d>
>> Host: 127.0.0.1:1344<0d>
>> Date: Sun, 08 Jun 2014 12:29:32 GMT<0d>
>> Encapsulated: req-hdr=0, null-body=87<0d>
>> Preview: 0<0d>
>> Allow: 204<0d>
>> X-Client-IP: 127.0.0.1<0d>
>> <0d>
>> CONNECT googleads.g.doubleclick.net:443 HTTP/1.0<0d>
>> User-Agent: Wget/1.12 (linux-gnu)<0d>
>> <0d>
>>
>> ICAP daemon responds:
>> ICAP/1.0 200 OK<0d>
>> Server: ufdbICAPd/1.0<0d>
>> Date: Sun, 08 Jun 2014 12:29:32 GMT<0d>
>> ISTag: "5394572c-4567"<0d>
>> Connection: keep-alive<0d>
>> Encapsulated: res-hdr=0, null-body=193<0d>
>> X-Next-Services: <0d>
>> <0d>
>> CONNECT blockedhttps.urlfilterdb.com:443 HTTP/1.0<0d> --
>> NOTE: also fails: CONNECT https://blockedhttps.urlfilterdb.com HTTP/1.0<0d>
>> Host: blockedhttps.urlfilterdb.com<0d>
>> User-Agent: Wget/1.12 (linux-gnu)<0d>
>> X-blocked-URL: googleads.g.doubleclick.net<0d>
>> X-blocked-category: ads<0d>
>> <0d>
>> 0<0d>
>> <0d>
>>
>> and Squid in the end responds to wget:
>> HTTP/1.1 500 Internal Server Error
>> Server: squid/3.4.5
>> Mime-Version: 1.0
>> Date: Sun, 08 Jun 2014 13:59:27 GMT
>> Content-Type: text/html
>> Content-Length: 2804
>> X-Squid-Error: ERR_ICAP_FAILURE 0
>> Vary: Accept-Language
>> Content-Language: en
>> X-Cache: MISS from XXX
>> X-Cache-Lookup: NONE from XXX:3128
>> Via: 1.1 XXX (squid/3.4.5)
>> Connection: close
>>
>> A fragment of cache.log is below.
>> I think that the line
>> HttpReply.cc(460) sanityCheckStartLine: HttpReply::sanityCheckStartLine:
>> missing protocol prefix (HTTP/) in 'CONNECT blockedhttps.urlfilterdb.com:443
>> HTTP/1.0<0d>
>> indicates where the problem is.
Received on Sun Jun 08 2014 - 19:20:10 MDT