Re: [squid-users] HTTPS forward proxy?

On 2014-01-22 11:44, David Deller wrote:
> Hi Amos, thanks for your reply.
>
> On Jan 21, 2014, at 4:29 PM, Amos Jeffries <squid3_at_treenet.co.nz>
> wrote:
>
>> On 2014-01-22 09:25, David Deller wrote:
>>> Hello,
>>> I’m trying to set up Squid as an HTTPS forward proxy, but I’m having
>>> trouble getting it to work.
>>> Here's some background about my problem:
>>> * I have a web service running on Heroku, with a dynamic IP address.
>>> Static IPs on Heroku are not an option.
>>> * I need to connect to an external web service which is behind a
>>> firewall. The people who operate the external web service will only
>>> open their firewall to a specific static IP.
>>> My attempted solution is to use Squid on a separate server with a
>>> static IP to forward-proxy requests from Heroku to the external
>>> service. That way, the external service always sees the proxy
>>> server's
>>> static IP, instead of the Heroku service's dynamic IP.
>>> Since my proxy server can't rely on an IP address for authentication
>>
>> You mean authorization, not authentication. A number (IP address)
>> cannot own credentials, only an entity can have credentials and
>> credentials are required for authentication.
>
> Sorry - I really do know the difference between the two auths, but it
> sometimes comes out wrong.
>
>>> (that's the problem to begin with!), it must rely on a username and
>>> password. Further, the username and password cannot be transmitted in
>>> clear text,
>>
>> Meaning: avoid Basic authentication type.
>>
>>> because if an attacker were to intercept that clear text,
>>> then they could connect to my proxy pretending to be me, make
>>> outbound
>>> requests using my proxy's static IP, and thus evade the external web
>>> service's firewall.
>>
>> Squid is not limited to the Basic authentication protocol. I see you
>> are already using Digest authentication, which already does what you
>> want even in unencrypted HTTP.
>>
>>> Therefore, the Squid proxy must only accept connections over HTTPS,
>>> not HTTP. (The connection to the external web service might be HTTP
>>> or
>>> HTTPS.)
>>
>> No. Given you already use Digest auth the SSL layer is not necessary
>> to prevent that attack type. It is just an extra layer to obscure the
>> content being transferred. I think it is a good idea to use it of
>> possible, but am aware there are some big troubles (namely that modern
>> browsers still do not support connecting to a proxy over SSL/TLS).
>
> I am distrustful of Digest because it uses MD5. The Wikipedia article
> about it seems to suggest that this may not in fact be a problem,
> despite MD5’s known weaknesses. Some quick Googling does not suggest a
> widespread consensus.
>
>>> I'm running Squid 3.1.10 on CentOS 6.5.x,
>>
>> Please try an upgrade. You can find details about newer versions for
>> CentOS at http://wiki.squid-cache.org/KnowledgeBase/CentOS
>
> I will look into this. I have had problems with third-party
> repositories in the past.
>
>> <snip>
>>> Using this setup, HTTP proxying works fine, but HTTPS proxying does
>>> not.
>>> Here's an HTTP proxy request from a local box:
>>> $ curl --proxy http://my-proxy-server.example:3128 \
>>> --proxy-anyauth --proxy-user redacted:redacted -w '\n' \
>>> http://urlecho.appspot.com/echo?body=OK
>>> OK
>> <snip>
>>> Here's another request, this time with HTTPS:
>>> $ curl --proxy https://my-proxy-server.example:3129 \
>>> --proxy-anyauth --proxy-user redacted:redacted -w '\n' \
>>> http://urlecho.appspot.com/echo?body=OK
>>> curl: (56) Recv failure: Connection reset by peer
>>> Nothing in `access.log` after this one, but in `cache.log`:
>>> 2014/01/20 20:46:15| clientNegotiateSSL: Error negotiating SSL
>>> connection on FD 10: error:1407609C:SSL
>>> routines:SSL23_GET_CLIENT_HELLO:http request (1/-1)
>>
>> See the serverfault response. curl is connecting to the proxy using
>> plain-text instead of SSL.
>
> I want to be able to connect to HTTP servers over the proxy as well,
> so changing the protocol of the URL I’m trying to access (per the
> serverfault comments) will not help me in cases where the remote web
> server does not support HTTPS. Are you saying that Squid does not
> support proxying plain HTTP requests over an HTTPS proxy? I could not
> find any documentation one way or the other about this.
>
> In any case, I tried it with an HTTPS request to be sure, and it still
> failed with the same error (even though the remote web server in this
> case does support HTTPS).

This is because your curl test is using plain-TCP to connect to the
proxy. The URL scheme has nothing to do with the curl<->proxy
connection.

>
>>> Here's the above again, more verbosely:
>>> $ curl -v --proxy https://my-proxy-server.example:3129 \
>>> --proxy-anyauth --proxy-user redacted:redacted -w '\n' \
>>> http://urlecho.appspot.com/echo?body=OK
>>> * Adding handle: conn: 0x7f9a30804000
>>> * Adding handle: send: 0
>>> * Adding handle: recv: 0
>>> * Curl_addHandleToPipeline: length: 1
>>> * - Conn 0 (0x7f9a30804000) send_pipe: 1, recv_pipe: 0
>>> * About to connect() to proxy my-proxy-server.example port 3129
>>> (#0)
>>> * Trying proxy.server.IP.redacted...
>>> * Connected to my-proxy-server.example (proxy.server.IP.redacted)
>>> port 3129 (#0)
>>
>> Note the absence of any mention about SSL/TLS here.
>
> I did notice this and wondered if it might be a problem with curl
> itself. So I also tried similar tests with Google Chrome and a Ruby
> HTTP library called excon, both of which specifically mention support
> of HTTPS proxies. I also tried a few other HTTP libraries that have
> HTTP proxy support but don’t specifically mention HTTPS. Since I saw
> the same failing result with all of them, I went back to trying to
> troubleshoot Squid as the likely source of the problem.

You are using "--proxy https://..." but the documentation I find for
curl only mentions http:// and various socks*:// proxy schemes supported
for that directive. These tests have all strong signs of curl not
supporting https:// in that parameter.

Also, please be clear on this. The connection to the proxy is *not*
HTTP. It is SSL. With a second-layer HTTP connection happening inside
the SSL. Anything which attempts to send bare/plain HTTP requests to the
proxy https_port will get this same termination and log message. You can
confirm that with telnet:

   echo "GET http://example.com/ HTTP/1.0\n\n" | telnet -p 3129
my-proxy-server.example

>
>>> > GET http://urlecho.appspot.com/echo?body=OK HTTP/1.1
>>> > User-Agent: curl/7.30.0
>>> > Host: urlecho.appspot.com
>>> > Accept: */*
>>> > Proxy-Connection: Keep-Alive
>>> >
>>> * Recv failure: Connection reset by peer
>>> * Closing connection 0
>>> curl: (56) Recv failure: Connection reset by peer
>>> Looks like an SSL error.
>>
>> It is. Try using --ssl option for curl. You will also need
>> --proxy-digest to get the Digest auth against the proxy working for
>> curl.
>
> --proxy-anyauth works fine with HTTP, as in my first example. It
> correctly detects that it needs to use Digest auth.
>
> It seems like --ssl refers to the inner HTTP request rather than the
> proxy, but the docs are somewhat unclear.
>
> Regardless, I tried with --ssl --proxy-digest just now, and it failed
> with the same error.

Okay, that seems to confirm your suspicion about --ssl. It seems that
curl is not capable of doing these tests any more than the web browsers.

>
>>> However, I'm reusing a subdomain-wildcard SSL
>>> certificate, shown in the above config as `cert.pem` and `key.pem`,
>>> that I've successfully deployed on other web servers. Moreover,
>>> accessing the proxy server directly with curl works, or at least
>>> establishes a connection past the SSL stage:
>>> $ curl https://my-proxy-server.example:3129
>>> <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
>>> "http://www.w3.org/TR/html4/strict.dtd">
>>> [--SNIP--]
>>
>> This tells us that the test machine can access the web serve. Please
>> run the same test directly on the squid machine. That will tell
>> whether or not connectivity for Squid is working properly.
>
> There is no web server running on that machine other than Squid. The
> request in this example was to Squid directly. Note the (bowdlerized)
> hostname and port which match the proxy in the above examples. It was
> meant to show that there doesn’t seem to be a problem with the SSL
> certificate I have configured with Squid.

You misunderstand. This test is whether curl, running on the same
machine as Squid can perform the same URL request as Squid will be
making on that machine.

Your second test earlier only tested that the web server worked. Not the
connectivity between Squid's machine and that server.

>
>>> Any ideas what I'm doing wrong? Is what I'm attempting even possible?
>>
>> It is possible and being done already by some. Although to get the
>> broken browsers to work properly a VPN/stunnel from the client machine
>> to proxy port is often needed.
>
> I am not particularly concerned about browsers since I am accessing
> the proxy programmatically from a server. I only need an HTTP library
> that can talk to Squid correctly.
>
> I take it, then, that you don’t see any obvious problems with my Squid
> config for what I’m trying to do?

No, not in the squid config. And the connection is not getting far
enough to identify any cert errors either.

You might actually be better going straight to trying your server app or
whatever will be accessing Squid normally. If it can start opening an
SSL connection to the proxy it is already one step better than your
curl.

Alternatively you could try using stunnel to open a SSL tunnel directly
to the proxy https_port and sending curl requests with that tunnel
IP:port as if it were a plain-HTTP proxy.

Amos
Received on Wed Jan 22 2014 - 00:07:05 MST