Re: [squid-users] Squid and Jesred configuration on Ubuntu Hardy Heron

On Wed, 9 Dec 2009 08:01:59 -0800, N <ng.subs_at_gmail.com> wrote:
> Hello,
>
> Would appreciate your help with my Ubuntu EC2 instance Squid
> configuration. Until recently I had a single dedicated Fedora server
> for my development needs. Currently I am trying to migrate to Amazon
> EC2 based Ubuntu Hardy Heron instance for my server requirements.
>
> Background:
> - Single Ubuntu EC2 instance to manage multiple services/domains
> running on different ports
> - Squid is configured on port 80, to intercept and manage all
> HTTP/HTTPS requests. Port 80 is the only publicly accessible port in
> this EC2 instance.
> - Squid utilizes jesred to redirect all incoming HTTP requests to the
> appropriate services in the instance. Some services/domains are Apache
> websites, and some are Rails and JBoss applications.
>

Firstly take a read through these. It covers the exemplar 'correct'
virtual-hosting setup:
  http://wiki.squid-cache.org/ConfigExamples/Reverse/VirtualHosting

and this covers the extension of handling multiple web servers behind one
Squid:
  http://wiki.squid-cache.org/ConfigExamples/Reverse/MultipleWebservers

Note that ACLs are used for fine-grained request routing which goes beyond
the simple old domain name per cache_peer.

I seriously advise you take a good look at converting your redirector
ruleset to a set of Squid ACL and using the cache_peer_access directive to
route requests around. It will simplify the software tool chain and prevent
all the side effects of domain information leakage from URLs in the HTML
and other headers.

> Configuration so far:
> - installed Apache2
> - install squid (sudo apt-get install squid). squid/2.6.STABLE18 is
> installed at this time.
> - install jesred (sudo apt-get install jesred)
> - edited the squid.conf, jesred.conf & jesred.rules files
>
> Unfortunately all my HTTP requests result in "Unable to forward this
> request at this time" error.
>
> Here are some log snippets when I receive the above error. Just as a
> FYI - when I run 'curl' on local ports the appropriate pages are
> retrieved.
>
> ========= BEGIN log snippets
>
> ==> /var/log/squid/access.log <==
> 08/Dec/2009:16:52:06 0000 5 171.xx.xx.xx TCP_MISS/503 1653 GET
> http://ec2-67-202-5-12.compute-1.amazonaws.com/ - NONE/- text/html
> 08/Dec/2009:16:52:07 0000 0 171.xx.xx.xx TCP_MISS/503 1675 GET
> http://ec2-67-202-5-12.compute-1.amazonaws.com/... - NONE/- text/html
>
> ==> /var/log/squid/cache.log <==
> 2009/12/08 16:52:06| Failed to select source for
'http://127.0.0.1:10001/'

NOTE: the "domain name" 127.0.0.1:10001 is not listed as a valid domain
anywhere in the config you posted.

Jesred appears to be creating it.

> 2009/12/08 16:52:06| always_direct = 0
> 2009/12/08 16:52:06| never_direct = 0
> 2009/12/08 16:52:06| timedout = 0
> 2009/12/08 16:52:07| Failed to select source for
> 'http://127.0.0.1:10001/favicon.ico'
> 2009/12/08 16:52:07| always_direct = 0
> 2009/12/08 16:52:07| never_direct = 0
> 2009/12/08 16:52:07| timedout = 0
>
> ==> /var/log/squid/jesred-rewrite.log <==
> 1260291126.849 171.xx.xx.xx/-
> http://ec2-67-202-5-12.compute-1.amazonaws.com/
> http://127.0.0.1:10001/ 1
> 1260291127.032 171.xx.xx.xx/-
> http://ec2-67-202-5-12.compute-1.amazonaws.com/...
> http://127.0.0.1:10001/favicon.ico 1
>
> ========== END log snippets
>
> Following is my current squid.conf definition
>
> ========== BEGIN /etc/squid/squid.conf definition
>
> http_port 80 accel vhost
> defaultsite=ec2-174-129-18-120.compute-1.amazonaws.com vport
>
> # ec2 instance hostname command value is used
> visible_hostname domU-12-31-39-0A-50-C7.compute-1.internal
>
> tcp_outgoing_address 127.0.0.1
>
> # error_directory /usr/share/squid/errors/English
> error_directory /etc/squid/errors/English
>
> hierarchy_stoplist cgi-bin ?
> hierarchy_stoplist banner
>
> acl QUERY urlpath_regex cgi-bin \?
> cache deny QUERY
> cache_store_log none
>
> acl apache rep_header Server ^Apache
> broken_vary_encoding allow apache
>
> cache_mem 128 MB
>
> #maximum_object_size 4096 KB
> #uncomment next line and comment previous to disable caching:
> maximum_object_size 0 KB
>
> minimum_object_size 0 KB
> maximum_object_size_in_memory 8 KB
>
> cache_dir ufs /var/spool/squid 1024 16 256

On Ubuntu you get faster caching with AUFS.
(Nothing else beyond a reconfigure needed to change it)

> logformat squid %tl %6tr %>a %Ss/%03Hs %<st %rm %ru %un %Sh/%<A %mt

The "squid" default log format is pre-defined and does not need to be
added to the config.
If you have not changed it please use the internal form. If you have
changed it please use another name.

> access_log /var/log/squid/access.log squid
>
> url_rewrite_program /usr/lib/squid/jesred
> url_rewrite_children 10
> url_rewrite_host_header on
>
> refresh_pattern . 0 20% 4320
>
> ############### begin access controls
>
> acl MyNetwork src localhost
> acl myLocalhost src 127.0.0.1/255.255.255.255

acl myLocalhost src 127.0.0.1

> acl to_localhost dst 127.0.0.0/8

We now advise for extra security with some broken client softwares:
 acl to_localhost dst 127.0.0.0/8 0.0.0.0/32

> acl hostsToBlock src 204.9.65.186
> acl all src 0.0.0.0/0.0.0.0

acl all src all

>
> acl ValidDomains dstdomain .domain1.com .domain2.com .amazonaws.com
>
> acl HTTPports port 80
> acl Safe_ports port 80
> acl Safe_ports port 443
> acl SSL_ports port 443
>
> acl CONNECT method CONNECT
> acl Safe_proto proto HTTP HTTPS
> acl buggy_server url_regex ^http://....
>
> redirector_access deny !ValidDomains
> redirector_access deny hostsToBlock
>
> http_access deny !Safe_proto
> http_access deny !ValidDomains
> http_access deny hostsToBlock
> http_access deny !Safe_ports
> http_access allow MyNetwork
> http_access allow CONNECT SSL_ports
> http_access allow myLocalhost
> http_access deny CONNECT
>
> http_access allow ValidDomains
>
> # no difference with or without the following http_access
> # http_access allow all
>
> ############################ end access controls
>
> nonhierarchical_direct off
> broken_posts allow buggy_server
>
> icp_access allow all
>
> coredump_dir /var/spool/squid
>
>
> ############# Start of cache_peer definitions
>
> cache_peer 127.0.0.1 parent 10001 0 no-query originserver name=tuesday
> default
> cache_peer_domain tuesday ec2-67-202-45-182.compute-1.amazonaws.com
>
> cache_peer 127.0.0.1 parent 10002 0 no-query originserver name=dom1
> cache_peer_domain dom1 www.domain1.com
>
> cache_peer 127.0.0.1 parent 10003 0 no-query originserver name=dom2
> cache_peer_domain dom2 www.domain2.com
>
> ========== END /etc/squid/squid.conf definition
>
> Thanks a bunch.
>
> Cheers.
Received on Thu Dec 10 2009 - 00:05:21 MST