RE: [squid-users] Multisite ICP peering

OK here are my configs for anyone interested.

Just to explain.

Primary site:
Web -> Squid primary caches -> Check all Squid siblings (primary site only) -> Primary Webservers

Secondary site:
Web -> squid dr caches -> Check all Squid siblings (both sites as this site is only a warm cache) -> dr webservers

Hardware: IBM x3650 M2
74gb memory
10 x 50gb SSD drives (one for each of /var/spool/squid0 to squid9)

So any request that hits primary checks the 3 local squid caches and then the origin servers
Any request that hits secondary check both secondary servers, then the 3 primary squid cache (dark fibre DC links) and then the DR origin servers.

Been nailing these boxes all morning with httperf log replays. Initally I got a lot of misses but now I am getting around 75-85% hit ratio based on last weeks apache logs.

Here are my configs:

This is the initial configuration that successfully fulfilled all criteria. The secondary site section is substituted out for the DR site.

============
PRIMARY SITE
============

acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32
acl localnet src 172.x.0.0/12 # RFC1918 possible internal network

acl admins src 172.x.x.x
acl admins src 172.x.x.x

acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT

http_access allow manager localhost
http_access allow manager admins
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny to_localhost

http_access allow localnet

acl allowed_sites dstdomain "/etc/squid/allowed_sites.conf"
http_access allow allowed_sites

http_access allow localhost
http_access deny all

icp_access allow localnet
icp_access deny all

http_port 80 accel vhost defaultsite=mywebsite.com

cache_peer 172.x.x.245 parent 80 0 no-query originserver round-robin name=webserver011 monitorurl=http://mywebsite.com/img/noPhoto.gif monitortimeout=10 connect-timeout=5 monitorinterval=5 no-digest
cache_peer 172.x.x.247 parent 80 0 no-query originserver round-robin name=webserver021 monitorurl=http://mywebsite.com/img/noPhoto.gif monitortimeout=10 connect-timeout=5 monitorinterval=5 no-digest
cache_peer 172.x.x.248 parent 80 0 no-query originserver round-robin name=webserver031 monitorurl=http://mywebsite.com/img/noPhoto.gif monitortimeout=10 connect-timeout=5 monitorinterval=5 no-digest
cache_peer 172.x.x.249 parent 80 0 no-query originserver round-robin name=webserver041 monitorurl=http://mywebsite.com/img/noPhoto.gif monitortimeout=10 connect-timeout=5 monitorinterval=5 no-digest

cache_peer 172.x.x.238 sibling 80 3130 name=pri-squid011 connect-timeout=5 multicast-siblings
cache_peer 172.x.x.188 sibling 80 3130 name=pri-squid021 connect-timeout=5 multicast-siblings
cache_peer 172.x.x.205 sibling 80 3130 name=pri-squid031 connect-timeout=5 multicast-siblings

cache_peer 239.128.0.112 multicast 80 3130 ttl=16

cache_peer_access webserver011 allow allowed_sites
cache_peer_access webserver021 allow allowed_sites
cache_peer_access webserver031 allow allowed_sites
cache_peer_access webserver041 allow allowed_sites
cache_peer_access pri-squid011 allow allowed_sites
cache_peer_access pri-squid021 allow allowed_sites
cache_peer_access pri-squid031 allow allowed_sites

hierarchy_stoplist cgi-bin

cache_mem 64 GB
maximum_object_size_in_memory 100 KB
memory_replacement_policy lru
cache_replacement_policy heap LFUDA

cache_dir aufs /var/spool/squid0 36864 16 256
cache_dir aufs /var/spool/squid1 36864 16 256
cache_dir aufs /var/spool/squid2 36864 16 256
cache_dir aufs /var/spool/squid3 36864 16 256
cache_dir aufs /var/spool/squid4 36864 16 256
cache_dir aufs /var/spool/squid5 36864 16 256
cache_dir aufs /var/spool/squid6 36864 16 256
cache_dir aufs /var/spool/squid7 36864 16 256
cache_dir aufs /var/spool/squid8 36864 16 256
cache_dir aufs /var/spool/squid9 36864 16 256

maximum_object_size 50 MB

cache_swap_low 90
cache_swap_high 95

logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh access_log /var/log/squid/squid.log combined all

access_log /var/log/squid/access.log squid

pid_filename /var/run/squid.pid

strip_query_terms off
buffered_logs on

refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320

quick_abort_min 0 KB
quick_abort_max 0 KB

negative_ttl 0 minutes
positive_dns_ttl 5 minutes

acl shoutcast rep_header X-HTTP09-First-Line ^ICY.[0-9]
upgrade_http0.9 deny shoutcast

acl apache rep_header Server ^Apache
broken_vary_encoding allow apache

collapsed_forwarding on
refresh_stale_hit 10 seconds

read_timeout 1 minute
request_timeout 1 minute
half_closed_clients off
shutdown_lifetime 5 seconds

cache_mgr unix_at_mywebsite.com

httpd_suppress_version_string on

icp_port 3130
log_icp_queries on
icp_hit_stale on

mcast_groups 239.128.0.112

max_filedescriptors 16384
hosts_file /etc/hosts
memory_pools off
forwarded_for on
client_db off
coredump_dir /var/spool/squid

==============
SECONDARY SITE
==============

This section of the configuration is substantially different to the primary site as the squid caches will reference all siblings across both sites before going to the secondary site backend webservers.

cache_peer 172.x.1.166 parent 80 0 no-query originserver round-robin name=dr-webserver011 monitorurl=http://mywebsite.com/img/noPhoto.gif monitortimeout=10 connect-timeout=5 monitorinterval=5 no-digest default
cache_peer 172.x.1.167 parent 80 0 no-query originserver round-robin name=dr-webserver021 monitorurl=http://mywebsite.com/img/noPhoto.gif monitortimeout=10 connect-timeout=5 monitorinterval=5 no-digest default
cache_peer 172.x.1.168 parent 80 0 no-query originserver round-robin name=dr-webserver031 monitorurl=http://mywebsite.com/img/noPhoto.gif monitortimeout=10 connect-timeout=5 monitorinterval=5 no-digest default
cache_peer 172.x.1.169 parent 80 0 no-query originserver round-robin name=dr-webserver041 monitorurl=http://mywebsite.com/img/noPhoto.gif monitortimeout=10 connect-timeout=5 monitorinterval=5 no-digest default

cache_peer 172.26.22.152 multicast 80 3130 name=dr-squid011 connect-timeout=5 multicast-siblings
cache_peer 172.26.22.153 multicast 80 3130 name=dr-squid021 connect-timeout=5 multicast-siblings
cache_peer 172.26.26.238 multicast 80 3130 name=pri-squid011 connect-timeout=5 multicast-siblings
cache_peer 172.26.26.188 multicast 80 3130 name=pri-squid021 connect-timeout=5 multicast-siblings
cache_peer 172.26.26.205 multicast 80 3130 name=pri-squid031 connect-timeout=5 multicast-siblings

cache_peer 239.128.0.112 multicast 80 3130 ttl=16

cache_peer_access dr-squid011 allow allowed_sites
cache_peer_access dr-squid021 allow allowed_sites
cache_peer_access pri-squid011 allow allowed_sites
cache_peer_access pri-squid021 allow allowed_sites
cache_peer_access pri-squid031 allow allowed_sites
cache_peer_access dr-webserver011 allow allowed_sites
cache_peer_access dr-webserver021 allow allowed_sites
cache_peer_access dr-webserver031 allow allowed_sites
cache_peer_access dr-webserver041 allow allowed_sites

-----Original Message-----
From: Amos Jeffries [mailto:squid3_at_treenet.co.nz]
Sent: Tuesday, 9 November 2010 5:35 PM
To: squid-users_at_squid-cache.org
Subject: Re: [squid-users] Multisite ICP peering

On 03/11/10 21:53, Chris Toft wrote:
> Thanks for the reply, I actually fixed it. Removed the multicast-responder option and just left multicast-sibling.
>
> Man this thing flies on 5 boxes with 64gb memory and 10x 50gb solid state drives for the cache :-)
>
> I will post working config tomorrow for anyone interested.
>

Interested :) please post.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.9
   Beta testers wanted for 3.2.0.3
The information contained in this e-mail message and any accompanying files is or may be confidential. If you are not the intended recipient, any use, dissemination, reliance, forwarding, printing or copying of this e-mail or any attached files is unauthorised. This e-mail is subject to copyright. No part of it should be reproduced, adapted or communicated without the written consent of the copyright owner. If you have received this e-mail in error please advise the sender immediately by return e-mail or telephone and delete all copies. Fairfax does not guarantee the accuracy or completeness of any information contained in this e-mail or attached files. Internet communications are not secure, therefore Fairfax does not accept legal responsibility for the contents of this message or attached files.