Re: [squid-users] Re: how distribute squid loads to cpus and memories using SMP feature??

From: Amos Jeffries <squid3_at_treenet.co.nz>
Date: Sat, 26 Oct 2013 16:16:54 +1300

On 26/10/2013 7:21 a.m., firecold wrote:
> Mi squid.conf lo tengo de esta manera:
>
> #========================== Squid 3.x Conf ===========================#
> #----------------------------------------------------------------------
> # Opciones de SQUID 3.x
> #----------------------------------------------------------------------
> http_port 3128 intercept
> http_port 3129
> cache_mgr Firecold
> visible_hostname proxy.os.com
> append_domain .proxy.os.com
> pinger_enable off
> workers 4
> dns_v4_first on
> cpu_affinity_map process_numbers=1,2,3,4 cores=1,2,3,4
> #----------------------------------------------------------------------
> # Servidor DNS y Politica de Cambios
> #----------------------------------------------------------------------
> dns_nameservers 127.0.0.1 200.49.160.35 8.8.8.8
> dns_retransmit_interval 5 seconds
> dns_timeout 2 minutes
> #----------------------------------------------------------------------
> acl Safe_ports port 80 82 84 86 # http
> acl Safe_ports port 21 # ftp
> acl SSL_ports port 443
> acl Safe_ports port 443 # https
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl Safe_ports port 631 # cups
> acl Safe_ports port 873 # rsync
> acl Safe_ports port 901 # SWAT
> acl Safe_ports port 1863 # MSN
> #acl SSL_ports port 443 # https
> acl SSL_ports port 563 # snews
> acl SSL_ports port 873 # rsync
> acl CONNECT method CONNECT
> #----------------------------------------------------------------------
> acl accesototal src "/etc/squid3/accesototal.txt"
> acl mired src "/etc/squid3/mired.txt"
> acl denegados url_regex -i "/etc/squid3/denegados.lst"
> acl magic_words1 url_regex -i 192.168.1
> acl magic_words2 url_regex -i .exe .mp3 .zip .rar .avi .mpeg .mpe .mpg .wav
> .mov .3gp .mov .flv .mp2 .mp5 .aac .wma .ogg .mka .asf .iff .amv

Problem #1: These regex do not do what you think.

The 3-letter patterns match *anywhere* in the URL from the second byte
onwards.
For example: http://example.com/movies/difflv.ico?jpg=no&rarify"
matches any one of: .mov .iff .rar

If you are wanting this to match "file" extensions use patterns like
those in your refresh_pattern lines.

> #----------------------------------------------------------------------
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access allow manager localhost
> http_access deny manager all
> #----------------------------------------------------------------------
> http_access allow localhost
> http_access allow accesototal
> http_access allow mired !denegados
> http_access deny all
> reply_body_max_size 200 MB mired
> #----------------------------------------------------------------------
> coredump_dir /home/squid3/squid
> #----------------------------------------------------------------------
> # Memoria reservada para cache
> # Se recomienda que dedique aprox. 5MB de RAM por cada 1GB asignado a
> cache_dir

Problem #2: cache_dir line has no options.

> #----------------------------------------------------------------------
> cache_mem 8192 MB
> maximum_object_size_in_memory 200 MB
> minimum_object_size 0 KB
> maximum_object_size 10 MB
> #----------------------------------------------------------------------
> # Sustituir archivos de cache cuando llegue a 96%
> #----------------------------------------------------------------------
> cache_swap_low 92
> cache_swap_high 96
> #----------------------------------------------------------------------
> # Total de espacio en HD a ser usado por el cache, numero de carpetas,
> # numero de subcarpetas en cache
> # 100000 = 100 GB
> #----------------------------------------------------------------------
> cache_dir aufs /var/spool/squid3/squid0${process_number} 10000 16 256
> min-size=3100 max-size=90000
> cache_dir aufs /var/spool/squid3/squid0${process_number} 10000 16 256
> min-size=3100 max-size=90000
> cache_dir aufs /var/spool/squid3/squid0${process_number} 10000 16 256
> min-size=3100 max-size=90000
> cache_dir aufs /var/spool/squid3/squid0${process_number} 10000 16 256
> min-size=3100 max-size=90000
> #----------------------------------------------------------------------
> # Estandar de actualización de cache
> # 1 mes = 10080 mins, 1 dia = 1440 mins
> #----------------------------------------------------------------------
> refresh_pattern -i \.jpg$ 14400 80% 43200 refresh-ims
> refresh_pattern -i \.gif$ 14400 80% 43200 refresh-ims
> refresh_pattern -i \.png$ 14400 80% 43200 refresh-ims
> refresh_pattern -i \.jpeg$ 14400 80% 43200 refresh-ims
> refresh_pattern -i \.bmp$ 14400 80% 43200 refresh-ims
> refresh_pattern -i \.tif$ 14400 80% 43200 refresh-ims
> refresh_pattern -i \.tiff$ 14400 80% 43200 refresh-ims
>
> refresh_pattern -i \.swf$ 14400 80% 43200 refresh-ims

NP: if you are not debugging the pattern matching it will make your
Squid a lot faster to ccombine the above lines with a single pattern.
Same for the following rules in groups by the directive parameters.

> refresh_pattern -i \.html$ 10 20% 4320 refresh-ims
> refresh_pattern -i \.htm$ 10 20% 4320 refresh-ims
> refresh_pattern -i \.shtml$ 10 20% 4320 refresh-ims
> refresh_pattern -i \.shtm$ 10 20% 4320 refresh-ims
> refresh_pattern -i \.nub$ 2880 80% 21600 refresh-ims
> refresh_pattern -i \.exe$ 14400 80% 43200
> refresh_pattern -i \.zip$ 14400 80% 43200
> refresh_pattern -i \.mov$ 14400 80% 43200
> refresh_pattern -i \.mpe?g?$ 14400 80% 43200
> refresh_pattern -i \.avi$ 14400 80% 43200
> refresh_pattern -i \.qtm?$ 14400 80% 43200
> refresh_pattern -i \.viv$ 14400 80% 43200
> refresh_pattern -i \.wav$ 14400 80% 43200
> refresh_pattern -i \.aiff?$ 14400 80% 43200
> refresh_pattern -i \.au$ 14400 80% 43200
> refresh_pattern -i \.ram?$ 14400 80% 43200
> refresh_pattern -i \.snd$ 14400 80% 43200
> refresh_pattern -i \.mid$ 14400 80% 43200
> refresh_pattern -i \.mp2$ 14400 80% 43200
> refresh_pattern -i \.mp3$ 14400 80% 43200
> refresh_pattern -i \.sit$ 14400 80% 43200
> refresh_pattern -i \.zip$ 14400 80% 43200
> refresh_pattern -i \.hqx$ 14400 80% 43200
> refresh_pattern -i \.arj$ 14400 80% 43200
> refresh_pattern -i \.lzh$ 14400 80% 43200
> refresh_pattern -i \.lha$ 14400 80% 43200
> refresh_pattern -i \.cab$ 14400 80% 43200
> refresh_pattern -i \.rar$ 14400 80% 43200
> refresh_pattern -i \.tar$ 14400 80% 43200
> refresh_pattern -i \.gz$ 14400 80% 43200
> refresh_pattern -i \.z$ 14400 80% 43200
> refresh_pattern -i \.a[0-9][0-9]$ 14400 80% 43200
> refresh_pattern -i \.r[0-9][0-9]$ 14400 80% 43200
> refresh_pattern -i \.txt$ 14400 80% 43200
> refresh_pattern -i \.pdf$ 14400 80% 43200
> refresh_pattern -i \.doc$ 14400 80% 43200
> refresh_pattern -i \.rtf$ 14400 80% 43200
> refresh_pattern -i \.tex$ 14400 80% 43200
> refresh_pattern -i \.latex$ 14400 80% 43200
> refresh_pattern -i \.class$ 14400 80% 43200
> refresh_pattern -i \.js$ 14400 80% 43200
> refresh_pattern -i \.ico$ 14400 80% 43200
> refresh_pattern -i \.css$ 10 20% 4320
> #----------------------------------------------------------------------
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
> #refresh_pattern (Release|Package(.gz)*)$ 0 20% 2880

Yes the above Debian-specific rule is useless for Squid-3. Debian
repository content is cacheable and Squid-3 does cache it by default.

> refresh_pattern . 0 20% 4320
> #----------------------------------------------------------------------
> # Log de acessos por el cache o para SARG
> #----------------------------------------------------------------------
> logfile_rotate 7
> access_log stdio:/var/log/squid3/access.log
> cache_log /var/log/squid3/cache.log
> cache_store_log none
> #----------------------------------------------------------------------
> # Otras configuraciones
> #----------------------------------------------------------------------
> server_persistent_connections off
> client_persistent_connections off
> quick_abort_min 0 KB
> quick_abort_max 0 KB
> quick_abort_pct 95
> fqdncache_size 65535
> cache_effective_user proxy
> cache_effective_group proxy
> ipcache_size 65535
> ipcache_low 98
> ipcache_high 99
> #
> -----------------------------------------------------------------------------
> # TIMEOUTS
> #
> -----------------------------------------------------------------------------
> forward_timeout 240 seconds
> connect_timeout 60 seconds
> peer_connect_timeout 30 seconds
> read_timeout 900 seconds
> request_timeout 120 seconds
> persistent_request_timeout 60 seconds
> client_lifetime 60 minutes
> half_closed_clients off
> pconn_timeout 60 seconds
> shutdown_lifetime 20 seconds
> #----------------------------------------------------------------------
> # Manteniendo objetos recientes y pequeños en memoria
> #----------------------------------------------------------------------
> memory_replacement_policy heap GDSF
> cache_replacement_policy heap LFUDA
> #----------------------------------------------------------------------
> # Sitios que se les niega el cache
> #----------------------------------------------------------------------
> acl nocache dstdomain .4shared.com .youtube.com .windowsupdate.com .gl$
> .yimg.com .cemaco.com 192.168.0.254 internet.tigo.com.gt
> no_cache deny nocache
> #----------------------------------------------------------------------
> # Negar cache para archivos con extension .asx e .asf |streaming|
> #----------------------------------------------------------------------
> acl asx url_regex -i \.asx$
> cache deny asx
> acl asf url_regex -i \.asf$
> cache deny asf

You may as well combine these into one pattern to halve the CPU consumed
by these rules. urlpath_regex also restricts the matching location to
the path portion of URLs, for better accuracy and even faster match.

   acl asfx urlpath_regex -i \.as[fx]$
   cache deny asfx

> #
> -----------------------------------------------------------------------------
> # Qos
> #
> -----------------------------------------------------------------------------
> qos_flows local-hit=0x30
> qos_flows parent-hit=0x32
> qos_flows disable-preserve-miss
> #
> -----------------------------------------------------------------------------
> # SNMP
> #
> -----------------------------------------------------------------------------
> snmp_port 3401
> acl snmppublic snmp_community public
> snmp_access allow snmppublic all
> snmp_incoming_address 0.0.0.0
> snmp_outgoing_address 255.255.255.255
> #
> -----------------------------------------------------------------------------
> # DELAY POOL PARAMETERS
> #
> -----------------------------------------------------------------------------
> delay_pools 3
> delay_class 1 2
> delay_parameters 1 -1/-1 -1/-1
> delay_access 1 allow accesototal

This pool is not useful. All it does is waste memory and CPU cycles
tracking pool usage that is never needed.

Instead of this, adjust the other pools access rules as below... note
that the deny is done before the allow rules below.

> #
> -----------------------------------------------------------------------------
> delay_class 2 2
> #-1/-1 significa que no hay limites.
> delay_parameters 2 -1/-1 -1/-1
delay_access 2 deny accesototal

> delay_access 2 allow mired magic_words1
> #
> -----------------------------------------------------------------------------
> delay_class 3 2
> delay_parameters 3 55000/55000 55000/55000

delay_access 3 deny accesototal

> delay_access 3 allow mired magic_words2
> #----------------------------------------------------------------------
> acl raptor_lst url_regex -i "/etc/raptor/raptor.lst"
> cache deny raptor_lst
> cache_peer 192.168.1.1 parent 8080 0 proxy-only no-digest
> dead_peer_timeout 2 seconds
> cache_peer_access 192.168.1.1 allow raptor_lst
> cache_peer_access 192.168.1.1 deny all
> #----------------------------------------------------------------------

Amos
Received on Sat Oct 26 2013 - 03:17:06 MDT

This archive was generated by hypermail 2.2.0 : Sat Oct 26 2013 - 12:00:06 MDT