Dear Amos,
Thank you for your reply, check my comments:
Amos Jeffries wrote:
> Haytham KHOUJA (devnull) wrote:
>> Hello,
>> The purpose of this thread is to join forces to have the best Squid
>> configuration for generic affordable Intel machines available by
>> major vendors (Dell/HP...) specifically for ISPs and corporations
>> that want a basic setup but with optimal response and throughput and
>> maximizing bandwidth savings.
>> I work for an important ISP and I currently replaced 2 NetApp
>> NetCache with 3 Dell 2950 hooked up on a Foundry Switch for Load
>> Balancing.
>> I used tproxy to enable IP Spoofing to IP spoofing the outgoing
>> address with some configurations on the Cisco core router, I had to
>> compile iptables and tproxy on a Debian kernel source (2.6.18)
>>
>> I've read almost every single thread on Optimizing Squid and Linux
>> and want to share my setup with you.
>> I do have some questions, clarifications and bugs but overall the
>> performance is pretty impressive. (Yes, much better than the NetApps)
>>
>> What i want to do is since i have 8 GB of RAMs, i want to store more
>> hot objects in the RAMs to maximize Memory hit ratio, but with my
>> setup, Squid doesn't
>> go above 2GB~3GB of usage. (Remember, that there are no other heavy
>> processes on the machine).
>
> You will need a 64-bit enabled squid to go higher than 2GB.
>
Yea, I hope i'll be able to replace the CPUs
>>
>> If i knew beforehand that Squid doesn't make use of SMP, i wouldn't
>> have bought Dual Quad Core and would have invested in Intel CPUs with
>> 8mb of Cache, but what's done is done :)
>>
>> Before i had Squid go down because of File Delimiters and maximum
>> open files and ip_conntrac fill up, i fixed both with some iptables
>> and sysctl configuration.
>> Now i'm hitting a "Oct 14 01:17:06 proxy4 squid[8883]: assertion
>> failed: diskd/store_io_diskd.c:384:
>> "!diskdstate->flags.close_request" Error, so Squid kills and restarts
>> (which flushes the Memory cache).
>>
>> I'm looking forward for some contributions, idea sharing, knowledge
>> correcting to make this setup a standard setup for large scale, well
>> optimized and high performant Squid for future tweakings. I hope this
>> configuration would be then uploaded to the Squid wiki.
>
> Post your squid.conf to
> http://squid.treenet.co.nz/cf.check/
> and review the results. I've pointed out the biggest worries below.
>
>
>> Here's my setup:
>> Dell 2950
>> Dual Quad Core 2.4Ghz / 8 GB Rams / 4x 136 GB 15000 RPM drives
>>
>> I have 3 cache_dir on separate drives and I formated the 3 disks with
>> ReiserFS:
>> /dev/sdb1 /CACHE1 reiserfs notail,noatime 0 0
>> /dev/sdc1 /CACHE2 reiserfs notail,noatime 0 0
>> /dev/sdd1 /CACHE3 reiserfs notail,noatime 0 0
>>
>> I run Debian GNU/Linux Etch and compiled Squid with the following:
>> Squid Cache: Version 2.6.STABLE16
>> configure options: '--bindir=/usr/bin' '--sbindir=/usr/sbin/'
>> '--sysconfdir=/etc' '--enable-icmp' '--enable-snmp'
>> '--enable-async-io' '--enable-linux-netfilter'
>> '--enable-linux-tproxy' '--with-dl' '--with-large-files'
>> '--enable-large-cache-files' '--with-maxfd=1000000'
>> '--enable-storeio=diskd,ufs' '--with-aio' '--enable-epoll'
>> '--disable-ident-lookups' '--enable-removal-policies=heap'
>> 'CFLAGS=-DNUMTHREADS=120'
>>
>> As you can see i have the following modules enabled: linux-tproxy,
>> diskd, epoll, and removal policies.
>> /dev/epoll improves network I/O performance, Diskd separates disk I/O
>> to separate processes (which reduces process locking from Squid to
>> write on disks), and read benchmarks for memory and disk removal
>> policies.
>
> aufs does a better job, particularly where threads are available and
> is not quite so broken as diskd.
I will recompile, use aufs and do more testing
>> My /etc/squid.conf is composed of the following:
>>
>> http_port 80 transparent tproxy
>> tcp_outgoing_address IP of the Machine
>> :: Those are for IP Spooding and Transparency
>>
>> via off
>> forwarded_for off
>> :: Those are for total transparency, remote hosts will never guess
>> that the request came from a proxy
>
> IIRC, theres more than this needed for complete silence. They just
> replace the Via and Forwarded-For with text 'unknown'. still leaving
> the headers in place for anon-proxy identification.
>
True, but this is used with tproxy for ip spoofing
>>
>> cache_mem 600 MB
>> :: A bit confused about this, When i go higher than 2GB, Squid kills
>> with a "out of memory" error. I have 8GB and want to maximize the use
>> of it.
>>
>> cache_effective_user nobody
>> cache_effective_group nogroup
>> :: Security and bla bla
>
So i can leave it to 2GB maximum? The rest of the OS will have the rest
of the RAM for OS purposes.
> This is the default UID. If this is going to be a standard config
> these MUST not be explicitly set.
> Also when GID is configured as above, will in fact cause a
> squid-specific deviation from the configured OS-level security policy.
>
> They are no longer to be used, unless the machine-specific setup
> requires it AND the admin knows how to setup for them properly.
>
>> cache_replacement_policy heap LFUDA
>> memory_replacement_policy heap GDSF
>> :: Very objective, you can google about them
>>
>> cache_dir diskd /CACHE1 61440 16 256 Q1=144 Q2=128
>> cache_dir diskd /CACHE2 61440 16 256 Q1=144 Q2=128
>> cache_dir diskd /CACHE3 61440 16 256 Q1=144 Q2=128
>> :: DISKD configuration, i'm only using 60GB of each disk
>>
>> cache_access_log /var/log/squid/access.log
>
> Obsolete option. Use access_log with same parameters instead.
>
Which is obsolete?
>> cache_log /var/log/squid/cache.log
>> cache_store_log none
>> :: No need to log cache_store, so minimizing the Disk I/O
>>
>> fqdncache_size 51200
>> ipcache_size 51200
>> :: Caching IPs/Domain Name and whatnot
>>
>> pipeline_prefetch on
>> :: Performance enhancement
>>
>> shutdown_lifetime 1 second
>> :: Tired to wait whenever i restart my Squids (Only on testing)
>>
>> read_ahead_gap 60 KB
>> maximum_object_size 2 GB
>> minimum_object_size 0 KB
>> maximum_object_size_in_memory 128 KB
>> cache_swap_high 80%
>> cache_swap_low 70%
>> half_closed_clients off
>> memory_pools on
>> positive_dns_ttl 24 hours
>> negative_dns_ttl 30 seconds
>> request_timeout 60 seconds
>> connect_timeout 30 seconds
>> pconn_timeout 30 seconds
>> ie_refresh on
>> dns_nameservers DNS1 DNS2
>> emulate_httpd_log off
>> log_ip_on_direct on
>> debug_options ALL, 9
>
> performance enhancements above to minimize disk IO yet you log
> everything at full-debug? this *,9 could cause extremely high disk
> usage under load. Try *,1 (minimal) or *,5 (detailed overview) instead.
>
Will do, thanks
>> pid_filename /var/run/squid.pid
>>
>> My IPtables/sysctl and startup file:
>> #!/bin/sh
>> iptables -t tproxy -A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j
>> TPROXY --on-port 80
>> :: I run Squids on port 80 so that i can forward all incoming
>> requests on port 80 to the Squids on the Cisco router level
>>
>> echo 1 > /proc/sys/net/ipv4/ip_forward
>> echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
>> echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
>> echo 1024 65535 > /proc/sys/net/ipv4/ip_local_port_range
>> echo 102400 > /proc/sys/net/ipv4/tcp_max_syn_backlog
>> echo 1000000 > /proc/sys/net/ipv4/ip_conntrack_max
>> echo 1000000 > /proc/sys/fs/file-max
>> echo 60 > /proc/sys/kernel/msgmni
>> echo 32768 > /proc/sys/kernel/msgmax
>> echo 65536 > /proc/sys/kernel/msgmnb
>> :: Maximizing Kernel configuration
>>
>> ulimit -HSn 1000000
>> /etc/init.d/squid stop
>> /etc/init.d/squid start
>> :: Re-enforcing ulimit parameters for the Squid process.
>>
>> Thank you
>
> No, thank you.
>
> Amos
>
Received on Sun Oct 14 2007 - 07:25:26 MDT
This archive was generated by hypermail pre-2.1.9 : Thu Nov 01 2007 - 13:00:01 MDT