[squid-users] Problems with squid in a campus setup

Hello everyone,

First of all, my practical experience with squid are as of yet rather limited,
so please bear with me. I couldn't find my specific problem in the FAQ, or
rather if it is in the FAQ I couldn't recognize it as my problem, and google
wasn't helpful either.

Some weeks ago I've been given the task to setup and operate a squid proxy for
roundabout 1500 users. We are managing internet connections for several student
dormitories on a university campus and recently switched from an old-fashioned
volume-based fee to a flat fee. However we misjudged the change in user
behaviour and our 100 MBit uplink was soon congested. The main motivation for
using squid is saving bandwidth and to make the user experience better on
average. For a minimal invasive approach we decided to use an intercept
configuration.

And it's been a rocky ride. Mostly because of a hard to find hardware fault.
The hardware has been replaced and it seemed we have a normal operation now.
Until yesterday that is.

First some information about the setup: the hardware itself is a Xeon E3110
server with 8 GB of RAM and lots of diskspace. OS is CentOS 6.2, a derivate of
Red Hat Enterprise Linux and I'm using the CentOS flavour of Squid, version
squid-3.1.10-1.el6_2.2.x86_64.

Half a TB is planned for squid webobjects with the following line:

cache_dir diskd /var/cache/proxy/squid 512000 16 256 Q1=72 Q2=64

Additional memory for storing objects is 2048 MB:

cache_mem 2048 MB

Squid works in combination with an NGINX proxy setup for caching youtube video
content, as this is probably the greatest bandwith hog. It is configured as a
cache_peer and a regexp acl:

acl youtube_videos url_regex -i ^http://[^/]+(\.youtube\.com|\.googlevideo\.com|\.video\.google\.com)/(videoplayback|get_video|videodownload)\?
acl range_request req_header Range .
acl begin_param url_regex -i [?&]begin=
acl id_param url_regex -i [?&]id=
acl itag_param url_regex -i [?&]itag=
acl sver3_param url_regex -i [?&]sver=3
cache_peer 127.0.0.1 parent 8081 0 proxy-only no-query connect-timeout=5 no-digest
cache_peer_access 127.0.0.1 allow youtube_videos id_param itag_param sver3_param !begin_param !range_request
cache_peer_access 127.0.0.1 deny all

Squid seemed to be in an infinite restarting loop and the following excerpts
from cache.log seem relevant.

The first restart had the following line in cache.log after about 2 weeks of
operation:

2012/03/25 11:23:45| assertion failed: filemap.cc:76: "fm->max_n_files <= (1 << 24)"

After that we have a rinse and repeat of squid restarting until after cache
validation and then:

2012/03/26 09:16:30| storeLateRelease: released 0 objects
2012/03/26 09:16:30| IpIntercept.cc(137) NetfilterInterception: NF getsockopt(SO_ORIGINAL_DST) failed on FD 17: (2) No such file or directory
2012/03/26 09:16:30| IpIntercept.cc(137) NetfilterInterception: NF getsockopt(SO_ORIGINAL_DST) failed on FD 19: (2) No such file or directory
[..several more of the same..]
2012/03/26 09:16:30| IpIntercept.cc(137) NetfilterInterception: NF getsockopt(SO_ORIGINAL_DST) failed on FD 132: (2) No such file or directory
2012/03/26 09:16:30| IpIntercept.cc(137) NetfilterInterception: NF getsockopt(SO_ORIGINAL_DST) failed on FD 137: (2) No such file or directory
2012/03/26 09:16:32| assertion failed: filemap.cc:76: "fm->max_n_files <= (1 << 24)"

this line again.

I'm not sure what exactly happened. Judging from the name of the assert it had
something to do with a maximum number of files. But is it a squid limitation or
is it a filesystem limitation? Amount of filedescriptors is set to 4096.
Filesystem type is ext4.

So finally here are my questions:
1) What exactly happened and how can I fix it?
2) From your experience, are the ressources used adequate for the use case
    given?
3) Is there a better way to cache video content with Squid 3.1 aside from using
    a cache_peer proxy?
4) Are there other hints and tips that you could share regarding such a setup?

Thanks in advance and best regards,
- Christian Loth
Received on Mon Mar 26 2012 - 11:35:55 MDT