Re: [squid-users] Problems with squid in a campus setup

Hello,

first of all, thank you for your recommendations.

On Monday 26 March 2012 16:34:21 Marcus Kool wrote:
> Youtube may be hogging your pipe but it is better to know than to guess.

Of course, before we decided for the proxy setup we investigated bandwidth
usage. HTTP Traffic was about 60-70% of our traffic, and a good chunk of that was
youtube. That's why we decided, that we would try a squid setup with youtube
caching. However it took a while before we found a solution for caching
youtube, as 3.1 hasn't implemented the necessary features yet.

> The access.log shows content sizes so with a simple awk script it should
> be easy to find out.
>
> I have also seen many sites where advertisements and trackers consume 15%
> bandwidth. This may vary. So blocking ads and trackers is a thing to
> consider.

Thanks for this insight! This would of course be a welcome saving of bandwidth
in my personal opinion. I'm just not sure if we're allowed to do this, as the
patron of the proxy is a public-law institution and as such bound to anti-
censorship laws. Need to check with a Legalese translator.

>
> Do not expect too much from web caching. More and more websites
> intentionally make their sites not cacheable. Look at the percentage of
> TCP_MISS in access.log or use a second awk script to find out more about
> cacheability.

Every bit counts. Before we apply for an increase of (expensive) uplink
bandwidth we want to play every trick we have up our sleeve. At the moment our
cache is still cold, because for getting the proxy running again I had to
completely wipe the cache. At the moment we have a hit:miss ratio of about
1:5. For youtube caching we have a saved bandwidth around 100 GB for the 27th
of march (one video in particular had a size of 768 MB and was watched 19
times). Online lectures are currently en vogue.

>
> I recommend going for a newer Squid: 3.1.19 is stable and fixes issues that
> 3.1.10 has.

Will do so.

> On Linux, aufs has a better performance than diskd

Thanks again for this tip!

>
> > Additional memory for storing objects is 2048 MB:
> >
> > cache_mem 2048 MB
>
> Seems right. But you also need virtual memory for Squid being able to
> fork processes without issues. Do have have 8 GB swap ?

Yes. 10 GB actually.

>
> But read the FAQ about memory usage and a large disk cache:
> http://wiki.squid-cache.org/SquidFaq/SquidMemory
> Squid uses an additional 512*14 MB = 7.1 GB for the index of the disk
> cache. I suggest to downsize to 1 GB in-memory index which implies to
> use only 73 GB disk cache.

Ah okay, here's one of my initial mistakes. I used only 10 MB for my
calculation but of course we use a 64bit squid. Out of curiosity and because I
want to learn: the reasoning for shrinking the disk cache from 512 GB to 73 GB
is because a big cache as we have it at the moment only leads to lots of stale
objects in cache which additionally burden the CPU and RAM because of in-
memory metadata?

Best regards,
 - Christian Loth
 
Received on Tue Mar 27 2012 - 10:26:09 MDT