Re: [squid-users] High load server Disk problem

On Tue, 17 Aug 2010 22:43:33 -0430, Jose Ildefonso Camargo Tolosa
<ildefonso.camargo_at_gmail.com> wrote:
> Hi!
>
> In my own personal opinion: your hard drive alone is not enough to
> handle that much traffic (110MBytes/s, ~1Gbps). See, most SATA hard
> drives (7200rpm) gives around 50~70MB/s *sequential* read speed, your
> cache reads are *not* sequential, so, it will be slower. In my
> opinion, you need something like a 8 drives RAID10 array, and/or use
> faster disks (10k), or maybe 15k SAS disks.
>
> Also, I would put a minimum object size for disk of 1M, and a maximum
> object size of whatever you want (this depends on your network, but
> usually ~150MB is enough to fit almost any upgrade download). And for
> RAM, I would put a maximum object size of 1M, with no minimum. Thus,
> keeping small files out of the disk cache.

The COSS storage type he has setup already does this very efficiently with
added disk-backing of the COSS chunks for cross-restart recovery of the
cache.

>
> Also, other questions: How many clients/connections are you handling?
> what are your server's specifications? and how is the system load over
> time? (maybe zabbix or any other monitoring system will let you know
> your system load over time).
>
> I hope this helps,
>
> Ildefonso Camargo
>
> On Tue, Aug 17, 2010 at 10:26 PM, Robert Pipca <robertpipca_at_gmail.com>
> wrote:
>> Hi.
>>
>> I'm using squid on a high speed network (with 110M of http traffic).
>>
>> I'm using 2.7.STABLE7 with these cache_dir:
>>
>> cache_dir aufs /cache 756842 60 100

Whats missing appears to be min-size=1048576 on the AUFS to push all the
small objects into the better COSS directories. (NOTE: the value is COSS
max-size+1)

>> cache_dir coss /cache/coss1 65520 max-size=1048575
>> max-stripe-waste=32768 block-size=4096 membufs=15
>> cache_dir coss /cache/coss2 65520 max-size=1048575
>> max-stripe-waste=32768 block-size=4096 membufs=15
>> cache_dir coss /cache/coss3 65520 max-size=1048575
>> max-stripe-waste=32768 block-size=4096 membufs=15
>>
>> Everything works fine most of the day, but on peak hours, I got these:
>>
>> 2010/08/17 20:06:59| squidaio_queue_request: WARNING - Disk I/O
>> overloading
>> 2010/08/17 20:06:59| squidaio_queue_request: Queue Length:
>> current=981, high=1488, low=321, duration=170
>>
>> After a while, I got a few of these, with "duration" increasing, until:
>>
>> 2010/08/17 20:23:09| squidaio_queue_request: WARNING - Disk I/O
>> overloading
>> 2010/08/17 20:23:09| squidaio_queue_request: Queue Length:
>> current=558, high=2177, low=321, duration=531
>>
>> The web browsing started to get very slow, which is when the support
>> team took squid down.
>>
>> All cache_dir are on a single sata-2 7200RPM 1TB hard drive.
>>
>> Is there a way to know which cache_dir is the problem and what I can
>> so this doesn't happen?

AIO is the I/O method preferred by AUFS. That aufs dir is also listed
first, which may affect the default choice.

>>
>> I tried using both 16 and 32 AIO threads, but didn't help.
>>
>> cache manager tells me that I have around 10 million objects:
>>
>> Average HTTP requests per minute since start: 18851.1
>>
>> Storage Swap size: 693535688 KB
>> Storage Mem size: 30872 KB
>> Mean Object Size: 64.50 KB
>>
>> Internal Data Structures:
>> 10752896 StoreEntries
>> 49 StoreEntries with MemObjects
>> 26 Hot Object Cache Items
>> 10752847 on-disk objects

*49* of 10 million objects are in-transit? that is very low. Though it
could be a result of the queue overload.

Amos
Received on Wed Aug 18 2010 - 03:36:15 MDT