Re: [squid-users] Re: Splitting objects by size into different cache_dir not working for me

On 02/01/2013 03:52 PM, Eliezer Croitoru wrote:
> On 2/1/2013 8:04 PM, Luciano Ruete wrote:
>> I've already tested and the above seams to be true. How can I know for
>> shure if there are or not objects in the cache_dir greater than
>> maximum_object_size_in_memory?
>>
>> I can ran more tests if you give me further instructions, or can try a
>> patch if provided.
>>
>> Regards.
>
> We need a set of http cachable objects in different sizes:
> tiny:
> http://repo.ngtech.co.il/rpm/centos/6/x86_64/squid-debuginfo-3.2.5-1.el6.x86_64.rpm.asc
>
>
> avg:
> http://repo.ngtech.co.il/squid/cachecluster.png
>
> avg download:
> http://repo.ngtech.co.il/rpm/centos/6/others/openssh-6.1p1-81.el6.x86_64.rpm
>
>
> big download:
> http://repo.ngtech.co.il/rpm/centos/6/x86_64/squid-debuginfo-3.2.6-1.el6.x86_64.rpm
>
> large download:
> http://dl.fedoraproject.org/pub/fedora/linux/releases/18/Fedora/x86_64/iso/Fedora-18-x86_64-netinst.iso
>
>
> very large download:
> http://dl.fedoraproject.org/pub/fedora/linux/releases/18/Fedora/x86_64/iso/Fedora-18-x86_64-DVD.iso
>
>
> All the above should have proper cache directive that can be checked
> using:
> http://redbot.org
> (ignore my testing Link headers)
>
> You can set these settings in squid.conf
> store_dir_select_algorithm round-robin
> #^^default is: "least-load" which can cause your problem.
> cache_dir rock /var/spool/squid/rock 1000 min-size=1024 max-size=31000
> max-swap-rate=250 swap-timeout=350
> cache_dir aufs /var/spool/squid/aufs 30000 16 256 min-size=209715
> max-size=734003200
> maximum_object_size_in_memory 512 KB #default
> minimum_object_size 0 KB #default
> maximum_object_size 300 MB #non default
> ##end conf
>
> First try only the round-robin to make sure this is not the basic
> cause for problems.
> reload\restart check the cache_dir sizes before and after getting into
> a website like yahoo movies or any site with many objects in it.

With this conf, objects larger than 512KB does not get cache into AUFS
cache_dir, example:

requesting
1359747785.360 97216 127.0.0.1 TCP_MISS/200 9225893 GET
http://repo.ngtech.co.il/rpm/centos/6/x86_64/squid-debuginfo-3.2.6-1.el6.x86_64.rpm
- HIER_DIRECT/58.28.153.233 application/x-rpm

results in this in store log:
1359747785.360 RELEASE -1 FFFFFFFF E8A98EBEE591A2878F4795D16C3A434D 200
1357962518 1357827095 1389498518 application/x-rpm 9225364/9225364 GET
http://repo.ngtech.co.il/rpm/centos/6/x86_64/squid-debuginfo-3.2.6-1.el6.x86_64.rpm

And no change in AUFS size at all.

File with less than 512KB (and above 205K min-size) does goes to AUFS,
example

requesting
1359747696.788 8644 127.0.0.1 TCP_MISS/200 231025 GET
http://repo.ngtech.co.il/rpm/centos/6/others/openssh-6.1p1-81.el6.x86_64.rpm
- HIER_DIRECT/58.28.153.233 application/x-rpm

results in this in store log:
1359747696.784 SWAPOUT 01 00000000 9641F2C5678BA0B610DC647C16AE8EAF 200
1359744929 1357195218 1391280929 application/x-rpm 230420/230420 GET
http://repo.ngtech.co.il/rpm/centos/6/others/openssh-6.1p1-81.el6.x86_64.rpm

And AUFS dir size gets incremented acording.

ROCK storage also grows if small objects are requested.

> Then try to change only the maximum_object_size_in_memory from 512
> default to 8 KB which will cause almost no ram cache and mostly dir
> cache to make sure that the reason for the low counters is not because
> of mem caching.
> Check the cache_dir before and after...
>

If i lower maximum_object_size_in_memory to 8 KB then

requesting:
1359749023.713 1683 127.0.0.1 TCP_MISS/200 1466 GET
http://repo.ngtech.co.il/rpm/centos/6/x86_64/squid-debuginfo-3.2.5-1.el6.x86_64.rpm.asc
- HIER_DIRECT/58.28.153.233 text/plain
1359749024.590 2561 127.0.0.1 TCP_MISS/200 33255 GET
http://repo.ngtech.co.il/squid/cachecluster.png -
HIER_DIRECT/58.28.153.233 image/png
1359749028.613 6583 127.0.0.1 TCP_MISS/200 231025 GET
http://repo.ngtech.co.il/rpm/centos/6/others/openssh-6.1p1-81.el6.x86_64.rpm
- HIER_DIRECT/58.28.153.233 application/x-rpm
1359749121.203 99173 127.0.0.1 TCP_MISS/200 9225894 GET
http://repo.ngtech.co.il/rpm/centos/6/x86_64/squid-debuginfo-3.2.6-1.el6.x86_64.rpm
- HIER_DIRECT/58.28.153.233 application/x-rpm

results in:
1359749023.712 SWAPOUT 00 000031E8 F104D7F44EDABE94F758EC16BD02ADA0 200
1359744786 -1 1360608786 text/plain 876/876 GET
http://repo.ngtech.co.il/rpm/centos/6/x86_64/squid-debuginfo-3.2.5-1.el6.x86_64.rpm.asc
1359749024.590 RELEASE -1 FFFFFFFF 5BFF306024844E48B8ED6A3BD89312FF 200
1359744790 -1 1360608790 image/png 32719/32719 GET
http://repo.ngtech.co.il/squid/cachecluster.png
1359749028.613 RELEASE -1 FFFFFFFF 45F6D81746DB2C8703359DEF1494F04D 200
1359744929 1357195218 1391280929 application/x-rpm 230420/230420 GET
http://repo.ngtech.co.il/rpm/centos/6/others/openssh-6.1p1-81.el6.x86_64.rpm
1359749121.203 RELEASE -1 FFFFFFFF 64FDA3C6D9598638709C7D5188D99793 200
1357962518 1357827095 1389498518 application/x-rpm 9225364/9225364 GET
http://repo.ngtech.co.il/rpm/centos/6/x86_64/squid-debuginfo-3.2.6-1.el6.x86_64.rpm

Only the smallest object goes to cache and if I flood the proxy with
user's traffic only the rock storage grows. And only object that are
less than 8k get a SWAPOUT in store_log.

> If there is no straight answer if it's ok or not and if no clean
> answer try to use all the directives I gave you together and try to
> download each of the set of files I gave you.
>
> The tiny should be cached in mem.
> all the others should be cached in a cache_dir by the size.
> the large ISO file should be cached in any case in the UFS cache_dir.
>
> Feel free to ask me anything.
> I am also in squid IRC channel here and there.
>

Let me know if you need any further test.
Received on Fri Feb 01 2013 - 20:13:31 MST