Re: [squid-users] Is ignore-private option from refresh_pattern broken?

Thank you Amos,

On 09/12/2013 02:53 PM, Amos Jeffries wrote:
> On 12/09/2013 10:51 p.m., Pavel Kazlenka wrote:
>> Hi gentlemen,
>>
>> I'm trying to cache youtube videos following
>> http://wiki.squid-cache.org/Features/StoreID guide.
>> But seems like squid rejects to cache the content because original
>> server returns header 'Cache-control:private' and 'refresh pattern
>> ... ignore-private' doesn't take effect. Here is debug log entries
>> that are interesting here (I think):
>>
>> 2013/09/11 19:24:24.842 kid1| helper.cc(419) helperSubmit:
>> buf[975]=http://r16---sn-4g57ln7e.c.youtube.com/videoplayback?ratebypass=yes&itag=43&ip=212.98.189.159&key=yt1&upn=6KCvBVLs-yM&mt=1378980214&fexp=919118%2C924606%2C929117%2C929121%2C929906%2C929907%2C929922%2C929127%2C929129%2C929131%2C929930%2C936403%2C92
>> 5726%2C936310%2C925720%2C925722%2C925718%2C925714%2C929917%2C906945%2C929933%2C920302%2C906842%2C913428%2C920605%2C919811%2C935020%2C935021%2C913563%2C919373%
>>
>> 2C930803%2C908536%2C932211%2C938701%2C931924%2C934005%2C936308%2C909549%2C900816%2C912711%2C904494%2C904497%2C939903%2C900375%2C900382%2C934507%2C907231%2C936
>>
>> 312%2C906001&id=2c5e89c0af6c8804&expire=1379004381&sver=3&ipbits=8&cp=U0hWTlBLUl9NS0NONl9IRVZEOjNVQko4ZGFaMGcz&ms=au&source=youtube&sparams=cp%2Cid%2Cip%2Cipbits%2Citag%2Cratebypass%2Csource%2Cupn%2Cexpire&mv=m&cpn=Ph7LcSRYt1STlsoQ&signature=BB797D0EFC5182670EF89E95EFBB6E5D12F49B8F.6404F37419A96004F8DDCA2CAB901101A
>>
>> 30082CA&ptk=youtube_none&pltype=contentugc 192.168.10.8/- - GET
>> myip=192.168.10.245 myport=3128
>> ...
>> 2013/09/11 19:24:24.843 kid1| helper.cc(919) helperHandleRead:
>> accumulated[77]=OK
>> store-id=http://video-srv.youtube.com.squid.internal/43&2c5e89c0af6c8804"
>> 2013/09/11 19:24:24.843 kid1| store_dir.cc(786) get: none of 1
>> cache_dirs have 028E8844ECA93A634459175C8C0D463D
>> 2013/09/11 19:24:24.843 kid1| store.cc(840) storeCreateEntry:
>> storeCreateEntry:
>> 'http://video-srv.youtube.com.squid.internal/43&2c5e89c0af6c8804'
>> 2013/09/11 19:24:25.032 kid1| http.cc(396) cacheableReply: NO because
>> server reply Cache-Control:private
>>
>> My squid.conf:
>>
>> acl rewritedoms dstdomain .c.youtube.com
>>
>> store_id_program /home/tester/squid/libexec/storeid_file_rewrite
>> /home/tester/squid/db.txt
>> store_id_children 40 startup=10 idle=5 concurrency=0
>> store_id_access allow rewritedoms
>> store_id_access deny all
>> ...
>> refresh_pattern ^http://video-srv\.youtube\.squid\.internal/.* 10080
>> 80% 79900 override-lastmod override-expire ignore-reload
>> ignore-must-revalidate ignore-private
>> refresh_pattern ^ftp: 1440 20% 10080
>> refresh_pattern ^gopher: 1440 0% 1440
>> #refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
>> refresh_pattern . 0 20% 4320
>>
>> Squid version is squid-3.4.0.1-20130910-r12989.
>>
>> So my questions are:
>>
>> 1) Am I right that refresh_pattern ... ignore-private should work
>> here and help to cache reply with CC:Private?
>
> Yes the exact code is:
> if ( Cache-Control is present AND contains "private" AND
> ignore-private is *absent*)
> display that "NO because server reply Cache-Control:private"
> message etc.
>
>> 2) Is there another (may be more correct) way to force squid to cache
>> replies with CC:Private?
>
> There is no _correct_ way to abuse the RFC protocol standard. This
> ignore-private explicitly forbidden behaviour for any cache shared by
> more than 1 person.
> It is officially only made available in Squid to allow single-person
> installations to operate a proxy between multiple devices.
>
Sure. I meant correct from squid's internals point of view (may be
dedicated directive).

>> 3) How can I check that 'refresh_pattern' config string is parsed
>> correctly and all options are remembered by squid?
>
> To check it is parsed correctly use:
> squid -k parse
>
>
> To check what the running config is you can produce a config file dump
> using the cache manager interface:
>
> * ensure that you have a cachemgr_passwd (or "none") explicitly
> defined for either the "config" or the "all" reports.
> http://www.squid-cache.org/Doc/config/cachemgr_passwd/
>
> * ensure that your http_access rules involving "manager" ACL permit
> you access to the proxy management interface.
>
> * fetch http://example.com/squid-internal-mgr/config in your browser
>
> The result should be a TXT format listing of all the squid.conf
> settings (including the defaults) which Squid is using.
>
My config obtained in this way includes 'refresh_pattern
^http://video-srv\.youtube\.squid\.internal/.* 10080 80% 79900
override-expire override-lastmod ignore-reload ignore-must-revalidate
ignore-private', so I guess that this is definitely a bug. I'm going to
open new defect on bugzilla. Will see if this will be fixed in 3.4.0.2
when 'unknown_cfg_function' patch is accepted. Any objections?

> Amos
Received on Thu Sep 12 2013 - 12:13:14 MDT