Re: [squid-users] Is ignore-private option from refresh_pattern broken?

On 09/12/2013 03:12 PM, Pavel Kazlenka wrote:
> Thank you Amos,
>
> On 09/12/2013 02:53 PM, Amos Jeffries wrote:
>> On 12/09/2013 10:51 p.m., Pavel Kazlenka wrote:
>>> Hi gentlemen,
>>>
>>> I'm trying to cache youtube videos following
>>> http://wiki.squid-cache.org/Features/StoreID guide.
>>> But seems like squid rejects to cache the content because original
>>> server returns header 'Cache-control:private' and 'refresh pattern
>>> ... ignore-private' doesn't take effect. Here is debug log entries
>>> that are interesting here (I think):
>>>
>>> 2013/09/11 19:24:24.842 kid1| helper.cc(419) helperSubmit:
>>> buf[975]=http://r16---sn-4g57ln7e.c.youtube.com/videoplayback?ratebypass=yes&itag=43&ip=212.98.189.159&key=yt1&upn=6KCvBVLs-yM&mt=1378980214&fexp=919118%2C924606%2C929117%2C929121%2C929906%2C929907%2C929922%2C929127%2C929129%2C929131%2C929930%2C936403%2C92
>>> 5726%2C936310%2C925720%2C925722%2C925718%2C925714%2C929917%2C906945%2C929933%2C920302%2C906842%2C913428%2C920605%2C919811%2C935020%2C935021%2C913563%2C919373%
>>>
>>> 2C930803%2C908536%2C932211%2C938701%2C931924%2C934005%2C936308%2C909549%2C900816%2C912711%2C904494%2C904497%2C939903%2C900375%2C900382%2C934507%2C907231%2C936
>>>
>>> 312%2C906001&id=2c5e89c0af6c8804&expire=1379004381&sver=3&ipbits=8&cp=U0hWTlBLUl9NS0NONl9IRVZEOjNVQko4ZGFaMGcz&ms=au&source=youtube&sparams=cp%2Cid%2Cip%2Cipbits%2Citag%2Cratebypass%2Csource%2Cupn%2Cexpire&mv=m&cpn=Ph7LcSRYt1STlsoQ&signature=BB797D0EFC5182670EF89E95EFBB6E5D12F49B8F.6404F37419A96004F8DDCA2CAB901101A
>>>
>>> 30082CA&ptk=youtube_none&pltype=contentugc 192.168.10.8/- - GET
>>> myip=192.168.10.245 myport=3128
>>> ...
>>> 2013/09/11 19:24:24.843 kid1| helper.cc(919) helperHandleRead:
>>> accumulated[77]=OK
>>> store-id=http://video-srv.youtube.com.squid.internal/43&2c5e89c0af6c8804"
>>> 2013/09/11 19:24:24.843 kid1| store_dir.cc(786) get: none of 1
>>> cache_dirs have 028E8844ECA93A634459175C8C0D463D
>>> 2013/09/11 19:24:24.843 kid1| store.cc(840) storeCreateEntry:
>>> storeCreateEntry:
>>> 'http://video-srv.youtube.com.squid.internal/43&2c5e89c0af6c8804'
>>> 2013/09/11 19:24:25.032 kid1| http.cc(396) cacheableReply: NO
>>> because server reply Cache-Control:private
>>>
>>> My squid.conf:
>>>
>>> acl rewritedoms dstdomain .c.youtube.com
>>>
>>> store_id_program /home/tester/squid/libexec/storeid_file_rewrite
>>> /home/tester/squid/db.txt
>>> store_id_children 40 startup=10 idle=5 concurrency=0
>>> store_id_access allow rewritedoms
>>> store_id_access deny all
>>> ...
>>> refresh_pattern ^http://video-srv\.youtube\.squid\.internal/.* 10080
>>> 80% 79900 override-lastmod override-expire ignore-reload
>>> ignore-must-revalidate ignore-private
>>> refresh_pattern ^ftp: 1440 20% 10080
>>> refresh_pattern ^gopher: 1440 0% 1440
>>> #refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
>>> refresh_pattern . 0 20% 4320
>>>
>>> Squid version is squid-3.4.0.1-20130910-r12989.
>>>
>>> So my questions are:
>>>
>>> 1) Am I right that refresh_pattern ... ignore-private should work
>>> here and help to cache reply with CC:Private?
>>
>> Yes the exact code is:
>> if ( Cache-Control is present AND contains "private" AND
>> ignore-private is *absent*)
>> display that "NO because server reply Cache-Control:private"
>> message etc.
>>
>>> 2) Is there another (may be more correct) way to force squid to
>>> cache replies with CC:Private?
>>
>> There is no _correct_ way to abuse the RFC protocol standard. This
>> ignore-private explicitly forbidden behaviour for any cache shared by
>> more than 1 person.
>> It is officially only made available in Squid to allow single-person
>> installations to operate a proxy between multiple devices.
>>
> Sure. I meant correct from squid's internals point of view (may be
> dedicated directive).
>
>>> 3) How can I check that 'refresh_pattern' config string is parsed
>>> correctly and all options are remembered by squid?
>>
>> To check it is parsed correctly use:
>> squid -k parse
>>
>>
>> To check what the running config is you can produce a config file
>> dump using the cache manager interface:
>>
>> * ensure that you have a cachemgr_passwd (or "none") explicitly
>> defined for either the "config" or the "all" reports.
>> http://www.squid-cache.org/Doc/config/cachemgr_passwd/
>>
>> * ensure that your http_access rules involving "manager" ACL permit
>> you access to the proxy management interface.
>>
>> * fetch http://example.com/squid-internal-mgr/config in your browser
>>
>> The result should be a TXT format listing of all the squid.conf
>> settings (including the defaults) which Squid is using.
>>
> My config obtained in this way includes 'refresh_pattern
> ^http://video-srv\.youtube\.squid\.internal/.* 10080 80% 79900
> override-expire override-lastmod ignore-reload ignore-must-revalidate
> ignore-private', so I guess that this is definitely a bug. I'm going
> to open new defect on bugzilla. Will see if this will be fixed in
> 3.4.0.2 when 'unknown_cfg_function' patch is accepted. Any objections?
>
Sorry, I checked with 'refresh_pattern ^.* 10080 80% 79900
override-expire override-lastmod ignore-reload ignore-must-revalidate
ignore-private' and file considered cacheable. So problem is in
incorrect regexp, I guess.

>
>> Amos
>
Received on Thu Sep 12 2013 - 12:39:39 MDT