RE: [squid-users] Caching youtube videos problem/ always getting TCP_MISS

Hi All

This is what I had to do for successfully caching youtube video. I tested with this URL http://www.youtube.com/watch?v=7M-jsjLB20Y

Please follow the instruction given on http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube . Then I had these following lines in squid.conf. Also pasting below my storeurl.pl

###Following refresh pattern and store url rewrite config was used

acl store_rewrite_list1 dstdomain .youtube.com .video.google.com
acl store_rewrite_list urlpath_regex \/(get_video\?|videodownload\?|videoplayback\?|watch\?)
#acl store_rewrite_list urlpath_regex \/(get_video\?|videodownload\?|videoplayback\?|watch\?|generate_204\?|docid=)
storeurl_access allow store_rewrite_list store_rewrite_list1
storeurl_access allow all
cache allow all
storeurl_rewrite_program /tmp/squid/storeurl.pl
storeurl_rewrite_children 1
storeurl_rewrite_concurrency 10

#youtube's videos
#refresh_pattern -i (get_video\?|videoplayback\?|videodownload\?|watch\?|generate_204\?|docid=)$ 5259487 99999999% 5259487 ignore-no-cache override-lastmod override-expire ignore-reload negative-ttl=0
refresh_pattern -i (get_video\?|videoplayback\?|videodownload\?|watch\?) 5259487 99999999% 5259487 ignore-private ignore-no-cache override-expire ignore-reload negative-ttl=0

##squid.conf portion ends

##contents of storeurl.pl
#!/usr/bin/perl
$|=1;
while (<>) {
    chomp;

    @X = split;
    $x = $X[0];
    $_ = $X[1];
    if (m/^http:(.*)\.youtube\.com\/videoplayback\?(.*)id=(.*?)&/) {
                print $x."http://video-srv.youtube.com.SQUIDINTERNAL/get_video?video_id=" . $3 . "\n";
    }elsif (m/^http:(.*)\.youtube\.com\/generate_204\?(.*)id=(.*)/) {
                print $x."http://video-srv.youtube.com.SQUIDINTERNAL/get_video?video_id=" . $3 . "\n";
    }elsif (m/^http:(.*)\.video\.google\.com\/(.*)docid=(.*?)&/) {
                print $x."http://video-srv.youtube.com.SQUIDINTERNAL/get_video?video_id=" . $3 . "\n";
    } elsif (m/^http:\/\/(.*)\.youtube\.com\/get_video\?video_id=(.*)/) {
                print $x."http://video-srv.youtube.com.SQUIDINTERNAL/get_video?video_id=" . $2 . "\n";
    } elsif (m/^http:(.*)\.youtube\.com\/watch.*v=(.*)&/) {
                print $x."http://video-srv.youtube.com.SQUIDINTERNAL/get_video?video_id=" . $2 . "\n";
    } elsif (m/^http:(.*)\.youtube\.com\/watch.*v=(.*)$/) {
                print $x."http://video-srv.youtube.com.SQUIDINTERNAL/get_video?video_id=" . $2 . "\n";
    } elsif (m/^http:(.*)\.youtube\.com\/get_video\?(.*)video_id=(.*?)&/) {
                print $x."http://video-srv.youtube.com.SQUIDINTERNAL/get_video?video_id=" . $3 . "\n";
    } elsif (m/^http:\/\/74\.125(.*?)\/get_video\?video_id=(.*?)&origin=(.*?)\.youtube\.com /) {
                print $x."http://video-srv.youtube.com.SQUIDINTERNAL/get_video?video_id=" . $2 . "\n";
    } else {
        print $x.$_ . "\n";
    }
##storeurl.pl ends

Thank You.

Regards,
Saurabh

-----Original Message-----
From: Saurabh Agarwal [mailto:Saurabh.Agarwal_at_citrix.com]
Sent: Friday, November 26, 2010 3:10 PM
To: Amos Jeffries
Cc: squid-users_at_squid-cache.org
Subject: RE: [squid-users] Caching youtube videos problem/ always getting TCP_MISS

Hi Amos

It works fine now. Youtube videos are being cached. There was a mistake in refresh_pattern.

Regards,
Saurabh

-----Original Message-----
From: Amos Jeffries [mailto:squid3_at_treenet.co.nz]
Sent: Tuesday, November 23, 2010 4:47 PM
To: squid-users_at_squid-cache.org
Subject: Re: [squid-users] Caching youtube videos problem/ always getting TCP_MISS

On 23/11/10 23:50, Saurabh Agarwal wrote:
> Thanks Amos.
>
> I fixed the channel id problem by fixing storeurl rewriter perl script but still the video/x-flv response is getting RELEASED in store.log instead of SWAPOUT. Can you please read below cache.log and suggest what is still going wrong? Now "Rewrote to" message prints the right transformed URL. After this store tries to look up for the "6B2E83D66FC215C27ECFBA432AB7B5F6" key which returns a TCP MISS for the same key lookup for 2nd and third tries as well. Then one hash entry gets inserted with another key for the same big URL with different hash key for 2nd and 3rd time as well. This new key is "04BE27CFF614A3315F5CEB008464C453". I have also pasted HTTP response header for video/x-flv content from cache.log. After this I see "Deferring starting swapping out" message in cache.log. Can you please suggest why it is not swapping out?
>
> I am pasting below a section of squid.conf file as well. After this is the cache.log output.
>

I'm thinking you don't want to ignore the last-modified header. It seems
to be usefully far in the past to cause caching for some period.
Ignoring it may cause the refresh_pattern to fail in extending the
storage time (99999999% of last-modified age).

Other than that I don't really know. I have not spent time working out
how to port storeurl* feature yet so don't know its internals well.

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.9
   Beta testers wanted for 3.2.0.3
Received on Fri Nov 26 2010 - 09:53:21 MST