Re: [squid-users] Caching files from Amazon S3

Annoyingly, why the hell is the request from the client a range request?
Squid can't easily cache those unless it somehow fetches the entire object
first.

Adrian

On Sat, Mar 15, 2008, Abidoon Nadeem wrote:
> Hi,
>
> We are stuck with a unique situation and require some help.
>
> We are trying to use squid as a reverse proxy. I have been able to
> configure it to cache normal images such as abc.gif residing on the local
> hard drive.
>
> I run the webserver on a virtual ip and have squid running our internet
> facing ip. The request comes to squid from the client and it sees if it has
> the relevant files in cacche otherwise it talks to the webserver on the
> backend and fetches it.
>
> The issue is that we want to hide where the images are coming from since
> they can come from S3 or our local system. So we used mod rewrite and wrote
> a rule which would pass this URL
> http://oursite.com/Utilities/feeds/getimages/display/thumb/142358.jpg from
> the client to squid and from squid to apache. Apache then passes this
> string to getimages.php? where we break this string, decode it and then
> return the jpg file.
>
> Our response is an http response given below that the client receives
>
> !!!! ----Header Recv----
> HTTP/1.0 206 Partial Content
> Date: Sat, 15 Mar 2008 16:42:21 GMT
> Server: Apache/2.2.6 (Fedora)
> X-Powered-By: PHP/5.2.4
> Cache-Control: private, must-revalidate, max-age=0
> Last-Modified: Sat, 15 Mar 2008 16:42:21 GMT
> Accept-Ranges: bytes
> ETag: "9518df2e0531022b1413d21978d6fb80"
> Content-Type: image/jpeg
> Content-Range: bytes 0-14673899/14673900
> Content-Length: 14673900
> X-Cache: MISS from ratchet.vistaclick.lahore
> X-Cache-Lookup: MISS from ratchet.vistaclick.lahore:80
> Via: 1.0 ratchet.vistaclick.lahore:80 (squid/2.6.STABLE17)
> Connection: close
>
> the issue is that when I look at the squid access log against this response
> it is as follows when we press F5 :
>
> 1205517662.101 81 192.168.0.241 TCP_MISS/200 16932 GET
> http://oursite.com/Utilities/feeds/getimages/display/thumb/142361.jpg -
> FIRST_UP_PARENT/192.168.0.237 image/jpeg.
>
> we get a TCP_MISS/304 when we try with CTRL F5
>
> Now the issue is that we have manipulated the http headers as much as we
> can using the php http library however we cannot get squid to cache these
> images. It caches normal images just fine however when we process the image
> though our getimages.php script we only get the entries above in the log
> file.
>
> Please let me know if there is anyway to make squid images that we process
> through PHP.
>
> Any help you can offer would be greatly appreciated by a squid newbie :)
>
> --
> Abidoon
>
> SQUID CONFIGURATION FILE BELOW
> ------------------------------------------------------------------------------------------------------------------
>
> # $Id: squid.conf,v 1.1 2005/07/16 22:24:57 jmates Exp $
>
> #visible_hostname
>
> #http_port 80 vhost
> http_port 192.168.0.236:80 defaultsite=www.mysite.com vhost
> cache_peer 192.168.0.237 parent 80 0 no-query originserver
>
>
> maximum_object_size 15360 KB
> # minimum_object_size 0 KB
>
> maximum_object_size_in_memory 1500 KB
>
> cache_replacement_policy lru
> memory_replacement_policy lru
>
> logfile_rotate 3
>
> cache_log none
> cache_store_log none
>
> cache_access_log /var/log/squid/access_log
>
> # use local caching name server, avoid /etc/hosts
> dns_nameservers 208.67.222.222
> hosts_file none
>
> #auth_param basic children 5
> #auth_param basic realm Squid proxy-caching web server
> #auth_param basic credentialsttl 2 hours
>
> request_header_max_size 16 KB
>
> negative_ttl 1 minutes
> negative_dns_ttl 1 minutes
>
> # connect_timeout 1 minutes
> # peer_connect_timeout 30 seconds
> # read_timeout 10 minutes
> # request_timeout 5 minutes
> # persistent_request_timeout 2 minute
> # half_closed_clients on
>
> acl all src 0.0.0.0/0.0.0.0
> acl manager proto cache_object
> acl localhost src 127.0.0.1/255.255.255.255
> acl to_localhost dst 127.0.0.0/8
> acl SSL_ports port 443 563
> acl Safe_ports port 80 # http
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 563 # https, snews
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl CONNECT method CONNECT
>
> http_access allow manager localhost
> #http_access deny manager
> # Deny requests to unknown ports
> #http_access deny !Safe_ports
> # Deny CONNECT to other than SSL ports
> #http_access deny CONNECT !SSL_ports
>
> #http_access deny to_localhost
>
> #acl okdomains dstdomain www2.noog.com
> #http_access deny !to_localhost
> http_access allow all
>
> # And finally deny all other access to this proxy
> #http_access deny all
>
> http_reply_access allow all
>
> #icp_access deny all
> miss_access allow all
>
> ident_lookup_access deny all
>
> cache_mgr webmaster@mysite.com
>
> cache_effective_user squid
>
> #httpd_accel_host 192.168.0.5
> #httpd_accel_port 81
> #httpd_accel_uses_host_header off
> #httpd_accel_single_host on
> #httpd_accel_with_proxy on
>
> forwarded_for on
>
> log_icp_queries on
>
> #snmp_port 0
> #snmp_access deny all
>
> # offline_mode off
> coredump_dir none
>
> pipeline_prefetch on
>
> mime_table /etc/squid/mime.conf
>
>

-- 
- Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support -
- $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -