Re: [squid-users] Mime.conf

Looking at client_side.cc, line 1152 of the source for 3.0.19 that was
used to build my copy from. It's a comment (/* delete old Content-Type,
add ours */). If I'm reading this right squid is reading the headers and
checking multipart content-range headers. Something breaks or is
actually invalid and it tries to guess the content headers. I'll have to
sit down and try and backtrack this later. Guessing this a bug.

On 1/18/2010 11:37 PM, Jason Spegal wrote:
> On 1/18/2010 8:55 PM, Amos Jeffries wrote:
>> On Mon, 18 Jan 2010 13:18:20 -0500, Jason Spegal<jspegal_at_comcast.net>
>> wrote:
>>> Alrighty. Did some more research and found a solution to my problem
>>> which leads to another issue.
>>>
>>> My problem: I was trying to serve a proxy auto configuration file
>>> (wpad.dat) from an internal webserver (http://wpad/). When the client
>>> down the pipe after squid picked it up the file was served with the
>>> mime
>>> type chemical/x-mopac-input. When I went direct to the
>>> webserver it
>>> served the correct mime type (which I had forced it to).
>>>
>>> Solution: On Gentoo squid is using the /etc/mime.types file to guess
>>> the
>>> mime type instead of what the remote webserver is saying the
>>> file is. I
>> Point 1: Squid does not do that. Does not use mime.types at all.
>>
>> Content-Type headers are passed through unchanged from what is received
>> unless administratively changed by header_replace.
> Taken from access.log
>
> Before changing mime.types
>
> 1263657638.249 0 10.10.122.248 TCP_MEM_HIT/200 670 GET
> http://wpad/wpad.dat - NONE/- chemical/x-mopac-input
> 1263661679.834 0 10.10.122.239 TCP_MEM_HIT/200 670 GET
> http://wpad/wpad.dat - NONE/- chemical/x-mopac-input
> 1263662648.054 9 10.10.122.248 TCP_CLIENT_REFRESH_MISS/200 654
> GET http://wpad/wpad.dat - DIRECT/10.10.122.250 chemical/x-mopac-input
> 1263662742.482 4 10.10.122.248 TCP_CLIENT_REFRESH_MISS/200 654
> GET http://wpad/wpad.dat - DIRECT/10.10.122.250 chemical/x-mopac-input
> 1263662752.973 0 10.10.122.248 TCP_IMS_HIT/304 264 GET
> http://wpad/wpad.dat - NONE/- chemical/x-mopac-input
> 1263664740.203 0 10.10.122.248 TCP_MEM_HIT/200 669 GET
> http://wpad/wpad.dat - NONE/- chemical/x-mopac-input
>
> After changing mime.types
>
> 1263834369.649 1 10.10.122.241 TCP_REFRESH_UNMODIFIED/200 647 GET
> http://wpad/wpad.dat - DIRECT/10.10.122.250
> application/x-ns-proxy-autoconfig
> 1263834539.719 0 10.10.122.241 TCP_MEM_HIT/200 657 GET
> http://wpad/wpad.dat - NONE/- application/x-ns-proxy-autoconfig
> 1263834791.576 0 10.10.122.241 TCP_MEM_HIT/200 657 GET
> http://wpad/wpad.dat - NONE/- application/x-ns-proxy-autoconfig
> 1263834822.423 0 10.10.122.241 TCP_MEM_HIT/200 657 GET
> http://wpad/wpad.dat - NONE/- application/x-ns-proxy-autoconfig
>
> I just double checked that (ForceType
> application/x-ns-proxy-autoconfig) in my apache vhost config is
> working correctly. Also apache's mime.types file is setup correctly
> for this particular item.
>>> fixed the file which I also noticed has several other issues answering
>>> my other other issue, my is 95% of my data being caught in the catch
>>> all
>>> refresh_pattern instead of the mime type ones.
>> Point 2: Squid does not accept mime types in the refresh_pattern
>> directive.
> This explains a few things.
>> Are you _sure_ that:
>> * the PAC file is not cached with old headers from before your
>> changes?
> Yes
>> * the PAC file is actually being fetched from the web server you are
>> expecting?
> Yes
>> * this is an official build of Squid?
> Yes, see below.
>> * nobody has applied third-party patches to it?
>> (none of the official Gentoo patches change mime.types.
>> http://sources.gentoo.org/viewcvs.py/gentoo-x86/net-proxy/squid/files/)
>>
> Fairly sure.
>> What headers does this produce when run on the Squid box?
>> squidclient -v -h wpad -p 80 /wpad.dat
>>
>>
> I'm posting version and configuration at the bottom of this email.
> Refresh patterns will be changed after this email is sent. This is a
> standard gentoo install with the epoll USE flag.
>
> [ebuild R ] net-proxy/squid-3.0.19 USE="caps epoll ldap mysql pam
> samba sqlite ssl -icap-client (-ipf-transparent) -kerberos -kqueue
> -logrotate* -nis (-pf-transparent) -postgres -radius -sasl (-selinux)
> -snmp -zero-penalty-hit" 0 kB
>
> (squidclient -v -h wpad -p 80 /wpad.dat) yeilds
>
> headers: 'GET /wpad.dat HTTP/1.0
> Accept: */*
>
> '
> HTTP/1.1 404 Not Found
> Date: Tue, 19 Jan 2010 03:27:19 GMT
> Server: Apache
> Content-Length: 265
> Connection: close
> Content-Type: text/html; charset=iso-8859-1
>
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>404 Not Found</title>
> </head><body>
> <h1>Not Found</h1>
> <p>The requested URL /wpad.dat was not found on this server.</p>
> <hr>
> <address>Apache Server at localhost Port 80</address>
> </body></html>
>
>
> So I used GET instead.
>
> (GET http://wpad/wpad.dat -USed)
>
> GET http://wpad/wpad.dat
> User-Agent: lwp-request/5.827 libwww-perl/5.831
>
> GET http://wpad/wpad.dat --> 200 OK
> Connection: close
> Date: Tue, 19 Jan 2010 03:28:59 GMT
> Accept-Ranges: bytes
> Age: 412
> ETag: "736a9e-119-47d6be3f06d80"
> Server: Apache
> Content-Length: 281
> Content-Type: application/x-ns-proxy-autoconfig
> Last-Modified: Mon, 18 Jan 2010 08:10:46 GMT
> Client-Date: Tue, 19 Jan 2010 03:28:59 GMT
> Client-Peer: 10.10.122.250:80
> Client-Response-Num: 1
>
>>> Of note for other Gentoo& Debian users: From mime.types # This
>>> file is
>>> part of the app-misc/mime-types package, which is based on debian's
>>> "mime-support".
>>>
>>> So my question is now; how do I force squid to use the mime-type
>>> delivered by the remote webserver without killing mime.types and thus
>>> breaking my system in new and unexpected ways?
>> The official releases of Squid pass content-type headers through
>> unchanged. Something is broken.
>>> On 1/15/2010 8:22 PM, Amos Jeffries wrote:
>>>> Jason Spegal wrote:
>>>>> Is mime.conf what is used by refresh_pattern when mime types are used
>>>>> for the regex?
>>>> No.
>>>>
>>>> refresh_pattern uses a text regex against the requested URL string.
>>>>
>>>> mime.conf is used by FTP and Gopher directory display to show the
>> icons.
>> Amos
> Squid Cache: Version 3.0.STABLE19
> configure options: '--prefix=/usr' '--build=i686-pc-linux-gnu'
> '--host=i686-pc-linux-gnu' '--mandir=/usr/share/man'
> '--infodir=/usr/share/info' '--datadir=/usr/share' '--sysconfdir=/etc'
> '--localstatedir=/var/lib' '--sysconfdir=/etc/squid'
> '--libexecdir=/usr/libexec/squid' '--localstatedir=/var'
> '--datadir=/usr/share/squid' '--with-default-user=squid'
> '--enable-auth=basic,digest,negotiate,ntlm'
> '--enable-removal-policies=lru,heap'
> '--enable-digest-auth-helpers=password'
> '--enable-basic-auth-helpers=DB,PAM,LDAP,SMB,multi-domain-NTLM,getpwnam,NCSA,MSNT'
> '--enable-external-acl-helpers=ldap_group,wbinfo_group,ip_user,session,unix_group'
> '--enable-ntlm-auth-helpers=SMB,fakeauth'
> '--enable-negotiate-auth-helpers=' '--enable-useragent-log'
> '--enable-cache-digests' '--enable-delay-pools' '--enable-referer-log'
> '--enable-arp-acl' '--with-large-files' '--with-filedescriptors=8192'
> '--enable-caps' '--disable-snmp' '--enable-ssl'
> '--disable-icap-client' '--enable-http-violations' '--with-pthreads'
> '--with-aio' '--enable-storeio=ufs,diskd,aufs,null'
> '--enable-linux-netfilter' '--enable-epoll'
> 'build_alias=i686-pc-linux-gnu' 'host_alias=i686-pc-linux-gnu'
> 'CC=i686-pc-linux-gnu-gcc' 'CFLAGS=-march=pentium4m -O2 -pipe
> -fomit-frame-pointer' 'LDFLAGS=-Wl,-O1' 'CXXFLAGS=-march=pentium4m -O2
> -pipe -fomit-frame-pointer'
>
>
> From squid.conf:
>
> acl allowed_sites url_regex
> "/etc/squid/filtered_sites/allow/exceptions.allow"
> acl blocked_sites url_regex "/etc/squid/filtered_sites/deny/ads.blocked"
> acl blocked_sites url_regex "/etc/squid/filtered_sites/deny/porn.blocked"
> acl blocked_sites url_regex
> "/etc/squid/filtered_sites/deny/spyware.blocked"
> http_access allow allowed_sites
> http_access deny blocked_sites
> acl manager proto cache_object
> acl localhost src 127.0.0.1/32
> acl to_localhost dst 127.0.0.0/8
> acl localnet src 10.10.0.0/12
> acl SSL_ports port 443
> acl Safe_ports port 80 # http
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 # https
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl Safe_ports port 901 # SWAT
> acl purge method PURGE
> acl CONNECT method CONNECT
> http_access allow manager localhost
> http_access allow manager localnet
> http_access deny manager all
> http_access allow purge localhost
> http_access allow purge localnet
> http_access deny purge
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access allow localnet
> http_access allow localhost
> http_access deny all
> acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\?
> acl videocache_allow_url url_regex -i
> \.googlevideo\.com\/videoplayback \.googlevideo\.com\/videoplay
> \.googlevideo\.com\/get_video\?
> acl videocache_allow_url url_regex -i \.google\.com\/videoplayback
> \.google\.com\/videoplay \.google\.com\/get_video\?
> acl videocache_allow_url url_regex -i
> \.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/videoplay
> \.google\.[a-z][a-z]\/get_video\?
> acl videocache_allow_url url_regex -i
> (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplayback\?
>
> acl videocache_allow_url url_regex -i
> (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplay\?
>
> acl videocache_allow_url url_regex -i
> (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/get_video\?
>
> acl videocache_allow_url url_regex -i
> proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
> acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
> acl videocache_allow_url url_regex -i
> [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
> acl videocache_allow_url url_regex -i
> bitcast\.vimeo\.com\/vimeo\/videos\/
> acl videocache_allow_url url_regex -i
> va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
> acl videocache_allow_url url_regex -i
> \.files\.youporn\.com\/(.*)\/flv\/
> acl videocache_allow_url url_regex -i
> \.msn\.com\.edgesuite\.net\/(.*)\.flv
> acl videocache_allow_url url_regex -i
> media[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
> mobile[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
> acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
> acl videocache_allow_url url_regex -i
> \.video[a-z0-9]?[a-z0-9]?\.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram)
>
> acl videocache_allow_url url_regex -i
> video\.break\.com\/(.*)\.(flv|mp4)
> acl videocache_allow_dom dstdomain .mccont.com dl.redtube.com
> .cdn.dailymotion.com
> acl videocache_deny_url url_regex -i
> http:\/\/[a-z][a-z]\.youtube\.com http:\/\/www\.youtube\.com
> icp_access allow localnet
> icp_access deny all
> htcp_access allow localnet
> htcp_access deny all
> http_port 10.10.122.250:3128 transparent
> http_port 10.10.123.1:3128 transparent
> hierarchy_stoplist cgi-bin ?
> cache_mem 1024 MB
> maximum_object_size_in_memory 8192 KB
> memory_replacement_policy heap LRU
> cache_replacement_policy heap LFUDA
> cache_dir aufs /mnt/archive3/squid_cache 65536 64 256
> store_dir_select_algorithm round-robin
> max_open_disk_fds 0
> minimum_object_size 0 KB
> maximum_object_size 8192000 KB
> cache_swap_low 97
> cache_swap_high 99
> logformat squid %ts.%03tu %6tr %>a %Ss/%03Hs %<st %rm %ru %un %Sh/%<A %mt
> logformat squidmime %ts.%03tu %6tr %>a %Ss/%03Hs %<st %rm %ru %un
> %Sh/%<A %mt [%>h] [%<h]
> logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %Hs %<st %Ss:%Sh
> logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %Hs %<st
> "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
> logformat superlog %tg.%03tu %14>a %-15>A %14<A %7tr %5rm HTTP/%3rv
> %-4Hs %-23Ss %8Sh %10<st %43mt URL::%'-130ru RESPONSE HEADER::%"<h
> access_log /var/log/squid/access.log squid
> access_log /var/www/websites/admin/logs/squid_superlog.txt superlog
> logfile_rotate 24
> strip_query_terms on
> buffered_logs on
> acl dynamic_content urlpath_regex -i
> \.(asp|aspx|php|pl|xml|rss|kml|cgi|py|pyc) #(\?.*)?$
> acl dynamic_content urlpath_regex -i http://audio*pandora.com/*.mp*
> acl dynamic_content urlpath_regex -i cgi-bin
> cache deny dynamic_content
> cache allow all
> refresh_pattern -i kh*.google.com/? 43200 80% 259200 ignore-no-cache
> ignore-private ignore-no-store ignore-auth override-expire
> override-lastmod ignore-reload
> refresh_pattern -i virtualearth.net/? 43200 80% 259200 ignore-no-cache
> ignore-private ignore-no-store ignore-auth override-expire
> override-lastmod ignore-reload
> refresh_pattern application/* 43200 80% 259200 ignore-no-cache
> ignore-private ignore-no-store ignore-auth
> refresh_pattern audio/* 43200 80% 259200 ignore-no-cache
> ignore-private ignore-no-store ignore-auth
> refresh_pattern images/* 10080 16% 259200 ignore-no-cache
> ignore-private ignore-no-store ignore-auth override-expire
> override-lastmod
> refresh_pattern text/* 0 16% 259200 refresh-ims
> refresh_pattern video/* 43200 80% 259200 ignore-no-cache
> ignore-private ignore-no-store ignore-auth
> refresh_pattern . 0 80% 259200 ignore-no-cache ignore-private
> ignore-no-store ignore-auth
> quick_abort_min -1 KB
> quick_abort_max 16384000 KB
> quick_abort_pct 5
> read_ahead_gap 7 KB
> negative_ttl 60 seconds
> positive_dns_ttl 1 hours
> negative_dns_ttl 5 minutes
> range_offset_limit -1 KB
> minimum_expiry_time 60 seconds
> vary_ignore_expire on
> reply_header_access Pragma deny all
> reply_header_access Cache-Control deny all
> shutdown_lifetime 3 seconds
> cache_mgr jspegal
> httpd_suppress_version_string on
> visible_hostname ichiban
> client_persistent_connections on
> server_persistent_connections on
> persistent_connection_after_error off
> detect_broken_pconn on
> icp_port 3130
> log_icp_queries on
> short_icon_urls on
> deny_info about:blank blocked_sites
> nonhierarchical_direct on
> prefer_direct off
> check_hostnames on
> dns_retransmit_interval 1 seconds
> dns_timeout 1 minutes
> dns_nameservers 10.10.122.250
> hosts_file /etc/hosts
> ipcache_size 12000
> ipcache_low 90
> ipcache_high 95
> fqdncache_size 12000
> memory_pools on
> memory_pools_limit 256 MB
> forwarded_for off
> cachemgr_passwd none all
> client_db on
> refresh_all_ims off
> maximum_single_addr_tries 3
> retry_on_error on
> offline_mode off
> coredump_dir /var/cache/squid
> balance_on_multiple_ip off
> pipeline_prefetch on
>
>
> Hopefully this is helpful.
Received on Tue Jan 19 2010 - 05:33:47 MST