Re: [squid-users] Strange Facebook Problems

Jason wrote:
> Jason wrote:
>> Amos Jeffries wrote:
>>> Jason wrote:
>>>> Amos Jeffries wrote:
>>>>> On Wed, 19 Aug 2009 20:58:19 -0700, Jason <jason_at_azii.net> wrote:
>>>>>
>>>>>> Everyone,
>>>>>>
>>>>>> I am running squid 3.1.0.6, transparent/intercepting (non tproxy,
>>>>>> non wpad, etc), nat'ted network, and users are reporting problems
>>>>>> using the uploaders at the facebook website. When I explored
>>>>>> this, here is what I found:
>>>>>>
>>>>>> Facebook has two upload methods, a newer java based one, and an
>>>>>> older html forms (i think) based one.
>>>>>>
>>>>>> 1. Both uploaders work perfectly when I bypass squid.
>>>>>>
>>>>>> 2. With internet explorer, the old uploader works fine
>>>>>>
>>>>>> 3. With Internet Explorer, the new uploader fails at first. If
>>>>>> you immediately hit the "Upload" button after the failure, it works.
>>>>>>
>>>>>> 4. With Firefox, the old uploader gives this error from squid:
>>>>>> ERROR
>>>>>> The requested URL could not be retrieved
>>>>>> The following error was encountered while trying to retrieve the
>>>>>> URL: http://upload.facebook.com
>>>>>> /photos_upload.php
>>>>>> Connection to upload.facebook.com failed.
>>>>>> The system returned: (110) Connection timed out
>>>>>> The remote host or network may be down. Please try the request again.
>>>>>> Your cache administrator is yours truly.
>>>>>>
>>>>>> 5. With Firefox, the new uploader fails at first. If you
>>>>>> immediately hit the "Upload" button after a failure, it works
>>>>>> (just like the IE
>>>>>>
>>>>> case).
>>>>>
>>>>>> On the proxy machine:
>>>>>> tcp_window_scaling is off
>>>>>> tcp_ecn is off
>>>>>> .facebook.com is in the "always direct" list I maintain.
>>>>>>
>>>>>>
>>>>>> Any help in solving this would be great!
>>>>>>
>>>>>> Jason
>>>>>>
>>>>>> Below is my Config:
>>>>>> qos_flows local-hit=0x30
>>>>>> acl manager proto cache_object
>>>>>> acl localhost src 127.0.0.1/32
>>>>>> acl to_localhost dst 127.0.0.0/8
>>>>>> acl Safe_ports port 80 # http
>>>>>> acl CONNECT method CONNECT
>>>>>> http_access allow manager localhost
>>>>>> http_access deny manager
>>>>>> acl our_networks src 10.0.0.0/16
>>>>>> http_access allow our_networks
>>>>>> http_access allow localhost
>>>>>> acl directlist dstdomain "/etc/squid/directsites"
>>>>>> always_direct allow directlist
>>>>>> http_access deny all
>>>>>> http_reply_access allow our_networks
>>>>>> http_reply_access allow localhost
>>>>>> http_reply_access deny all
>>>>>> icp_access deny all
>>>>>> htcp_access deny all
>>>>>> htcp_clr_access deny all
>>>>>> miss_access allow our_networks
>>>>>> miss_access allow localhost
>>>>>> miss_access deny all
>>>>>> http_port 10.0.0.1:3594 transparent
>>>>>> disable-pmtu-discovery=transparent
>>>>>> http_port 127.0.0.1:3594 transparent
>>>>>> disable-pmtu-discovery=transparent
>>>>>> cache_mem 128 MB
>>>>>> memory_replacement_policy heap GDSF
>>>>>> cache_replacement_policy heap LFUDA
>>>>>> cache_dir aufs /squida 21760 16 256
>>>>>> cache_dir aufs /squidb 21760 16 256
>>>>>> cache_dir aufs /squidc 21760 16 256
>>>>>> max_open_disk_fds 0
>>>>>> minimum_object_size 0 KB
>>>>>> maximum_object_size 10 MB
>>>>>> cache_swap_low 95
>>>>>> cache_swap_high 97
>>>>>> access_log /var/log/squid/access.log
>>>>>> cache_log /var/log/squid/cache.log
>>>>>> cache_store_log /var/log/squid/store.log
>>>>>> mime_table /etc/squid/mime.conf
>>>>>> pid_filename /var/run/squid.pid
>>>>>> log_fqdn off
>>>>>> strip_query_terms off
>>>>>> unlinkd_program /usr/lib/squid/unlinkd
>>>>>> url_rewrite_program /usr/bin/squidGuard
>>>>>> url_rewrite_children 32
>>>>>> url_rewrite_concurrency 0
>>>>>> url_rewrite_host_header on
>>>>>> url_rewrite_bypass off
>>>>>> refresh_pattern (cgi-bin|\?) 0 0% 0
>>>>>> refresh_pattern . 0 20% 4320
>>>>>> quick_abort_min 50 KB
>>>>>> quick_abort_max 50 KB
>>>>>> quick_abort_pct 50
>>>>>> read_ahead_gap 16 KB
>>>>>> negative_ttl 0 minutes
>>>>>> positive_dns_ttl 5 minutes
>>>>>> negative_dns_ttl 10 seconds
>>>>>> range_offset_limit 0 KB
>>>>>> request_header_max_size 128 KB
>>>>>> reply_header_max_size 128 KB
>>>>>> ie_refresh on
>>>>>> request_entities on
>>>>>> forward_timeout 1 minutes
>>>>>> connect_timeout 20 seconds
>>>>>> shutdown_lifetime 3 seconds default
>>>>>> cache_mgr support_at_azii.net
>>>>>> cache_effective_user proxy
>>>>>> cache_effective_group proxy
>>>>>> visible_hostname integrityinternet.net
>>>>>> snmp_port 45656
>>>>>> snmp_access allow our_networks
>>>>>> snmp_access allow localhost
>>>>>> snmp_access deny all
>>>>>> snmp_incoming_address 10.0.0.1
>>>>>> icon_directory /usr/share/squid/icons
>>>>>> dns_nameservers 127.0.0.1
>>>>>> ipcache_size 5120
>>>>>> ipcache_low 95
>>>>>> ipcache_high 97
>>>>>> fqdncache_size 5120
>>>>>> memory_pools_limit 512 MB
>>>>>> client_db off
>>>>>> uri_whitespace strip
>>>>>> coredump_dir /squida
>>>>>> pipeline_prefetch off
>>>>>> client_persistent_connections off
>>>>>> server_persistent_connections off
>>>>>>
>>>>>
>>>>>
>>>>> Please note that "always_direct" does not mean the URLs bypass
>>>>> Squid. It
>>>>> means that squid will not pass those requests to a cache_peer
>>>>> server. Of
>>>>> which you have none, meaning the always_direct is merely wasting
>>>>> CPU time.
>>>>>
>>>>> Please try these:
>>>>>
>>>>> * a current release of 3.1
>>>>>
>>>>> * turning persistent connections ON.
>>>>> client_persistent_connections off
>>>>> server_persistent_connections off
>>>>>
>>>>>
>>>>> Amos
>>>>>
>>>>>
>>>>>
>>>> Amos,
>>>>
>>>> Thank you for replying. I've tried persistent_connections both
>>>> ways, with no difference. Next, I'll try the latest squid 3.1. Also
>>>> thanks for the tip regarding always_direct. I thought it meant that
>>>> squid would not look in its cache for that site.
>>>>
>>>> Jason
>>>>
>>>
>>> Ah, to not use the local storage its the "cache" directive with ACLs
>>> describing what not to store.
>>> http://www.squid-cache.org/Doc/config/cache/
>>>
>>> Amos
>> I've compiled and am running on squid 3.1.0.13, still having the
>> problems with facebook. The 3.1.0.13 error pages in squid are nicer
>> to look at than the 3.1.0.6 pages were, however!
>>
>> Anyone have any other ideas? Could it be that something in the http
>> communication is breaking down?
>>
>> Jason
>>
>>
> More information: When I manually inform the web browsers of the proxy,
> the facebook error is still there. So I don't think it has anything to
> do with being transparent/intercepting.
>
> Jason

The problem is definitely this:
    "The system returned: (110) Connection timed out "
when running the first request the TCP link takes too long to establish.

On following requests it seems to have underlying system routing states
already found (ie DNS caches in the resolver or Squid or persistent TCP
connections already opened or firewall state about MTU and routing tabel
already stored)

It might be as simple as these being too short:

  forward_timeout 1 minutes - total time allowed to search for a
working destination (opening, writing and receiving reply back from
multiple TCP links).

  connect_timeout 20 seconds - maximum time allowed to open a TCP link.

Yes the ideal for web stuff is 10 sec or much less for a full forwarding
and reply. Sometimes it can take a while though when sending data.

upload.facebook.com seems to be having DNS trouble from what I can tell
right now.

host upload.facebook.com
-->
   upload.facebook.com has address 69.63.178.32
   ;; connection timed out; no servers could be reached
   Host upload.facebook.com not found: 3(NXDOMAIN)

Takes a total of 12 seconds to reply with two contradictory answers.

Your DNS settings in squid.conf may also be compounding the problem.
You are enforcing a minimum storage time for positive DNS results inside
Squid of 5 minutes (3600 seconds). Negative results ar only stored for a
minimumof 10 seconds. The facebook IPs change every 30 to 900 seconds
on some load balancing.
Playing with DNS TTLs on domains you don't own will I expect cause the
following behaviour for you:

Some DNS lookup for facebook:
  -> some DNS answer comes back with an IP
  --> connection made and client request happens nicely...
  -> 30 seconds or so later the IP changes.

Some other request for facebook (less than 5 minutes from first)..
  -> Squid checks for IPs. Forced to use the cached one.
    ~~> it _may_ or may not still actually work.
   -> if it has not you are lucky and things work.
   -> on failure Squid sends out a "connection failed" error to the
client due to bad IPs.
   -> Squid marks the one IP as bad and needing replacement.

Next request for facebook arrives...
  -> Squid sees its IPs are all bad. Checks negative TTL.
     --> sees negative TTL as passed and new lookup needed.
  -> repeat from step #1.

This is not limited to facebook, but any domain with TTL shorter than
your cache minimums. Luckily DNS best-practice is to set TTL of 24 hours
or so. Most domains will be doing that and not cause you problems.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE6 or 3.0.STABLE18
   Current Beta Squid 3.1.0.13