Re: [squid-users] CARP Failover behavior - multiple parents chosen for URL

>
> On May 6, 2009, at 8:14 PM, Amos Jeffries wrote:
>
>>> Hi,
>>>
>>> I've noticed a behavior in CARP failover (on 2.7) that I was
>>> wondering
>>> if someone could explain.
>>>
>>> In my test environment, I have a non-caching squid configured with
>>> multiple CARP parent caches - two servers, three per box (listening
>>> on
>>> ports 1080/1081/1082, respectively, for a total of six servers.
>>>
>>> When I fail a squid instance and immediately afterwards run GETs to
>>> URLs that were previously directed to that instance, I notice that
>>> the
>>> request goes to a different squid, as expected, and I see the
>>> following in the log for each request:
>>>
>>> May 6 11:43:28 cdce-den002-001 squid[1557]: TCP connection to http-
>>> cache-1c.den002 (http-cache-1c.den002:1082) failed
>>>
>>> And I notice that the request is being forwarded to a different, but
>>> consistent, parent.
>>>
>>> After ten of the above requests, I see this:
>>>
>>> May 6 11:43:41 cdce-den002-001.den002 squid[1557]: Detected DEAD
>>> Parent: http-cache-1c.den002
>>>
>>> So, I'm presuming that after ten failed requests, the peer is
>>> considered DEAD. So far, so good.
>>>
>>> The problem is this: During my test GETs, I noticed that immediately
>>> after the "Detected DEAD Parent" message was generated, the parent
>>> server that the request was being forwarded to changed - as if
>>> there's
>>> an "interim" decision made until the peer is officially declared
>>> DEAD,
>>> and then another hash decision made afterwards. So while consistent
>>> afterwards, it's apparent that during the failover, the parent server
>>> for the test URL changed twice, not once.
>>>
>>> Can someone explain this behavior?
>>
>> Do you have 'default' set on any of the parents?
>> It is entirely possible that multiple paths are selected as usable and
>> only the first taken.
>>
>
> No, my cache_peer config options are
>
> cache_peer http-cache-1a.den002 parent 1080 0 carp http11 idle=10
> <repeat for each hostname>
>
>
>
>> During the period between death and detection the dead peer will
>> still be
>> attempted but failover happens to send the request to another
>> location.
>> When death is detected the hashes are actual re-calculated.
>>
>
> OK, correct me if I misread, but my understanding of the spec is that
> each parent cache gets its own hash value, each of which is then
> combined with the URL's hash to come up with a set of values. The
> parent cache corresponding with the highest result is the cache
> chosen. If that peer is unavailable, the next-best peer is selected,
> then the next, etc etc.
>
> If that is correct, what hashes are re-calculated when a dead peer is
> detected? Any why would those hashes result in different results than
> the pre-dead peer run of the algorithm
>
> And more importantly, will that recalculation result in URLs being re-
> mapped that weren't originally pointed to the failed parent? I thought
> avoiding such an arbitrary re-mapping was the whole point of the CARP
> algorithm.
>
> -C
>
>> If anyone wants a task it may be useful to see whether leaving dead
>> peers
>> in the existing hash and omitting the dead peers at the selection time
>> instead of connection time is more responsive like this while
>> reducing the
>> double-change.
>>
>
> Again, I'm not clear on the difference between the two - educate me
> please :)

If you have actually read the CARP specs you are one up on me. I've only
read the peer selection code.
Given a few minutes to think about it yes, your point above about URL
'stickiness' must be right.

I was thinking that the peer hash of the pairing gets re-calculated since
a peer #n has dropped of the possible list. Maybe thats not true. In which
case one of the peer choices you have noticed should not be happening at
all. My thought (without any backing) is that the second choice (after
death) is probably the right one which should have been made at the first
failover point.

Amos
Received on Thu May 07 2009 - 02:13:27 MDT