AsyncCall API notes

Hello,

    As a part of Squid3 cleanup and planning, I promised to assemble
AsyncCall API-related notes from a few squid-dev threads and bring them
in sync with the current implementation. The first draft of that
documentation is below, FYI.

This is a work-in-progress and no formal review is requested at this
time. Moreover, please keep in mind that portions of the text refer to
fixes being implemented in the current trunk cleanup effort. Those
portions may not be in sync with the code in the trunk (I am posting
those fixes today though!).

If something jumps at you as totally wrong with respect to How Things
Should Be Done, please yell.

---
Terminology:
* Call: a dumb object meant to be delivered from the Caller to the
  Recipient.
* Recipient: the code being the destination of the Call. The specific
  function receiving the call is often called "handler".
* Caller: the code scheduling or placing the Call.
* Creator: the code creating the Call.
* Dialing: the final act of delivering the Call to the Recipient.
* Canceling: preventing the call from being dialed in the future.
* Pending call: the call which has been scheduled but no attempt to dial
  it has been made.
* Live call: the in-progress call that has been dialed and the Recipient
  (receiving) function has not finished yet.
Typical lifecycle:
1. Creator creates a Call object and stores its pointer, possibly in a
temporary variable. If Creator is the future Recipient of the Call, it
is customary to talk about a "callback". In such context, Creator gives
the Call pointer to some other code (the future Caller). Calls are
refcounted and the AsyncCall::Pointer class must be used to point and
share them.
2. Time passes and something triggers the need to dial the call.
3. Caller schedules (a.k.a, places) the Call. The Call is placed in the
AsyncCallQueue. The queue preserves the scheduling call order.
4. Time passes and it is now time to dial the call.
5. AsyncCallQueue checks whether the call can be dialed. If yes,
AsyncCallQueue dials the Call and the Recipient receives it. Otherwise,
only a debugging message is logged (if enabled). Dialing a call involves
several state checks at several stages, depending on the Call type.
6. The Call is alive while the Recipient is handling it. That state ends
when the Recipient code (a.k.a., handler) returns. At that point, the
AsyncCallQueue proceeds to handle other calls.
7. The Call object is destroyed when the last reference to it is gone.
One should not try to invalidate or delete the call.
Basic rules:
* Callbacks must be called: If a Caller has a Call pointer as a
callback, the Caller must make that Call when conditions are right
(possibly with a call-specific error indicator). Whether that Call will
be dialed is irrelevant here.
* Unwanted calls must be canceled: If Recipient may not be able to
handle the call back, then it must cancel it.  Whether that Call will be
scheduled is irrelevant here. If the Recipient has an AsyncCall pointer,
calling AsyncCall::cancel is sufficient, but the code should use
call-specific APIs when possible (e.g., comm_read_cancel or comm_close).
* Processed calls should be forgotten: If you scheduled, received, or
cancel the call, set its pointer to NULL. The Caller should forget the
call after scheduling it. the recipient should forget the call when it
receives or cancels it. These are just cleanup rules to avoid
double-scheduling or double-cancellation as well as situations where the
code assumes it can still change a pending Call (which will not work for
SMP code).
* The calls are received in the order they were scheduled.
* The Recipient is guaranteed to receive at most one Call at a time. An
attempt to dial call B while some call A is alive is a bug, and call B
will not be dialed.
Call Canceling:
* A call may be canceled at any time.
* A canceled call will not be dialed. 
* Cancellation has immediate effect (i.e., the call will never be dialed
after cancel() is called), but cancellation may be "late" if the call
has already been dialed (i.e., the call is or was alive).
* Cancellation by non-Recipients is discouraged because it may not work
in a SMP environment.
* It is not a bug to cancel something late, but the canceling code
should be aware of that possibility. Canceling something late is a
no-op.
---
HTH,
Alex.