Some may recall from IRC a while back discussing the fdc_table array and
how horribly it wasted memory.
This is a full re-working of the way AcceptFD are handled in Squid-3.
Previously:
a pre-defined fdc_table was initialized and left waiting just in case
any of the available FD were needed as a listening port. At which point
a handler was assigned and select was setup. Much action was wasted
initializing the array on startup and shutdown, and various unnecessary
maintenance references in comms every FD closure.
Now:
one merely opens a socket, defines the AsyncCall handler for the
listening FD and creates a ListenStateData object from the two. When a
listener socket is done with, delete the ListenStateData and the FD gets
closed. Callers are responsible for maintaining a pointer to the
ListenStateData.
ListenStateData silently does all the comm processing for setting up and
watching for new connections. Including whether accept() on a port is
delayed until more FD are available. Then when any accept() is completed
it sends the resulting FD and ConnectionDetails objects to the assigned
callback.
COMM_ERR_CLOSING and re-scheduling themselves is now not generally
relevant to layers higher than comm on listening sockets. The callbacks
never get called at all unless a real new connection has been accepted.
The old code is still used internally by the comm layer, but details of
error handling and re-scheduling of accept() never leak out into higher
level code now.
ListenStateData can be created in use-once or accept-many form.
* use-once are intended for short temporary listening sockets such as
FTP passive data links. A single inbound connection is allowed after
which the handler self-destructs.
* accept-many are for long term FD such as the http_port listeners
which continuously accept new connections and pass them all to the same
handler until something causes the ListenStateData to die (deletion, or
self destruct under fatal socket IO errors).
The previous existing AcceptLimiter is slightly remodeled to work with
ListenStateData* instead of an intermediary callback reference type. And
also altered from LIFO to FIFO behavior for better client response times
under load.
All the code relevant to comm layer processing of accept() is bundled
into libcomm-listener.la.
TODO:
* wrap the SSL handshake into ListenStateData. That way SSL on some
sockets can be treated as a real transport layer and for example
httpsAccept and httpAccept merged into one function.
* there is one bug uncovered, that when the accept() handler is deleted
and the port closed while it has been deferred. The deferred queue needs
to be cleared of all (multiple) deferral events, or they will be
scheduled uselessly and may segfault with FD errors.
This only gets hit if Squid is shutdown during active load limiting.
Amos
# Bazaar merge directive format 2 (Bazaar 0.90)
# revision_id: squid3_at_treenet.co.nz-20100102101700-lbtyg0y63czxez15
# target_branch: http://www.squid-cache.org/bzr/squid3/trunk/
# testament_sha1: 17fd946e8119c788d0a459a53e1a38c1b5873edd
# timestamp: 2010-01-03 00:14:35 +1300
# base_revision_id: squid3_at_treenet.co.nz-20100102053754-\
# z28ezqcpx71uylwo
#
# Begin patch
=== modified file 'configure.in'
--- configure.in 2009-12-19 11:56:02 +0000
+++ configure.in 2010-01-01 21:26:45 +0000
@@ -2572,6 +2572,7 @@
linux/types.h \
machine/byte_swap.h \
malloc.h \
+ map \
math.h \
memory.h \
mount.h \
@@ -4269,6 +4270,7 @@
src/adaptation/Makefile \
src/adaptation/icap/Makefile \
src/adaptation/ecap/Makefile \
+ src/comm/Makefile \
src/esi/Makefile \
src/eui/Makefile \
src/icmp/Makefile \
=== modified file 'src/Makefile.am'
--- src/Makefile.am 2010-01-01 21:16:57 +0000
+++ src/Makefile.am 2010-01-02 04:28:30 +0000
@@ -34,7 +34,7 @@
LoadableModules.h \
LoadableModules.cc
-SUBDIRS = base eui acl fs repl auth ip icmp ident log
+SUBDIRS = base comm eui acl fs repl auth ip icmp ident log
if USE_ADAPTATION
SUBDIRS += adaptation
@@ -532,6 +532,7 @@
squid_LDADD = \
$(COMMON_LIBS) \
+ comm/libcomm-listener.la \
eui/libeui.la \
icmp/libicmp.la icmp/libicmp-core.la \
log/liblog.la \
@@ -1220,8 +1221,10 @@
wordlist.cc
nodist_tests_testCacheManager_SOURCES = \
$(BUILT_SOURCES)
+# comm.cc only requires comm/libcomm-listener.la until fdc_table is dead.
tests_testCacheManager_LDADD = \
$(COMMON_LIBS) \
+ comm/libcomm-listener.la \
icmp/libicmp.la icmp/libicmp-core.la \
log/liblog.la \
$(REPL_OBJS) \
@@ -1402,6 +1405,7 @@
tests_testEvent_LDADD = \
$(COMMON_LIBS) \
icmp/libicmp.la icmp/libicmp-core.la \
+ comm/libcomm-listener.la \
log/liblog.la \
$(REPL_OBJS) \
${ADAPTATION_LIBS} \
@@ -1554,6 +1558,7 @@
tests_testEventLoop_LDADD = \
$(COMMON_LIBS) \
icmp/libicmp.la icmp/libicmp-core.la \
+ comm/libcomm-listener.la \
log/liblog.la \
$(REPL_OBJS) \
${ADAPTATION_LIBS} \
@@ -1701,6 +1706,7 @@
tests_test_http_range_LDADD = \
$(COMMON_LIBS) \
icmp/libicmp.la icmp/libicmp-core.la \
+ comm/libcomm-listener.la \
log/liblog.la \
$(REPL_OBJS) \
${ADAPTATION_LIBS} \
@@ -1853,6 +1859,7 @@
tests_testHttpRequest_LDADD = \
$(COMMON_LIBS) \
icmp/libicmp.la icmp/libicmp-core.la \
+ comm/libcomm-listener.la \
log/liblog.la \
$(REPL_OBJS) \
${ADAPTATION_LIBS} \
@@ -1927,11 +1934,11 @@
tests/testStoreHashIndex.h \
tests/TestSwapDir.cc \
tests/TestSwapDir.h \
- tests/stub_fd.cc \
tests/stub_HelperChildConfig.cc \
tests/stub_HttpReply.cc \
tests/stub_cache_manager.cc \
- $(STORE_TEST_SOURCES)
+ $(STORE_TEST_SOURCES) \
+ tests/stub_fd.cc
nodist_tests_testStore_SOURCES= \
$(TESTSOURCES) \
@@ -2220,6 +2227,7 @@
tests_testURL_LDADD = \
$(COMMON_LIBS) \
icmp/libicmp.la icmp/libicmp-core.la \
+ comm/libcomm-listener.la \
log/liblog.la \
$(REGEXLIB) \
$(REPL_OBJS) \
=== modified file 'src/ProtoPort.cc'
--- src/ProtoPort.cc 2009-01-21 03:47:47 +0000
+++ src/ProtoPort.cc 2009-12-31 02:35:01 +0000
@@ -1,7 +1,5 @@
-
/*
* $Id$
- *
*/
#include "squid.h"
@@ -18,6 +16,8 @@
http_port_list::~http_port_list()
{
+ delete listener;
+
safe_free(name);
safe_free(defaultsite);
safe_free(protocol);
=== modified file 'src/ProtoPort.h'
--- src/ProtoPort.h 2009-09-18 23:46:33 +0000
+++ src/ProtoPort.h 2009-12-31 02:35:01 +0000
@@ -6,6 +6,7 @@
//#include "typedefs.h"
#include "cbdata.h"
+#include "comm/ListenStateData.h"
struct http_port_list {
http_port_list(const char *aProtocol);
@@ -37,6 +38,13 @@
unsigned int timeout;
} tcp_keepalive;
+ /**
+ * The FD listening socket handler.
+ * If not NULL we are actively listening for client requests.
+ * delete to close the socket.
+ */
+ Comm::ListenStateData *listener;
+
#if USE_SSL
// XXX: temporary hack to ease move of SSL options to http_port
http_port_list &http;
=== modified file 'src/client_side.cc'
--- src/client_side.cc 2010-01-01 01:12:55 +0000
+++ src/client_side.cc 2010-01-01 21:26:45 +0000
@@ -82,30 +82,32 @@
*/
#include "squid.h"
+
+#include "acl/FilledChecklist.h"
+#include "auth/UserRequest.h"
+#include "ChunkedCodingParser.h"
#include "client_side.h"
+#include "client_side_reply.h"
+#include "client_side_request.h"
+#include "ClientRequestContext.h"
#include "clientStream.h"
-#include "ProtoPort.h"
-#include "auth/UserRequest.h"
-#include "Store.h"
#include "comm.h"
+#include "comm/ListenStateData.h"
+#include "ConnectionDetail.h"
+#include "eui/Config.h"
+#include "fde.h"
#include "HttpHdrContRange.h"
#include "HttpReply.h"
#include "HttpRequest.h"
#include "ident/Config.h"
#include "ident/Ident.h"
#include "ip/IpIntercept.h"
+#include "MemBuf.h"
#include "MemObject.h"
-#include "fde.h"
-#include "client_side_request.h"
-#include "acl/FilledChecklist.h"
-#include "ConnectionDetail.h"
-#include "client_side_reply.h"
-#include "ClientRequestContext.h"
-#include "MemBuf.h"
+#include "ProtoPort.h"
+#include "rfc1738.h"
#include "SquidTime.h"
-#include "ChunkedCodingParser.h"
-#include "eui/Config.h"
-#include "rfc1738.h"
+#include "Store.h"
#if LINGERING_CLOSE
#define comm_close comm_lingering_close
@@ -148,7 +150,6 @@
static CSD clientSocketDetach;
static void clientSetKeepaliveFlag(ClientHttpRequest *);
static int clientIsContentLengthValid(HttpRequest * r);
-static bool okToAccept();
static int clientIsRequestBodyValid(int64_t bodyLength);
static int clientIsRequestBodyTooLargeForPolicy(int64_t bodyLength);
@@ -2965,8 +2966,6 @@
#endif
}
-
-
static void
clientLifetimeTimeout(int fd, void *data)
{
@@ -2977,22 +2976,6 @@
comm_close(fd);
}
-static bool
-okToAccept()
-{
- static time_t last_warn = 0;
-
- if (fdNFree() >= RESERVED_FD)
- return true;
-
- if (last_warn + 15 < squid_curtime) {
- debugs(33, 0, HERE << "WARNING! Your cache is running out of filedescriptors");
- last_warn = squid_curtime;
- }
-
- return false;
-}
-
ConnStateData *
connStateCreate(const IpAddress &peer, const IpAddress &me, int fd, http_port_list *port)
{
@@ -3047,16 +3030,6 @@
http_port_list *s = (http_port_list *)data;
ConnStateData *connState = NULL;
- if (flag == COMM_ERR_CLOSING) {
- return;
- }
-
- if (!okToAccept())
- AcceptLimiter::Instance().defer (sock, httpAccept, data);
- else
- /* kick off another one for later */
- comm_accept(sock, httpAccept, data);
-
if (flag != COMM_OK) {
debugs(33, 1, "httpAccept: FD " << sock << ": accept failure: " << xstrerr(xerrno));
return;
@@ -3263,16 +3236,6 @@
https_port_list *s = (https_port_list *)data;
SSL_CTX *sslContext = s->sslContext;
- if (flag == COMM_ERR_CLOSING) {
- return;
- }
-
- if (!okToAccept())
- AcceptLimiter::Instance().defer (sock, httpsAccept, data);
- else
- /* kick off another one for later */
- comm_accept(sock, httpsAccept, data);
-
if (flag != COMM_OK) {
errno = xerrno;
debugs(33, 1, "httpsAccept: FD " << sock << ": accept failure: " << xstrerr(xerrno));
@@ -3308,7 +3271,6 @@
if (identChecklist.fastCheck())
Ident::Start(details->me, details->peer, clientIdentDone, connState);
}
-
#endif
if (s->http.tcp_keepalive.enabled) {
@@ -3384,6 +3346,8 @@
++bumpCount;
#endif
+ /* AYJ: 2009-12-27: bit bumpy. new ListenStateData(...) should be doing all the Comm:: stuff ... */
+
enter_suid();
if (s->spoof_client_ip) {
@@ -3397,9 +3361,10 @@
if (fd < 0)
continue;
- comm_listen(fd);
+ AsyncCall::Pointer call = commCbCall(5,5, "SomeCommAcceptHandler(httpAccept)",
+ CommAcceptCbPtrFun(httpAccept, s));
- comm_accept(fd, httpAccept, s);
+ s->listener = new Comm::ListenStateData(fd, call, true);
debugs(1, 1, "Accepting " <<
(s->intercepted ? " intercepted" : "") <<
@@ -3450,9 +3415,10 @@
if (fd < 0)
continue;
- comm_listen(fd);
+ AsyncCall::Pointer call = commCbCall(5,5, "SomeCommAcceptHandler(httpsAccept)",
+ CommAcceptCbPtrFun(httpsAccept, s));
- comm_accept(fd, httpsAccept, s);
+ s->listener = new Comm::ListenStateData(fd, call, true);
debugs(1, 1, "Accepting HTTPS connections at " << s->http.s << ", FD " << fd << ".");
@@ -3467,7 +3433,6 @@
{
clientHttpConnectionsOpen();
#if USE_SSL
-
clientHttpsConnectionsOpen();
#endif
@@ -3478,15 +3443,27 @@
void
clientHttpConnectionsClose(void)
{
- int i;
-
- for (i = 0; i < NHttpSockets; i++) {
- if (HttpSockets[i] >= 0) {
- debugs(1, 1, "FD " << HttpSockets[i] <<
- " Closing HTTP connection");
- comm_close(HttpSockets[i]);
- HttpSockets[i] = -1;
- }
+ for (http_port_list *s = Config.Sockaddr.http; s; s = s->next) {
+ if (s->listener) {
+ debugs(1, 1, "FD " << s->listener->fd << " Closing HTTP connection");
+ delete s->listener;
+ s->listener = NULL;
+ }
+ }
+
+#if USE_SSL
+ for (http_port_list *s = Config.Sockaddr.https; s; s = s->next) {
+ if (s->listener) {
+ debugs(1, 1, "FD " << s->listener->fd << " Closing HTTPS connection");
+ delete s->listener;
+ s->listener = NULL;
+ }
+ }
+#endif
+
+ // TODO see if we can drop HttpSockets array entirely */
+ for (int i = 0; i < NHttpSockets; i++) {
+ HttpSockets[i] = -1;
}
NHttpSockets = 0;
=== added directory 'src/comm'
=== modified file 'src/comm.cc'
--- src/comm.cc 2010-01-02 04:32:46 +0000
+++ src/comm.cc 2010-01-02 10:17:00 +0000
@@ -37,6 +37,9 @@
#include "comm.h"
#include "event.h"
#include "fde.h"
+#include "comm/AcceptLimiter.h"
+#include "comm/comm_internal.h"
+#include "comm/ListenStateData.h"
#include "CommIO.h"
#include "CommRead.h"
#include "ConnectionDetail.h"
@@ -48,7 +51,6 @@
#include "icmp/net_db.h"
#include "ip/IpAddress.h"
#include "ip/IpIntercept.h"
-#include "protos.h"
#if defined(_SQUID_CYGWIN_)
#include <sys/ioctl.h>
@@ -243,44 +245,15 @@
static PF commHandleWrite;
static IPH commConnectDnsHandle;
-static PF comm_accept_try;
-
-class AcceptFD
-{
-
-public:
- AcceptFD(int aFd = -1): fd(aFd), theCallback(0), mayAcceptMore(false) {}
-
- void subscribe(AsyncCall::Pointer &call);
- void acceptNext();
- void notify(int newfd, comm_err_t, int xerrno, const ConnectionDetail &);
-
- int fd;
-
-private:
- bool acceptOne();
-
- AsyncCall::Pointer theCallback;
- bool mayAcceptMore;
-};
-
typedef enum {
COMM_CB_READ = 1,
COMM_CB_DERIVED
} comm_callback_t;
-struct _fd_debug_t {
- char const *close_file;
- int close_line;
-};
-
-typedef struct _fd_debug_t fd_debug_t;
-
static MemAllocator *conn_close_pool = NULL;
-AcceptFD *fdc_table = NULL; // TODO: rename. And use Vector<>?
fd_debug_t *fdd_table = NULL;
-static bool
+bool
isOpen(const int fd)
{
return fd_table[fd].flags.open != 0;
@@ -1373,77 +1346,6 @@
return status;
}
-/* Wait for an incoming connection on FD. FD should be a socket returned
- * from comm_listen. */
-static int
-comm_old_accept(int fd, ConnectionDetail &details)
-{
- PROF_start(comm_accept);
- statCounter.syscalls.sock.accepts++;
- int sock;
- struct addrinfo *gai = NULL;
- details.me.InitAddrInfo(gai);
-
- if ((sock = accept(fd, gai->ai_addr, &gai->ai_addrlen)) < 0) {
-
- details.me.FreeAddrInfo(gai);
-
- PROF_stop(comm_accept);
-
- if (ignoreErrno(errno)) {
- debugs(50, 5, "comm_old_accept: FD " << fd << ": " << xstrerror());
- return COMM_NOMESSAGE;
- } else if (ENFILE == errno || EMFILE == errno) {
- debugs(50, 3, "comm_old_accept: FD " << fd << ": " << xstrerror());
- return COMM_ERROR;
- } else {
- debugs(50, 1, "comm_old_accept: FD " << fd << ": " << xstrerror());
- return COMM_ERROR;
- }
- }
-
- details.peer = *gai;
-
- if ( Config.client_ip_max_connections >= 0) {
- if (clientdbEstablished(details.peer, 0) > Config.client_ip_max_connections) {
- debugs(50, DBG_IMPORTANT, "WARNING: " << details.peer << " attempting more than " << Config.client_ip_max_connections << " connections.");
- details.me.FreeAddrInfo(gai);
- return COMM_ERROR;
- }
- }
-
- details.me.InitAddrInfo(gai);
-
- details.me.SetEmpty();
- getsockname(sock, gai->ai_addr, &gai->ai_addrlen);
- details.me = *gai;
-
- commSetCloseOnExec(sock);
-
- /* fdstat update */
- fd_open(sock, FD_SOCKET, "HTTP Request");
- fdd_table[sock].close_file = NULL;
- fdd_table[sock].close_line = 0;
- fde *F = &fd_table[sock];
- details.peer.NtoA(F->ipaddr,MAX_IPSTRLEN);
- F->remote_port = details.peer.GetPort();
- F->local_addr.SetPort(details.me.GetPort());
-#if USE_IPV6
- F->sock_family = AF_INET;
-#else
- F->sock_family = details.me.IsIPv4()?AF_INET:AF_INET6;
-#endif
- details.me.FreeAddrInfo(gai);
-
- commSetNonBlocking(sock);
-
- /* IFF the socket is (tproxy) transparent, pass the flag down to allow spoofing */
- F->flags.transparent = fd_table[fd].flags.transparent;
-
- PROF_stop(comm_accept);
- return sock;
-}
-
void
commCallCloseHandlers(int fd)
{
@@ -1541,7 +1443,6 @@
}
-
void
comm_close_complete(int fd, void *data)
{
@@ -1558,15 +1459,10 @@
close(fd);
- fdc_table[fd] = AcceptFD(fd);
-
statCounter.syscalls.sock.closes++;
/* When an fd closes, give accept() a chance, if need be */
-
- if (fdNFree() >= RESERVED_FD)
- AcceptLimiter::Instance().kick();
-
+ Comm::AcceptLimiter::Instance().kick();
}
/*
@@ -1626,9 +1522,6 @@
commio_finish_callback(fd, COMMIO_FD_READCB(fd), COMM_ERR_CLOSING, errno);
}
- // notify accept handlers
- fdc_table[fd].notify(-1, COMM_ERR_CLOSING, 0, ConnectionDetail());
-
commCallCloseHandlers(fd);
if (F->pconn.uses)
@@ -1941,10 +1834,8 @@
fd_table =(fde *) xcalloc(Squid_MaxFD, sizeof(fde));
fdd_table = (fd_debug_t *)xcalloc(Squid_MaxFD, sizeof(fd_debug_t));
- fdc_table = new AcceptFD[Squid_MaxFD];
- for (int pos = 0; pos < Squid_MaxFD; ++pos) {
- fdc_table[pos] = AcceptFD(pos);
- }
+ /* make sure the accept() socket FIFO delay queue exists */
+ Comm::AcceptLimiter::Instance();
commfd_table = (comm_fd_t *) xcalloc(Squid_MaxFD, sizeof(comm_fd_t));
for (int pos = 0; pos < Squid_MaxFD; pos++) {
@@ -1974,10 +1865,6 @@
safe_free(fd_table);
safe_free(fdd_table);
- if (fdc_table) {
- delete[] fdc_table;
- fdc_table = NULL;
- }
safe_free(commfd_table);
}
@@ -2217,169 +2104,6 @@
}
}
-/*
- * New-style listen and accept routines
- *
- * Listen simply registers our interest in an FD for listening,
- * and accept takes a callback to call when an FD has been
- * accept()ed.
- */
-int
-comm_listen(int sock)
-{
- int x;
-
- if ((x = listen(sock, Squid_MaxFD >> 2)) < 0) {
- debugs(50, 0, "comm_listen: listen(" << (Squid_MaxFD >> 2) << ", " << sock << "): " << xstrerror());
- return x;
- }
-
- if (Config.accept_filter && strcmp(Config.accept_filter, "none") != 0) {
-#ifdef SO_ACCEPTFILTER
- struct accept_filter_arg afa;
- bzero(&afa, sizeof(afa));
- debugs(5, DBG_CRITICAL, "Installing accept filter '" << Config.accept_filter << "' on FD " << sock);
- xstrncpy(afa.af_name, Config.accept_filter, sizeof(afa.af_name));
- x = setsockopt(sock, SOL_SOCKET, SO_ACCEPTFILTER, &afa, sizeof(afa));
- if (x < 0)
- debugs(5, 0, "SO_ACCEPTFILTER '" << Config.accept_filter << "': '" << xstrerror());
-#elif defined(TCP_DEFER_ACCEPT)
- int seconds = 30;
- if (strncmp(Config.accept_filter, "data=", 5) == 0)
- seconds = atoi(Config.accept_filter + 5);
- x = setsockopt(sock, IPPROTO_TCP, TCP_DEFER_ACCEPT, &seconds, sizeof(seconds));
- if (x < 0)
- debugs(5, 0, "TCP_DEFER_ACCEPT '" << Config.accept_filter << "': '" << xstrerror());
-#else
- debugs(5, 0, "accept_filter not supported on your OS");
-#endif
- }
-
- return sock;
-}
-
-void
-comm_accept(int fd, IOACB *handler, void *handler_data)
-{
- debugs(5, 5, "comm_accept: FD " << fd << " handler: " << (void*)handler);
- assert(isOpen(fd));
-
- AsyncCall::Pointer call = commCbCall(5,5, "SomeCommAcceptHandler",
- CommAcceptCbPtrFun(handler, handler_data));
- fdc_table[fd].subscribe(call);
-}
-
-void
-comm_accept(int fd, AsyncCall::Pointer &call)
-{
- debugs(5, 5, "comm_accept: FD " << fd << " AsyncCall: " << call);
- assert(isOpen(fd));
-
- fdc_table[fd].subscribe(call);
-}
-
-// Called when somebody wants to be notified when our socket accepts new
-// connection. We do not probe the FD until there is such interest.
-void
-AcceptFD::subscribe(AsyncCall::Pointer &call)
-{
- /* make sure we're not pending! */
- assert(!theCallback);
- theCallback = call;
-
-#if OPTIMISTIC_IO
- mayAcceptMore = true; // even if we failed to accept last time
-#endif
-
- if (mayAcceptMore)
- acceptNext();
- else
- commSetSelect(fd, COMM_SELECT_READ, comm_accept_try, NULL, 0);
-}
-
-bool
-AcceptFD::acceptOne()
-{
- // If there is no callback and we accept, we will leak the accepted FD.
- // When we are running out of FDs, there is often no callback.
- if (!theCallback) {
- debugs(5, 5, "AcceptFD::acceptOne orphaned: FD " << fd);
- // XXX: can we remove this and similar "just in case" calls and
- // either listen always or listen only when there is a callback?
- if (!AcceptLimiter::Instance().deferring())
- commSetSelect(fd, COMM_SELECT_READ, comm_accept_try, NULL, 0);
- return false;
- }
-
- /*
- * We don't worry about running low on FDs here. Instead,
- * httpAccept() will use AcceptLimiter if we reach the limit
- * there.
- */
-
- /* Accept a new connection */
- ConnectionDetail connDetails;
- int newfd = comm_old_accept(fd, connDetails);
-
- /* Check for errors */
-
- if (newfd < 0) {
- assert(theCallback != NULL);
-
- if (newfd == COMM_NOMESSAGE) {
- /* register interest again */
- debugs(5, 5, HERE << "try later: FD " << fd <<
- " handler: " << *theCallback);
- commSetSelect(fd, COMM_SELECT_READ, comm_accept_try, NULL, 0);
- return false;
- }
-
- // A non-recoverable error; notify the caller */
- notify(-1, COMM_ERROR, errno, connDetails);
- return false;
- }
-
- assert(theCallback != NULL);
- debugs(5, 5, "AcceptFD::acceptOne accepted: FD " << fd <<
- " newfd: " << newfd << " from: " << connDetails.peer <<
- " handler: " << *theCallback);
- notify(newfd, COMM_OK, 0, connDetails);
- return true;
-}
-
-void
-AcceptFD::acceptNext()
-{
- mayAcceptMore = acceptOne();
-}
-
-void
-AcceptFD::notify(int newfd, comm_err_t errcode, int xerrno, const ConnectionDetail &connDetails)
-{
- if (theCallback != NULL) {
- typedef CommAcceptCbParams Params;
- Params ¶ms = GetCommParams<Params>(theCallback);
- params.fd = fd;
- params.nfd = newfd;
- params.details = connDetails;
- params.flag = errcode;
- params.xerrno = xerrno;
- ScheduleCallHere(theCallback);
- theCallback = NULL;
- }
-}
-
-/*
- * This callback is called whenever a filedescriptor is ready
- * to dupe itself and fob off an accept()ed connection
- */
-static void
-comm_accept_try(int fd, void *)
-{
- assert(isOpen(fd));
- fdc_table[fd].acceptNext();
-}
-
void CommIO::Initialise()
{
/* Initialize done pipe signal */
@@ -2434,44 +2158,6 @@
}
}
-AcceptLimiter AcceptLimiter::Instance_;
-
-AcceptLimiter &AcceptLimiter::Instance()
-{
- return Instance_;
-}
-
-bool
-AcceptLimiter::deferring() const
-{
- return deferred.size() > 0;
-}
-
-void
-AcceptLimiter::defer (int fd, Acceptor::AcceptorFunction *aFunc, void *data)
-{
- debugs(5, 5, "AcceptLimiter::defer: FD " << fd << " handler: " << (void*)aFunc);
- Acceptor temp;
- temp.theFunction = aFunc;
- temp.acceptFD = fd;
- temp.theData = data;
- deferred.push_back(temp);
-}
-
-void
-AcceptLimiter::kick()
-{
- if (!deferring())
- return;
-
- /* Yes, this means the first on is the last off....
- * If the list container was a little more friendly, we could sensibly us it.
- */
- Acceptor temp = deferred.pop_back();
-
- comm_accept (temp.acceptFD, temp.theFunction, temp.theData);
-}
-
/// Start waiting for a possibly half-closed connection to close
// by scheduling a read callback to a monitoring handler that
// will close the connection on read errors.
=== modified file 'src/comm.h'
--- src/comm.h 2009-09-20 20:20:42 +0000
+++ src/comm.h 2009-12-31 02:35:01 +0000
@@ -37,7 +37,6 @@
/* comm.c */
extern bool comm_iocallbackpending(void); /* inline candidate */
-extern int comm_listen(int fd);
SQUIDCEXTERN int commSetNonBlocking(int fd);
SQUIDCEXTERN int commUnsetNonBlocking(int fd);
SQUIDCEXTERN void commSetCloseOnExec(int fd);
@@ -103,8 +102,6 @@
class ConnectionDetail;
typedef void IOACB(int fd, int nfd, ConnectionDetail *details, comm_err_t flag, int xerrno, void *data);
-extern void comm_accept(int fd, IOACB *handler, void *handler_data);
-extern void comm_accept(int fd, AsyncCall::Pointer &call);
extern void comm_add_close_handler(int fd, PF *, void *);
extern void comm_add_close_handler(int fd, AsyncCall::Pointer &);
extern void comm_remove_close_handler(int fd, PF *, void *);
@@ -133,33 +130,6 @@
inline void commMarkHalfClosed(int fd) { commStartHalfClosedMonitor(fd); }
inline bool commIsHalfClosed(int fd) { return commHasHalfClosedMonitor(fd); }
-/* Not sure where these should live yet */
-
-class Acceptor
-{
-
-public:
- typedef void AcceptorFunction (int, int, ConnectionDetail *, comm_err_t, int, void *);
- AcceptorFunction *theFunction;
- int acceptFD;
- void *theData;
-};
-
-class AcceptLimiter
-{
-
-public:
- static AcceptLimiter &Instance();
- void defer (int, Acceptor::AcceptorFunction *, void *);
- void kick();
-
- bool deferring() const;
-
-private:
- static AcceptLimiter Instance_;
- Vector<Acceptor> deferred;
-};
-
/* A comm engine that calls comm_select */
class CommSelectEngine : public AsyncEngine
=== added file 'src/comm/AcceptLimiter.cc'
--- src/comm/AcceptLimiter.cc 1970-01-01 00:00:00 +0000
+++ src/comm/AcceptLimiter.cc 2009-12-31 02:35:01 +0000
@@ -0,0 +1,32 @@
+#include "config.h"
+#include "comm/AcceptLimiter.h"
+#include "comm/ListenStateData.h"
+#include "fde.h"
+
+Comm::AcceptLimiter Comm::AcceptLimiter::Instance_;
+
+Comm::AcceptLimiter &Comm::AcceptLimiter::Instance()
+{
+ return Instance_;
+}
+
+void
+Comm::AcceptLimiter::defer(Comm::ListenStateData *afd)
+{
+ afd->isLimited++;
+ debugs(5, 5, HERE << "FD " << afd->fd << " x" << afd->isLimited);
+ deferred.push_back(afd);
+}
+
+void
+Comm::AcceptLimiter::kick()
+{
+ debugs(5, 5, HERE << " size=" << deferred.size());
+ if (deferred.size() > 0 && fdNFree() >= RESERVED_FD) {
+ debugs(5, 5, HERE << " doing one.");
+ /* NP: shift() is equivalent to pop_front(). Giving us a FIFO queue. */
+ ListenStateData *temp = deferred.shift();
+ temp->isLimited--;
+ temp->acceptNext();
+ }
+}
=== added file 'src/comm/AcceptLimiter.h'
--- src/comm/AcceptLimiter.h 1970-01-01 00:00:00 +0000
+++ src/comm/AcceptLimiter.h 2009-12-31 02:35:01 +0000
@@ -0,0 +1,41 @@
+#ifndef _SQUID_SRC_COMM_ACCEPT_LIMITER_H
+#define _SQUID_SRC_COMM_ACCEPT_LIMITER_H
+
+#include "Array.h"
+
+namespace Comm {
+
+class ListenStateData;
+
+/**
+ * FIFO Queue holding listener socket handlers which have been activated
+ * ready to dupe their FD and accept() a new client connection.
+ * But when doing so there were not enough FD available to handle the
+ * new connection. These handlers are awaiting some FD to become free.
+ *
+ * defer - used only by Comm layer ListenStateData adding themselves when FD are limited.
+ * kick - used by Comm layer when FD are closed.
+ */
+class AcceptLimiter
+{
+
+public:
+ /** retrieve the global instance of the queue. */
+ static AcceptLimiter &Instance();
+
+ /** delay accepting a new client connection. */
+ void defer(Comm::ListenStateData *afd);
+
+ /** try to accept and begin processing any delayed client connections. */
+ void kick();
+
+private:
+ static AcceptLimiter Instance_;
+
+ /** FIFO queue */
+ Vector<Comm::ListenStateData*> deferred;
+};
+
+}; // namepace Comm
+
+#endif /* _SQUID_SRC_COMM_ACCEPT_LIMITER_H */
=== added file 'src/comm/ListenStateData.cc'
--- src/comm/ListenStateData.cc 1970-01-01 00:00:00 +0000
+++ src/comm/ListenStateData.cc 2010-01-02 10:17:00 +0000
@@ -0,0 +1,285 @@
+/*
+ * DEBUG: section 5 Listener Socket Handler
+ * AUTHOR: Harvest Derived
+ *
+ * SQUID Web Proxy Cache http://www.squid-cache.org/
+ * ----------------------------------------------------------
+ *
+ * Squid is the result of efforts by numerous individuals from
+ * the Internet community; see the CONTRIBUTORS file for full
+ * details. Many organizations have provided support for Squid's
+ * development; see the SPONSORS file for full details. Squid is
+ * Copyrighted (C) 2001 by the Regents of the University of
+ * California; see the COPYRIGHT file for full details. Squid
+ * incorporates software developed and/or copyrighted by other
+ * sources; see the CREDITS file for full details.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111, USA.
+ *
+ *
+ * Copyright (c) 2003, Robert Collins <robertc_at_squid-cache.org>
+ */
+
+#include "squid.h"
+#include "CommCalls.h"
+#include "comm/AcceptLimiter.h"
+#include "comm/comm_internal.h"
+#include "comm/ListenStateData.h"
+#include "ConnectionDetail.h"
+#include "fde.h"
+#include "protos.h"
+#include "SquidTime.h"
+
+/**
+ * New-style listen and accept routines
+ *
+ * Listen simply registers our interest in an FD for listening,
+ * and accept takes a callback to call when an FD has been
+ * accept()ed.
+ */
+void
+Comm::ListenStateData::setListen()
+{
+ int x;
+
+ if ((x = listen(fd, Squid_MaxFD >> 2)) < 0) {
+ debugs(50, 0, HERE << "listen(FD " << fd << ", " << (Squid_MaxFD >> 2) << "): " << xstrerror());
+ errcode = x;
+ return;
+ }
+
+ if (Config.accept_filter && strcmp(Config.accept_filter, "none") != 0) {
+#ifdef SO_ACCEPTFILTER
+ struct accept_filter_arg afa;
+ bzero(&afa, sizeof(afa));
+ debugs(5, DBG_IMPORTANT, "Installing accept filter '" << Config.accept_filter << "' on FD " << fd);
+ xstrncpy(afa.af_name, Config.accept_filter, sizeof(afa.af_name));
+ x = setsockopt(fd, SOL_SOCKET, SO_ACCEPTFILTER, &afa, sizeof(afa));
+ if (x < 0)
+ debugs(5, DBG_CRITICAL, "SO_ACCEPTFILTER '" << Config.accept_filter << "': '" << xstrerror());
+#elif defined(TCP_DEFER_ACCEPT)
+ int seconds = 30;
+ if (strncmp(Config.accept_filter, "data=", 5) == 0)
+ seconds = atoi(Config.accept_filter + 5);
+ x = setsockopt(fd, IPPROTO_TCP, TCP_DEFER_ACCEPT, &seconds, sizeof(seconds));
+ if (x < 0)
+ debugs(5, DBG_CRITICAL, "TCP_DEFER_ACCEPT '" << Config.accept_filter << "': '" << xstrerror());
+#else
+ debugs(5, DBG_CRITICAL, "accept_filter not supported on your OS");
+#endif
+ }
+}
+
+Comm::ListenStateData::ListenStateData(int aFd, AsyncCall::Pointer &call, bool accept_many) :
+ fd(aFd),
+ theCallback(call),
+ mayAcceptMore(accept_many)
+{
+ assert(aFd >= 0);
+ debugs(5, 5, HERE << "FD " << fd << " AsyncCall: " << call);
+ assert(isOpen(aFd));
+ setListen();
+ commSetSelect(fd, COMM_SELECT_READ, doAccept, this, 0);
+}
+
+Comm::ListenStateData::~ListenStateData()
+{
+ comm_close(fd);
+ fd = -1;
+}
+
+/**
+ * This private callback is called whenever a filedescriptor is ready
+ * to dupe itself and fob off an accept()ed connection
+ *
+ * It will either do that accept operation. Or if there are not enough FD
+ * available to do the clone safely will push the listening FD into a list
+ * of deferred operations. The list gets kicked and the dupe/accept() actually
+ * done later when enough sockets become available.
+ */
+void
+Comm::ListenStateData::doAccept(int fd, void *data)
+{
+ debugs(5, 2, HERE << "New connection on FD " << fd);
+
+ assert(isOpen(fd));
+ ListenStateData *afd = static_cast<ListenStateData*>(data);
+
+ if (!okToAccept()) {
+ AcceptLimiter::Instance().defer(afd);
+ }
+ else {
+ afd->acceptNext();
+ }
+ commSetSelect(fd, COMM_SELECT_READ, Comm::ListenStateData::doAccept, afd, 0);
+}
+
+bool
+Comm::ListenStateData::okToAccept()
+{
+ static time_t last_warn = 0;
+
+ if (fdNFree() >= RESERVED_FD)
+ return true;
+
+ if (last_warn + 15 < squid_curtime) {
+ debugs(5, DBG_CRITICAL, "WARNING! Your cache is running out of filedescriptors");
+ last_warn = squid_curtime;
+ }
+
+ return false;
+}
+
+bool
+Comm::ListenStateData::acceptOne()
+{
+ /*
+ * We don't worry about running low on FDs here. Instead,
+ * doAccept() will use AcceptLimiter if we reach the limit
+ * there.
+ */
+
+ /* Accept a new connection */
+ ConnectionDetail connDetails;
+ int newfd = oldAccept(connDetails);
+
+ /* Check for errors */
+ if (newfd < 0) {
+
+ if (newfd == COMM_NOMESSAGE) {
+ /* register interest again */
+ debugs(5, 5, HERE << "try later: FD " << fd << " handler: " << *theCallback);
+ commSetSelect(fd, COMM_SELECT_READ, doAccept, this, 0);
+ return false;
+ }
+
+ // A non-recoverable error; notify the caller */
+ debugs(5, 5, HERE << "non-recoverable error: FD " << fd << " handler: " << *theCallback);
+ notify(-1, COMM_ERROR, errno, connDetails);
+ return false;
+ }
+
+ debugs(5, 5, HERE << "accepted: FD " << fd <<
+ " newfd: " << newfd << " from: " << connDetails.peer <<
+ " handler: " << *theCallback);
+ notify(newfd, COMM_OK, 0, connDetails);
+ return true;
+}
+
+void
+Comm::ListenStateData::acceptNext()
+{
+ assert(isOpen(fd));
+ debugs(5, 2, HERE << "connection on FD " << fd);
+ mayAcceptMore = acceptOne();
+}
+
+void
+Comm::ListenStateData::notify(int newfd, comm_err_t errcode, int xerrno, const ConnectionDetail &connDetails)
+{
+ // listener socket handlers just abandon the port with COMM_ERR_CLOSING
+ // it should only happen when this object is deleted...
+ if (errcode == COMM_ERR_CLOSING) {
+ return;
+ }
+
+ if (theCallback != NULL) {
+ typedef CommAcceptCbParams Params;
+ Params ¶ms = GetCommParams<Params>(theCallback);
+ params.fd = fd;
+ params.nfd = newfd;
+ params.details = connDetails;
+ params.flag = errcode;
+ params.xerrno = xerrno;
+ ScheduleCallHere(theCallback);
+ if (!mayAcceptMore)
+ theCallback = NULL;
+ }
+}
+
+/**
+ * accept() and process
+ * Wait for an incoming connection on FD.
+ */
+int
+Comm::ListenStateData::oldAccept(ConnectionDetail &details)
+{
+ PROF_start(comm_accept);
+ statCounter.syscalls.sock.accepts++;
+ int sock;
+ struct addrinfo *gai = NULL;
+ details.me.InitAddrInfo(gai);
+
+ if ((sock = accept(fd, gai->ai_addr, &gai->ai_addrlen)) < 0) {
+
+ details.me.FreeAddrInfo(gai);
+
+ PROF_stop(comm_accept);
+
+ if (ignoreErrno(errno)) {
+ debugs(50, 5, HERE << "FD " << fd << ": " << xstrerror());
+ return COMM_NOMESSAGE;
+ } else if (ENFILE == errno || EMFILE == errno) {
+ debugs(50, 3, HERE << "FD " << fd << ": " << xstrerror());
+ return COMM_ERROR;
+ } else {
+ debugs(50, 1, HERE << "FD " << fd << ": " << xstrerror());
+ return COMM_ERROR;
+ }
+ }
+
+ details.peer = *gai;
+
+ if ( Config.client_ip_max_connections >= 0) {
+ if (clientdbEstablished(details.peer, 0) > Config.client_ip_max_connections) {
+ debugs(50, DBG_IMPORTANT, "WARNING: " << details.peer << " attempting more than " << Config.client_ip_max_connections << " connections.");
+ details.me.FreeAddrInfo(gai);
+ return COMM_ERROR;
+ }
+ }
+
+ details.me.InitAddrInfo(gai);
+
+ details.me.SetEmpty();
+ getsockname(sock, gai->ai_addr, &gai->ai_addrlen);
+ details.me = *gai;
+
+ commSetCloseOnExec(sock);
+
+ /* fdstat update */
+ fd_open(sock, FD_SOCKET, "HTTP Request");
+
+ fdd_table[sock].close_file = NULL;
+ fdd_table[sock].close_line = 0;
+
+ fde *F = &fd_table[sock];
+ details.peer.NtoA(F->ipaddr,MAX_IPSTRLEN);
+ F->remote_port = details.peer.GetPort();
+ F->local_addr.SetPort(details.me.GetPort());
+#if USE_IPV6
+ F->sock_family = AF_INET;
+#else
+ F->sock_family = details.me.IsIPv4()?AF_INET:AF_INET6;
+#endif
+ details.me.FreeAddrInfo(gai);
+
+ commSetNonBlocking(sock);
+
+ /* IFF the socket is (tproxy) transparent, pass the flag down to allow spoofing */
+ F->flags.transparent = fd_table[fd].flags.transparent;
+
+ PROF_stop(comm_accept);
+ return sock;
+}
=== added file 'src/comm/ListenStateData.h'
--- src/comm/ListenStateData.h 1970-01-01 00:00:00 +0000
+++ src/comm/ListenStateData.h 2009-12-31 04:07:20 +0000
@@ -0,0 +1,54 @@
+#ifndef SQUID_LISTENERSTATEDATA_H
+#define SQUID_LISTENERSTATEDATA_H
+
+#include "config.h"
+#include "base/AsyncCall.h"
+#include "comm.h"
+#if HAVE_MAP
+#include <map>
+#endif
+
+class ConnectionDetail;
+
+namespace Comm {
+
+class ListenStateData
+{
+
+public:
+ ListenStateData(int fd, AsyncCall::Pointer &call, bool accept_many);
+ ListenStateData(const ListenStateData &r); // not implemented.
+ ~ListenStateData();
+
+ void subscribe(AsyncCall::Pointer &call);
+ void acceptNext();
+ void notify(int newfd, comm_err_t, int xerrno, const ConnectionDetail &);
+
+ int fd;
+
+ /// errno code if any happened so far.
+ int errcode;
+
+ /// whether this socket is delayed and on the AcceptLimiter queue.
+ int32_t isLimited;
+
+private:
+ /// Method to test if there are enough file escriptors to open a new client connection
+ /// if not the accept() will be postponed
+ static bool okToAccept();
+
+ /// Method callback for whenever an FD is ready to accept a client connection.
+ static void doAccept(int fd, void *data);
+
+ bool acceptOne();
+ int oldAccept(ConnectionDetail &details);
+
+ AsyncCall::Pointer theCallback;
+ bool mayAcceptMore;
+
+ void setListen();
+};
+
+}; // namespace Comm
+
+#endif /* SQUID_LISTENERSTATEDATA_H */
=== added file 'src/comm/Makefile.am'
--- src/comm/Makefile.am 1970-01-01 00:00:00 +0000
+++ src/comm/Makefile.am 2009-12-31 02:35:01 +0000
@@ -0,0 +1,13 @@
+include $(top_srcdir)/src/Common.am
+include $(top_srcdir)/src/TestHeaders.am
+
+noinst_LTLIBRARIES = libcomm-listener.la
+
+## Library holding listener comm socket handlers
+libcomm_listener_la_SOURCES= \
+ AcceptLimiter.cc \
+ AcceptLimiter.h \
+ ListenStateData.cc \
+ ListenStateData.h \
+ \
+ comm_internal.h
=== added file 'src/comm/comm_internal.h'
--- src/comm/comm_internal.h 1970-01-01 00:00:00 +0000
+++ src/comm/comm_internal.h 2009-12-31 02:35:01 +0000
@@ -0,0 +1,16 @@
+#ifndef SQUID_COMM_COMM_INTERNAL_H
+#define SQUID_COMM_COMM_INTERNAL_H
+
+/* misc collection of bits shared by Comm code, but not needed by the rest of Squid. */
+
+struct _fd_debug_t {
+ char const *close_file;
+ int close_line;
+};
+
+typedef struct _fd_debug_t fd_debug_t;
+extern fd_debug_t *fdd_table;
+
+extern bool isOpen(const int fd);
+
+#endif
=== modified file 'src/fde.h'
--- src/fde.h 2009-12-21 12:05:22 +0000
+++ src/fde.h 2010-01-01 21:26:45 +0000
@@ -126,4 +126,6 @@
};
+SQUIDCEXTERN int fdNFree(void);
+
#endif /* SQUID_FDE_H */
=== modified file 'src/ftp.cc'
--- src/ftp.cc 2009-12-11 14:15:28 +0000
+++ src/ftp.cc 2010-01-01 21:26:45 +0000
@@ -33,28 +33,30 @@
*/
#include "squid.h"
-#include "Store.h"
-#include "HttpRequest.h"
-#include "HttpReply.h"
+#include "comm.h"
+#include "comm/ListenStateData.h"
+#include "ConnectionDetail.h"
#include "errorpage.h"
#include "fde.h"
-#include "comm.h"
+#include "forward.h"
+#include "HttpHdrContRange.h"
#include "HttpHeaderRange.h"
-#include "HttpHdrContRange.h"
#include "HttpHeader.h"
-#if DELAY_POOLS
-#include "DelayPools.h"
-#include "MemObject.h"
-#endif
-#include "ConnectionDetail.h"
-#include "forward.h"
+#include "HttpRequest.h"
+#include "HttpReply.h"
+#include "MemBuf.h"
+#include "rfc1738.h"
#include "Server.h"
-#include "MemBuf.h"
-#include "wordlist.h"
+#include "SquidString.h"
#include "SquidTime.h"
+#include "Store.h"
#include "URLScheme.h"
-#include "SquidString.h"
-#include "rfc1738.h"
+#include "wordlist.h"
+
+#if DELAY_POOLS
+#include "DelayPools.h"
+#include "MemObject.h"
+#endif
/**
\defgroup ServerProtocolFTPInternal Server-Side FTP Internals
@@ -140,11 +142,21 @@
/// called after the socket is opened, sets up close handler
void opened(int aFd, const AsyncCall::Pointer &aCloser);
- void close(); /// clears the close handler and calls comm_close
- void clear(); /// just resets fd and close handler
+ /** Handles all operations needed to properly close the active channel FD.
+ * clearing the close handler, clearing the listen socket properly, and calling comm_close
+ */
+ void close();
+
+ void clear(); /// just resets fd and close handler. does not close active connections.
int fd; /// channel descriptor; \todo: remove because the closer has it
+ /** Current listening socket handler. delete on shutdown or abort.
+ * FTP stores a copy of the FD in the field fd above.
+ * Use close() to properly close the channel.
+ */
+ Comm::ListenStateData *listener;
+
private:
AsyncCall::Pointer closer; /// Comm close handler callback
};
@@ -435,6 +447,11 @@
void
FtpStateData::dataClosed(const CommCloseCbParams &io)
{
+ if (data.listener) {
+ delete data.listener;
+ data.listener = NULL;
+ data.fd = -1;
+ }
data.clear();
failed(ERR_FTP_FAILURE, 0);
/* failed closes ctrl.fd and frees ftpState */
@@ -2697,7 +2714,7 @@
int on = 1;
int x = 0;
- /// Close old data channel, if any. We may open a new one below.
+ /// Close old data channels, if any. We may open a new one below.
ftpState->data.close();
/*
@@ -2741,7 +2758,12 @@
return -1;
}
- if (comm_listen(fd) < 0) {
+ typedef CommCbMemFunT<FtpStateData, CommAcceptCbParams> acceptDialer;
+ AsyncCall::Pointer acceptCall = asyncCall(11, 5, "FtpStateData::ftpAcceptDataConnection",
+ acceptDialer(ftpState, &FtpStateData::ftpAcceptDataConnection));
+ ftpState->data.listener = new Comm::ListenStateData(fd, acceptCall, false);
+
+ if (!ftpState->data.listener || ftpState->data.listener->errcode < 0) {
comm_close(fd);
return -1;
}
@@ -2896,8 +2918,8 @@
char ntoapeer[MAX_IPSTRLEN];
debugs(9, 3, "ftpAcceptDataConnection");
- if (io.flag == COMM_ERR_CLOSING)
- return;
+ // one connection accepted. the handler has stopped listening. drop our local pointer to it.
+ data.listener = NULL;
if (EBIT_TEST(entry->flags, ENTRY_ABORTED)) {
abortTransaction("entry aborted when accepting data conn");
@@ -2918,17 +2940,20 @@
io.details.peer << "), expecting " <<
fd_table[ctrl.fd].ipaddr);
+ /* close the bad soures connection down ASAP. */
comm_close(io.nfd);
+
+ /* we are ony accepting once, so need to re-open the listener socket. */
typedef CommCbMemFunT<FtpStateData, CommAcceptCbParams> acceptDialer;
AsyncCall::Pointer acceptCall = asyncCall(11, 5, "FtpStateData::ftpAcceptDataConnection",
acceptDialer(this, &FtpStateData::ftpAcceptDataConnection));
- comm_accept(data.fd, acceptCall);
+ data.listener = new Comm::ListenStateData(data.fd, acceptCall, false);
return;
}
}
if (io.flag != COMM_OK) {
- debugs(9, DBG_IMPORTANT, "ftpHandleDataAccept: comm_accept(" << io.nfd << "): " << xstrerr(io.xerrno));
+ debugs(9, DBG_IMPORTANT, "ftpHandleDataAccept: FD " << io.nfd << ": " << xstrerr(io.xerrno));
/** \todo XXX Need to set error message */
ftpFail(this);
return;
@@ -3059,7 +3084,7 @@
AsyncCall::Pointer acceptCall = asyncCall(11, 5, "FtpStateData::ftpAcceptDataConnection",
acceptDialer(this, &FtpStateData::ftpAcceptDataConnection));
- comm_accept(data.fd, acceptCall);
+ data.listener = new Comm::ListenStateData(data.fd, acceptCall, false);
} else {
debugs(9, DBG_IMPORTANT, HERE << "Unexpected reply code "<< std::setfill('0') << std::setw(3) << code);
ftpFail(this);
@@ -3195,7 +3220,7 @@
AsyncCall::Pointer acceptCall = asyncCall(11, 5, "FtpStateData::ftpAcceptDataConnection",
acceptDialer(ftpState, &FtpStateData::ftpAcceptDataConnection));
- comm_accept(ftpState->data.fd, acceptCall);
+ ftpState->data.listener = new Comm::ListenStateData(ftpState->data.fd, acceptCall, false);
/*
* Cancel the timeout on the Control socket and establish one
* on the data socket
@@ -3256,7 +3281,7 @@
typedef CommCbMemFunT<FtpStateData, CommAcceptCbParams> acceptDialer;
AsyncCall::Pointer acceptCall = asyncCall(11, 5, "FtpStateData::ftpAcceptDataConnection",
acceptDialer(ftpState, &FtpStateData::ftpAcceptDataConnection));
- comm_accept(ftpState->data.fd, acceptCall);
+ ftpState->data.listener = new Comm::ListenStateData(ftpState->data.fd, acceptCall, false);
/*
* Cancel the timeout on the Control socket and establish one
* on the data socket
@@ -3848,7 +3873,7 @@
/**
* Did we close all FTP server connection(s)?
*
- \retval true Both server control and data channels are closed.
+ \retval true Both server control and data channels are closed. And not waitigng for a new data connection to open.
\retval false Either control channel or data is still active.
*/
bool
@@ -3927,7 +3952,15 @@
void
FtpChannel::close()
{
- if (fd >= 0) {
+ // channels with active listeners will be closed when the listener handler dies.
+ if (listener) {
+ delete listener;
+ listener = NULL;
+ comm_remove_close_handler(fd, closer);
+ closer = NULL;
+ fd = -1;
+ }
+ else if (fd >= 0) {
comm_remove_close_handler(fd, closer);
closer = NULL;
comm_close(fd); // we do not expect to be called back
=== modified file 'src/protos.h'
--- src/protos.h 2010-01-01 21:16:57 +0000
+++ src/protos.h 2010-01-01 21:26:45 +0000
@@ -144,7 +144,6 @@
SQUIDCEXTERN void fd_note(int fd, const char *);
SQUIDCEXTERN void fd_bytes(int fd, int len, unsigned int type);
SQUIDCEXTERN void fdDumpOpen(void);
-SQUIDCEXTERN int fdNFree(void);
SQUIDCEXTERN int fdUsageHigh(void);
SQUIDCEXTERN void fdAdjustReserved(void);
=== modified file 'src/store.cc'
--- src/store.cc 2009-12-03 02:38:11 +0000
+++ src/store.cc 2010-01-01 21:26:45 +0000
@@ -35,6 +35,7 @@
#include "squid.h"
#include "event.h"
+#include "fde.h"
#include "Store.h"
#include "CacheManager.h"
#include "StoreClient.h"
=== modified file 'src/tests/stub_fd.cc'
--- src/tests/stub_fd.cc 2009-01-21 03:47:47 +0000
+++ src/tests/stub_fd.cc 2009-12-31 02:35:01 +0000
@@ -32,7 +32,8 @@
*
*/
-#include "squid.h"
+#include "config.h"
+#include "fde.h"
int
fdNFree(void)
# Begin bundle
IyBCYXphYXIgcmV2aXNpb24gYnVuZGxlIHY0CiMKQlpoOTFBWSZTWS/DOW8AKTlfgHXwf///////
/+6/////YDt+93gPjmA8zmxw9AefA5dZy1RIfbvrd1z72rebq8q+58991L27fdhnvvffPe8Ovm3W
iHV25UOuugdzO7F2DNhlpp197Pe1b755tCdnK5dLPWc3uOude5pdGvb0Na5e3qxcp3tr7j3Dyc27
T613XbPN3Sqa0099j0e+vvY9Tq+7nVap6LbO1NxqdNvvrHni3ex1JKPqzZvZ3wkkCaABNARqegJo
0J6A1DKaTT9KNqeU9R6g09T1DQ9T0glBDQETQIQ0mmhT8h6kjxQDENAyA9QAAADTIEIgJqYnqkya
MgNAAaaNAAAAGhoNAJNKJBGgmhppqemRNTw0mEJ6EyYQGhptRoAAGgIlCCaBNGgmGiPUmUb0mmaS
eRplPInlTwTSHpAADygkRCAICaCYk8jKeUZopmimhtIbU2kPUaMTQAAdHXnBaBFVXP7CH5QW3xTr
WQnjD9eyki/mhMIsiiqupr9q4/Tj+Bzavj8rva9Ma+B1x8nXxZJ3/r3d4riQa9O2Pr9Psk249IFw
0Jvb7Qyion4vBui8iHMQ8Aj9iDuT1/lXpqAWos2ffLPVOMejZ8EQfH5sOSTLFEeXbrIb+BGNh71+
fbAy8eJ84d4mvfSXn/YRB+CiRcDWU2COg2Vm3XqOXnr66HMd15AEP6WG3q7OT9c8T6nT7+ZPSrP9
zuRHEEsi429lC8VS3MiEUkOTqIckF1se7og0EsYdzt+SIHaW8SPd7TxgG07pNxde4XhkNjW/vVic
dVnDJDkiW5pAEi29wkjHspu3as7uPYW8SHLDmoNvDhES4cF3BCVmgbIofrkmSBH5yJtSZL7iAoIg
zDrz76Qb2G3qrxp/uc1N5FwRl5Vv773ptnCuUA2Lm9I1z/TOueGE9E0p5MXI5/eudeqfC6hKmI6u
nuwpYXwZ5OjY2ZphFUw1DHKZwKWzGv9XNxZ93roGH2u/08kk603CxEjwWjGtIMk1wZLhwp4tPNfL
pHZ4oYB4eEHVxDxvhVfXfG+wkdxzS9gg/CT+s9csNIAAzsiQSw3I4JBukquSw8PZmemYyQ5gYt5p
NRxzQyLUDCCNVq4YE4pBnLAanTgLKssFlCjogCSTJAGgTq3aC4U4w4Rik4kLN5hYlXXg7MzDuT4r
YZO6SYgko2QNxAZE7cIJ4GU5eRchcA5U1cyBhcHtSxsYEWDKxVwJ0wyHQMzq6zrKjGKInArCyMZR
pLBkZsrW35ajMUk5MlvMZL6pJm1cvKjs6ZIDeb1FAzwUUCgApo5KAodkACwc8C1UTCLiyhWBt/v4
fKHi8VjCKYaTyATfASZCxGVoIIWwg2yNjQrNs5eTvupsahffJuFxgqef7My30Je1rU09slW2sBv5
oAm0HOkaMELnfmOFb05fHke6tg0teS1bFavaR8eR9xYuLibHLPxJl0V6s4bXrDyAJ2+gU6+eSJIy
UCLFCIgoAsrCqxRQIsFQYjIoKCMIosgpwmzpsqMpOL82J/Ykk9DPIelXe826+kx6ba30Vzz8cqmt
LSAACCxIJ5Nm8zo69qMzCZ2VvB4WrS3SBJpB6mYmVgwcknAe3pxTksLODCscFNrNsNGMCXRKhOgi
ggZlUc6SLYlWLcfjiDGzuKSFF3t3cpxW2X1EhEVFxjY3FiqUQ+LmIYFo71Vld3dbIiNdSl4nXjoG
xFT0PjuPW2tt+Ncipnlh9/bsBEDJJuzs86SdUcUgjkvOsRZF3l4lCJt4fF7MC+pU2H440djZBJPZ
3NK3O63t1rtDLdVjbOfVW02A8wIhuJAzp3AzKi05DiIh5myG3BazbphNigdnh1l3IdLEpqhDItZd
sjSma8OXYhhtquF8HiNPVS4dQ7UqW8KsGAH3J4XhZvK3eHxwuXJ1rVaGVAfjAd6I+Px31vLDZFTC
3Dqi5C1GRe5q8O9llBVwe3wB9+yDLH11B/P+Mg3IW75hhRC9aQjC7+uL/9LlPPndyatdS2cocfGz
UR77+v35x9vAySoXkhrOuXu4RSnAgPmzj2B1+z3dftt9IU1eYuPZzeU3TczUr5WDnhNb/Ci+o3yJ
Jjgxvs1fqBYO9bmvPAHt+EbDTDFoUAOpl+1nrWpM17D2MMXOdY46YFjWVt7quqU40uX4sO/NacMp
pK8ra2vFuDSBmlhZbE0R6T7+KjFu2VtNXO3PjLyX3jjjbXoCdVp8OqvH/vnX51EKj9+CxYhBnszs
WugPToRRjex65vGKCMicP+snKjzT/G7f1s3f2C5Wu+x19txtTMd+nLlZ10Tla/5pk/fe5Yikky5d
udvFDt9VfbS7FtH3HwfR4Tbp3txv84/eUP155OnejE5dvXXPbrrD8lKbSkuqWVubsOvalvTWRtoo
Uii+hwxQ9o4Rdy16XZGfNqWkNKzY1jd97ve37j0ot177NM9MoOfg+dxcV6DXpz7UERwvwL9cHlES
gMtBuWjin0Y8FhSOqp2odlTI1cJyvyutdaZ926ajCh/G64mjSYb5KZK3Hz4T89+Pe8/pj1fCTv2x
7mMYWUuUVb2DbdTO1u3kxDeLtSw967/vfwFfIgbpykc/kU4gyvmUYkJgtZ5kHHKLmNLH9sXubG3j
h6uw+TkN7BLt5ZE4uOhj/LIfO/nk60Yo41b2MptPl0hdg+PQa4dF0EjxCbuqRRdon2pTbha0D2UX
0B0x7JTxNWazXa2W2Pk115uhT+l5aqskccy2dWzF55O9o11ynsnhDXnDSqLRhuphHJZ92XY9R8mJ
nmrKOcTfTMat5umFfm7V3Znq9T5pVybfGZPchUoYEWll8CvPBVsHVkFrG+hWS9uqrE1vWXFOq+1n
4nsw8HHKTopT8y68/v8HI53+xVU+rFNzyl0o8k0npHay762rz8/u1Yz9Gzd6+pERLS3zuw8azKST
NnePbtmbw1PUvl9EsPbHa1S9p0WxVuSAGBRVFFGYZiDyfAxOcibyA7KufTdIKttFIejGM+jr9Vmu
9sshWtvzoZpFMDr3Kq6EuEIuYzGfJpS7LeVuhXK6rTvfv14H2FXw7AAMypdU9teM04kKvT11ebYU
kK9XlnF4/Schbsw3Wr4vY/HfuLfKwa9s76rc9R4W+bqIefIvXeXx0m0NSNcpSlzv5rVSlJeHBU8z
Pvx5rqvvMZfcyHSu4p+jXayrW94evJW3pZarxU5s1ZHL3y5d214ML8oGuuF1M4chShqgsgc+H4dZ
63qBH1e77Pm977KEtd8fqlF32v+bPd4O6MYVSav7banx48ssmD2gx9rHFkCeMLnQz+1mCN6MjWQ2
0WwGUJOJQ5FcOzyOlNuIKo2kaWprOrVYCHK1EmEr08Hwut4n3HdtDrGunpcFaOAhkuBjOHVp455T
s5bN9NI5TFK7n5Na6ml5ZB9+GGMIYlLZImx0+FWg763dXWUHFIQMGGjVq18KeVMVxVIhAkkUjCEk
kiMGQERkEYgqixggoMYCIwIwVCH8feih/fVPSRfkgn6pEObyFBg45cZB+UKOjursmGmhtPnSgghw
6VrNZmvGsA69slLuc82ANNK7G15H6OcoSfdp+6TbWkYfINcueATsMjlr8CEgRsupoDeWCoDTynFM
JcwElBzRB9ZB0YA4IMWGnt806Bq6BrcB72U193z+vkpffj7r+lvm3zxb54Ov/DjbxXuPG7azya2P
zZeyoX/jx40OwbgePy/CJ0VbZzL8i6jabXTHM8lGDNB1zC7JOVhtHy6SpL1o5w8C7GZ8rXk7L5N7
L7NsbIpJF7/CefZsuPBdC+CV04dDmRJ6awMEOhh57qBLdHbchznnBdm8h8x8/hkbRWpQJVRaj9j6
c4zK4GuEUbQtucVcJmyxznFxddCXLARpNpHMhiF2a23Sak7A3+UH1nHxXVCZD4Q+b8Rw8a0r1n6V
Dqhq3vQ9l1SHwcG/q0umKj6ea+CKvF3sbuIM1d6y0zeOMfwMZRr3dBai5mtPat5z3Crndz5Vd7OW
ijEhAmTIj80zuKD5kb/1+Qzexvr/b9oNBRIOuoh8uJu7kfOgHWEA9wawyC6DUeO4KHrfHDawb0bF
AtQuAgVFArFByDfETNBgDgPsEwdDGQwopLcSe7Y3XR8SqOn1+kOKXUFvuIXco8sC0I8xnzyEOsTF
MUqNUHaki8MwNwiL0CNqL8kkKFHBM4PugPCgurO4aTQb8Xz9HQ+qfbilaSlFG8Z9pflsx3h4vO1n
LxPA7T/BJT6b77Y2iJ+MUH7sjui11se9op5P2+HZznazENGm3okfRiabLAs7q8ohU306JiIhuzAk
JAkoKFBsDTfZEsGYAdokkAx53FZB9dNMK4z8uJxh4nyjiRwwxUaw9xspWDpsy7JzF4slg4QDii8v
CFh00Eqpghg/NSIoF6UHOE6sKOnwEJuSfASH0/Efxseg/niZpbBJnSpii0W6py5aJc/jICNi2fN7
uPfVKn029Dsa4R3Ta4gpzWrz5A169KmntX6oc4ySzNh39N0gW2W0CIiIg7Dla7Bz6/l9BB4e9+Pv
iqE/TC0w53x6WXWAK1rSJYsDFlWlRhi0YNFHRoDBitdGLBGQwmjJlk1MJowm8wDRldqSuE2ogWrS
TxBPk90KgFkMKyQ6BNESYEwnOVqQYYIxAmoITQmNVCHAWaQkovDiSSeIz0uVHLC8EbsYoNxdQci6
6xBXBAkSzR6CVCCEU8hgGobQIxKzvOeifwFhBMjAXzChAh6SFPR5MpKaaEAwMOgWg88eW8uobS80
MCi2Juv4ZCvoVSUTmVWCQxyI0PrQEORKQsIil1EJFTfeZgKUnNslVoPFQ2kjBNETKJQJYFKGiURM
JlgmQqTmJbqLLXQYwOYuJdRJqAIjulpNMm7VLF+CxA4gmlBL9C69TRxxY3d9xxKGxQgsMfEmbGTR
uQMDImVDhqiwsIntHGZeOOvLK4iMRLy0xCk6z2/3pIWYuXA02AbobkGGyG6sRdtqmpsJYsEDa52F
mNGkNSBqthhhtSdgYutJUNRgjGEwzSyYBE0glBfFKcpsgiiKIRYed4R9mYi5oOvDPA+Ja0xm8EkL
QdEr4zpAIXT25VAC2orAMbImSo2UKFi1oRWNJSxKPMmVJCN6u5tqSmz7tvrYhbnRSycbyBzRF9DS
5TLoOSMfaol3RBPUes6EvY9o1c6XDXOcBJjkNGQg76ZiSdsyFJAHOdG0pYOqBCzgsw0LtdTx5cd0
6rcdM7zCfqNHnVcTE1i972tF4zu5bcmZbpuCmqQjGKFAtVC0JKqwOpdJIaNdA8JwKmpYsWta2QBN
UXCQNGJWhPztXjSOA700msIWU3mXSRjbCgo1GZ2gRbEigwKPeMgk4FCibHhpTaVLx8AwRiUgolRi
xM7IJKkzAEAwj9hxnE0DuSJE/Bi5lhjNCFDagak3wUsQPUhcNC46kldNwS0MhG4voMjaUIIbEDop
znOGHakAtB5jusKikNhlSEUINhUSHm02lJgZk0lMedCBPOpIOQwpgltYsYLhIgweJ4niDEFj9RBI
8aoSwpK2JIWGqOSxMOBKVWDpunTUukeMW1IxYtg3zydG3zmLuHyQ9lWnSWXVKjjGhYjYb7LCtfpp
hE1Va36ZpKaFagNpFogSW4aBIsKCCSrPM4FQTvJ8CoSRhn0MMxcp3tM30+WQjEtEZtHPIQgW3SqB
HEohYl5qYkBYrGQUNxVG+o4rNpbQIxGjdOiBFbw2VrQnZSl2EJLExJKEvUqKxSkiQD2EmaSFVAhj
caJZ7j6vUzZXuPWYsMhQGu7jMszccck9YigiirRhnJjnI2JGOhdStlRESuDccdc3NDm6pfZblZXy
7Ciq+fEuUM1pCTtOV1WyCEuQUhjIyWygKVUcWBi3XrI7ptERXKAg9wk2Ib3ASY0GoJu1kSxIQwJE
MzAsNZSQUOhsVEmOdyZoccydRjucFyxU0MSGNy0sJEBwWjES01oOIP5svgItQedBmWhaIY6wX24E
t6GYg7Ul1ftDRlEpbsicgij1SxgUJ0o1JYikpu7sr3QwO8GLvQFQEFMSq9kCbBzzoOUgsV6pxmgy
+AvkgZxGQ7y4rIUxaSFDSTVROxnhm6Zc5hnYiFWrKFA9skOegZFHQEGzdvAu3gKUKCjfVU4ORMc6
nQIJgKBxKBYmDXaIgJQmMZBvIRhQjaNzmodDyORPYrihBk2SDENpyrvYonSYhGERBOzyuSEEJTQn
1J8sGRXd8nL2OWORXQuFktzRQyTOdLaRkF1ehMnJFcXsciHKLPCzKoI8w5kxzoTNhyRcs0GBuGJI
6kHOhSpBwRbkkkBUajINRWPIHjIzLdKi036y0GAYyJlh1FHNWFkKbkFOCxEzBsTGDqTHO9kOlo27
U054Q3awXPjGXmuhvkHMSminzKnL5EQcXVTbFLqqlcJR/HCYuTEVIaoBm7PtomM/4g2SiLtLZ0sk
m5gSDaZkks2RZm8neXrTZSby5szVHKD0KlJJDG8RFdyDE0Yq6kIHQh0LzvIaQyU4JI6ywPMvMHur
ESHQV/PysQSOChE85ZnbE3wgJZwepSghI0bDkEiBLKuGcXxExgJSwi82CFtyKG+xY4NkBCW2UuYZ
3tgmKKLsdzek6jM0VoXk5BFiUh9GhEBTqMSKFzZLFAySJEyCg4p1gVjgscyhFjobGgsLCQ2CS9CB
HaS/AOo4HVKBusyqvTy/WNso2ZNrl4xT3lHZSzQzQ5q86h6FS2pvIVl9U4zqbJug8Uneosy9LHmw
lt0woPqnGzfpb5AACYAtQQmgJRVDEFCbXcgSAkQUREEgTBDSM4FPFIxYq9S0vI+Seczi56zmesma
RdNY9j+wLu9EcMSZRiZnmI2gwDqhuBe7xLjabtu3OMHkgQ7UZEhEilgwVJAyIkis3GdJGdkoVFCY
lkSbO0HRrh7SxKNxbBsNuVLclRszN1ORtQ9zMdrci5PSyQoczmczlogWC5uePdKZU5qtegopk3Lh
g3FPgiIJQkI5PCGgc8jcsYJkjgucFtI1BgeUrVJyvM+F5x6/LZziEcQjzhiplA0MJ7x/ow69JZTm
PKPI+duxarEiY386BGUXvCdZjNRS1aDZaq1UZpxWSQxbRm4yIEL7z4g7ogbEUVVKoCGjzO44b+z5
/l5WXXPXPI5GDB0LlWe67oeTG57zgMYjcix4gKU1sbZNXQEV/MyYBRy1M4qX6iZKtdE2LBo5JkiR
RG4J6luTlQttn0EqSPcHlNzmsXXtpDYLE6kNc2Ha611cJkzAoKkkFR8qqBqTkVhQE4GJUVkCk+Gv
k817jMmPDE1FpgUFpiXFZArPILokhbOsLTJBbdw3Try8C/pfy6WaUo6d82TMjFxFSCcSNaegYzdC
JI6KKzSq0opOsWowlEGL1WtAOQp4nQyGQkL2HOw4wqG0CwECP7kxPdYQQyH1jzXaltFdChkkcGxQ
4kauoz183wZg5ou1qCszVJIx6/Xs9qslG3CbWJUpom7nIoEikp8upJjncscXmOuu3mZHRIQ8mDis
qLCREJZmZQPPRUZki8sHpioqNCBWONCQ8uCQUkCk1I60LQXpvEy4VQENlF5i65+HGptpjlgi9ZpN
y1GIa1ZRd0H0KOL1mpwQc0HWcJVo1URqJkFgwXPfO1EE9NEFZImjED1MoXGoVKMFTjQxoKGIMLWn
eRTGLYPLxtjYg2JBgUmulbZdvFpqKVLFL9ECIGJgMdYi+oqam1A+qY4WwqMBxTToPPIqdTPSiw9/
KossFhtk5n074oLMcY8CkuSJbR3yUM32JnckQSmr3XIVahYxWRCqcFHKsaFCtR+1GNLXa8syOJUg
Zawx4azToCu50a5h4Fho6FDsenCG6Qm5Y6CnS4uRTmWOZA4pkuQ1GBeVLjXVsZFQ+Xr6BO+U8qFO
qyo79Ssx29YeQMw/OVI6dggY8aUwO58LfMVUjJ1rOmVKc2LrSHmP6SkGEBCdwZ8YktEYSYKIzIhP
A4jXJBMnes4oqIhRRBEJ+IydSp4D2gZrrl8mIoiVEIO1mwoNpv46kHFW5W3ZlzDx4XDZGJzXUqPY
b+MiKklxbj7Qtq283WdUJFxTE4Fixg6dL7KSXAQc8bj7FRRYm52OCgpClzsUMC8bLpwb4k0ECZUw
k2nEgseYScCgxjJlTJByNDnsqdOjBBzLmxk4PegJuIVNSmQobJ3UHNn892PrGX4n84TPV+/XzSd/
jX4v9XMC2MIwALEMJzpJyAUpig8QWHu7SQmuIyIjFIcydMaJQUI8CiWp2Ch2KJcQFL7b+bQwP1jp
56PNDun8qUxmqPlphVeG4yqmMQ4554rv3c+tJHsYviz9LWP8sQ1hpFWizKbc40o3ey/z2zQzgh7m
pex3RhJl8YCe8wP5oogwH2P1NdG3eSYvcapw8pg7XJydPDTqfXy3i/bwUh88JDrkYKBGAe44bpQT
QkS4QTPho46Z70AuIl8V/NEJQPswfvcQwMxaQweyh9gwMjAnD2O6Sgs3oIVRBRQNxBlYMVRYnDBk
LOYQsESikEEFVEpKURhEQ5ZAYxBAYsRkhRFNhPUHwAJF9mVE5k4koJcmKVArCZkx2By/cVl8INEM
E8A5kggV6edv5z0epFA/7wPH0bvQdSR2r0/pOg/yD8ikP9++VtHoh+q8Zf4KlOppcTgnYh1Jj2+5
baeTmhk1WPZVnX3nwy59jDxB4/b6xNUNrsv5E9hvDeg/YbIepQgsBRIRQGRY/0OcD9n/XswiPSeP
MRssFyFokdxi/FeKV4mMDtE4DRYWVDl5fzqEQI3iHsRKK/N0kSyWkcx+/oJwNHtYTBiB6jIVgon+
pum9je6Gm0QRCe4gkxsp+u6BvuB+u03fvEjmYB1phzq8Cj5gn4lugLihG+OwHrig6jgzWAKl/Qr4
z4/NKvZ3rnsbfoXJg5773b4FKB2+RWjyD87mNYWduJqWwK+SlVv2O7fzePN9mvZvYcehsZMKAY/z
gwVTahhbw0cGWmgPgDBzrmXQHTAJr8k7sv7exd1AfTmaYLwAYZD3geuADceGsra/R8kyI/RNU9nn
49+SN4iLdoQzcwstqXsxbguBn5PIsuxfXkGNYzN6Nhid65mizKDH7hkcmOCyqXODCNOjiMsdusbX
I5jPULZipnTkDoRKRgcHhhSHRR/YlrISH+M327uZngGhcJb/POhQ15qajj2NFCB1qGMBSoMfWv5S
RgG8cKKnjhG+9GCJlxteCExqeNsjTRTQm7wHghHMdbrDS6dNjwm5Q+mSZYd0aSiQXN0kSBJgkefw
xXUirVIVo510KcOAW5HR1NkVEN6RU7S4J80c3dPw6YsB4619K4qM0yt0EOvL9jiEcfd5FLk0l0Lj
gX82PnUJuVDnWYMymgtDAhMJgVhrdl7Wej90Umx3XbtCmrzqa2NKblUqSVo5G6hTjgemJUcWZPC9
2q7A3Y7GbvFJGFNvGkgNOGME3ymAPWSLICyAiTJ154O147xZg4g9E4ZBqWtaFl1hBRW0P2pNipuC
56LKHVBTHabjuGZlXGjITPrXMo1d7JafJcZqRy/YRTBCvK6k6WjPLI5ke7aSS3wqzHE2iecO9mTk
eDJH3aBzDdVqG4PeoOC9CT4BqECkrr2DhuwdasevjmqV2vwCS80k5SsH/DUoKeI3cjQ+DHd+f7om
36liSTBY5z5w1GBtU8lBohEHyIFESjMBjQFM0KNUjc/1MruywYrhYtqxR7iASSCRAhrPanr3P17b
h5JwQ2TtDIcdktEWKkEhJT8LrIGxiEZCKh95xH9TMYH3H6EPl9fFop+B3EQPzm4yCAp1PwIL3kYN
yZAYNsBv/KTLD8jEoMj+UAifmdpUYGBaODgZ56H8hcQlQNrNZcXlZcVGooNyE7rcH2Ln57io3H53
lhgMSNZSxnOwnh0BQbnoIh6oPuSAd6ewpQ949MabY1YoczQ2nWFm1lmeOiW9SYRWVMB2j6FI1nOS
yKW849x55h2DB5RW6cw85hjM0Ni+5MRLajecj71eXfyh7b3y9JbwVEgKVOkHsDvodoRFrzH4n8jw
STE+KCBq3pn29vjJb1niRxOONBgwZajRixxJWpTyXWp0odJnIDMhUqdeSWA1hA82YM8ZchZKq2+B
CSEUJNfsobYl4bkwdu1pB/fe907cHZ/HtkSBwuQSJMLKiefnmRCVGeeFUWzU6Qr5Cdbo5Typc1nT
AQ76QXeDBILBikUiEViMjOuoEIKsgi9bO0kVZfwgyNGe5h7dRP1hvp5h2BuWzLpmYl4PR/DZ/Hw4
tyPecDwHnuPQUkPoN5VUSIEz3ESAOREEspMaR7yYxYyQFRjqOOXP8tDHsMn36KiRaUvOfvwJlBiV
lhcee4zGJkjMmRMxh3qTML1A4zMyg2o72S46JLdF5rcUFpgMVGo0MQnr1mogdiRGLy1KF5aWzCkr
KErxolpvMyksNovQTVJ8ZHv0SUJbRPdYP+5empM0lrRpLcBmhKwqDQuR8uzISEKqIH5RAwDj14en
ghFQA8zluMyZsN5U9MdRWYIFyCBrJHMFJEkXFZU9ox6ixgyQXLCnl7rGiQxkuOGiZgYkDic7BGo9
GoE7NYlY4YrN00NWc44NjfBMI3Mb+QyM5vPQCO08JyB0fbFkSMIxgRgwYsIiwVnCd45y8uMDcUTm
kSTjLGsvKlBSOGJJxxOmmRDeFIgShXtLTIqPIMXlKmzwQvLJ4cyfS6Om/XhoYATEQVIpCsgFYlWx
4A9ScpkhHwH1EDrt1RCXfVSlU8UMkkLUfL1WQkBAVTT2lSFlmxHSImAkGsgkh8FrvSPG/6zx7OFu
tDDMZpKmEy2UHSMKosM8MaJEtBwG2KRgbHcfcp8Q7goIbITD3bYtJ3Mhg+DOzeKY1h1AajZ00pFC
gT58F0mbgPPM2TqumLbc4HGC2YjitwrbjG23LDR0Q8Lo61NiOLDDC3ZjKBMesgKorQIaiQnsoGJ7
DAsZlTyPZ9BBNvpIPeHYUpKo2+c85aRGHldByLTFCQFg8pJES0mMMQLTzjzFJAxx6ZlB6NSQiszN
RWRM9YxEg0RBxNpQ4luHvCIY+ZAI6lBKkaGJosZhrjLUROKPL5Bul1Qux53jw7oeZoBFp83w2m83
m4fr3ETmNaOJzDjIZSOBzJepDLyiQo9rmT2DnGyNTcWLG3nDvHOc8SUYMCcBrcJwENG1b1Y81+xc
rJT/VFQzJW0KmSl+DB6Y9CqGDGLuSVWc5zHOajccwdh1HA2DjupzAqEg3u7Cxz7D8NJx9UFcvTzG
e9OiSLfA3zSPsiUjCy7snWHVC6SrEZ2h9fT61V63eDi8SFHF1Tvi80yjVBVUhsBzmsiSzURY5hO/
0HAusa4ODGnqcbPvSxSbGhlnctN6zKfurmcqOi7NJQAWrBbFP2yhJM1dHNbvZdvXrGputFJLQZt5
TOWJxJjlvZ92ZKGBh9fsj22lpQfKQs4tCabVTep92Pl+Xcv2mQzoyyL3l2aFglczbNJv0KRRK1t0
X7IoZYgUBUxbYMJcManOAYWaLM6EAYkjw4uMTxFeahtJv31+uzn9ZMgSIBzkSktJHYdh6zArIlxY
PNaKh5AYOYg6+H0e/h0Or1N5KX/w4ctfLXdfWwM2pi1u7hw1lBdrKyoiRGeURGGGGK9AgMEjgDG8
cYFRx4+UrOcYe8buTjObwWoW8G6de8EqkRwFConK6TPCkQKSQhBc+9C9sKtc2egaXdVqgGQKOJXE
6jM1mo2huO2+qpt5j3RDYYiHeXUYGheXmBchLjiPKDnInUcSs7C7ItNonnTkmOdQYeyEn4TbzXyP
Y/7/0/DntbJaRH9tuPfS61tq6+tll9a8XnQ2shTSsXCCPefAmTvw4h6jYN0mU0plfyMF3DyG851n
UXMqcQ9BZFA0DwjRZedJ1Gy2gzm4qJIsOv9Ecu/ahUBhnrPtfkGwrOGsopPmxx3pyCz6aO40hSZf
UO2t3Ts1+8KkT4Sb2+5fg8rd7O4iKMZnFpKpEgSTIlAVx1gV+MbKL+LDMigXkAfDJEi6QoI5bOhD
iNZNzQS0kmCj44U3JLJwhC8xY3Y0De8xAlykljWOmrwQN8uMoPNNrAwea8WG2OzGy4uYWTguVdSZ
02xCAd9DCSGUB3MoURViqId+ywSBRQwol1EKVaZEdrmPonmhg85xZbsW5HRdw+09276/XyvIX1Qp
iGr0oXMs3xOXJd74CjeI+TETb1o8M8Wcfp5XQdbVTQyRde3Ce1SYQ031MdCOySJxg0bzDocg8aYj
hsJAMfceONEOqBvKqJIsE1AYZ8h/1IFbYKuTkJnBuYgIMBaH1bLh5iSAQiP5RkQrQKG0UD7U1J1A
cqdTnHpGQnUnkExY0tx28mBQ9lN2oDJMCiTAyNR9B8CBYOHlp4g4idJeew954l58DxJAxcMGrxKD
A5is7gpPiVECwgWGBcVkCYxEHnsLDDmeQLwQYMnu/Iq8Dncnc3EJmTBgkVF57HyoH2imLEjwODoe
A5gpgXBAOSINy5I0e92+EpbK95n4Pe9+51bH39ZqziOJRqjnEohqCBUlSwbQznmV0FufnzYyZofs
kyJon2R0Gc+EVWl8xjM7Z3nISCk4Djac8bRHZ2uCskcTyHUbjWKFTADX0iP7g5bW+0AIFAe1ALVU
CQglQ1COzru79Kh3daup0rTDoYrlFLuXe8cJhWsQIHAY5DclT0n5zyCxwTQwfYhbUDoHgOw7yOav
XAugeKNI1ijVlRZAp5TxAckD4nSF516kYHYPLJsyn1owfASFvqkYVbLU2BeHy3WLHzYN+Jcg5GtM
B7PnO4oU7SsoQlCU7qWEEqlMUO31qjfdFELtbv+yne7xRYrDXIx101iTq6MGiT5GAFaklgwPB+gx
LWZPGhc/s1Eyxu3wMS60jELnGlgm0P20wKGGc4811BOzZ4HqQIVRw3nFyHsD6fcJ57WDr+su8U3n
qUr/SFirGEqHnC314pA0ldOpWECQgIhvQEgm0D6ryaO2KGKeTUJQoTznvpNlsrWIKYPxhk1pLgkB
mOMxshmBMwU6qjpgWHjS6AWLAK7x/GyLVHkL/DAdHKyCQiQi7kHbsJmoG+PgHiBF2jldQJDx1Ted
t89XqNC1xSeU7yykdi0NqLYRM333sVIiuY4Y0hwHl3clBkkiD3SDj2wqYKvSMuuTBKjXV6A0OyMS
CyBEkFCIQBgtxi/TsfOcLfuMHriyRIySQMGmiR5OI4wEIJ0wjFiiwGQEIhAjAHBNWfVKf1LcM4NR
djISXQjIaMycySNUaaVxJMqW0e/OrT0K2zKBrgzvh9Imwtw+nha4fHcGsuPb5A7acrRHnQ8ygcKC
eJB9ppsG0ydLVft5Wl1BaCBo+jtoeEFLOZrQMVR/gcP1Bo0psCh08EyGwzNEHaaaTNRZxsnfSGQ2
XSGYYVITxRAPBBfAQiECFjgQ4kGv4lNg56B8T52boJuCdYHJOTS7lwTo6iHLnhbGoQI533sJQkbe
QxpsmcayEoA3CM1qpNh6rWdZsoqZgcwkzAUiJTPeB3vRTYL3sCopmcwc5um53jTIjn1dWT6DSovT
0PoGImSCdOCs6SHyaObk12B25tiJeM7G3u4DuyxqrIQ6msKhiCeXXv+y9zTeKHMyLII57meXddeb
gkiaK0VkKjZDQe1LI3JmVHmw7G1PFedFNw6aXKd5xpg8h6Th04q4UoyiVZ8p5kFCjVSEEughcwE5
kEx7f3BdcimETjIBiRIRR5KEel6WTqa/z3v0cSlzd/A+ynb4eAOD1MkCCBIJURXjtmfmhbkaouEd
hdzFjEwsku1KqglEGtYX/zS+g9gQom0UPNXEIfr2Zly5NCrqafWYmwraMr3Cg+1hAVKqFJJLqSId
gjATgO4UF2Ckw8yDCdCYZygoGG5EMBN+MKqmATGKCgad/lknHsNAEviHfX+lRFzWQbh8vScgoXw+
c+AfHjB4RyiHpOJUdEZpeELkOeJcCljbtaaxLXJJYksJBpb0+EwiBAPKphXcKxo1UAC1+5qqgXgH
nhCB85lcQRwOFPMUVkUiWLVMx8kZKTMhG3ApH7b4WAlZiAlEmCnL49c53VVH0eCnj0JS/9/9iNh5
teyDSL5iwbGqadkUDJBuJiixgyCILFlajIUiKUQ4Yha1rQtFZWio3eFB0US5W7xOANvNCQDvHN6i
Ch8LPcZeSwFS57HPdcIOnxIoE+fvoU+S+/P7tEIlG3d+WjjJUhrBPxJIASMkErRPecAnN6vSGJt7
wTXoEOgZbgQK4tLn0pqkIpDL5AbdYdPvuuFWyrB5FjUauK/BDgtJsYXMISEBWIpEiEYRVkzjisWR
Ia6UjSOfWUHbAU/CIBlhiMMTipjO5OHRgRWIX3LRQqQIhFzCwC544hqCjig0oWBBjOuZ3+xyKEkQ
kjtCFhaNydErFnw79hlwldslRhIbujsz6tXMU1AS1N6Z7wdGOV5kHC8UoQi4SWHRBIkXZrYQn/GV
wL8IQu54otEur8norMWP7TXKVI7iU0Qzfaa3uKyGJa3Emz/52Q2fOZ3hzJIUDvt9Gt/SQHxSHEDK
o8usVUH8gfAUmCuQ4SRfgoCCejBmy3v7ks2VfCnuTyKJXi2OoUJphsab+R8Q1a+tEQ8N0ykLDzSQ
qBWUQTS+vHbGnni5uCAkmvG63uRzXeogm1NokH1SBCFbk4FTIiQiqGyU1pQOZhYyMDb7d7mJe0ev
WqPUcvtPowPBV9fzI7hLaI3RQvmkhfeQNaWiMD3huQzEkASIZnx9g7GO2iMBiBOelLvXCiI1lTha
iRPAmflzhwlS6YAMMZLLJ4ENYnjT3BIoaJKhEZDV56fl7RKGUAhEu0sSiuRDyXApR0uQPlTFwpfe
1OEhOLViTX59YVTWMowrdRiry0oGCOXK7jhtiNvQemMjnRQPlW2M2ZqHul2WUIhFWREahUh+2YDR
jpl2oj+ge/upQKSNf28yeVJQB8UNASl5GHxN2opIZ8NpvfQJ/kyZ5i6MC7F7u1z3kdCkweVKEYRk
SDiUpbvKU2Yt8CakjDXkWVMBPY0Ux4KVnTnudJ8KI6lGfSwDRWKKaMqixLaCEXNIJSEjVJYnpNuF
vqQawKl8NiOBF3b+Xxv8Ko+y6rudMnb9BDenBPBsgpTnjSqUxbKclpQVBllT+ervKlbh4GYfnxvn
NgbkdwdQszHGGgnEU5cbv1YOdMCSKcGRwrYZx/oPnPzkCVzQqLlkkDOcJ1mMTQQ0tCNVX6vtnAa6
wsFtzMJcaHZrVooIggUn4XIKol24iGU9npesdz16pI/V0kSWsQhjTFNN0qgsujnPrJyqxqERt4TQ
6NfzcBVvn3zuY2LCh4kQrxk9kSoEM5mme9JFAJeXXgBB336lEqHBISJfR/N85REuSByQ1vcl44fr
m19H2w7MlZPdKSDIFCJSMoVClkECglBsqUoJrMGIQwfXyG/ugkEj2uBErwXCYc5c3FGJimbPPTEG
Bf4+aokTTnndLGQChSByYldYKazEhW6dwFxHPOanLz011WELPNu/xdGas7Tf8Ejoyy7YPvjmZHze
6iyTQvh5fefEidpDNLIRQqqCsYeyoDULwM4DYt0Q41jWVnArCbWdbPU5VOU8RRE2JKvpXGud5Tzu
9L1hhpFvVmHowwviKKkksY3tUJQDKDRV3xAaxsWN+mFE00JAUjCshIW1DCMX6I3IuNgjMuSMzSta
BRhLQiFeggCIxoRrVSogZ2jpACoEACzeAjnAi4SDmhIPuEgGAYML9b83fLFMge1mSQVdOJTaY5Yu
U1Z5TATz7jTnQULtkMzRaFAlStKmBdZspa1FslrPBrSi4YW6d91QALkYLYboIokJd/0922W3RF3C
/g3RCY41KxnIrCKIpSSqRlwVduG/oAKvJptsE22JvFj2FkK1ypAX79LmQa4+EM7wF4UzGUaCD3iV
K51bl6R1BGDAPpCgYe/Koe8zep38Ex7USAaRv2wbBpFkux1UtdDKnORwqQDiFQSJddVRKpXjpZJ2
B94wDXmSyh7eQM+SZs9aVT6AMd/X5Og9So/A92hB9w9PZA6Xgffqu4AkDxqjONhH+VDsQ/yj+Bo5
A4ksUQfYhmaiAR5oDjpbhQi5yDlhpPyrpaSmPz9uZ0sRjlUh0jS6GCYmA1PMNMUD8BPDphldB5z7
E+R4bpBt3ALnmijnzL1oWo8nk9FqF/FAj+AiaSF0WBs8yITGaqIH3z1uRyQ+xJNbqHQVr21+IWqY
XgINEHgaB5CzDzBsPgXh1fRH6uGw0ANjY3n3o25tv57cuJDZpS2kPS2biDtmBrQ4wCojwJu8zltX
hHXqQJ2tI7GDFBO8Nnx3B4BTeG47yxbukJLelb4fL3EV5gW/cdwh50EQMZIBEELQRkanidPn2eLD
SQ9XD21B0mgvcVrMRF0+F0QhkQR4kmibOQ5dSrrOeAtFlSxhaLKHIshNzoEDagR5OsDUi0VSKkZA
4NZ1JKIG3pHetYmhxBW0rlQvxoUtZrQkByyA2jlXsWaSFTnn52SE2WVHgDn+BpA6Uc4aldo9qEgw
U/Uh55PC5gKDKDEsPox/8XckU4UJAvwzlvA=
Received on Sat Jan 02 2010 - 11:57:32 MST
This archive was generated by hypermail 2.2.0 : Sat Jan 02 2010 - 12:00:04 MST