Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

While trying to set up an advanced Squid-3.2.3 with 4 workers, a number
of rock stores for different objects and per-worker aufs caches for
larger files I ran into multiple issues.

- Quite frequent NULL pointer assertions failures in peer selection if
the client aborts before peer selection have completed. Guess this is
seen whenever there is peering enabled and sufficiently impatient
clients around.

- each worker restarting at startup due to rock store DB open failures
(timeout). This seems to settle once the rock store have completed
rebuilding, but not 100% sure that's the cause as logs are a bit
inclusive. Might also be related to the delay due to the kid rebuilding
it's own aufs cache_dir.

- kid registration failures at startup (timeout). This seem to be due to
the local cache rebuild in each kid taking more than 6 seconds to
complete, confusing the registration timeout management.

- cache.log completely flooded with memset() messages at startup with
full debug enabled.

The attached patch tries to address or work around the above issues. But
have not fully understood the timeouts. Suspect the timeours are false
and caused by each kid doing a store rebuild at startup which takes a
bit of time so those changes are likely only a workaround and not
actually addressing the root problem.

Will also set up bug reports for these, but sending here first so it
doesn't get lost by mistake.

Regards
Henrik

--- squid-3.2.3/src/DiskIO/IpcIo/IpcIoFile.cc 2012-10-20 15:39:49.000000000 +0300
+++ squid-3.2.3/src/DiskIO/IpcIo/IpcIoFile.cc 2012-11-29 00:47:11.451848433 +0200
@@ -30,7 +30,7 @@
 // TODO: make configurable or compute from squid.conf settings if possible
 static const int QueueCapacity = 1024;
 
-const double IpcIoFile::Timeout = 7; // seconds; XXX: ALL,9 may require more
+const double IpcIoFile::Timeout = 300; // seconds; XXX: ALL,9 may require more
 IpcIoFile::IpcIoFileList IpcIoFile::WaitingForOpen;
 IpcIoFile::IpcIoFilesMap IpcIoFile::IpcIoFiles;
 std::auto_ptr<IpcIoFile::Queue> IpcIoFile::queue;
@@ -218,7 +218,7 @@
 {
     bool ioError = false;
     if (!response) {
- debugs(79, 3, HERE << "error: timeout");
+ debugs(79, 1, HERE << "error: timeout");
         ioError = true; // I/O timeout does not warrant setting error_?
     } else {
         if (response->xerrno) {
--- squid-3.2.3/src/HttpHeader.cc 2012-10-20 15:39:49.000000000 +0300
+++ squid-3.2.3/src/HttpHeader.cc 2012-11-21 19:51:23.826807754 +0200
@@ -548,7 +548,7 @@
 
     char *nulpos;
     if ((nulpos = (char*)memchr(header_start, '\0', header_end - header_start))) {
- debugs(55, 1, "WARNING: HTTP header contains NULL characters {" <<
+ debugs(55, 2, "WARNING: HTTP header contains NULL characters {" <<
                getStringPrefix(header_start, nulpos) << "}\nNULL\n{" << getStringPrefix(nulpos+1, header_end));
         goto reset;
     }
--- squid-3.2.3/src/ipc/StoreMap.cc 2012-10-20 15:39:49.000000000 +0300
+++ squid-3.2.3/src/ipc/StoreMap.cc 2012-11-20 20:55:41.089500435 +0200
@@ -272,8 +272,8 @@
 
 Ipc::StoreMapSlot::StoreMapSlot(): state(Empty)
 {
- xmemset(&key, 0, sizeof(key));
- xmemset(&basics, 0, sizeof(basics));
+ memset(&key, 0, sizeof(key));
+ memset(&basics, 0, sizeof(basics));
 }
 
 void
--- squid-3.2.3/src/ipc/Strand.cc 2012-10-20 15:39:49.000000000 +0300
+++ squid-3.2.3/src/ipc/Strand.cc 2012-11-29 00:12:12.385439783 +0200
@@ -53,7 +53,7 @@
     TypedMsgHdr message;
     ann.pack(message);
     SendMessage(coordinatorAddr, message);
- setTimeout(6, "Ipc::Strand::timeoutHandler"); // TODO: make 6 configurable?
+ setTimeout(600, "Ipc::Strand::timeoutHandler"); // TODO: make 6 configurable?
 }
 
 void Ipc::Strand::receive(const TypedMsgHdr &message)
--- squid-3.2.3/src/peer_select.cc 2012-10-20 15:39:49.000000000 +0300
+++ squid-3.2.3/src/peer_select.cc 2012-11-21 12:01:06.030883260 +0200
@@ -241,7 +241,7 @@
                                (req->flags.intercepted || req->flags.spoof_client_ip);
     const bool useOriginalDst = Config.onoff.client_dst_passthru || !req->flags.hostVerified;
     const bool choseDirect = fs && fs->code == HIER_DIRECT;
- if (isIntercepted && useOriginalDst && choseDirect) {
+ if (isIntercepted && useOriginalDst && choseDirect && req->clientConnectionManager.valid()) {
         // construct a "result" adding the ORIGINAL_DST to the set instead of DIRECT
         Comm::ConnectionPointer p = new Comm::Connection();
         p->remote = req->clientConnectionManager->clientConnection->local;