Re: thoughts on memory usage...

--MimeMultipartBoundary
Content-Type: TEXT/PLAIN; charset=US-ASCII

On Wed, 20 Aug 1997, Stewart Forster wrote:
> Whaddya think. If I haven't missed the point I'll write it up (once
>I complete the direct "squid as a caching named" mods I'm in the middle of).

We still haven't seen a response from Duane on what effects he thinks URL
compression would have (and if the URL and key differ, should the key be
compressed, and so on). Since he knows the internals of squid best I was
hoping for a response about this. I think that something which will gain
~40% memory savings on URL strings is worth doing (especially in the NOVM
case with large caches where people are seeing 60-100M of URL strings in
RAM). As disks seem to become cheaper a lot quicker than RAM, and
hardware tends to be more limited in RAM expandability than disk, people
are often constrained even in the NOVM version on how much disk they can
use due to RAM requirements. The simple (and I believe fast enough)
scheme I've done in a few spare hours of programming saves ~33% (a fairly
good set of prefixes/suffixes deduced from my 66M of URLs didn't give me
much more savings); a bit-shuffling technique on top of that should get
this better again. This has the advantage that it can still be
strcmp()'d.

If the StoreEntry structure contains a reference to a compressed URL, or a
compressed URL, each interaction with this structure may or may not need
to uncompress the URL (ie, URLs can be compared in a compressed state but
can only be logged, recieved, transmitted, etc in an uncompressed state).
This is where I would see the large programming effort, implementing this
with a simple compression technique to see if it is feasible was what I
intended to do first (spending at most say 6-12hrs coding the compression
technique in case modifying squid to have hooks for using a compressed URL
in the StoreEntry was too much).

Maybe the StoreEntry should be tagged as having a compressed or
uncompressed URL and only compressed when the item is no longer "active"
(in memory/with pending connection).

Duane? How much work is is going to take me (or someone else) to add
hooks to allow for a compressed URL in the StoreEntry, or a "URL tag" in
the store entry referring to an index into a URL tree?

David.

PS: I had a lot of failures due to the tcp_up going down on a single
connection failure for 80 seconds and inside_firewall being set, and put a
more resiliant (must fail 10 times to be considered dead, if it fails once
start the testing-connections at 30 seconds for the first, 60 seconds for
subsequent, once one succeeds put the tcp_up counter back to 10) check for
tcp connectivity in. This level of persistancy is probably only
applicable to something like the inside_firewall single-parent-per-url
case, but something less fragile than the current situation would be
better IMHO. OTOH, multiple HTTP/1.1 pipelines between caches will make
the issue irrelevant.

--MimeMultipartBoundary--
Received on Tue Jul 29 2003 - 13:15:42 MDT