Re: squid rewrite

Andres Kroonmaa wrote:
>
>
> As your thoughts are quite similar to those I have thought of for
> quite some time, I would like you to take a look at these ideas and
> comment aggressively. As it appeares to be too big to include with a
> mail, here is a URL: http://cache.online.ee/cache/Devel/design.txt

Ok - from what I can tell there is one basic question:

Can squid get better?
I don't know... I think that it can, but I can't really be sure, except
for the basic problems like integrating ftp gets and support for ftp puts.

There are some problems with the scalability of squid though... these
include:

1) you are limited by filehandles
2) you are limited by having one process that does everything, so you
        can't do SMP nicely

These are listed (now that I look) in your document....
I can't see any other problems with squid, (though you say that there is
a problem with the expiration policy...) It basically does a little bit of
work at the beginning of each request and then just reads data from
disk or a socket and pumps it out another socket, only being limited by
the tcp connection's speed and how long the OS takes to do that.

Note that if you run multiple processes you effectively remove the
"max-filehandle" barrier... most OS's can change the "overall system limit"
of open files fairly easily, but can't change the "files per process" limit.
Doing this normally involves forking processes though, which is bad.

What I am basically proposing:

Create a process that does the listen on the incoming sockets. It also
functions as a database manager, in that it knows all of the objects
in the cache and where they are stored. This database will include
locking, so that as an object is being downloaded it isn't expired.

This database is stored as (?a seperate file?) a memory-mapped (man mmap)
file with the synchronous flags on so that one update immediately
becomes visible to all process that have the same memory-mapped
file. Note that these process can't change the file (they open
it with "PROT_READ" - only the "central-object-manager" has access
to "PROT_WRITE")

When something connects to the main process it hands the new filedescriptor
to a seperate (pre-forked) process over a unix-domain socket. The parent
then closes the fd, and creates an internal "job-number" for that child process.
The job number is so that the manager knows how loaded the squid is
and can decide as to if it should start sending requests to the second
squid-baby.
This child process then gets the "GET _url_" request. This process
then checks the object-map to see if the object is in the cache.

If an object is not in the cache it sends a request for a "storage space"
which will (at least now) be a position in the "cache_dir" directory.

This baby process essentially behaves exactly like the current (novm) squid,
being a "non-forking" process in a big select loop. The parent process
handles all the "cache-management" for it though.

If the baby-process gets too loaded (eg nears the fd limit) it means that
the parent process can then fork (possibly pre-fork?) a second
process that it can then pass all requests to. If there are multiple
CPUs in the machine it can send every 2nd request to one process, then
the other... thus we eliminate both problems at once.

The only way I can figure a way around both these problems is to
use shared memory (or some form of ICP - shared memory will be the most
efficient, I think, since it doesn't involve system calls and waiting
on unix-domain sockets etc)

If we want to get really fancy we could mmap the entire structure
that the data is stored in.... letting the OS handle all of the disk-io
without the lag caused by the filesystem.

Problems with this include: There is a maximum size of your allocated
memory (unsigned int on linux, at least). This means that you can have a
2.1gig cache max. I think that you can get away with multiple processes
mapping multiple segments of this size (not sure at all here) at once,
and then passing requests between the cache-processes. This is messy.
Creating an squid-FS like this means that you win on the number of
filehandles used (big time - you don't even use a fd when doing mmaps,
well you do for a second or two, but that's it). It means, though,
that you have the above problem and you would have to basically
"re-invent" the filesystem... coz you will have to be able to handle
lots of different sized objects without fragmentation etc)

        Oskar
Received on Tue Jul 29 2003 - 13:15:41 MDT