Re: squid store io modifications

On Wed, 12 Jan 2000, Adrian Chadd wrote:

>
> Henrik, Arjan de Vet and myself have been working on some changes to
> squid's store IO interface to support new file system designs.
>
> Currently the code in squid is still too UFS-style centric, so writing
> alternate filesystem models isn't too easy.

Funny this should come up, I've already done all this myself while I was
implementing COSS in squid2.3. I thought you might be interested in how
what I did compares to what you did.

> * creation of storeCreate() to create an object
I called it storeAllocate(), but its the same idea. There is a companion
routine called storeFree().

> * removal of object number allocation from the store routines
Did this.

> * moving storeOpen() to opening an existing object, not creating an object
Yup.

> * Allowing the swap file / dir allocation to occur after storeCreate()
> has returned (so you can assign filesystem location information after
> you have some more information about the object..)
This sounds good. I did it in a slightly different, COSS-specific way.
COSS really needs to know the complete size of the object to be swapped
out during its allocation, so storeAllocate() is delayed until the size
is known.

> * Replacing swap_file_number with swap_filen and swap_dirn variables
I renamed swap_file_number to swap_file_info and it is simply 32 bits
of non-fs specific data (but is persistant). For UFS its just the fileno,
but for COSS its the disk offset of the object.

> * Reorganisation of the fs code into src/fs/$DIRTYPE
Didn't do this, but its a fine idea.

> The configure script has been rewritten to take a list of store modules
> to add. It also takes the old --enable-async-io option. For a complete
> list of what has happened to the configure script (thanks Henrik :),
> please look at the patch.
>
> A few caveats:
>
> * A lot of code has been duplicated between the ufs and aufs code now, so
> it can be totally seperated, mostly being the swaplog file routines.
> Perhaps these should go in a single file which both reference, but this
> is up for discussion.
>
> * The interface is surely going to change, fast and soon, when Henrik/Arjan
> and myself get more time to talk about the store io code, and implement
> new filesystems. Only then will we truely know what we actually need. :-)
>
> One thing that has been discussed is the possibility of moving the store
> heap code out of the store routines and making it per-fs . That way,
> each fs has its own object replacement algorithms, which squid can 'alter'
> (though storeExpire() calls? Its been a week or so since I've looked at this
> particular bit of code, my hands are dirty elsewhere right now). You might
> be asking "Why?", but consider how you'd integrate a rotational filesystem
> in the current framework ..

In fact, this is exactly what I did. COSS (btw, COSS = Cyclical Object
Storage System) is indeed a rotational filesystem. As such, there is no
need to expire objects from it at all.. Objects are stored in LRU order on
disk, as new space is allocated old objects are simply overwritten.

/-----------------------------------------------------------------------/
/ Eric Stern - Industrial Code & Logic Inc. - (519) 249-0508 /
/ http://www.indcl.com /
/ WebSpeedWare - web caching doesn't have to be expensive! /
/-----------------------------------------------------------------------/
Received on Wed Jan 12 2000 - 09:16:24 MST