Re: async disk io

On Fri, Apr 07, 2000, Henrik Nordstrom wrote:
> Adrian Chadd wrote:
>
> > Well, I'm actually at a problem point :) See, I'm looking at the store
> > code in an attempt to reduce the number of list walks (and yes, there
> > are a *lot*) and memcpy()s in the storage manager. I'd also like to not
> > have another memcpy when putting the object on disk. Filesystems
> > like UFS are fine, but implementing write clustering with the
> > current way the code exists involves a memcpy in SD->obj.write to
> > cluster the writes together (look at COSS).
>
> My view is that data once read (from net or disk, no difference) should
> be put in a memory buffer block, and this buffer is then passed along
> (and locked/unlocked) by the parts needing that data. There shouldn't be
> any memcpy() at all, except possibly in header parsing and write
> colescaling in preparation to write to disk (remember disk/fs block
> boundaries, or you might be hit by a pagein to complete the write)

Well, the current idea is this - you can assume that for now, since this is
squid 2 and not the magical rewrite I'm sure Duane is busy drafting :),
you will only be appending to the mem_node chain, right? So you have
a list of clients, you track the smallest offset off all clients, and
you consider that block of memory non-wired (ie clean pages that are
able to be reclaimed.)

This means deciding which memory pages in the mem_node list per object
that are dirty (needing swapout), clean (having valid info on disk) and
wired (are in use by an open client) is easy. Which means you don't need
a refcount on eah page.

Now, this doesn't work for non-linear things like http range requests
its a close enough estimation for now. Later on for streaming objects
it will not be a valid assumption but that is later. :)

So, please, feedback. Ideas, etc. I'm going to work on improving the
storage manager swapin/outmemory. Some more async io ideas would be
neat.

Adrian
Received on Fri Apr 07 2000 - 16:00:16 MDT