Re: async disk io

On Sun, Apr 09, 2000, Henrik Nordstrom wrote:
> Adrian Chadd wrote:
>
> > you will only be appending to the mem_node chain, right?
>
> right.
>
> > So you have
> > a list of clients, you track the smallest offset off all clients, and
> > you consider that block of memory non-wired (ie clean pages that are
> > able to be reclaimed.)
>
> Yes. This is how it is done today.
>
> > This means deciding which memory pages in the mem_node list per object
> > that are dirty (needing swapout), clean (having valid info on disk) and
> > wired (are in use by an open client) is easy. Which means you don't need
> > a refcount on eah page.
>
> True, but in case of for example COSS, you might want to keep the data
> in memory some time after the last client has finished, and given the
> StoreIO isolation layer I don't think keeping the object locked until
> the filesystem layer has actually finished storing the data is a good
> approach. There is also the issue with requests being aborted while data
> are being sent/stored and such, and how we handle cache hits. For some
> (future) large objects it might also be required to allow holes in
> memory where data is only kept on disk until the slower clients catches
> up (or aborts, which is usually the case there).

This can be handled. I was looking at making the current storage manager
handle objects with 'holes' and deal with them somehow. More thought into
this over the upcoming week.

> It has been some time since I looked at how the Store I/O layer works,
> so maybe this isn't really a issue. I have some vauge memory of counters
> being updated when data was stored, and in that case the above shouldn't
> be any problem.

I thought about it over the weekend, and realised that yes, a refcount
per memnode would be nice. That way, we can implement write clustering
for things like COSS by doing something like a writev() of all the
(unordered) memory blocks. I was trying to think of a way around
a refcount, but I ouldn't think of anything.

> > Now, this doesn't work for non-linear things like http range requests
> > its a close enough estimation for now. Later on for streaming objects
> > it will not be a valid assumption but that is later. :)
>
> We really should not put in more limitations on how range requests can
> be handled. Range is becoming a quite commonly used standard feature of
> HTTP, and I expect there will be more plugins than Acrobat Reader which
> makes out-of-order range requests.

Very good point.

adrian
Received on Sun Apr 09 2000 - 23:56:44 MDT