Re: aio completion notification?

On ons, 2008-06-11 at 16:31 +0800, Adrian Chadd wrote:
> So perusing the aio code in preparation to disassocate it from the Squid
> fs code, I found that there's a pipe being used as a notification/wakeup
> mechanism.

Yes..

> Basically, thread completion writes a ! into the pipe; this wakes up the
> main squid loop and calls a PF callback.

Yes..

> All the callback does is read into a temp buffer, and re-schedules for
> read IO.

Yes..

The point of this wakeup is to break the select/poll/epoll/kqueue sleep.

> We then call aioCheckCallbacks() per storedir each time through the
> comm loop. This can be hundreds of times a second; add 10 + store_dir's
> to the mix and this ends up being thousands of times a second at minimum.

Yes.

It could obviously be optimized to only call this callback if there was
a wakeup event on the pipe (or whatever the store_dir is using). It's
done this way because the pipe was added after the storeDirCallback()
API framework was in place.

diskd should also have a similar event descriptor, not sure it has
today.

Proposal: Drop storeDirCallback() (and it's corresponding
SwapDir->callback), and let each store_dir (or driver if not having per
cache_dir queues) register it's own wakeup event to process the
callbacks.

> http://www.creative.net.au/diffs/20080711-cacheboy-aufs-callback.diff
>
> It should apply against 2.6, 2.7 and 2.HEAD as well as Cacheboy-HEAD.
> aioCheckCallbacks() is now being called much, much less frequently than
> it was before; more in line with how many operation requests there are.

If you clean it up to get rid of storeDirCallback() entirely then it's a
+1 from me.

The next step is to clean up aiops to have a set of worker threads per
store instead of the global pool it has today. Would open up for proper
load scheduling and also make it possibe to add fault management if a
disk fails..

Regards
Henrik