Re: memory mapped store entries

> How would it do this?

Lock the page with a semaphore.

> The kernel doesn't really have a way of arresting
> a process if it writes to a mmapped page it currently writes back to disk.
> Also, why would the kernel want to do this?

Perhaps to ensure consistency. Of course there is no guarantee anyway, so
maybe you're right. Maybe it does do just as you say below.

> The kernel simply sees a dirty page, schedules a write, and is done with
> it. The write will take effect in the background, grabbing whatever
> data is in there at that moment. If the userprogram scribbled into
> the page in the meantime, then so be it, the new scribbles jump over
> to the disk.
>

<snip>

> >So you're saying that you want to mlock() all these pages into RAM so they
> >will never get paged out and cause delays?
>
> It would be an option, but I don't think it's necessary.

What about when paging occurs due to memory shortage. Sure paging is bad
in any event. I was thinking mlock() would ensure the mmap()ed pages would
remain resident so they didn't require any page-faults to bring them in
again after being paged out.

> Correct me if I'm wrong, but didn't I just demonstrate that the page-in/out
> behaviour of the mmapped case is going to be better (marginally) than
> the behaviour without mmap?

Maybe. You have assumed ample memory to prevent buffer cache thrash for
your mmap()ed pages.

> > What about consistency in
> >the event of a crash?
>
> Works like a charm. I can "kill -9" my squid any time I want, it
> will come back smooth and fast with *no* false URLs and no race conditions.
> As to kernel-crash consistency, I have to admit that I don't have
> much experience with that due to the simple fact that my proxy server's
> kernel (Linux) has not crashed since more than a year.
> If you're concerned, you could build in checkpoints, where squid tells
> the kernel to sync mmap and disk content *now* (this will happen
> asynchronously again).

I'd do that with a threaded fsync() happening every N secs.

> > Are we
> >happy to mlock() pages into RAM to ensure mmap()'s performance?
>
> Like I said, I don't think this will make much of a difference.

You'd have to do comparisons first to say that.
 
> >I'm happy to admit that mmap() will provide some benefits under lowish loads.
> >My concern is always at the bleeding edge of performance and I'm ever happy
> >to sacrifice some low-end speed by 5-10% if it means high end speed is 10%
> >faster.
>
> I'm servicing 400 requests per minute average on peak time, using a
> Pentium 133 and 192MB of RAM
> Available buffercache: 118MB
> Resident size of squid: 29MB+46MB
> Mmapped storeSwapLogData + shared libs: 29MB
>
> Memory in use according to malloc: 39MB
> Allocated memory according to malloc: 46MB
> Total accounted for by squid: 35MB
>
> Storage swap size: 6.9GB
> Storage Mem size: 14MB
> StoreEntries in use: 568552 15MB (28 bytes each)
> StoreEntries allocated: 573117 15MB
> storeSwapLogData (mmapped): 568552 23MB (44 bytes each)
>
> I'm not sure if this qualifies as low-load or high-load. I can tell you
> that it works, and I can also tell you that also works with 96MB of RAM
> probably, but not without the mmap patch.

That's pretty low load. I'm talking about 5000 HTTP requests per minute at
peak loads across 80Gb cache swaps. The mmap() system you propose will
save a fair chunk of memory on that. I'd still want to run it up under
our production loads to see if it holds up before I trusted it. You can
see why I get a little paranoid about "performance" changes that rely on
non-consistent kernel extensions that have been known to cause problems
in the past. (Says me who introduced asyncio on non-consistent pthread
implementations. :-/ )

What I'm REALLY ranting about is that before mmap()ed indexes are committed
to squid 1.2 I want to be REALLY certain there isn't some major page-faulting
about to happen as opposed to the solution that's currently in squid under
high loads.

I propose that a side-patch for your stuff is generated and tested against
some real killer caches before committing it. If it's just a #define at
compile time, then I'm happy with that too.

        Stew.

-- 
Stewart Forster (Snr. Development Engineer)
connect.com.au pty ltd, Level 9, 114 Albert Rd, Sth Melbourne, VIC 3205, Aust.
Email: slf@connect.com.au   Phone: +61 3 9251-3684   Fax: +61 3 9251-3666