Linux filesystem speed comparison

We are running some large proxies in our Melbourne POP, and we graph the CPU
counters available in the 2.6 linux kernel to give us an idea of what the
CPU is doing. We noticed that the CPU was spending large amounts of time
(around 60%) in an I/O wait state, which is when the CPU is idle, but there
are pending disk i/o opeartions.

Some other recent tests have shown that on linux the aufs disk type gives us
the best performance, but I wanted to see if I could reduce the amount of
I/O wait time on the proxy servers by changing the filesystem.

In Perth we have 4 identical proxies (P3-500, 512Mb RAM, 3x9Gb cache disks,
linux 2.6.10 kernel, squid s2_5-epoll tree), which we were running with the
ext3 filesystem. I reformatted 3 of them with reiserfs, xfs and jfs to see
what difference each of these filesystems would have on the I/O wait. The
mount options for each are as follows:

/dev/sdb1 on /var/spool/squid/disk1 type reiserfs (rw,noatime,notail)
/dev/sdb1 on /var/spool/squid/disk1 type xfs (rw,noatime)
/dev/sdb1 on /var/spool/squid/disk1 type ext3 (rw,noatime,data=writeback)
/dev/sdb1 on /var/spool/squid/disk1 type jfs (rw,noatime)

Below is a single set of results from the daily averages of the graphs we
have. I have taken 10 samples of 5 mijnute averages over the past week, and
they come up with similar figures (the 5 minute samples are pasted at the
end of this e-mail):

Filesystem User Sys IO Req/sec U/R S/R I/R
Reiser 7.6 8.4 14.1 28 0.27 0.17 0.50
Xfs 8.4 5.3 4.4 27.3 0.31 0.19 0.16
Ext3 7.6 4.4 10.4 28.2 0.27 0.16 0.15
Jfs 7.3 4.1 15.8 26.6 0.27 0.15 0.59

The numbers are as follows:
User - %CPU user
Sys - %CPU system
IO - %CPU IO wait
Req/sec - Requests/sec for squid
U/R - User/(Req/sec)
S/R - Sys/(Req/sec)
I/R - IO /(Req/sec)

The interesting thing is that this test shows that in a 2.6.10 kernel, XFS
is the clear winner for I/O wait, followed by ext3 writeback. I was not
surprised to see reiser come off worse than ext3, as I have previously tried
to use reiser on our proxies (on a 2.2 kernel), and noticed that initially
the proxy was a lot quicker, but as the disk filled up, the cache
performance dropped.

I thought I'd post this to squid-dev for comments first, as I have read
other posts that say that squid+reiser is the recommended combination, and
was wondering if there are other tests that I should perform.

Steven

The

                User Sys IO Req/sec U/R S/R I/R
5/4 9:43am
reiser 11.5 7.3 12.2 54 0.21 0.14 0.23

xfs 12.3 7.4 4.5 49.4 0.25 0.15 0.09

ext3 12.2 8.8 10 48 0.25 0.18 0.21

jfs 9 5.3 10.2 40.9 0.22 0.13 0.25

5/4 8:23pm

reiser 11.8 8 13 46.1 0.26 0.17 0.28

xfs 12.9 8 6.2 56.5 0.23 0.14 0.11

ext3 13.4 8.8 14.3 59.7 0.22 0.15 0.24

jfs 12.4 7.8 12.8 56.2 0.22 0.14 0.23

6/4 7:23am

reiser 4.3 2.2 5.1 21.2 0.20 0.10 0.24

xfs 5.2 2.6 1.2 24.3 0.21 0.11 0.05

ext3 4.1 2.3 5.1 13.9 0.29 0.17 0.37

jfs 5.9 2.7 4.7 17 0.35 0.16 0.28

6/4 10:47am

reiser 10.9 7.6 12.3 48.1 0.23 0.16 0.26

xfs 11.5 7.5 5.6 50.7 0.23 0.15 0.11

ext3 11.1 7.4 14.1 51 0.22 0.15 0.28

jfs 10.2 6.2 13.2 42.1 0.24 0.15 0.31

6/4 12:02pm

reiser 10.1 6.2 12.5 49.7 0.20 0.12 0.25

xfs 11.6 8.3 6.4 49 0.24 0.17 0.13

ext3 12.3 8.2 14.4 48.8 0.25 0.17 0.30

jfs 10.1 6.2 11.2 41.3 0.24 0.15 0.27

6/4 15:20pm

reiser 10.2 6.8 12.5 47.8 0.21 0.14 0.26

xfs 13.9 9.9 7.6 58.9 0.24 0.17 0.13

ext3 11.9 7.9 13.5 46.9 0.25 0.17 0.29

jfs 13.4 6 13.4 41.9 0.32 0.14 0.32

7/4 07:54am

reiser 7.8 4.7 10.7 34.8 0.22 0.14 0.31

xfs 8.4 5.6 4.7 26.9 0.31 0.21 0.17

ext3 7.5 5.2 10.2 29.4 0.26 0.18 0.35

jfs 6 3.7 9.3 24.8 0.24 0.15 0.38

7/4 1:44pm

reiser 12 8.5 19.7 55.3 0.22 0.15 0.36

xfs 11.4 9.3 7.5 42.2 0.27 0.22 0.18

ext3 12.3 8.3 15.5 53 0.23 0.16 0.29

jfs 12.3 8.5 17.2 51.9 0.24 0.16 0.33

8/4 9:57am

reiser 12.4 9.2 23.4 46.4 0.27 0.20 0.50

xfs 11 7.1 6.5 40.2 0.27 0.18 0.16

ext3 11.4 7.9 12.3 50 0.23 0.16 0.25

jfs 10.1 6 21.6 45 0.22 0.13 0.48

11/4 8:45am

reiser 9.2 5.8 18.4 42.6 0.22 0.14 0.43

xfs 9.4 6.9 5.3 34.5 0.27 0.20 0.15

ext3 9.5 5.5 13.9 40.1 0.24 0.14 0.35

jfs 10.6 6.8 24.4 40.3 0.26 0.17 0.61

-- 
No virus found in this outgoing message.
Checked by AVG Anti-Virus.
Version: 7.0.308 / Virus Database: 266.9.5 - Release Date: 4/7/2005