anecdotal ccd interleave performance numbers on sparc64

[Erik E. Fair]

I'm setting up an UltraSPARC replacement for an older Sun SPARCstation 20
running NetBSD. After a little advice from ICB, I tried some quick testing of
different ccd interleave numbers for optimal raw read and write performance
in ccd, striped across two 7200rpm SCSI disks, on two separate controllers.

The optimal interleave number for this configuration appears to be 64.

The configuration is an Ultra Enterprise 2:

NetBSD 3.0.1_STABLE (UE2) #2: Sun Jul 23 20:02:55 PDT 2006
***@grandfather.clock.org:/usr/obj/sys/arch/sparc64/compile/UE2
total memory = 256 MB
avail memory = 242 MB
mainbus0 (root): SUNW,Ultra-2: hostid
cpu0 at mainbus0: SUNW,UltraSPARC-II @ 296.002 MHz, version 0 FPU
cpu0: 32K instruction (32 b/l), 16K data (32 b/l), 2048K external (64 b/l)
sbus0 at mainbus0 addr 0xfffc8000: clock = 25 MHz
[...]
esp0 at sbus0 slot 14 offset 0x8800000 vector 20 ipl 3: FAS366/HME, 40MHz, SCSI ID 7
scsibus0 at esp0: 16 targets, 8 luns per target
esp1 at sbus0 slot 0 offset 0x8800000 vector 3 ipl 3: FAS366/HME, 40MHz, SCSI ID 7
scsibus1 at esp1: 16 targets, 8 luns per target
esp2 at sbus0 slot 2 offset 0x8800000 vector 3 ipl 3: FAS366/HME, 40MHz, SCSI ID 7
scsibus2 at esp2: 16 targets, 8 luns per target
[...]
sd2 at scsibus1 target 0 lun 0: <SEAGATE, SX118273LC, 6367> disk fixed
sd2: 17366 MB, 7501 cyl, 20 head, 237 sec, 512 bytes/sect x 35566480 sectors
sd2: sync (100.00ns offset 15), 16-bit (20.000MB/s) transfers, tagged queueing
sd3 at scsibus2 target 1 lun 0: <SEAGATE, SX118273LC, 6367> disk fixed
sd3: 17366 MB, 7501 cyl, 20 head, 237 sec, 512 bytes/sect x 35566480 sectors
sd3: sync (100.00ns offset 15), 16-bit (20.000MB/s) transfers, tagged queueing

If you're interested in the precise specifications for those drives, a search
at seagate.com with the "SX" replaced with "ST" from the identify information
provided above will give results.

The disks have their on-HDA electronics write-back caches enabled, because I
have a UPS, and because NetBSD knows to send a cache flush on shutdown. This
has a significant effect on write performance.

The test commands were:

dd if=/dev/rccd0c of=/dev/null bs=1m count=1000 ; read
dd if=/dev/zero of=/dev/rccd0c bs=1m count=1000 ; write

The results, as reported by dd:

ileave read (MB/s) write (MB/s)
32 23.3 20.6
64 26.2 22.8
128 15.0 13.5
256 15.0 13.5

What can I say? I like powers of 2. This is all just FYI.

Erik <***@netbsd.org>

[Thor Lancelot Simon]

Post by Erik E. Fair
I'm setting up an UltraSPARC replacement for an older Sun SPARCstation 20
running NetBSD. After a little advice from ICB, I tried some quick testing of
different ccd interleave numbers for optimal raw read and write performance
in ccd, striped across two 7200rpm SCSI disks, on two separate controllers.
The optimal interleave number for this configuration appears to be 64.

Sure, if you only care about a single stream of I/O.

Much better performance for most applications can be had by using a large
prime number as the interleave factor, which will let the disks work in
parallel for most pairs of requests.

[Erik E. Fair]

Can you bound the range of said prime?

Further, can you perhaps review/revise the meager discussion of
interleave numbers in ccd(4)? Or give an appropriate reference?


Erik <***@netbsd.org>