Ideas for 2nd Fefe benchmarks?

[Hubert Feyrer]

Reading Fefe's (german language) weblog[1] he writes:

``I'm working on new benchmarks similar to my old ones[2], but I want to
measure more this time. I have taken about 30GB of image data from a
customer's webserver, and logged about 50.000 web requests, and wanted
to replay them as fast as possibl. To measure the filesystem I want to
keep 10% of the 30GB images seperately, and then use tar to unpack and
measure. But the tests so far are very negative for BSD (10 minutes
unpacking on Linux with ext3 against 26 minutes under FreeBSD 6 with
UFS2. I'm therefore looking for benchmark ideas in this category (so not
"install postgres and shove in the following selects"), which are
friendly to BSD and unfriendly to Linux. There has to be something?!
Please mail your hints to me!''

Given the last round of benchmarks and they improvements that were made to
NetBSD after them, this sounds very interesting. Ideas? What is it that
NetBSD can do better than Linux (and FreeBSD :-).


- Hubert


[1] http://blog.fefe.de/?ts=bde80e9c
[2] http://bulk.fefe.de/scalability/

[Pavel Cahyna]

Post by Hubert Feyrer
``I'm working on new benchmarks similar to my old ones[2], but I want to
measure more this time. I have taken about 30GB of image data from a
customer's webserver, and logged about 50.000 web requests, and wanted
to replay them as fast as possibl. To measure the filesystem I want to
keep 10% of the 30GB images seperately, and then use tar to unpack and
measure. But the tests so far are very negative for BSD (10 minutes
unpacking on Linux with ext3 against 26 minutes under FreeBSD 6 with
UFS2. I'm therefore looking for benchmark ideas in this category (so not
"install postgres and shove in the following selects"), which are
friendly to BSD and unfriendly to Linux. There has to be something?!
Please mail your hints to me!''
Given the last round of benchmarks and they improvements that were made to
NetBSD after them, this sounds very interesting. Ideas? What is it that
NetBSD can do better than Linux (and FreeBSD :-).

I also measured unpacking with tar (in my case, it was pkgsrc.tar.gz).

I found that it is helpful to:
- raise the kern.maxvnodes sysctl (probably to the number of files going
to be created by tar)

- do it on LFS instead of FFS

Probably it would also help building kernel with options UFS_DIRHASH (but
I don't know if it is compatible with LFS and FreeBSD probably already has
it by default). Also using UFS1 instead of UFS2 (if you don't want to use
LFS) might help.

Bye Pavel

[Hubert Feyrer]

Post by Pavel Cahyna
Also using UFS1 instead of UFS2 (if you don't want to use
LFS) might help.

Why/how?


- Hubert

[Pavel Cahyna]

Post by Pavel Cahyna
Also using UFS1 instead of UFS2 (if you don't want to use
LFS) might help.

Because UFS2 has inodes twice as large than UFS1 IIRC, so it means more
data going to/from disks and more buffer space wasted. But I never
actually did a performance comparison.

Bye Pavel

[Hubert Feyrer]

Post by Pavel Cahyna

Post by Pavel Cahyna
Also using UFS1 instead of UFS2 (if you don't want to use
LFS) might help.

Because UFS2 has inodes twice as large than UFS1 IIRC, so it means more
data going to/from disks and more buffer space wasted. But I never
actually did a performance comparison.

Let us know when you did. I'm sure the 4 bytes per file have a huge
performance impact.


- Hubert

[Mike Cheponis]

Post by Hubert Feyrer

Post by Pavel Cahyna

Post by Pavel Cahyna
Also using UFS1 instead of UFS2 (if you don't want to use
LFS) might help.

Because UFS2 has inodes twice as large than UFS1 IIRC, so it means more
data going to/from disks and more buffer space wasted. But I never
actually did a performance comparison.

Let us know when you did. I'm sure the 4 bytes per file have a huge
performance impact.
- Hubert

It probably won't have a huge difference, but I have seen instances where just a few bytes over a cache line or just a few bytes over some buffer boundary does, indeed, cause poor performance.

But, we'll know when Pavel completes his testing.

-Mike

[matthew green]

Post by Pavel Cahyna

Post by Pavel Cahyna
Also using UFS1 instead of UFS2 (if you don't want to use
LFS) might help.

Because UFS2 has inodes twice as large than UFS1 IIRC, so it means more
data going to/from disks and more buffer space wasted. But I never
actually did a performance comparison.

Let us know when you did. I'm sure the 4 bytes per file have a huge
performance impact.


this is not 64 bit ino_t. this is 256 bytes -> 512 bytes IIRC.

[Sten Daniel Sørsdal]

Post by Pavel Cahyna

Post by Pavel Cahyna
Also using UFS1 instead of UFS2 (if you don't want to use
LFS) might help.

Because UFS2 has inodes twice as large than UFS1 IIRC, so it means more
data going to/from disks and more buffer space wasted. But I never
actually did a performance comparison.
Bye Pavel

I am no FS hacker but i was under the impression that the smallest
writable unit is sector(512 bytes). If UFS2 adds more sector
reads/writes then sure, it would be a little slower but only noticable
if it's an scattered read/write.

How large is an inode and why does this make a difference?

[Matthias Scheler]

Post by Hubert Feyrer
``I'm working on new benchmarks similar to my old ones[2], but I want to
measure more this time. I have taken about 30GB of image data from a
customer's webserver, and logged about 50.000 web requests, and wanted
to replay them as fast as possibl. To measure the filesystem I want to
keep 10% of the 30GB images seperately, and then use tar to unpack and
measure. But the tests so far are very negative for BSD (10 minutes
unpacking on Linux with ext3 against 26 minutes under FreeBSD 6 with
UFS2. I'm therefore looking for benchmark ideas in this category (so not
"install postgres and shove in the following selects"), which are
friendly to BSD and unfriendly to Linux. There has to be something?!
Please mail your hints to me!''
Given the last round of benchmarks and they improvements that were made to
NetBSD after them, this sounds very interesting. Ideas? What is it that
NetBSD can do better than Linux (and FreeBSD :-).

Wouldn't it be much more interesting to focus on what we do worse than
Linux (e.g. the FS part in the above) and fix that?

Kind regards

[Wolfgang S. Rupprecht]

Post by Hubert Feyrer
Given the last round of benchmarks and they improvements that were
made to NetBSD after them, this sounds very interesting. Ideas? What
is it that NetBSD can do better than Linux (and FreeBSD :-).

One thing that all open-source distributions stumble on to some extent
is 64-bit cleanliness. On some distributions programs like (cpio,
rdist) seem to be plagued with 32-bit file size limits. Some
distributions have 32-bit file-size limits on NFS. Others have
problems when running on 64-bit CPU's of dealing with > 32-bit bss or
data segments when either malloc-ing the memory or mmap-ing it. This
should be a fun bake-off because everybody has these problems, and in
many cases they are relatively simple to fix which will lead to a lot
of frenzied fixing which will be good for the open-source community as
a whole.

(Trying to deal with creating, moving and archiving an 8GByte data
file has been an eye opener. Nobody seems to get all aspects of this
right.)

-wolfgang