benchmark

[Igor Shmukler]

Hubert,

In general I would not bother to reply to a message to that died without respose a while ago.
However, I am tired of you plain lying and trying to seed conflicts between BSD projects.
Everything you say is pretty much BS.

1. nobody called benchmarksuseless. I and other simply pointed out that a microbenchmarks do not show performance. That's why they are called microbenchmarks. They do however show that NetBSD seriously looked at number of low level allocators and scalability is now excellent.
2.FreeBSD people do respect NetBSD team and other software developers do project do borrow from one another be it commercial or open source. 
3. If I would be a mail admin for either BSD I would have banned you a while ago.

What you are doing is very wrong. You are not helping either project.

It's sad what kind of person you are. BSDs are already very segmented. For number of reasons we now have four project FreeBSD, NetBSD, Op
enBSD and Dragonfly. There is already a serious lack of resources and a significant amount of time is being spent on tracking progress of other projects so that things that do improve performance or other aspects find it's way in every OS.

What is your contribution to BSD cause? What are you trying to do? Do you want to hurt BSD effort? If you are not doing this on purpose, I think you are just too dumb to be helpful. 

IS.

-----
When I posted Gregory McGarry's benchmark recently, a lot of (FreeBSD :)
people cried that microbenchmarks are useless, and "real life" benchmarks 
should be used. Well, as sort of an answer, I've collected a few data 
points on this whole issue:

 * "Micro"benchmark between NetBSD 2.0 and FreeBSD 5.3:
 http://www.feyrer.de/NetBSD/gmcgarry/
 * Memory/File performance comparison between NetBSD 2.0, OpenBSD 3.6
 and Fedora Core 3:
 http://bsd.slashdot.org/comments.pl?sid=1352
40&threshold=-1&commentsort=0&tid=190&tid=7&mode=thread&pid=11309939#11319773
 * Internet2 speed records:
 http://proj.sunet.se/LSR2/
 http://proj.sunet.se/LSR3-s/
 http://proj.sunet.se/LSR3-m/
 * Fefe's Benchmark of Linux, NetBSD, FreeBSD and OpenBSD:
 http://bulk.fefe.de/scalability/


 - Hubert

[David Maxwell]

Post by Igor Shmukler
Hubert,



If you want to flame someone, could you at least use ASCII?

Thanks.

[Igor Shmukler]

I am sorry about not using ascii.
I use this as a special email (webmail) account for mailing lists and conferences, so that my work
and/or school email does not get bombarded with spam.
I am using a Mac as a workstation and there are some bugs in Mac's IE. Recently I started using
Firefox after trying Opera and Safari, but apparently it has its problems too. I should probably
file this as a bug and hope Mozilla folks will fix this soon.
Sorry. For now I am composing my webmail under IE again. Messages written with it used to appear
alomost the same with[/out] HTML formatting.

About flaming. I am not a fan of publically humiliating anyone. It's a wrong thing to do. In fact
when the person who started this sent an email to FreeBSD performance list I merely pointed out
that microbenmark != performance. It does not mean that NetBSD performs worse than FreeBSD. In fact
it is possible that NetBSD is better, although I want to see a proof before such a statement could
be made. I would think if dirhash optimizations get incorporated into the kernel performance will
get another serious boost. I understand this is on its way?!

However, I don't like when an entire community gets crapped upon. In this particular instance
without any merit.

If the author thinks that admistrating FTP for an amiga port and writing a networking FAQ gives him
a license to lie and spread FUD, I wonder why did not anyone came to his defence. Perhaps I am not
totally alone.

Personally, I would be very happy if NetBSD will the very best OS in the world. Hardware
abstraction makes porting a breeze. As far as license goes FreeBSD and NetBSD are equivalent.

I am speaking only for myself, but I believe most FreeBSD people would agree to some degree. I do
not dislike NetBSD project! I wish anyone success. I work for a kernel development shop and I
personally lifted some code from NetBSD in a past. I have a reason to be thankful to NetBSD
developers.

What I do not like is FUD and dishonesty.

If someone has nothing better to, it is possible to configure Oracle to run under a BSD and run
TPC-C. Results will reflect database "performance". I am sure there are benchmarks that could be
used to evaluate which system performs better.

However even that is not so simple. For instance system could be optimized for high-load, medium-
load, low-latency etc.

Either way.

I have lots of work to do. So I am bowing out, unless someone would find that something I have said
is not accurate.

igor.

> Hubert,


If you want to flame someone, could you at least use ASCII?
Thanks.
--
If you don't spend energy getting what you want,
You'll have to spend it dealing with what you get.
- Unknown

[Dries Schellekens]

Post by Igor Shmukler
About flaming. I am not a fan of publically humiliating anyone.
It's a wrong thing to do. In fact when the person who started this
sent an email to FreeBSD performance list I merely pointed out that
microbenmark != performance. It does not mean that NetBSD performs
worse than FreeBSD. In fact it is possible that NetBSD is better,
although I want to see a proof before such a statement could be made.

Did you read the conclusions of the paper?

"Microbenchmarks are not always the best indicators to make judgments on
the overall performance of one operating system over another. However,
they are useful to infer an understanding of the architectural decisions
that go into building an operating system. For many applications, the
results presented in the paper may never affect performance. For others,
the scalability of the operating system may simply not permit the
application to run suitably."

and

"Although NetBSD 2.0 has outperformed FreeBSD 5.3 in most of the
benchmarks presented here, FreeBSD 5.3 has made significant developments
with its symmetric multiprocessor (SMP) architecture, particularly in
the area of scalability with fine-grained locking. NetBSD 2.0 continues
to use a single lock to serialize access to kernel mode. Additionally,
the performance of the thread implementation on multiprocessor systems,
where thread concurrency can be achieved, would be worth investigating.
Benchmarks for these areas are the objective of future research."


Cheers,

Dries

[Igor Shmukler]

Dries,
If you read this thread of the one from FreeBSD performance carefully, you will notice following.
Nobody EVER argued with the paper or its content.
Actually, as far as I remember original article on OSNews had a normal title too - NetBSD scales
better (or something like this).
So to answer your questions - YES.
BTW, there are problems with the benchmark suite. We used for an internal project but after
changing some of the benchmarks, otherwise results MAY not be correct.
I contacted author to provide details (back when this was originally published), but he did not
follow up. Perhaps spam filter swollowed my email.
The acticle is NOT a rant at all. It simply an analisys of a scalability test-suite.
Igor.

Post by Dries Schellekens

Post by Igor Shmukler
About flaming. I am not a fan of publically humiliating anyone.
It's a wrong thing to do. In fact when the person who started this
sent an email to FreeBSD performance list I merely pointed out that
microbenmark != performance. It does not mean that NetBSD performs
worse than FreeBSD. In fact it is possible that NetBSD is better,
although I want to see a proof before such a statement could be made.

Did you read the conclusions of the paper?

Post by Igor Shmukler
Microbenchmarks are not always the best indicators to make judgments on

the overall performance of one operating system over another. However,
they are useful to infer an understanding of the architectural decisions
that go into building an operating system. For many applications, the
results presented in the paper may never affect performance. For others,
the scalability of the operating system may simply not permit the
application to run suitably."
and

Post by Igor Shmukler
Although NetBSD 2.0 has outperformed FreeBSD 5.3 in most of the

benchmarks presented here, FreeBSD 5.3 has made significant developments
with its symmetric multiprocessor (SMP) architecture, particularly in
the area of scalability with fine-grained locking. NetBSD 2.0 continues
to use a single lock to serialize access to kernel mode. Additionally,
the performance of the thread implementation on multiprocessor systems,
where thread concurrency can be achieved, would be worth investigating.
Benchmarks for these areas are the objective of future research."
Cheers,
Dries

[Hubert Feyrer]

Post by Igor Shmukler
The acticle is NOT a rant at all. It simply an analisys of a scalability test-suite.

And that justifies you calling me a liar... how?


- Hubert

[Igor Shmukler]

Post by Hubert Feyrer

Post by Igor Shmukler
The acticle is NOT a rant at all. It simply an analisys of a scalability test-suite.

And that justifies you calling me a liar... how?

That has nothing to do with me calling you a liar.

You wrote "When I posted Gregory McGarry's benchmark recently, a lot of (FreeBSD :)
people cried that microbenchmarks are useless...", that did not happen as anyone interested could
see from freebsd-performance mailing list archive. That's why I call[ed] you a liar.

Nobody said benchmarks are useless. Microbenchmarks have a place, but they do not reflect
performance of an operating system. In particular, the suite in question shows scalability of some
resource allocators. That's what you were told.

Also, you asked me to google who you are. I did. Now what?

I feel sorry for you, but you started this. First by posting an email "Benchmark: NetBSD 2.0 beats
FreeBSD 5.3 in server performance" to freebsd-performance, then by claiming that LOTS of folks told
you microbenchmarks are USELESS.

Don't spread FUD, don't lie and you won't be a target of stupid flame-threads that waste eveyone's
time.

Again, you still did not provide anything to back your claim about NetBSD beating FreeBSD (besides
LSR). LSR is a nice thing and large frames are helpful, but it's not something I would call server
performance. In fact, be it a server or a workstation NetBSD will be faster on LSR. However, a
typical server opens more than one connection (often over multiple interfaces) and does more than
just pipe data through. Hence, LSR is similar to a microbenchmark. If I would be a part of project
that won LSR, I would too promote this as an achievement, there is nothing wrong with that.
However, it does not prove that YOUR point.

The funny part that it's possible that NetBSD would beat FreeBSD [at least] in UP configuration.
But you NEVER BOTHERED to run normal benchmarks such as SPEC, TPC etc.

Posting something stupid and starting a flame war is that much easier.

Do you understand why I called a liar now?

Igor.

[Frank van der Linden]

Post by Igor Shmukler
Don't spread FUD, don't lie and you won't be a target of stupid flame-threads that waste eveyone's
time.

That's right. This thread is a big waste of time, so please stop.

Thanks,

- Frank

[Hubert Feyrer]

Post by Igor Shmukler
You wrote "When I posted Gregory McGarry's benchmark recently, a lot of
(FreeBSD :) people cried that microbenchmarks are useless...", that did
not happen as anyone interested could see from freebsd-performance
mailing list archive. That's why I call[ed] you a liar.

You're ignoring the mail I got in private, and quite some of which were
written in the same tone of voice that your initial posting was in.
You're aparently also ignoring the discussion that arose on slashdot.
I cannot tell what happened on the freebsd-performance list (which I was
told to spam, btw), as I'm not on the list.

That said, I think an apology is in order from your side.


- Hubert

[Igor Shmukler]

Post by Hubert Feyrer
You're ignoring the mail I got in private, and quite some of which were
written in the same tone of voice that your initial posting was in.

Well, if it was in the same tone as my initial posting, there should be no problem. I merely, in a
polite way, pointed out that (performance != microbenchmark) which is correct. I pasted that email
below, so anyone could see there is nothing wrong with it. Hence, I am sure everyone was on the
ball.

Post by Hubert Feyrer
You're apparently also ignoring the discussion that arose on slashdot.
I cannot tell what happened on the freebsd-performance list (which I was
told to spam, BTW), as I'm not on the list.

You were to told to spam freebsd-performance. By whom? If someone told you to kill FreeBSD core,
would that be an excuse?

I did not read slashdot. I am not even subscribed to NetBSD lists, but I look through kernel
related traffic for ideas.

Post by Hubert Feyrer
That said, I think an apology is in order from your side.

Apology for what exactly? Is anything that I have written so far inaccurate?

How about we continue this in private? Frank asked to discontinue this thread I think it relates to
both of us.

PS If you were not the one who started thread on FreeBSD, why did you not tell anyone that someone
impersonated you? People (including myself) have written you. The email address work (as we know
now). In fact there are other reasons for me to believe you were not impersonated, but I believe
it's beyond the scope of this thread. Obviously, it's not totally impossible. If you started that
thread, then it's irrelevant whether you are subscribed or not. You said many FreeBSD people told
you benchmarks are useless. You did not mention up until now that ALL of these many people were
afraid to post to a public mailing list. Otherwise, they have been naturally told. Microbenchmarks
are NOT useless. They benchmark performance of some aspect of a system. That's why people run them.
However, we can hardly accept that these misterious cowards represent FreeBSD community.
PPS That was rather large post-scriptum


---------------------------------------
Hubert,
The benchmark suite in question is not bad, but it's a microbenchmark suite.
BTW, when we used the suite to tune kernel for ourt needs we had to change many things.
As of a year ago some tests had to be modified, otherwise we might start tuning for the wrong
goals. The title of this email and the article is NetBSD beats FreeBSD in performance should be
changed to NetBSD beats FreeBSD on comprehensive microbenchmark suite. I don't remember who it was,
but someone posted results of real server tests. These were in favor of FreeBSD. I doubt that much
changed since last year but who knows ... Maybe NetBSD is better, but this is not something that
could be concluded from a microbenchmark.
IMHO.
igor.



;-----Original Message-----
From: Hubert Feyrer <hubert at feyrer.de>
To: freebsd-performance at FreeBSD.org, freebsd-advocacy at FreeBSD.org
Date: Thu, 6 Jan 2005 02:19:37 +0100 (CET)
Subject: Benchmark: NetBSD 2.0 beats FreeBSD 5.3 in server performance

Post by Hubert Feyrer
Abstract: ``With the recent releases of NetBSD 2.0 and FreeBSD 5.3 operating
system, many new and exciting features have been implemented. Both criticism
and commendation on performance, reliability and scalability have been directed
towards these releases.
This paper presents a suite of benchmarks and results for comparing the
performance of these operating systems. The benchmarks target core operating
system functionality, server scalability and thread implementation. These
benchmarks are useful server-based criteria for demanding applications such as
loaded webservers, databases, and voice-over-IP (VoIP) media relays. The
results indicate that NetBSD has surpassed FreeBSD in performance on nearly
every benchmark and is poised to grab the title of the best operating system
for the server environment.''
URL: http://www.feyrer.de/NetBSD/gmcgarry/
- Hubert
--
NetBSD - Free AND Open! (And of course secure, portable, yadda yadda)

[David Maxwell]

Post by Igor Shmukler
I am sorry about not using ascii.

Thanks - the lines in this post are still rather long, but at least
they're not all strung together. The previous mail was so difficult to
read that I couldn't be bothered.

Post by Igor Shmukler
About flaming. I am not a fan of publically humiliating anyone. It's
a wrong thing to do. In fact when the person who started this sent an

The problem I see, is that instead of asking a question about Hubert's
intentions, you came out and said that he's a liar. That's a very
accusatory tone to start a conversation with, and really doesn't
encourage people to work together to sort out issues.

Post by Igor Shmukler
However, I don't like when an entire community gets crapped upon. In
this particular instance without any merit.

I think you are overreacting. One email, saying that a new benchmark
shows that NetBSD outperforms FreeBSD is not 'crapping' on an entire
community. Discussion of the benchmark is reasonable, but if any
comparison between OSs is discouraged, everyone (on both sides) loses.

Post by Igor Shmukler
If the author thinks that admistrating FTP for an amiga port and
writing a networking FAQ gives him a license to lie and spread FUD, I
wonder why did not anyone came to his defence. Perhaps I am not totally
alone.

I think Hubert was pointing out that as a NetBSD port-master and
developer for ~12 years, he has more than a little bit of credibility as
a responsible and reasonable person. Accusing him of lies and FUD is not
likely to be taken well, when you are not known in _this_ community, and
he is.

Post by Igor Shmukler
What I do not like is FUD and dishonesty.

Slow down. Here's what I see:

Hubert posted a link to some benchmarking that was done. He didn't do
the benchmarking, and he didn't extend conclusions from it, he just
copied the statements in the report, essentially.

So, as far as that part goes, it seems you're shooting the messenger.

I fail to see any FUD or dishonesty on his part here. Certainly, the
applicability of any particular benchmark to any real-world workload can
be argued (or tested).

As for further comments about what feedback Hubert got on the post - I
certainly don't have access to his inbox. I'm guessing you don't either,
so how are you justified in calling him a liar for summarizing the
responses he received?

I'm inclined to agree with Hubert (and not just because of the respect
he's earned from prior behaviour), and suggest that you went way
overboard, and do owe him an apology.

[Hubert Feyrer]

On Mon, 24 Jan 2005, Igor Shmukler wrote:
[unreadable glibberish deleted]

Sir,

I'd strongly suggest you either back up your insult with some facts+proof,
or voice your apology.

As to the fact of what I do and who I am, ask google. If you take personal
offense of a text I did not write, I'll let that be your personal problem,
maybe read the first line of http://www.feyrer.de/NetBSD/gmcgarry/ again.


- Hubert