Is it time to stop adding features to soaring software? Is it timeto focus on reliability?

We made unit tests to check complicated stuff like OLC realtime
calculations, FAI triangle calculations and such, in the development phase.
But generally I called "unit tests" the people doing individual checking of
each beta versions, and the experience shew that you need at least 300 of
them for 3 months to be relatively sure everything is ok. This is why I have
brought the beta phase to almost 12 months.
One way or another, you still need beta testing because obvious problems are
easy to fix, while the nasty stuff is always obfuscated and for Murphy's
laws will pass all unit tests, because tests did not consider the problem
(otherwise, you would have fixed it already).
Best would be to have both, of course. Xcsoar and LK can have hundreds of
betatesters, and dozens of eyes checking at the code and spotting problems.
But in the end, people doing debugging are just a few around the world, for
both projects. You can count people doing this work on xcsoar and lk8000
with fingers of one hand.




"Tobias Bieniek" wrote in message
...

We're all using our free time in a way which makes sense and fun. Finding
bugs, correcting them and even rewriting code just because once in the
past we took some shortcuts and now we're seeing the unwanted effects is
not fun.

Well... actually... I've been doing exactly that for three years on the
XCSoar project now and let me tell you that this can be fun too. For me it
was a learning experience that ultimately got me my current job and a few
other things before that.

and @Paolo: why do you have unit tests if you don't even trust them?

On Tuesday, December 11, 2012 5:18:29 PM UTC+1, pcool wrote:
We made unit tests to check complicated stuff like OLC realtime
calculations, FAI triangle calculations and such, in the development phase.

Your use of the plural "tests" implies that there is more than one. However, that's an exaggeration, there's only one program (TestContest), and it's not even a unit test.

But generally I called "unit tests" the people doing individual checking of
each beta versions, and the experience shew that you need at least 300 of
them for 3 months to be relatively sure everything is ok. This is why I have
brought the beta phase to almost 12 months.

People are not unit tests. I think you misunderstand the meaning of the word "unit test", which is what this thread is about. You dismiss them as "useless" which you know too little about.

One way or another, you still need beta testing because obvious problems are
easy to fix, while the nasty stuff is always obfuscated and for Murphy's
laws will pass all unit tests, because tests did not consider the problem
(otherwise, you would have fixed it already).

Not quite. We XCSoar developers fix a lot of bugs that are found by unit tests. By the time new code gets published, these bugs are fixed already. Unit tests help a lot during development, and save a lot of time.

Just look how many bugs you had to fix last week, that would not have happened with unit tests.

You can count people doing this work on xcsoar and lk8000
with fingers of one hand.

Hm. 4,638 pilots have installed XCSoar 6.5 preview releases on Android alone (number of unique Google accounts, no duplicates). The stable 6.4 version has been installed on Android by 22,005 pilots. Not counting all those people on Linux, Windows, WinCE, Mac OS X. Our bug tracker has 415 user accounts and 2,400 bug reports in the past 3 years. Lots of eyes, lots of bugs & bug fixes!

What makes me wonder is why you rejected the bug fixes I sent you today: https://github.com/LK8000/LK8000/pull/307

You wont find on github the test procedures for Contest and FAI, the latest
I remember. The contest test you mention is not the one we used.
In either cases I did not make them.
However this is not the point. I agree that having internal tests is better
than not having them! Of course.
Our 289 internal checks made with assertions can help, and did help, but
cannot be compared to your unit tests.
Honestly I cannot judge your code because I dont know it at all, but I am
sure it is well thought for this part as well.

You know the reason why I dont merge your code already, and it is not worth
discussing it here for a simple reason, which I think you agree on.
Some software manufacturers are just upset, to use a minimalistic word, by
the fact free software is now at a quality standpoint that is making a real
alternative to commercial products. Having one free software is already a
pain, having two is simply killing someone business.
It looks pretty funny to them, and not only , to read yours and mine
argumentations about how good or how bad one software is.



"Max Kellermann" wrote in message
...

On Tuesday, December 11, 2012 5:18:29 PM UTC+1, pcool wrote:
We made unit tests to check complicated stuff like OLC realtime
calculations, FAI triangle calculations and such, in the development
phase.

Your use of the plural "tests" implies that there is more than one. However,
that's an exaggeration, there's only one program (TestContest), and it's not
even a unit test.

But generally I called "unit tests" the people doing individual checking
of
each beta versions, and the experience shew that you need at least 300 of
them for 3 months to be relatively sure everything is ok. This is why I
have
brought the beta phase to almost 12 months.

People are not unit tests. I think you misunderstand the meaning of the word
"unit test", which is what this thread is about. You dismiss them as
"useless" which you know too little about.

One way or another, you still need beta testing because obvious problems
are
easy to fix, while the nasty stuff is always obfuscated and for Murphy's
laws will pass all unit tests, because tests did not consider the problem
(otherwise, you would have fixed it already).

Not quite. We XCSoar developers fix a lot of bugs that are found by unit
tests. By the time new code gets published, these bugs are fixed already.
Unit tests help a lot during development, and save a lot of time.

Just look how many bugs you had to fix last week, that would not have
happened with unit tests.

You can count people doing this work on xcsoar and lk8000
with fingers of one hand.

Hm. 4,638 pilots have installed XCSoar 6.5 preview releases on Android alone
(number of unique Google accounts, no duplicates). The stable 6.4 version
has been installed on Android by 22,005 pilots. Not counting all those
people on Linux, Windows, WinCE, Mac OS X. Our bug tracker has 415 user
accounts and 2,400 bug reports in the past 3 years. Lots of eyes, lots of
bugs & bug fixes!

What makes me wonder is why you rejected the bug fixes I sent you today:
https://github.com/LK8000/LK8000/pull/307