On Thursday, August 6, 2015 at 5:40:12 PM UTC-7, wrote:
On Thursday, August 6, 2015 at 8:28:24 PM UTC-4, Andy Blackburn wrote:
On Thursday, August 6, 2015 at 7:40:21 AM UTC-7, Andy Blackburn wrote:

We now have at least one contest we can compare the results from to see if pilots scored significantly differently with stealth vs not - there is normally a lot of variance in the scoring even over a long contest so it may or may not show anything but at least we will have an indication if anything significant is going on.


I ran some numbers to test the theory that less skilled pilots use Flarm as a way to up their performance - and that Stealth mode helps negate this strategy. I compare the last two 15 Meter Nationals - 2014 at Montague (Flarm without Stealth Mode) and 2015 at Harris Hill (Flarm with Stealth Mode). I compared each pilot's performance as measured by final score as a percentage of winner's score to their PRL percentage, which is the longer-term average of the same metric and is used for US Team selection so it's the best available measure of overall pilot skill. The metric I used was average absolute error for each contest - on average how closely did contest performance match PRL percentage. I dropped from the analysis any pilots who withdrew from either contest.

The theory we are testing is that with unrestricted Flarm you should get more inexperienced pilots higher in the scoresheet with higher scores and potentially some of the more highly skilled pilots dropping down as a result.

The average absolute error for the 2014 15-Meter Nationals at Montague was 7.4%. That is, on average pilots scored only 7.4% off of their PRL. Four pilots were competing in Standard Class gliders at Montague without handicaps. If I give them each a 4% handicap the average absolute error drops to 7.1%.

The 2015 15-Meter Nationals at Harris Hill had an average absolute error of 11.8%. That a significant difference and not in the direction you'd expect. It turns out that at Harris Hill a number of highly ranked pilots under-performed their PRL and a few lower-ranked pilots out-performed their PRL.

This is 35 pilots flying a total of 259 contest flights, so it's not a super big sample size, but not terrible given the stark difference. I also looked for any potential sources of sample bias - average pilot skill, contest length or devalued days. The two contests were pretty similar. Average PRL for Montague was .899 and Harris Hill was .906. Montague was 8 days and Harris Hill was 7 days. The average daily winner's score at Montague was 869 points and Harris Hill was 844 points. Not perfect, but pretty close.

So, if the theory is that Flarm allows lower skilled pilots to leech their way to the top of the score sheet and Stealth mode puts a lid on this behavior, the data suggests the opposite. In fact a number of long-time (30-40 years) highly ranked pilots under-performed their PRL with stealth mode. One potential reason for this is that more experienced pilots are actually better at making use of Flarm information than inexperienced pilots. The other theory is that Flarm helps reduce the random landouts that tend to scramble the scoresheet and actually reinforces stability and order.

Discuss.

9B

P.S. I already said that it's only 259 data points.

Weather variability is most likely a bigger factor than Flarm. To wit- SM, who never lands out got snookered multiple times at Elmira. Same with KS. Only 3 pilots completed all tasks assigned. I suspect the weather had a much bigger effect on performance variability from the expected than Stealth- or not- ever would.
It is also worth noting that FS, who has a lot of Flarm expertise relative to the rest of us, won 15M in Stealth. Would he have enhanced his advantage? I think so.
UH

Well SM landed out on a 486 point day at Harris Hill and P7 landed out on a 850 point day at Montague (and scored even lower) so I'd say that was worse. Also, the landout day that caught the guys you mentioned also got the guy who won the contest, so I'd say that bit of weather variability didn't really scramble the field much and therefore wasn't much of a factor. Just to be sure, I took out the roughly 200 point benefit the finishers got over the landouts that day. The average standard error went from 11.8% to 12.9% - so it made it even worse.

There are always exceptional circumstances on any given day, but things tend to average out with enough days and the effects aren't as strong as people seem to remember.

The puzzling thing is that lower-ranked pilots out-performed with Stealth on and HH and despite a fair amount of weather randomness at Montague and a lot of low scratching in the mountains and blue holes - the middle ranked pilots weren't able to out-perform, so not only is the leeching theory not reflected in the data we have, but the data tilts a bit in the opposite direction - Stealth mode is actually associated with less consistent results.

Yes, only two contests - but if there was something seriously going wrong from using Flarm you'd not expect these results.

9B