Some good ideas on how to balance objective and subjective inputs. Worth considering.

As someone who tilts to the analytical I find myself drawn to the precision of mathematical decision frameworks. Mathematical solutions are precise, transparent and not prone to things like block voting.

However, I'd observe that when it comes to calculating our way to a WGC team there is a difference between precision and accuracy. Just because we can calculate something to three decimal places doesn't mean that the precision signifies anything particularly accurate about the mix of the many personal attributes that makes someone a viable contender to be world champion.

We calculate UST scores in part based on performance at prior Nationals in each class. Given the inevitable random pilot performance and luck factors, this has two effects. First, people who have the personal flexibility to put in the travel to attend multiple sequential Nationals in a class have a better shot and second, people who have the personal flexibility to fly in Nationals in multiple classes get multiple bites at the apple. We only count the best scores, not the duds, so flying more contests ups your chances -at least if you are a good enough pilot to score near the top some of the time. There are a handful of pilots who get good scores based on only two qualifying contests flown - they are some of the best and most consistent pilots I know. Personally I look at both the average score across ALL contests flown AND the top two scores when attempting to judge a pilot's skill level and consistency. Neither is perfect by itself.

Also, we count, small, short and devalued Nationals the same as long competitive ones - even Nationals that are three days and get a 5% discount for PRL purposes get 100% credit for UST purposes - at least as I read it. We give credit for WGC performance and Continental contests in equal proportion even though the former is likely more competitive than a typical US Nationals and the latter could be the same or less - depending on participation. Adding these contests give pilots yet more scores to put into the mix and the WGC number gets counted across multiple classes, not just the one flown. More bites at the apple.

I'm not saying I necessarily disagree with the way things are calculated for the most part, but I would caution that just because you can calculate a score to three decimal places doesn't mean that those decimal places tell you something that's very accurate. How you arrange the formula has a big impact.

IMO a system that adds in peer review voting in a way that limits movement up or down the list to two or three places would be appealing. (BTW, it's my understanding that this is a primary purpose of the UST committee - to weed out voting monkey business). Tim's suggestion that we use a 1-10 scale where pilots in contention would typically earn a 9 or 10 seems like an opportunity for a small number of ill intentioned voters to blackball a pilot by giving them a 1 or 2. It's a good idea in general, but probably not implemented exactly that way.

9B



On Thursday, November 9, 2017 at 11:57:21 AM UTC-5, Tim Taylor wrote:
The system selection process could be improved by utilizing a hybrid system of objective (hard numbers from contests) with subjective (pilot votes). The current system used objective values to select those eligible but then used subjective voting to create the final list.

The forced choice ranking was subject to large variance and the absolute values were used to rank the pilots without looking to see if they were statistically significantly different. Rather than forced choice ranking a value of 0 to 10 could be given to each pilot during the voting. Remember all of these pilots were above 88 percent to make the list. Most should be getting an equavelnt score between 9 and 10 in voting.