Thank you to all who submitted feedback on the 2020 Judging System. We have considered everyone’s feedback and made changes where we thought appropriate. We would like to take this opportunity to explain generally the feedback we have received, the changes we have made in response, and any areas where we did not think any change was necessary.

 

 

As a starting point, we would like to address a few of the questions raised about specific results from Frisbeer and Holyjam. Although the committee takes no position on whether the results of those tournaments were “correct,” we have looked into each result raised by commentators and looked to see whether the changes made by the 2020 Judging System caused those results. What we found was that, in general, those results can be explained entirely by the raw scores. In other words, the results would be the same even if we input the raw scores into the Existing Judging System. In other other words, any “error” in the results was judge error, not judging system error.

 

 

For instance, many commentators asked about the placement (4th) of Fabio Nizzio and Tom Leitner in the Open Pairs Finals at Frisbeer. Their placement accords entirely with their raw scores. Even though they won execution by nearly two points, they received raw scores in the middle or bottom half in difficulty, variety, and AI. If those raw scores are put into the Existing Judging System, Fabio and Tom place no higher. To be clear, we are not saying they were or were not judged correctly, but the 2020 Judging System did not itself cause the outcome.

 

 

Many commentators also asked about the placement (6th) of Andrea Rimatori and Stefan Dunkel in the Holy Jam Open Pairs Finals. As before, their placement accords entirely with their raw scores. They received raw scores in the bottom half of difficulty, variety, and AI, and even their execution score was in the middle. Simply put, there is no multiplier, weighting change, or other system change that can take raw scores in the bottom half of every category and put that team on the podium.

 

 

We do acknowledge, however, that, the system interface may have contributed to some judging error. Some judges complained that it was difficult to see their prior scores for reference while entering new scores. Many more did not realize that they could see their prior scores by swiping up on their screens. The next iteration of the 2020 Judging System will make it much easier for judges to reference prior scores. Indeed, subject to other design constraints, we hope that in the future prior scores will always be visible.

 

 

Others have raised concern about judges feeling disconnected from their scores because of how some scores are affected by multipliers. Again, we note that, based on our analysis, none of the results commented on from Frisbeer or Holyjam would have changed in the absence of these multipliers given the raw scores. We will ensure, however, that future iterations of the judging system make it clearer to judges both what their raw and adjusted scores are. But we remind everyone that the multipliers exist precisely because, for some categories, judge’s raw scores do not reflect community consensus on how freestyle should be judged. For instance, the original 1.5 difficulty multiplier was introduced precisely because the community felt that the spread in difficulty scores did not reflect actual differences in the level of difficulty displayed by teams, which effectively diluted the weight of difficulty vis-a-vis the other divisions. To a certain degree, then, the multipliers exist to “correct” judges, and so we do not want judges overly focused on “adjusted” scores because allowing that undermines the purpose of the multipliers.

 

 

Another concern that has been raised is the weight of “AI” (as understood in the existing system) in the 2020 Judging System, particularly now that variety has been separated into its own category. Some have questioned whether four categories–general impression, music choreography, teamwork, and variety–deserve so much weight. Others have noted that while these categories may be extremely important, they are also the most subjective. We agree with these criticisms. The next iteration of the 2020 Judging System will reduce the weight of AI relative to execution and difficulty. Further testing will be required to find the “best” weight, but our goal will be to ensure that the spread of Difficulty and AI is, on average, about the same (we recognize, however, that sometimes the spread of one will actually be significantly smaller or larger than the spread of the other based on how the teams played).

 

 

The next iteration will also require every judge to input a general impression score at the end of each team. The weight of general impression will remain the same, but we hope that asking all 9 judges about their general impression will reduce some of the randomness and variance of what is arguably the most subjective evaluation in the judging system.

 

 

Finally, the next iteration of AI will also include form, which will reduce the weight of the other individual components of AI.

 

 

In difficulty, we have modified the difficulty gradient to give teams more flexibility in allocating their difficulty. At the end of the routine, all diff scores will be sorted from highest to lowest. In a 3 minute routine, the top 12 scores will first be scaled nonlinearly, then averaged together to get a final score of 0 to roughly 15. Extra difficulty scores will also be processed in a similar manner, but at a gradually reducing rate. This means that in a 3 minute routine, the 13th highest score will add roughly .3 times its value, the 14th highest score will add roughly .28 times its value and so on. This ensures that teams will continue to earn difficulty marks beyond their top 12 but at a much lower rate to disincentivize teams from “packing” difficulty by performing as many combinations as possible.

 

 

On the technical side, we expect the following improvements in the next iteration. First, the judging system will have an LAN/Offline mode so that it does not rely on internet to operate. Second, calculation sheets will be much easier to read and understand. Third, we will add short “primer” videos that judges will be required to watch before every round reminding them of things to remember and what to look for while judging each category (for instance, the execution video may include brief examples of the different types of deductions to look for). This last change will be part of a judging education effort and should also reduce certain cognitive biases.

 

 

The 2020 Judging System will be used next at Jammers and the Italian Open. We will continue to make revisions and improvements through each iteration. We value everyone’s continued feedback. You can send it to fpa-judging-committee@googlegroups.com

 

 

Thank you.

 

The 2020 Judging System was tested at Frisbeer and HolyJam. The Judging Committee would like to thank the tournament directors for the opportunity to use the 2020 Judging System and for the players and spectators for their patience while we tested and improved the system.

We are working on the issues that arose and we plan to produce a thorough write-up explaining how the system worked, why it produced any aberrational results, and the changes we are implementing going forward. 

To help us make the right changes, we would like to solicit more feedback from the players, spectators, and tournament organizers. We think it would be especially helpful for responses that point out any results that they may have been unfair or incorrect. All feedback will be kept completely anonymous.

Feedback email: fpa-judging-feedback@googlegroups.com

Please respond with feedback by Monday April 8th. We will issue another response addressing messages sent by April 8th.

______________________________

New changes and detailed explanations

Mid Frisbeer Changes

Difficulty averages based on min phrase count

• DESCRIPTION – In the case where a team performed fewer phrases than the minimum required, zero is added for the missing difficulty scores.

• Example: A 3 minute routine requires 12 difficulty scores. A team with 10 scores of 6 would score a 5 instead of a 6. 10 X 6 = 60. 60 / 12 = 5.

• REASON – This is to reduce the effectiveness of re revving to combine 2 separate combos into a single phrase to artificially gain a higher difficulty score. Long combos can be very difficult, but a combo being long does not automatically make it difficult. This change is intended to discourage breaks in consecutivity to create a longer combo.

Difficulty weight increased

• DESCRIPTION – The number of points gained from difficulty increase by about 50%.

• REASON – Compared to AI results after the first day, difficulty had a significantly lower impact. The intent is to create results where a team that does just as well in difficulty and AI, gets the same amount of points in difficulty and AI. Difficulty needed to be raised to match a similar score in AI. This change also addresses concerns raised about three AI categories (music choreography, teamwork, general impression) counting for too much and to balance out the increased weight given to variety.

EX ratio reduced

• DESCRIPTION – The benefit of phrase count decreasing the impact of execution deductions is reduced. The effectiveness has been reduced by about 50%. This means execution deduction are bigger.

• Example: In a 3 minute routine. Before a team with 25 phrases would reduce their deduction by about 50%. After the change, a team with 25 phrases would reduce their deduction by about 25%.

• REASON – Execution deductions were not playing a big enough role in final results. This change helps avoid a scenario where a team wins by dropping hard combos instead of catching medium combos.

Pre HolyJam Changes

Difficulty now includes partial credit for phrases after the minimum phrase count.

• DESCRIPTION – Extra phrases after the minimum phrase count have been met will now be added to the difficulty score, but at a reduced amount. The amount contributed will decrease for each additional score.

• Example: In a 3 minute routine, the top 12 are average normally. If a team has 15 phrases, the lowest 3 phrase scores are added at a reduced amount. If the 3 lowest phrase scores are 2, 2, 1, then that would add (2 X .3) + (2 X .28) + (1 X .26) = 1.42 points

• REASON – We saw 2 teams at Frisbeer have similar difficulty scores, but one team performed several additional medium difficulty phrases outside of the min phrase count. This slightly rewards teams that perform many difficult combos more than a team that just performs the minimum amount of difficulty combos.

Execution deductions increased

• DESCRIPTION – Raw deduction values are doubled.

• Example: .1 deduction is subtracting .2 points from the team’s final score.

• REASON – The weight of execution in the results at Frisbeer was too small. Teams could get away with several times more execution errors before it became a deciding factor. The overall weight of execution is still less than the current system. This is a partial step back to the current system’s execution impact.

Preliminary Additions/Changes (to be tested in next iteration):

• Adding form back into AI
• Every judge gives a General Impression score (overall weight unchanged)
• Difficulty weight increase
• Primer videos for judges