Date: Tue, 15 Nov 1994 11:32:21 -1000 From: sasaki@netopd.harvard.edu (Marty Sasaki) Message-Id: <3ab9d5$ggn@necco.harvard.edu> Organization: Harvard OIT Network Services Subject: Scoring Systems Currently, the AKA, and other folks like STACK, use a scoring system that go like this: - The score for a performance is divided into a number of parts o As an example, ballet has two portions that correspond to the choreography and the execution of a routine. o Each of the above portions has a weight, i.e.choreography is worth 60, execution is workth 40 I'm not sure about the actual numbers, but you get the idea. - A panel of judges (usually 3 or 5) scores a performance by the different parts. Each part is scored from 0 to 10 points in 0.1 point increments. - If 5 or more judges are on the panel, the highest and the lowest scores for each part are thrown out and the resulting scores are averaged. - The weighting factor is applied to each part, and the sum of each weighted part is the score for the performance. - the score is used to determine how well someone does in a competition. The person with the highest score is first, the next highest score is second, etc. A number of sports use this system or something very similar to it. It has a number of advantages including simplicity, and ease of implementation. Many events use computers to calculate the scores, but it can easily be done using a calculator, abacus, or someone who is good at arithmetic. This scoring system has two major disadvantages: - In order for it to be fair and effective, judges must give similar scores. The judging staff must "calibrate" their scores against each other. - Scores get thrown out. An ordinal system has been advocated by several folks. An ordinal system would go something like this: - The score for a performance is divided into a number of parts. o Use the sub parts from above. - A panel of judges scores a performance by the different parts. Each part is scored from 0 to 10 points in 0.1 point increments. - The weighting factor is applied to each part and the sum of the weighted part is the score for the performance. This calculation is done for each competitor and for each judge. - For each judge, the scores are ordered from highest to lowest. The competitor with the highest score receives the ordinal value 1, the  second highest score gets an ordinal of 2, etc. - The competitor with the most number of 1's is the winner. The competitor with the most number of 2's is second, etc. There are all sorts of variations on ordinals (each part is ordinaled separately, weighting factors done before ordinaling on the raw scores, weighting factors applied to ordinals, etc), but I've presented this one since it is the simplist. There are also different ways of counting the number of 1's, 2's, etc, as well as how to break ties. The advantages of ordinals are that all scores are used, no score is thrown out, and there is no need to calibrate scores. The primary disadvantage is that calculating the finish can get complicated, especially if it is being done by hand. Ice skating and roller skating use ordinal systems. Diving, freestyle skiing, and gynmastics use a system similar to what stunt kites currently use. I personally favor the ordinal system since calibration of judging on panels is difficult, especially with international panels, or with judges from parts of the US. What do you think? -- Marty Sasaki Harvard University Sasaki Kite Fabrications sasaki@noc.harvard.edu Network Services Division 26 Green Street 617-496-4320 10 Ware Street Jamaica Plain, MA 02130 Cambridge, MA 02138-4002 phone/fax: 617-522-8546 = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = Date: Tue, 15 Nov 1994 19:07:00 -1000 From: SWAGNER@LAW.FORDHAM.EDU (DECA BOY) Message-Id: <941116000700.2020f03c@LAW.FORDHAM.EDU> Organization: Division of Applied Sciences, Harvard University Subject: Scoring systems As a competitor, the idea of using ordinals at individual events wouldn't make t too much difference, except that the top score for my performance, as well as everyone else's would still count. As a judge, knowing that my scores would not be thrown out would be comforting. Also knowing that if some judge was more generous than I was, as long as we were both sor of consistant, our scores would be more compatable than the current system. You also mentioned the judging seminars across the Easten League this past year. I have attended a good number of them and have found them invaluable as a judge. Attendance at most of these was good and I do believe that the judging on the East Coast has improved as a direct result. There has been at least one seminar at every Eastern League event and both competitors, judges as well as those interested in judging in the future were welcome to attend. Special thanx to all who have helped me in judging. Sue Edison, Roger Chewning, Dave Arnold, Dodd Gross, and many other experienced flyers and judges have done a wonderful job to improve the judging. I hope it keeps up. These seminars have also been helpful to me as a competitor and I recommend them to all if these seminars continue next season. Sandy "Deca Boy" Wagner = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = Date: Wed, 16 Nov 1994 04:16:50 -1000 From: Mr.Nasty@ix.netcom.com (Frank Kenisky) Message-Id: <3ad48i$1at@ixnews1.ix.netcom.com> Organization: Netcom Subject: Re: Scoring Systems In <3ab9d5$ggn@necco.harvard.edu> sasaki@netopd.harvard.edu (Marty Sasaki) writes: >I personally favor the ordinal system since calibration of judging on >panels is difficult, especially with international panels, or with >judges from parts of the US. > >What do you think? I think that since you brought it up the AKA will probably go ahead and adopt it. Personally, I feel it is possible, but because all events are not equal it would a bit difficult, like you mention to calculate the scores timely and accuractly if you were not using some sort of data processing device. The method we now use seems suited to our environment for several reasons of which have NO logic. 1) This is the system we have been using in the past and although not the best, it has worked. 2) Making this sort of change needs not only to be discussed it needs to be tested. The results need to be provided to everyone and then discuss or debated. (Well maybe this is logical.) Mr.Nasty = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = Date: Wed, 16 Nov 1994 10:38:07 -1000 From: travell@comics.enet.dec.com (I want to Go Fly a Kite!) Message-Id: <9411162038.AA28596@vbormc.vbo.dec.com> Organization: Division of Applied Sciences, Harvard University Subject: RE: Scoring Systems Marty wrote : > - For each judge, the scores are ordered from highest to lowest. The > competitor with the highest score receives the ordinal value 1, the > second highest score gets an ordinal of 2, etc. > - The competitor with the most number of 1's is the winner. The > competitor with the most number of 2's is second, etc. I suggest simply adding all ordinal scores for each user, the lowest wins, and use the ordinal distribution to break ties. i.e. compare lowest ordinal, then next lowest, etc. if there is still an exact match, for example both competitors have 1,1,2,2,3 (maybe 1,3,1,2,2 & 2,1,2,3,1) then use the scores >From the judge that gave scores closest to the otherwise finished results. By this I mean count the tied competitors as tied, calculate the overall order, then compare to the order that each judge would have given alone. I expect that this would also help to improve judging, there could be a prize for the judge who got the order most nearly right. John Travell. = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = Date: Tue, 22 Nov 1994 10:15:18 -1000 From: d1ckerson@aol.com (D1CKERSON) Message-Id: <3atjgm$pne@newsbf01.news.aol.com> Organization: America Online, Inc. (1-800-827-6364) Subject: Re: Scoring Systems In article <3ab9d5$ggn@necco.harvard.edu>, sasaki@netopd.harvard.edu (Marty Sasaki) writes: I personally favor the ordinal system since calibration of judging on panels is difficult, especially with international panels, or with judges >From parts of the US. I must disagree with Marty. Ordinal systems are subject to the same kinds of "surprizes" that preferential voting systems are. Most voting countries have abandoned preferential voting. In this kind of a system, someone who nobody voted to win come in first place. Imagine I am an evil judge and there are only three competitors and only three judges. The competitors are Marty Sasaki, Dodd Gross and Dave Arnold. The judges are myself, Jenny Gross and Sherri Arnold. If Sherri scores the performances as Dave, Dodd, Marty and Jenny scores the performances as Dodd, Dave, Marty and I would have scored them as Marty, Dodd, Dave but I suspect that the other judges are placing their husbands first, then I could vote for Dodd to come in first in order to avoid a tie and keep Dave out of first place. Judges aren't necessarily evil and any voting system can be abused but preferential voting lends itself to such abuses much more readily than many other systems. Books on mathematical puzzles are full of problems involving preferential voting for this very reason. Before the AKA takes a recommendation seriously to make a change, I would hope that they spend some time investigating the system that certain sports and certain countries use in their preferential voting systems and try to consider how this would impact our events when we only use 5 judges instead of a dozen or more like these other sports. David Dickerson = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = Date: Sat, 26 Nov 1994 05:45:20 -1000 From: d1ckerson@aol.com (D1CKERSON) Message-Id: <3b7l6g$mvn@newsbf01.news.aol.com> Organization: America Online, Inc. (1-800-827-6364) Subject: Re: Scoring Systems In article <3ab9d5$ggn@necco.harvard.edu>, sasaki@netopd.harvard.edu (Marty Sasaki) writes: "I personally favor the ordinal system since calibration of judging on panels is difficult..." I responded to this already but there is more to what I have to say and to top it all off, I don't think my response made it past my own telephone. Ordinal scoring systems are similar to the preferential voting systems that have been abandoned by most voting countries as well as most counties, cities and states here in the USA. It would work best if you were only trying to pick first place but when you try to order eight performances by quality, it breaks down quickly. By "break down," I mean that there could be some unfortunate surprises. Before we abandon our current system, we should look at other sports that use ordinal systems and try to see in advance how their systems would effect us when we use less than half the number of judges that they use and when we use amature judges (peers really). Secondly, I feel that it is in our best interest as competitors to remove the top and bottom scores of our performances. CASE IN POINT: John Beressi (I hope I spelled it correctly) came in first place at AKA in Open Individual Quadline Ballet. His performance was stunning. As always, his execution was near perfect and his correography was first rate. He used Bugs Bunny's "Barber of Seville" opera as his music. 4 of the 5 judges scored him 9's all around. There was one judge (nameless) who scored him and 7's claiming that his routine was not ballet at all but just a bunch of manuvers set to sound effects. OK, I can kind of see his point. But if this judges score was not thrown out, John would not be the national champion right now. Suppose this judge did this to settle a personal score. We can see how the whims of a single judge can dramatically affect the outcome of an event. As a judge, knowing that my scores will be deleted from the sample if they are outrageous would be a comfort to me. How many times have I seen a performance that scored way above or way below than what I would have given it! Until we increase the numbers of our judging staffs AND PAY THEM, I would resist going to an ordinal system. Dave Dickerson = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =