Collaborative Referee Test
May 4th, 2009Now open for business is a simple Collaborative Referee Test.
It works like this: it displays a random question. You pick and submit your answer, and it adds your answer in to the answer pool. Then, on the next page, the previous question is re-displayed with a count next to each answer showing how many people have picked that answer as the correct answer.
The test works like this because there is no official (public?) answer key to the referee test. So, there’s no “right” answer publicly available. The collaborative nature of the test is to try and point people in the right direction, as well as show which answers are most confusing.
The test itself is currently not designed to track which questions have been taken already. I’m definitely willing to take questions/comments/etc. and will improve the test based on input I receive. I think the top two requests so far are “right answers”, which may or may not be possible/reliable, and tracking for questions to ensure all are taken with minimal repeats.
May 27th, 2009 at 2:38 pm
Thank you ever so much for creating the biggest time-sink I’ve stumbled upon in eons. I submit an answer, tell myself I’m done, can’t help but do another… and so on and so on. A little frustrating to see the same questions several times in a row (but hey, it’s random). Also frustrating to not see an officially-accepted answer, but as I found when I took the exams a year ago, a lot of the answers can be successfully argued. (The exam writers are hopelessly vague at times.) It’s probably a good thing you don’t allow comments on specific questions or you’d see a full-scale war break out.
June 16th, 2009 at 4:32 am
yeah, i’m definitely going to improve the system in the future. will implement a process so that repeat questions are less frequent. i probably won’t guarantee a full run through the guide, but possibly a length of maybe 25 unique questions or so.
its also pretty interesting to see the problem questions — questions who have a split decision, or the majority of people pick the wrong answer. really highlights the problem areas with respect to reffing. if everyone always got them right, reffing would be easy.