It’s just a short little blurb, but report is that starting this year the MCATs are going to be graded automatically using an artificial intelligence system, rather than human graders. The MCAT, like the GRE, moved to a computerized format recently to allow more frequent administration. The essay section, though, was graded in much the same way with multiple readers scoring the writing samples and a degree of consensus being required for the final score. The claim is that the AI system is more consistent than human scorers. I’ve certainly read that human scoring can vary widely. The AI system, IntelliMetric has been used in a variety of settings so far, including a range of domains. I’ve not read through the papers on their site in detail, but it seems to be a basic learning algorithm that works from a corpus of pre-graded writing samples to match essays to be graded to various scoring catagories. I’ll definitely be reading more about this, both for reference for my own research and possibly for inclusion in my spring class. I want to look into what types of errors they are getting compared to the mistakes made by expert raters, if they have that data available….
Glimpses of the websurfing habits of a computer science professor, obsessive crafter, occasional gamer, and all around geek
I’d welcome the opportunity to explain a bit more about IntelliMetric. It’s a very sophisticated tool-set that has over 12 years in research. It is also the engine that is used for the GMAT as well as the Official SAT Online Course offered by the College Board. On top of the engine sits an instructional layer that is called MY Access!. MY Access! is used by over 1 million K-12 students and some college freshman.