SOPHIE MANN FROM DAILYMAIL
Not as smart as we thought: ChatGPT averaged a C+ when University of Minnesota law professors used it to generate answers in four law-school exams – while humans averaged a B+
- The popular and powerful AI robot was administered law school tests by instructors at the University of Minnesota
- The bot passed each exam, albeit without flying colors, and averaged a C+ across four subject area tests
- The authors of the paper say the bot has a long way to go before it is a convincing law student, but its potential utility should not be ignored
A group of University of Minnesota Law School instructors administered popular artificial intelligence bot ChatGPT four law school exams that it took alongside real students, and discovered that the bot was a C+ student on paper.
The ChatGPT bot is currently considered the world’s most advanced example of generative intelligence.
The instructors were curious about ways the bot might be used both to help students cheats and professors teach, so they decided to test its abilities.
Jon Choi, an associate professor of law, ran exam questions through ChatGPT and mixed the resultant exams in with students’ tests.
The bot ended up with a ‘low but passing grade’ in Constitutional Law, Employee Benefits, Taxation, and Torts. All told, it wound up with a C+ average.
According to the paper, the bot came in 36th out of 40 students in Con Law, 18th out of 19 in Employee Benefits, 66th out of 67 in Tax, and dead-last out of 75 in Torts.
Ultimately, the bot performed better on essays than on multiple choice questions, and performed particularly poorly at multiple choice questions involving math, which the paper’s authors say it ‘bombed.’
The paper’s authors explained that in essay questions, the bot was able to adequately recite legal rules and correctly describe cases (without proper citations), but was minimally skilled at ‘spotting issues’ and providing in-depth analysis.
The three human graders, who are listed as the co-authors of the paper, graded the tests blind but said they had suspicions about which were AI generated exams.
In all cases, they were correct about which ones were not written by humans.
