Studie von Forscher:innen der University of Minnesota, 25.1.2023 (engl. Original)
How well can AI models write law school exams without human assistance? To find out, we used the widely publicized AI model ChatGPT to generate answers on four real exams at the University of Minnesota Law School. We then blindly graded these exams as part of our regular grading processes for each class. Over 95 multiple choice questions and 12 essay questions, ChatGPT performed on average at the level of a C+ student, achieving a low but passing grade in all four courses.
Overall, ChatGPT’s performance on law school exams, while currently uneven at best, suggests considerable promise and peril. We expect such language models to be important tools for practicing lawyers going forward; we also expect them to be very helpful to students using them (licitly or illicitly) on law school exams. Although ChatGPT would have been a mediocre law student, its performance was sufficient to successfully earn a JD degree from a highly selective law school, assuming its work remained constant throughout law school (and ignoring other graduation requirements that involve different skills). In an era where remote exam administration has become the norm, this could hypothetically result in a struggling law student using ChatGPT to earn a JD that does not reflect her abilities or readiness to practice law.
Die vollständige Studie auf Englisch finden Sie hier.