You've heard of AI models taking the bar exam, but this time, AI also helped write the questions.
The State Bar of California revealed on Monday that it used AI to develop a portion of its exam questions, according to the LA Times. The AI-generated exam questions were created by an independent psychometrician called ACS Ventures hired by the State Bar. The questions were "developed with the assistance of AI and subsequently reviewed by content validation panels and a subject matter expert in advance of the exam," announced the State Bar in a statement addressing technical glitches and question errors that test takers had previously complained about.
The LA Times reported that 23 out of the 171 multiple choice questions were made by ACS Ventures. The majority of the multiple choice questions were developed by Kaplan, and a "small subset" were taken from the First-Year Law Students' Exam. This past year, the bar was offered remotely to California-based test takers. Students and educators alike were already outraged about the remote test platform crashing and being riddled with bugs. But now, the discovery that some of the exam questions were created with AI has further fueled that outrage.
"I'm almost speechless. Having the questions drafted by non-lawyers using artificial intelligence is just unbelievable," Mary Basick, assistant dean of academic skills at UC Irvine School of Law told the Times. "It's a staggering admission," Katie Moran, an associate professor at the University of San Francisco School of Law told the outlet.
Moran also pointed out the fact that ACS Ventures, the firm used to craft the AI questions, was the same firm to approve the questions.
Alex Chan, who chairs the State Bar's Committee of Bar Examiners, told the outlet that the California Supreme Court had pressured the State Bar to look into "new technologies, such as artificial intelligence" as a means of improving reliability or cost-efficiency.
Automating tasks with AI has surged since the rise of generative AI — and not just simple tasks or low-stakes work, but critical work that has very real consequences. Some suspect that the formula used to calculate the Trump Administration's tariff rates was created by ChatGPT or something similar. In 2023, two New York lawyers were sanctioned for using ChatGPT in a legal brief, which cited fake cases. And academic journals are flooded with papers that include AI-generated text.
And those are just a few examples of the ones that got caught. Generative AI's ability to rapidly write, summarize, and source information has been an irresistible way for workers to save time and effort. But it has innate hallucination problems and poses ethical issues by outsourcing work to a bot — especially when it comes to law students whose entire career rests on passing the bar.
