A lawyer caught using AI citations and quotes in a Supreme Court legal case has been called out by a judge for defending themselves with, err, AI citations and quotes

Dall E impression of what a robot lawyer would look like in court. — (Image credit: Dall E)

For fans of real-life legal drama, I've got a doozy for you. An attorney in a New York Supreme Court legal case was accused by a plaintiff and their lawyers of providing "inaccurate citations and quotations" that appeared to be "hallucinated" by an AI tool.

When said lawyer opposed the claims, they were then accused of defending their use of AI with, well, AI (via 404 Media). Or, to put it more succinctly, in the words of judge Joel Coehn:

"Counsel relied upon unvetted AI—in his telling, via inadequately supervised colleagues—to defend his use of unvetted AI."

The case itself involved a dispute between family members and a defaulted loan, usually a relatively straightforward proceeding. However, once the plaintiff brought to the court's attention the apparently AI-generated citations and quotes in the defense counsel's brief, the defense claimed that "these passages were intended as paraphrases or summarized statements of the legal principles established in the cited authorities."

Phoenix Wright holds his head in his hands in the courtroom, clearly in distress. — (Image credit: Capcom)

After being called out on the potential use of AI by the plaintiff, the defense then provided opposition to the claim, which was also later deemed to be made with the help of AI. According to the judge:

"This time, [the] plaintiff has identified more than double the number of mis-cites, including four citations that do not exist, seven quotations that do not exist in the cited cases, and three that do not support the propositions for which they are offered."

The judge called the case "yet another unfortunate chapter to the story of artificial intelligence misuse in the legal profession," and stated that, while some of the citations happened to contain "arguably correct statements of law", it made them "no less frivolous."

Well, that must have made for an awkward courtroom. Still, as the judge references, this is far from the first time the use of AI has reared its ugly head in legal proceedings. Earlier this year, a judge ordered a ChatGPT-using lawyer to not only pay $5,500 in sanctions, but to attend a plenary session entitled "Smarter Than Ever: The Potential and Perils of Artificial Intelligence" for their transgressions.

It appears that courts look unfavourably upon AI avatars for personal representation, too. Back in April, a plaintiff in an employment dispute attempted to use just such an AI stand-in to deliver their opening presentation, as they felt their tendency to mumble would negatively affect their case, only for the artificial representative to be shut down by the judge before it could finish its first sentence.

Judge gavel, scales of justice and law books in court. — (Image credit: Getty Images / Brian A. Jackson/South Florida Sun Sentinel)

All of this comes as no surprise, I reckon. The legal profession is a complicated business, and given that AI tools are often touted for their ability to scythe through dense information to pull out the relevant facts, it's no great wonder that some are taking AI-based shortcuts in legal proceedings.

And so far, they seem to usually come unstuck. However, part of me wonders how much AI-generated content has already made its way into legal cases without being noticed. In this case, the plaintiff and their counsel appear to have been on the ball, and as a result the judge was able to call out the discrepancy.

My favourite law, though, the law of averages, suggests that at least some AI-generated arguments, citations, and quotations might have made it through the cracks. Pessimistic? Me? Never. I'm just a happy-go-lucky bundle of joy.

Best gaming rigs 2025

1. Best gaming laptop: Razer Blade 16

2. Best gaming PC: HP Omen 35L

3. Best handheld gaming PC: Lenovo Legion Go S SteamOS ed.

4. Best mini PC: Minisforum AtomMan G7 PT

5. Best VR headset: Meta Quest 3

👉Check out our list of guides👈

Andy built his first gaming PC at the tender age of 12, when IDE cables were a thing and high resolution wasn't—and he hasn't stopped since. Now working as a hardware writer for PC Gamer, Andy spends his time jumping around the world attending product launches and trade shows, all the while reviewing every bit of PC gaming hardware he can get his hands on. You name it, if it's interesting hardware he'll write words about it, with opinions and everything.

You must confirm your public display name before commenting

Please logout and then login again, you will then be prompted to enter your display name.