Due diligence reveals undue intelligence as federal judge withdraws ruling due to AI-like errors

A New Jersey federal judge has withdrawn his decision in a pharmaceutical securities case after lawyers identified fabricated quotes and false case citations in his ruling — errors that mirror the hallucination patterns commonly seen in AI-generated legal content. The withdrawal highlights growing concerns about artificial intelligence’s reliability in legal research, as attorneys increasingly turn to tools like ChatGPT despite their tendency to generate convincing but inaccurate information.

What happened: Judge Julien Xavier Neals pulled his decision denying CorMedix’s lawsuit dismissal request after attorney Andrew Lichtman identified a “series of errors” in the ruling.

• The opinion contained misstated outcomes from three other legal cases and “numerous instances” of fabricated quotes falsely attributed to judicial decisions.
• A court notice published Wednesday stated “that opinion and order were entered in error” and promised a corrected version would follow.
• While minor post-ruling corrections are common, major modifications like withdrawing entire decisions are rare in federal courts.

The AI connection: Though no official confirmation exists that AI was used, the citation errors display classic signs of AI hallucinations that have plagued other legal cases.

• Attorneys defending MyPillow founder Mike Lindell were fined earlier this month for submitting AI-generated citations in court filings.
• Anthropic’s own legal team blamed their Claude AI chatbot for making an “embarrassing” erroneous citation in the company’s battle with music publishers.
• These incidents represent just a fraction of cases where large language models have generated convincing but entirely fictional legal references.

Why this matters: The pattern of AI-generated legal errors demonstrates that current language models remain unreliable for professional legal research, despite their sophisticated output.

• The fabricated quotes and case outcomes in Judge Neals’ decision mirror the exact types of hallucinations that AI tools are known to produce when asked to cite legal precedents.
• As lawyers increasingly integrate AI tools into their research workflows, the legal system faces mounting challenges in distinguishing between legitimate citations and AI-generated fiction.
• The incident underscores that while AI can assist with legal work, it won’t be replacing human lawyers anytime soon — particularly given the profession’s reliance on accurate case law and precedent.