GPT-4 Can Make Court Rulings Easier to Read. It Can Also Lie to You About Them, Confidently.
TL;DR Italian Constitutional Court rulings are written for lawyers. Most citizens can’t follow them. Can an LLM fix that? We ran a 75-person human study comparing four versions of the same legal content: original judgments, expert “massime” summaries, GPT-4o summaries, and a fine-tuned LLaMA 2 7B. Comprehension rates: expert summaries 45%, GPT-4o 38%, raw judgments 33%, fine-tuned LLaMA 30%. GPT-4o really does make legal text more readable. It also produces a worrying pattern: confident, fluent, wrong — readers leave with strongly held but incorrect understandings. Use LLMs for legal summarization. Don’t use them without human review. A democratic problem dressed up as a technical one Constitutional Court rulings are some of the most important documents a country produces. They define what your government can and can’t do. They shape what your rights are. They are also written by lawyers, for lawyers, in dense Italian legal prose that the average citizen has no chance of understanding. ...