Benchmark · Legal retrieval

Measuring embedding models with real Supreme Court rulings

Judges already wrote the ground truth. We turned it into a reproducible benchmark for legal retrieval in Spanish — and found a dataset far more valuable than any single leaderboard.

dvsglobal lab · July 2026 · every case links to the official ruling

Honduras' judiciary publishes its Supreme Court rulings online, each at a stable URL. A ruling contains the question and the answer, written by the same person: the magistrate narrates the facts and cites the article of law that applies. That's a retrieval benchmark waiting to happen — with ground truth authored by judges, not LLMs and not annotators.

We built it for our first target, the Honduran Traffic Act:

20,074 published rulings, each with an official URL

~3,000 mention traffic somewhere

23 explicitly cite a Traffic Act article

21 usable cases

The funnel from corpus to benchmark. Bar widths are on a square-root scale — on a linear scale the last two bars would be invisible, which is rather the point of this post.

The method

ruling (median: 33,000 characters)
   │   locate “artículo N de la Ley de Tránsito”
   ▼
window: ±1,200 chars around each citation
   │   · edges snapped to paragraph boundaries
   │   · overlapping windows merged, 4,000-char cap per query
   │   · mask: the law’s name, decree, cited article numbers
   │   · drop: paragraphs that are verbatim copies of the law
   │           (≥30% of their 8-grams appear in the corpus)
   ▼
query ──embed──► 127 articles of the Traffic Act
                    │   cosine similarity, top-10 articles
                    ▼
        did the judge’s article appear? at which rank?

One query per ruling: the passage where the magistrate actually invoked the article, stripped of everything that gives the answer away.

Four decisions matter, and each came from getting it wrong first:

Mask the giveaways — but only the giveaways. A ruling contains its own answer. Left unmasked, the retriever "wins" by string-matching "artículo 104 de la Ley de Tránsito" against article 104 — measuring nothing. References to the Traffic Act are replaced:

…no obstante lo dispuesto en el [CITA OMITIDA], que establece la responsabilidad del conductor…

Citations to other statutes stay visible: the corpus only contains Traffic Act articles, so they can't leak the answer. An unmasked control run showed the gap is smaller than we feared — judges cite far more often than they transcribe. Knowing that required running the control.

Window, don't truncate. Rulings run to 33k characters; a phone-sized embedder reads ~8k before the runtime silently cuts — and every model cuts at a different point, so feeding whole rulings means each model reads a different document. The window guarantees every model reads exactly the same text.

The law has versions. Our 21 rulings span 2013–2025; the Traffic Act was amended six times in that span. We reconstructed every amendment from the original gazettes, stored each article as a chain of dated versions, and by default each case searches the law as it stood on its ruling date. Each version is embedded once; a historical snapshot is just a row mask over the vector matrix.

Recall only, never precision. The judge's citations are confirmed positives. Articles the judge didn't cite are unknowns, not negatives. Every number below is a lower bound, and any precision figure would be fiction.

What the models said

phone-class (≤ 1.1B params) needs a GPU (4B–8B) ★ ships in our products

Mean reciprocal rank across the 21 cases, searched against the law as in force on each ruling's date. 1.0 would mean the cited article always ranks first. Full per-case results, recall@k and the strictest single-article cut are in the repo.

With n=21, one case moves recall ~5 points — so read gently. Still:

Bigger is not necessarily better — not even within one family. Qwen3-Embedding's 0.6B beats its own 4B and 8B siblings here, the reverse of their MTEB ordering. General leaderboards shortlist candidates; they don't pick winners on 127 articles of Honduran traffic law.
The 0.3–1B class forms a knot near the top — and that class runs offline on a cheap phone.
Domain tuning shows: NVIDIA's days-old, legal-tuned Nemotron-3-Embed-8B takes the crown, by about one case.

The real finding: a gold mine

The models aren't the story. The funnel is: we used 21 rulings out of 20,074, because one law is all we've curated so far. The rest cite the Civil Code, the Labor Code, the Criminal Code — all with the same judge-authored structure of facts → applicable law. That's a gold mine of evals and training data in legal Spanish, and we've drilled one hole.

Reproduce it — everything is open

Code github.com/Turupawn/transito-hn-retrieval-eval — one script, ollama + numpy, no API keys. Apache-2.0.

Dataset huggingface.co/datasets/DVSGlobal/transito-hn-retrieval-eval — the 21 cases with their masking audit trail, plus the Traffic Act versioned article-by-article from the original gazettes. CC BY 4.0.

ollama pull embeddinggemma
pip install numpy
python run_eval.py          # as-of by default; --latest for today's text
python run_eval.py --only-single

We don't redistribute ruling texts — every case links to the official ruling on the judiciary's site. Don't trust our labels: click and read what the magistrate wrote.

First post from the dvsglobal lab. We solve problems with AI in the conditions Central America actually has: offline, in Spanish, on cheap hardware. If you work on multilingual embeddings or legal NLP, come compare notes. More soon.