No trade-off.
- 23×cheaper than the typical LLM-as-a-judge195× vs Opus 4.7
- +18%more accurate than the average LLM-as-a-judge
- 19×faster~200 ms a call
- σ = 0deterministicsame input, same score
The fabric architecture
A library of trained metrics.
Each one a single criterion, learned — the RAG, quality, and safety checks you’d otherwise hand to an LLM judge. Open any lens to see it score, or bring your own content alongside.
Whether every claim in an answer holds up against its retrieved context — the RAG hallucination check.
The agreeable reflex RLHF trains in — flattery and opinion-matching over telling you the truth.
Information density over word count — the preambles, hedging, and filler that pad model output.
Concrete detail over generic AI prose — proper nouns, numbers, named particulars, not filler.
Whether a line actually lands the joke — surprise, economy, specificity — not just attempts one.
Whether an answer addresses the question that was actually asked — on-topic, not just fluent.
qed-bench · four tasks, one pattern
Measured against the strongest baseline for each task.
Every metric is compared to its task-appropriate baseline — trained human raters, gold labels, or an eight-model LLM-as-judge panel. It matches the best of them, at a fraction of the cost.
Seriously — talk to the MLE.
Name’s Taras. He built this, and he’s interested in scaling your evals. No sign-up, no sales call — just the person who wrote the code, one message away.
FAQ
How do you evaluate without an LLM?
DLM (Discriminative Language Model) runs on an encoder-only architecture and drops a generative layer. Put simply, reads meaning straight from text and scores it in one pass.
Isn’t this just embedding similarity?
No. Cosine similarity tells you how close two texts are, not whether one is good. It has no notion of the criterion you’re scoring, like faithfulness, relevance, tone. A DLM learns that criterion.
When should I still use an LLM?
When you need the judge to argue — a written rationale, a paragraph defending the verdict. U/=22A8 gives you a score and its confidence, not prose; for that explanation, an LLM is still the right tool.
Huh?
Fair — it’s a strange idea the first time you meet it. The MLE is notoriously happy to explain it in more detail than you probably want. Talk to the MLE and ask anything.