Embedding trust: a new model for detecting LLM hallucinations
A geometric signal — semantic isotropy — flags unreliable long-form output with no labels, no fine-tuning, and no access to model internals.
In medicine, law, policy, and science, organizations are beginning to rely on LLMs to summarize documents, answer complex questions, and support decisions. The problem: LLMs hallucinate, and most detection methods check factual claims one by one — decompose the response into atomic statements, retrieve evidence for each, aggregate. That costs many LLM calls per response, compounds errors on long-form text, and breaks down at scale.
Vijil Chief Scientist Tim G. J. Rudner, with Dhrupad Bhardwaj and Julia Kempe, published a different approach at ICML 2026: Embedding Trust: Semantic Isotropy Predicts Nonfactuality in Long-Form Text Generation. Instead of evaluating the text, evaluate the geometry of its embeddings.
The method samples a few long-form responses to the same prompt — as few as two or three — embeds them with an open-weight embedding model, and scores the angular dispersion of the embeddings on the unit sphere. Across datasets, generator models, and embedding models, greater dispersion correlates strongly with hallucination and factual inconsistency.
Model-agnostic, label-free, hyperparameter-free — and under two seconds on an A100.
Because it needs only output embeddings, the technique applies to closed models whose internals are inaccessible. And because it needs no golden dataset, it suits high-stakes domains where users need a trust signal on this specific response, not an average over a benchmark.
The broader shift matters as much as the technique: rather than treating hallucination detection purely as fact-checking, an LLM’s uncertainty can be inferred from its own behavior. Vijil already evaluates reliability with dedicated small language models beyond LLM-as-judge, and semantic isotropy is a candidate for a formal evaluation technique in Diamond.
