LLM Hallucination Detection via Probability Geometry

Central Project Idea

I proposed and led this group project to investigate whether the shape of a language model's next-token probabilities could help predict incorrect responses. The goal was to look for a measurable signal in the model's own probability patterns in order to be able to predict a wrong answer prior to seeing the response itself. The central hypothesis was that when a model was uncertain it would exhibit a higher-entropy token distribution without a clear (high-probability) stream of tokens. We ran Qwen2.5 on two separate types of questions (math and logic), recording the top 1,000 token probabilities at each generation step, and conducted an analysis to find which features contained predictive power in assessing model correctness.

What we found

Our results partially supported the hypothesis. Using only the model’s probability patterns we could distinguish correct from incorrect answers better than chance across both mathematical and commonsense reasoning tasks (0.714 and 0.631 AUROC for the best models on each type of question). This suggests that uncertainty leaves a measurable signature in a model’s internal probability distribution.

However, the learned patterns did not transfer reliably between different types of questions. Probability geometry therefore appears to provide a useful, task-dependent warning signal, but our results did not establish a universal hallucination detector. More broadly, the project showed that a model’s probability distributions can reveal meaningful information about the likely quality of its answer before the final response is evaluated.