This study tests how often LLMs invent answers when they should rely only on supplied documents.
Rohan Paul Twitter · Rohan Paul (@rohanpaul_ai) · 2026-06-25
A study challenges the assumption that retrieval-augmented LLM systems are inherently safe by measuring how frequently models hallucinate answers even when restricted to supplied source documents.
Extraction
Topics: llm-hallucinationretrieval-augmented-generationllm-reliabilitydocument-grounding
Claims
- LLMs frequently invent answers even when they are explicitly constrained to rely only on supplied documents.
- Companies commonly assume document-grounded LLM systems are safer than open-ended generation, but this assumption is tested and questioned by the study.
- Retrieval-augmented generation does not eliminate hallucination risk as widely believed.
Key quotes
This study tests how often LLMs invent answers when they should rely only on supplied documents.
companies often use LLMs to answer questions from documents and they assume document-based LLM systems are safer because the model is given source material.