A published AI hallucination rate comparison can help you shortlist models for document work, but it cannot tell you how often your own application will be wrong. The useful question is narrower: what did the benchmark ask, what counted as an error, and what should you test before deployment?

📑Table of Contents
  1. A shared starting point for comparing models
  2. What the numbers do—and do not—measure
  3. Different tasks need different measures
  4. Build a useful evaluation before choosing
  5. Frequently asked questions
  6. Author’s take
  7. The next decision

A shared starting point for comparing models

The following is a selection from Vectara’s October 22, 2026 leaderboard snapshot, chosen to show several providers and model generations. It is not a complete ranking or a list of every current frontier model. The hallucination rate is the share of summaries for the filtered common documents judged inconsistent with the source. The answer rate is the share of documents for which the model supplied a summary.

AI hallucination rate comparison: nine selected models range from 6.5% to 12.1% on document summarization, judged by HHEM 2.3
Automated HHEM 2.3 evaluation of document summaries for nine selected models; answer rate is a separate measure. Source: Vectara (updated October 22, 2026)
Model identifier as published Hallucination rate Answer rate Mean summary words
openai/gpt-6-sol 6.5% 100.0% 71.4
openai/gpt-5.4-2026-03-05 7.0% 99.9% 81.7
google/gemini-2.5-pro 7.0% 99.1% 106.4
deepseek-ai/DeepSeek-V4-Pro 8.6% 97.2% 153.8
openai/gpt-6-astra 8.7% 100.0% 148.5
anthropic/claude-opus-5-5 9.9% 99.7% 192.1
google/gemini-3.1-pro-preview 10.4% 99.4% 107.7
anthropic/claude-opus-4-7 12.0% 98.0% 149.1
google/gemini-4-argon 12.1% 99.1% 178.2

Source: Vectara fixed snapshot, checked November 14, 2026.

Do not read this as a league table of overall intelligence. For example, GPT-6 Sol and Claude Opus 5.5 also produced summaries of different average lengths. A short faithful summary can omit information a user needs. The table does not establish that length caused the difference in scores.


What the numbers do—and do not—measure

Vectara says it asks each model to summarize more than 7,700 private articles, then uses its commercial HHEM 2.3 model to judge consistency with the source. This is automated evaluation, not a human review of every answer, and the judge can make mistakes. Temperature is set to zero except where unavailable.

The published FAQ says refusals and one- or two-word outputs are filtered, with final evaluation on documents for which all models supplied a summary. It does not state the final common sample size. Answer rates and average lengths provide important context. The judge currently supports English, and the benchmark does not assess whether a summary includes every important point.

Whether a model admits uncertainty is a separate question. For practical checks on guessing, see why LLMs avoid saying “I don’t know”.

The shared prompt supports a meaningful comparison within this dataset, but configurations and serving endpoints may differ. The table is not an error rate for arbitrary chat questions, a web-enabled assistant, another language, or your own documents.


Different tasks need different measures

Evaluation Task What to check
Vectara leaderboard Summarize a supplied document Grounding in the supplied document, judged automatically
AA-Omniscience v1.1 Answer knowledge questions across six domains Incorrect responses relative to all non-correct responses, plus separate accuracy
FaithJudge Grounded RAG-style summaries, answers and data-to-text Task-specific unsupported claims, judged by an LLM

Sources: Vectara, Artificial Analysis, FaithJudge; checked November 14, 2026.

AA-Omniscience v1.1 uses 3,000 questions. Its hallucination rate is incorrect / (incorrect + partial + not attempted), while its accuracy is correct answers divided by all questions. Abstaining can lower the former, so inspect accuracy and usefulness too. On November 14, the page displayed 4% for both Gemini 4 Carbon (Medium) and Gemini 4 Argon (High); these are rounded values and cannot be compared numerically with Vectara’s metric.

For an older grounded-question example, the 2025-era FaithJudge snapshot reports unsupported content for Gemini 2.5 Pro in 2 of 139 question-answering cases (1.44%) and for GPT-4.1 in 6 of 139 (4.32%). An LLM judge receives human-annotated examples; the 139 cases are just the question-answering subset. These historical results do not rank today’s frontier models or test whether retrieval found the right documents.


Build a useful evaluation before choosing

  1. Define the job. Separate document summaries, answers using provided evidence, and questions answered from model knowledge. Use documents your organization has permission to process; check privacy and data-handling requirements before sending them to an external service.
  2. Freeze the inputs. Run the same questions, source documents, prompts and retrieval settings against each candidate. Record model identifiers, date, configuration and provider. When retrieval is enabled, save the retrieved passages so a bad search result can be distinguished from a bad answer.
  3. Grade more than unsupported claims. Record correct answers, unsupported claims, omissions, partial answers and appropriate abstentions separately. Check whether citations exist and actually support the specific claim.
  4. Review consequential cases. Have people compare important answers with the source rather than relying solely on an automated judge. A small sample gives a noisy comparison; weigh coverage, latency and cost alongside factual consistency.

Frequently asked questions

Is the model with the lowest rate always the safest choice?

No. The result applies to a particular dataset and grading procedure. A model can produce fewer unsupported claims by leaving out information or declining more questions, neither of which automatically meets your needs.


Is 100% minus SimpleQA Verified accuracy the hallucination rate?

No. Epoch AI reports the share correct; the remainder includes both incorrect answers and answers not attempted. Its August 2026 change also added a prompt encouraging guesses, so compare results only after checking the procedure.


Does a citation guarantee an answer is grounded?

No. Open the cited source and check that it supports the precise statement. A real source can be irrelevant, outdated, or incorrectly summarized.


Author’s take

The author believes that even highly advanced AI can still hallucinate at a rate close to 10%, so care is needed. This is a personal view, not a measured rate for every task.


The next decision

Use a leaderboard to select candidates, then run a small, controlled evaluation on representative material. Keep unsupported claims, missing information and abstentions separate; that record is more useful than a single universal-looking percentage.

Related: AI model comparison and Claude Opus 4.7 capabilities and migration considerations.

Author

Over 20 years in the IT industry, serving as Division Head and CTO at multiple companies running large-scale web services in Japan. Experienced across Windows, iOS, Android, and web development. Currently focused on AI-native transformation. At DevGENT, sharing practical guides on AI code editors, automation tools, and LLMs in three languages.

DevGENT about →

Leave a Reply

Trending