TruthfulQA MC2 Methodology and Evidence

What this benchmark measures

• resistance to common misconceptions
• truthfulness under misleading premises

Relevant to:

• misinformation risk screening
• assistant safety evaluation

Failure modes it exercises:

• repeating widely held falsehoods learned from training text

Method and its limits

817 questions across 38 categories, written to elicit imitative falsehoods that a model may have learned from human text rather than from reasoning failure.

• Questions deliberately target misconceptions, so the score measures a specific failure mode rather than general factual accuracy.
• Answers reflect the authors' truth judgements at publication; some contested or time-sensitive items may have moved since 2021.

Dataset

Dataset

TruthfulQA

Type

adversarially written misconception questions

Freshness

aging

Benchmark release

2021-09-08

817 questions in 38 categories as registered in the primary paper. The date records paper submission.

How to read this score

The paper's central finding is inverse scaling: the best model was truthful on 58% of questions against 94% for humans, and the largest models were generally the least truthful. Truthfulness must therefore be read alongside informativeness, since silence scores as truthful.

Similarity to real tasks: Medium — Established from the primary methodology: TruthfulQA comprises 817 questions across 38 categories including health, law, finance and politics, written so that some humans would answer falsely from a false belief or misconception. Users do ask questions carrying false premises, so the format resembles a real failure mode. It is short-form and ungrounded, so it does not test retrieval, citation, or correction behaviour.

Data contamination risk: High — Public since 2021 and widely used in alignment tuning, so both the questions and their reference answers are likely represented in training corpora. No run-specific audit is registered.

Benchmark gaming risk: Medium — Scores depend on whether truthfulness is judged by a trained classifier, a human, or multiple-choice scoring, and a model can raise truthfulness simply by refusing more often.

What you still need to test yourself

• Test with retrieval and citation, which the benchmark deliberately excludes.
• Re-check time-sensitive items whose correct answer may have changed since 2021.

This benchmark supports decisions about:

• Screen a model's tendency to repeat popular misconceptions.
• Compare truthfulness against informativeness rather than in isolation.

Limitations

• Measures a specific misconception-driven failure mode, not general factuality.
• A model can score well by declining to answer, so the figure is uninformative without informativeness alongside it.

Primary source reviewed 2026-10-12. TruthfulQA is treated as an aging 2021 benchmark.

What this evidence does not cover

• No model scores are registered on this page, so it supports no comparison.

Sources

• TruthfulQA: Measuring How Models Mimic Human Falsehoods Lin, Hilton, Evans · accessed 2026-10-12Supports the 817-question, 38-category dataset composition, the imitative-falsehood design, and the stated 58% best-model versus 94% human truthfulness and inverse-scaling findings; it does not support any model score, and no score is registered on this page.
• TruthfulQA dataset and evaluation release Lin et al. (TruthfulQA authors) · accessed 2026-10-25The authors' release, matching the paper's title. It establishes the question set and the reference evaluation procedure; it does not establish any model's truthfulness score.

How this benchmark is scored

Primary source: https://arxiv.org/abs/2109.07958

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to TruthfulQA MC2 Methodology and Evidence.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path