ModelRefs / AGIEval Methodology and Evidence
AGIEval Methodology and Evidence
Human-centric standardized exams (SAT, GRE, GMAT, LSAT, civil service). Methodology, dataset provenance, and citation sources for AGIEval.
Overview
Human-centric standardized exams (SAT, GRE, GMAT, LSAT, civil service).
How it is measured: Zero/few-shot accuracy across English & Chinese exams.
What this benchmark measures
- academic and professional exam reasoning
- bilingual knowledge application
Relevant to:
- broad reasoning evaluation portfolios
- English and Chinese exam-task screening
Failure modes it exercises:
- knowledge and reasoning errors across standardized exam formats
Method and its limits
Zero- and few-shot evaluation over public admission, qualification, law, civil-service, and math-competition exams in English and Chinese.
- Aggregate accuracy can hide large differences by language, exam, prompt, and answer format.
- Public exam content creates model-specific training-overlap and contamination uncertainty.
Dataset
- Dataset
- AGIEval
- Type
- human standardized admission and qualification exam questions
- Freshness
- aging
- Benchmark release
- 2023-04-13
The paper describes 20 public exams spanning English and Chinese tasks; it is not a current-world or workplace-performance dataset.
How to read this score
Use matched prompts and exam-level breakdowns; do not treat one aggregate score as general intelligence or professional qualification.
Similarity to real tasks: Low — Tasks originate in human exams, but static questions and exact grading do not reproduce most open-ended production work.
Data contamination risk: High — The source material is public and no model-specific training-overlap audit is registered. ModelRefs cannot rule out training overlap for any specific model run, so contamination exposure should be treated as high unless a score source documents decontamination.
Benchmark gaming risk: Medium — No benchmark-specific optimization or prompt-tuning audit is registered.
What you still need to test yourself
- Test open-ended, current, domain-specific tasks with the target language and output format.
- Evaluate calibration, safety, cost, latency, and model-specific training overlap separately.
This benchmark supports decisions about:
- Compare broad exam-style reasoning under the same AGIEval task set and prompt protocol.
- Identify languages or exam domains requiring targeted evaluation.
Limitations
- Exam accuracy does not prove professional competence, tool use, safety, or production reliability.
- Cross-report scores may use different subsets, prompts, languages, and answer extraction.
Dataset freshness is not fully confirmed from available source coverage. The paper date is not treated as a knowledge cutoff.
What this evidence does not cover
- No authoritative dataset knowledge cutoff or current maintenance snapshot is registered.
- No model-specific contamination analysis or universal prompt protocol is established.
Sources
- AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models AGIEval authors / Findings of NAACL 2024 · accessed 2026-06-29Supports benchmark composition, evaluation protocols, and limitations; it does not establish production-task validity.
- AGIEval code, data, and model outputs AGIEval authors · accessed 2026-06-29Official implementation and dataset reference; exact revision, subset, and prompt settings remain necessary for comparison.
How this benchmark is scored
| Category | reasoning |
|---|---|
| Maximum score | 100 % accuracy |
| Direction | Higher is better |
Primary source: https://arxiv.org/abs/2304.06364
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to AGIEval Methodology and Evidence.
Frequently asked questions
What does the AGIEval benchmark measure?
Human-centric standardized exams (SAT, GRE, GMAT, LSAT, civil service).
What is the methodology?
Zero/few-shot accuracy across English & Chinese exams.