JailbreakBench Methodology and Evidence

What this benchmark measures

• jailbreak resistance
• refusal robustness
• overrefusal

Relevant to:

• safety evaluation
• red-team planning

Failure modes it exercises:

• complying with misuse requests under adversarial prompting

Method and its limits

The JBB-Behaviors dataset of 100 misuse behaviours across ten policy-aligned categories, about 55% original and the remainder drawn from AdvBench and TDC/HarmBench, plus 100 benign behaviours for overrefusal, evaluated through a standardised library with required jailbreak artifacts for reproducibility.

• The official page does not describe how harmfulness is judged, and states that open-ended harmfulness evaluation is not yet standardised, so the scoring method is not established from it.
• Roughly 45% of behaviours are inherited from earlier benchmarks, so results partly reflect those sets rather than independent construction.

Dataset

Dataset

JBB-Behaviors

Type

misuse and benign behaviour prompts for jailbreak evaluation

Freshness

aging

Benchmark release

2024-03-28

100 misuse behaviours across ten categories aligned to published usage policies, with about 55% original and the rest from AdvBench and TDC/HarmBench, plus 100 benign behaviours. The page states no release date; the freshness date recorded here is the primary paper's publication date.

How to read this score

Attack success rate must be read together with the benign overrefusal set: a model can appear robust simply by refusing more, which is why the benchmark ships 100 benign behaviours alongside.

Similarity to real tasks: Medium — Established from the official source: JailbreakBench evaluates attacks against 100 distinct misuse behaviours across ten categories corresponding to published usage policies, and pairs them with 100 benign behaviours to measure overrefusal. Testing both compliance and overrefusal matches how a deployed safety system is judged. It is a research harness rather than an audit of a production stack with its own filters.

Data contamination risk: Medium — Behaviours and jailbreak artifacts are published by design, so models trained after release may have been hardened against these exact prompts. No run-specific audit is registered.

Benchmark gaming risk: High — Because the artifacts are public, a model can be hardened against the specific published jailbreaks without becoming generally more robust, which the benchmark's own reproducibility requirement makes easier.

What you still need to test yourself

• Test your own deployed stack including filters and system prompts, which this harness does not model.
• Establish your own harmfulness judging criteria, since the benchmark does not fix one.

This benchmark supports decisions about:

• Compare jailbreak robustness alongside overrefusal rather than in isolation.
• Plan red-teaming coverage against policy-aligned categories.

Limitations

• Published artifacts mean robustness may be specific to known jailbreaks.
• Not an evaluation of a production safety stack.

Official project source reviewed 2026-10-12. The source states no release date; the freshness date recorded here is the publication date of the primary paper attached 2026-10-25.

What this evidence does not cover

• No model scores are registered on this page, so it supports no comparison.
• The harmfulness judging method and the release date are not established from the official source reviewed.

Sources

• JailbreakBench — official project page JailbreakBench authors · accessed 2026-10-12Supports the 100 misuse behaviours across ten policy-aligned categories, the roughly 55% original composition with the remainder from AdvBench and TDC/HarmBench, the 100 benign overrefusal behaviours, and the artifact-submission requirement; it does not describe the harmfulness judging method and does not support any model score, and no score is registered on this page.
• JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models Chao et al. · accessed 2026-10-25The primary methodology paper, establishing the benchmark's attack and judge design and its release date. It does not establish any model's score.

How this benchmark is scored

Primary source: https://jailbreakbench.github.io/

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to JailbreakBench Methodology and Evidence.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path