One candidate, one paired evaluation
ASR = successful attacks / valid, clean-correct pairs. Multiple attacks from the same original each count separately. Accuracy is candidate-weighted on valid pairs; unsupported and invalid pairs are excluded.
A BENCHMARK FOR DECISION MODEL RELIABILITY
Decision models.
Stress tested.
How reliably do decision models respond when their inputs change? Explore curated challenges across text, structured inputs, and images.
Paper in preparation. Dataset and evaluation results available now.
Loading release…
01 / MEASURE
The leaderboard
Attack success rate (ASR) measures how often a correct clean decision becomes incorrect under attack. Lower is better — interpret it alongside clean accuracy and coverage.
RESULTS / VISUALIZED
Model ranking
Choose models
Lower ASR is better. † Selection model · ○ Partial coverage. Ranks use the eligible models in each dataset; inspect clean accuracy and coverage in the table below. Missing results remain gaps.
† Used in selecting this suite. Partial coverage. “—” means not measured or no eligible denominator; it is never 0%. Select model checkboxes to compare.
Model versions and evaluation bindings
02 / INSPECT
A closer look at the data
Browse the actual released examples. Compare the original input with its attacked counterpart, then inspect the full native request.
03 / INTERPRET
What these results mean
These are deliberately difficult, model-selected sets. They measure robustness on a curated challenge, not an unbiased real-world error rate.
Read the full evaluation protocol ↗Complete inputs. Explicit scoring.
Keep all questions in the request, including those without ground truth. Score only task.question_ids. Choice uses the native label; Noul normalizes affirmative probabilities; Score uses the probability-weighted expected value with the published tolerance.
Coverage and adapters matter
Not every model supports every task or image input. Historical image results include different inference and scoring adapters. See model bindings and the protocol before making strict model-to-model comparisons.
USE DECISIONROBUST
Build on the benchmark.
Public data and results on Hugging Face. Website source on GitHub; evaluation code is being prepared for release.
Hugging Face ↗