A new study evaluating 67 frontier models from 21 providers, including GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro, found that combining multiple AI models does not create the safety net most enterprises assume. On the MATH-500 benchmark, standard correlation metrics predicted a 2.3% “co-failure rate,” the share of prompts where every model in a pool fails simultaneously, but the actual rate was 5.2%, an underestimate of roughly 2.25 times. When researchers converted graduate-level science questions from multiple-choice to free-response format, that all-wrong tail expanded to 12.7%.
For BFSI and collections operations increasingly building multi-model architectures, routing simple queries to cheap models and escalating complex ones to premium models, this finding challenges a core design assumption. The whole justification for routers, cascades, and Mixture-of-Agents setups is that models with low “pairwise error correlation” cover each other’s blind spots. The study shows this logic breaks down precisely where BFSI teams need it most: on open-ended, high-stakes generation such as drafting a dispute response or interpreting an ambiguous customer query, rather than on tasks with a single checkable answer. The researchers also found that naive majority voting across unequal models can actively hurt performance, with weaker models outvoting the strongest one by a margin the paper describes as a 10-point negative swing on hard prompts.
What the coverage understates is how domain-specific this ceiling is. The study distinguishes “ceiling-bound” environments, where the task is genuinely too hard and all models fail together, from “realizability-bound” environments, where at least one model usually knows the answer but disagreement is too subtle for a router to resolve without an oracle. Most enterprise deployments, including collections workflows involving account-specific context and regulatory nuance, sit closer to the realizability-bound case, meaning the failure is not a capability gap but a routing-signal gap that adding more models cannot close. As the paper’s author put it, adding a twentieth model does not buy tail coverage, because the tail of failures is shared across the market, not spread across providers.
This aligns with what we see in production deployments across collections: the fix is not more models stacked together, it is measuring the actual ceiling before building the architecture. The study’s proposed method, a Clopper-Pearson bound applied to a held-out set of real queries, gives a free, mathematically grounded answer to whether orchestration will pay off at all. Their own worked example is directly relevant to BFSI: a team testing five agents on 50 sample queries might assume 96% production accuracy after seeing two shared failures, when the statistically guaranteed ceiling could be as high as 12%. Operationalizing this means building a held-out benchmark from real customer interactions, ideally with human-verified resolutions, before investing engineering effort into multi-model routing, and reserving that investment for tasks where answers can be objectively checked rather than open-ended communication where the co-failure ceiling is hardest to close.