RetireBench: how accurate are AI models at retirement math?
RetireBench asks leading AI models the retirement questions people actually face (Social Security claiming, RMDs, Roth conversions, IRMAA surcharges, ACA subsidies, federal tax) and scores the answers against exact, rule-based math.
Preliminary development results. The numbers below come from development runs on RetireBench's public question set (latest run 2026-07-18), rescored under a tolerant answer-extraction policy. They are directionally useful but not official: the audited run on the private held-out set is in preparation, and this page will switch to those results when it lands.
Leaderboard: Core split
Each model answered the same 160 questions: eight domains, twenty questions each, spanning single-rule lookups (T1) through multi-step calculations (T3). Accuracy is equal-weighted across domains. Costs are the metered API cost of the full run.
| # | Model | 95% CI | ||||
|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 Anthropic | 95.6% | 91%–98% | $4.26 | $0.028 | 10.9 s |
| 2 | Claude Opus 4.8 Anthropic | 64.4% | 57%–71% | $1.52 | $0.015 | 5.9 s |
| 3 | GPT-5 OpenAI | 55.0% | 47%–63% | $4.53 | $0.052 | 36.6 s |
| 4 | Gemini 2.5 Pro Google | 47.5% | 40%–55% | $3.73 | $0.049 | 21.8 s |
| 5 | DeepSeek R1 DeepSeek | 45.6% | 38%–53% | $2.62 | $0.036 | 265.6 s |
| 6 | GPT-5 Mini OpenAI | 43.1% | 36%–51% | $0.47 | $0.0068 | 19.6 s |
| 7 | GPT-5 Nano OpenAI | 36.3% | 29%–44% | $0.27 | $0.0046 | 35.3 s |
| 8 | DeepSeek V3.1 DeepSeek | 21.3% | 16%–28% | $0.11 | $0.0032 | 8.6 s |
| 9 | Llama 3.3 70B Meta | 18.8% | 13%–26% | $0.020 | $0.0007 | 1.8 s |
| 10 | Claude Haiku 4.5 Anthropic | 18.1% | 13%–25% | $0.18 | $0.0062 | 2.5 s |
| 11 | Gemini 2.5 Flash Google | 16.3% | 11%–23% | $0.060 | $0.0023 | 1.1 s |
| 12 | Qwen3 Max Alibaba | 16.3% | 11%–23% | $0.022 | $0.0009 | 44.0 s |
| 13 | Qwen3 235B Alibaba | 13.8% | 9%–20% | $0.0046 | $0.0002 | 1.2 s |
= on the accuracy-vs-cost Pareto frontier: no cheaper run scored higher. 95% CI is the Wilson interval on 160 questions. Cost / correct = run cost ÷ correct answers.
Accuracy vs. cost
Getting retirement math right is worth paying for, but the price spread is three orders of magnitude. Gold points are on the Pareto frontier; hollow points are dominated (a cheaper model scored at least as well).
Where models fail: accuracy by domain
Aggregate accuracy hides the shape of the failures. Models that look passable overall can be coin-flips on IRMAA tiers or ACA subsidy math, the domains where a wrong answer has a real price tag.
| Model | ACA subsidies | Early withdrawal | Federal tax | IRMAA | RMDs | Roth conversion | SS claiming | Taxable SS |
|---|---|---|---|---|---|---|---|---|
| Claude Fable 5 | 85% | 100% | 95% | 95% | 100% | 95% | 95% | 100% |
| Claude Opus 4.8 | 75% | 85% | 70% | 5% | 65% | 20% | 100% | 95% |
| GPT-5 | 15% | 100% | 25% | 0% | 100% | 0% | 100% | 100% |
| Gemini 2.5 Pro | 5% | 100% | 20% | 0% | 75% | 5% | 80% | 95% |
| DeepSeek R1 | 5% | 100% | 20% | 0% | 65% | 0% | 85% | 90% |
| GPT-5 Mini | 5% | 100% | 20% | 0% | 35% | 0% | 100% | 85% |
| GPT-5 Nano | 10% | 100% | 25% | 5% | 30% | 0% | 70% | 50% |
| DeepSeek V3.1 | 5% | 60% | 5% | 0% | 20% | 0% | 50% | 30% |
| Llama 3.3 70B | 5% | 55% | 0% | 0% | 15% | 0% | 25% | 50% |
| Claude Haiku 4.5 | 0% | 50% | 10% | 10% | 35% | 0% | 25% | 15% |
| Gemini 2.5 Flash | 0% | 70% | 5% | 0% | 10% | 0% | 30% | 15% |
| Qwen3 Max | 5% | 70% | 0% | 5% | 15% | 0% | 10% | 25% |
| Qwen3 235B | 0% | 70% | 0% | 0% | 10% | 0% | 20% | 10% |
Twenty questions per domain. Cells below 50% are marked. SS = Social Security.
The Hard split: joint planning problems
Real retirement decisions interact. A Roth conversion changes IRMAA, which changes which claiming age produces the highest lifetime after-tax outcome. The Hard split tests exactly that: T4 chains multiple rules inside one domain; T5 requires jointly optimizing across domains. So far one model has run it.
| Claude Fable 5 | Accuracy | Correct |
|---|---|---|
| T4: chained, single-domain | 94.4% | 68 / 72 |
| Chained conversion | 83% | |
| Chained decumulation | 100% | |
| Distractor | 92% | |
| Multi-year | 100% | |
| Optimization | 92% | |
| Traps | 100% | |
| T5: joint, cross-domain | 27.8% | 10 / 36 |
| Conversion × claiming | 50% | |
| Spend × sequencing | 8% | |
| Triple joint | 25% |
The gap is the finding. Claude Fable 5 scores 95.6% on the Core split and 94.4% on chained single-domain problems, then drops to 27.8% when domains have to be optimized jointly. That is the part of retirement planning where software that prices every interaction on one ledger, rather than reasoning step by step, has a structural advantage.
What RetireBench measures
- Eight domains, 160 questions (Core). Federal tax, Social Security claiming, taxable Social Security, RMDs, Roth conversion, IRMAA, ACA subsidies, and early-withdrawal rules. Twenty questions each, tiers T1–T3 from single-rule lookups to multi-step calculations.
- Exact answers, no partial credit. Every question has a rule-derived answer with an explicit numeric tolerance. An answer is correct or it isn't.
- Refusals and truncations count against accuracy. A model that won't answer a retirement question hasn't answered it.
- Cost and speed are measured, not estimated. Cost is the metered API cost of the run; latency is the per-question median.
- Uncertainty is reported. Every headline accuracy carries a Wilson 95% confidence interval. On 160 questions, differences of a few points are noise.
How to read these numbers
- These are development runs on the public question set, rescored under a tolerant extraction policy that credits correct values in non-conforming formats. The official run uses a private held-out set with full provenance.
- Models were accessed through a routing provider; the specific serving backend can affect latency and occasionally behavior.
- One run per model. Models also change under stable names, so treat this as a snapshot dated 2026-07-18, not a permanent verdict on any vendor.
- Benchmark questions have exact answers by construction. Real retirement situations add ambiguity these scores don't capture. A high score here is necessary for a trustworthy answer, not sufficient.
Why RetireGolden runs this benchmark
RetireGolden's planner doesn't ask a language model to do retirement math. Every number comes from a deterministic, auditable ledger with tested, sourced rules. RetireBench exists to measure the alternative people actually use: asking a chatbot. The results so far say what you'd hope and fear. Frontier models are getting genuinely good at single-rule questions, and still fail the joint optimization problems that decide real money.
Next up: the audited official run, and the flagship comparison, running the same models with and without RetireGolden's planning tools connected, to measure what changes when a model can delegate the math.
RetireBench is a RetireGolden project. Nothing on this page is financial, tax, or legal advice.