96 Test Runs for $9.58: He Had Claude Code Build a Trap-Filled CRM to See Which Claude Model Answers Sales Questions Right for the Least Money
- Level Advanced
- Setup1–3 hrs (estimated)
- Evidence Some evidence
- CostNot stated in source
- No-codeNo
What did they do with Claude Code?
Before picking a model for a sales-data agent, he had Claude Code build a test: an invented CRM full of traps (four currencies, archived duplicate deals, deals with two CFOs, deals that entered a stage twice), 8 questions with answers computed two ways, and an agent with one read-only SQL tool. Three Claude models ran 96 times in total for $9.58 (as of 2026-11-12). No model fell for a summing trap. Sonnet and Opus matched the answer key 32 of 32 times and Fable 29 of 32, and the cost per right answer was 2.7 cents for Sonnet, 9.3 for Opus and 19.9 for Fable. His advice: test on questions you already know the answers to, and divide cost by right answers, not tokens.
How it went
dev.to article by Kim Olsen (Nov 10, 2026) with code excerpts: invented Danish CRM with deliberate traps; 3 Claude models, 8 questions, 2 conditions, 96 runs costing $9.58 (as of 2026-11-12) at list price; Sonnet 5.5 and Opus 5.5 32/32, Fable 5.2 29/32; cost per right answer 2.7, 9.3 and 19.9 cents. Test data is invented, so results are one small test
Worth knowing
- dev.to article by Kim Olsen (Nov 10, 2026) with code excerpts: invented Danish CRM with deliberate traps; 3 Claude models, 8 questions, 2 conditions, 96 runs costing $9.58 at list price; Sonnet 5.5 and Opus 5.5 32/32, Fable 5.2 29/32; cost per right answer 2.7, 9.3 and 19.9 cents. Test data is invented, so results are one small test.
Prompt
Core command is original; setup is cleaned from the source.
Help me test which AI model to put behind a read-only agent for my [CRM or database]. Write 5 to 10 business questions I already know the answers to, and compute each true answer two ways (in code and in SQL) so they must agree. Give each model one read-only SQL tool and a submit tool that returns the number, the SQL and a short explanation. Run each question at least twice per model, count an answer right if it is within 0.5% of the truth, label each miss with what caused it, and report the cost per right answer for each model.
Recommended
Try it in Grok Bot
- Copy the prompt (button below).
- Open Grok Bot chat and paste it. Replace any [bracketed] placeholders.
- Connect the listed tools (as needed) as connectors when Grok Bot asks.
- If the task repeats, ask: "Make this a daily routine."
Alternatives: the same prompt works in Claude Code, Claude, ChatGPT Agent, OpenClaw.
Set it up
Stack: Claude Code
- Read the original source for full context.
- Connect the tools your agent needs.
- Paste the prompt and replace [bracketed] placeholders.
- Run once manually and check the output before scheduling it.
Generic suggested steps, not from the source.
Evidence
dev.to article by Kim Olsen (Nov 10, 2026) with code excerpts: invented Danish CRM with deliberate traps; 3 Claude models, 8 questions, 2 conditions, 96 runs costing $9.58 (as of 2026-11-12) at list price; Sonnet 5.5 and Opus 5.5 32/32, Fable 5.2 29/32; cost per right answer 2.7, 9.3 and 19.9 cents. Test data is invented, so results are one small test
Source: Web · Verified by us: not yet · Report broken or outdated