Research

96 Test Runs for $9.58: He Had Claude Code Build a Trap-Filled CRM to See Which Claude Model Answers Sales Questions Right for the Least Money

  • Level Advanced
  • Setup1–3 hrs (estimated)
  • Evidence Some evidence
  • CostNot stated in source
  • No-codeNo
Claude CodeSource-linked

What did they do with Claude Code?

Before picking a model for a sales-data agent, he had Claude Code build a test: an invented CRM full of traps (four currencies, archived duplicate deals, deals with two CFOs, deals that entered a stage twice), 8 questions with answers computed two ways, and an agent with one read-only SQL tool. Three Claude models ran 96 times in total for $9.58 (as of 2026-11-12). No model fell for a summing trap. Sonnet and Opus matched the answer key 32 of 32 times and Fable 29 of 32, and the cost per right answer was 2.7 cents for Sonnet, 9.3 for Opus and 19.9 for Fable. His advice: test on questions you already know the answers to, and divide cost by right answers, not tokens.

How it went

dev.to article by Kim Olsen (Nov 10, 2026) with code excerpts: invented Danish CRM with deliberate traps; 3 Claude models, 8 questions, 2 conditions, 96 runs costing $9.58 (as of 2026-11-12) at list price; Sonnet 5.5 and Opus 5.5 32/32, Fable 5.2 29/32; cost per right answer 2.7, 9.3 and 19.9 cents. Test data is invented, so results are one small test

Worth knowing

  • dev.to article by Kim Olsen (Nov 10, 2026) with code excerpts: invented Danish CRM with deliberate traps; 3 Claude models, 8 questions, 2 conditions, 96 runs costing $9.58 at list price; Sonnet 5.5 and Opus 5.5 32/32, Fable 5.2 29/32; cost per right answer 2.7, 9.3 and 19.9 cents. Test data is invented, so results are one small test.

Prompt

Core command is original; setup is cleaned from the source.

Help me test which AI model to put behind a read-only agent for my [CRM or database]. Write 5 to 10 business questions I already know the answers to, and compute each true answer two ways (in code and in SQL) so they must agree. Give each model one read-only SQL tool and a submit tool that returns the number, the SQL and a short explanation. Run each question at least twice per model, count an answer right if it is within 0.5% of the truth, label each miss with what caused it, and report the cost per right answer for each model.

Recommended

Try it in Grok Bot

  1. Copy the prompt (button below).
  2. Open Grok Bot chat and paste it. Replace any [bracketed] placeholders.
  3. Connect the listed tools (as needed) as connectors when Grok Bot asks.
  4. If the task repeats, ask: "Make this a daily routine."

Alternatives: the same prompt works in Claude Code, Claude, ChatGPT Agent, OpenClaw.

Set it up

Stack: Claude Code

  1. Read the original source for full context.
  2. Connect the tools your agent needs.
  3. Paste the prompt and replace [bracketed] placeholders.
  4. Run once manually and check the output before scheduling it.

Generic suggested steps, not from the source.

Evidence

dev.to article by Kim Olsen (Nov 10, 2026) with code excerpts: invented Danish CRM with deliberate traps; 3 Claude models, 8 questions, 2 conditions, 96 runs costing $9.58 (as of 2026-11-12) at list price; Sonnet 5.5 and Opus 5.5 32/32, Fable 5.2 29/32; cost per right answer 2.7, 9.3 and 19.9 cents. Test data is invented, so results are one small test

Source: Web · Verified by us: not yet · Report broken or outdated

Related use cases

ContentX · 2026-01

Turn YouTube channels into a weekly EPUB magazine

Every week, pulls new videos from a list of YouTube channels, rewrites their transcripts as magazine articles, and emails them to you as a newsletter plus an EPUB file.

  • Level Intermediate
  • Setup1–3 hrs*
  • Evidence Strong evidence
Claude CodeSource-linked
ResearchReddit

Schedule Reddit, email and Search Console digests

Isolated cron jobs scan Reddit, check inboxes, and pull Google Search Console data, then send summaries to Telegram.

  • Level Beginner
  • Setup~20 min per job (source)
  • Evidence Strong evidence
OpenClawSource-linkedBeginner-friendly
ResearchX · 2026-10

Scrape X bookmarks into a Notion database every few hours

Bookmarks pile up and get forgotten. Jeston Lu built a Grok Bot that scrapes saved X posts every 6 hours and auto-categorizes them into a Notion database so the archive stays searchable.

  • Level Intermediate
  • Setup1–3 hrs*
  • Evidence Strong evidence
Grok BotSource-linked