- Without custom tools. The agent started cold, with only the provider tools (Slack, Linear, GitHub).
- With custom tools. The agent could also call the custom tools that Coral saved from earlier runs.
−37% tokens
−31% cost
−29% time
34 / 34 correct
Tasks
Each cell shows the result without custom tools → the result with custom tools. The percentage is the change. Each task is a summary of the prompt. Every answer was correct in both runs, except where a task says otherwise.| Task | Tokens | Cost | Time |
|---|---|---|---|
In benchmark mode, do running queries see newly planted flag rows, does a replant replace or add rows, and what does a corrupt flag file do? github | 346,520 → 355,907+3% | $0.36 → $0.34−5% | 72 s → 72 s0% |
A reviewer said a new states table gave the wrong rows. Was the table fixed or dropped, in which commit, and who gave the final approval? github | 197,929 → 185,666−6% | $0.26 → $0.20−23% | 45 s → 33 s−27% |
A deploy script fails on a second run within one minute. The bug report was closed for another ticket. Which issue should I follow? linear | 314,996 → 286,839−9% | $0.27 → $0.25−10% | 44 s → 42 s−3% |
I reported that a deploy script fails when run twice in one minute, and it was closed for another ticket. Which issue should I follow? linear | 317,157 → 266,456−16% | $0.28 → $0.22−19% | 42 s → 36 s−15% |
A report that a deploy fails on a rerun within one minute was closed as a duplicate. Which issue is the canonical one? linear | 330,393 → 244,283−26% | $0.31 → $0.24−22% | 50 s → 37 s−26% |
A deploy bug (it fails on a rerun within one minute) was filed twice, apart from a similar CI bug. Which issue is the canonical one? linear | 326,817 → 266,878−18% | $0.28 → $0.29+1% | 69 s → 49 s−29% |
An auth bug let a corrupt token scope give an agent unrestricted access. What was the issue ID, and which pull request fixed it? github+linear | 595,074 → 240,775−60% | $0.42 → $0.20−53% | 95 s → 61 s−36% |
Benchmark runs failed when an image tag was re-pushed. Which issue stopped the build trusting mutable tags, and which PR fixed it? github+linear | 325,055 → 283,273−13% | $0.23 → $0.26+11% | 63 s → 60 s−5% |
The nightly benchmark digest stopped comparing with only the previous run. What was the issue ID, and which baseline window did it use? github+linear | 342,979 → 267,356−22% | $0.31 → $0.25−19% | 50 s → 49 s−2% |
The nightly benchmark digest got its link to the experiment run back. What was the issue ID, and which PR shipped it? github+linear | 482,900 → 246,864−49% | $0.34 → $0.21−38% | 61 s → 45 s−26% |
A cleanup ticket made the benchmark load its test cases once. Which issue was at the start of the chain of blockers that held it up? linear | 253,122 → 239,451−5% | $0.23 → $0.20−14% | 46 s → 55 s+19% |
In a thread about a benchmark regression, someone mentioned a graph bug. Which ticket tracked the fix, and which PR shipped it? github+linear+slack | 602,256 → 351,791−42% | $0.46 → $0.37−19% | 79 s → 78 s0% |
Someone asked in the engineering channel why the cron alerts were noisy and got an answer. Who were the two people? slack | 270,707 → 154,637−43% | $0.19 → $0.14−28% | 58 s → 38 s−34% |
Who asked and who answered, in the engineering channel, how the cron alerts were set up and why they were noisy? slack | 376,155 → 155,076−59% | $0.30 → $0.14−53% | 53 s → 40 s−25% |
Someone asked if anyone objected to turning off some database metrics to cut monitoring cost. Who were the two people in that exchange? slack | 260,764 → 200,611−23% | $0.19 → $0.16−19% | 60 s → 75 s+23% |
Who were the two people who discussed the cloud account for desktop app code signing in the engineering channel? slack | 315,449 → 182,634−42% | $0.22 → $0.14−36% | 64 s → 41 s−37% |
One of our team and one customer worked to connect the customer’s dashboards for the first time. Who were the two people? slack | 310,920 → 189,488−39% | $0.28 → $0.18−37% | 44 s → 75 s+70% |
When we added rating buttons to investigations, what did we decide to rate when an investigation has several follow-up responses? slack | 368,112 → 295,645−20% | $0.30 → $0.24−21% | 54 s → 101 s+87% |
A thread proposed to move prompts out of the codebase because deploys were slow. Which option did we choose, and how did it end? github+slack | 259,713 → 216,735−17% | $0.22 → $0.19−16% | 50 s → 52 s+2% |
A thread compared two infrastructure-as-code tools before a cloud resource import. What did we decide, and when would we revisit it? slack | 265,311 → 330,849+25% | $0.21 → $0.29+35% | 50 s → 118 s+133% |
A thread complained about the cost of our infrastructure-as-code tool. What did we decide, and which other options came up? slack | 274,821 → 192,666−30% | $0.22 → $0.18−16% | 71 s → 72 s0% |
A thread listed places to keep prompts outside the codebase to avoid slow deploys. Which option was tried, and what was the final call? slack | 207,450 → 161,123−22% | $0.21 → $0.15−28% | 41 s → 48 s+17% |
After a complaint about our infrastructure-as-code tool’s yearly cost, what did we decide, and which other option did we not choose? linear+slack | 398,130 → 340,896−14% | $0.34 → $0.26−22% | 100 s → 112 s+12% |
Someone asked about self-hosted docs for a customer. Which PR took the work, and did we publish the docs at once or wait? slack | 321,103 → 173,440−46% | $0.23 → $0.18−24% | 56 s → 43 s−23% |
Someone asked for a backend engineer to fix a slow endpoint that blocked mentions. Who took it, and what was their draft PR number? github+slack | 248,045 → 255,342+3% | $0.22 → $0.19−16% | 37 s → 87 s+133% |
Someone reported a failed release workflow after a change to its push steps. Which pull request fixed it? slack | 423,710 → 173,032−59% | $0.36 → $0.13−65% | 78 s → 34 s−56% |
An editor migration needed backend changes to convert saved documents to markdown. Which pull request shipped that fix? github+slack | 506,996 → 243,002−52% | $0.43 → $0.19−55% | 91 s → 83 s−9% |
The release workflow started to fail during desktop packaging work, and a fix was promised. Which pull request fixed it? github+slack | 323,958 → 203,697−37% | $0.23 → $0.15−36% | 49 s → 48 s−1% |
In which channel did someone share cheap flights for the team that went to the offsite? slack | 268,707 → 188,696−30% | $0.26 → $0.17−36% | 46 s → 89 s+96% |
In which channel did someone complain that a vendor merged its desktop chat app into its coding agent? slack | 309,061 → 170,667−45% | $0.26 → $0.18−29% | 141 s → 61 s−57% |
In which channel did someone offer to clean up suppressed lint warnings? slack | 238,641 → 124,227−48% | $0.22 → $0.10−51% | 130 s → 34 s−74% |
In which channel did someone share an AI agent’s diagnosis of why our release automation failed? Answer: wrong without custom tools, correct with them slack | 1,178,574 → 127,995−89% | $0.73 → $0.11−84% | 516 s → 27 s−95% |
In which channel did someone ask for help with a list of prompts to track if AI assistants recommend us? slack | 355,140 → 233,702−34% | $0.23 → $0.17−27% | 158 s → 83 s−48% |
In which channel did someone complain that our public example agent repo needed a dependency update? slack | 251,346 → 135,489−46% | $0.25 → $0.13−46% | 149 s → 31 s−79% |