Skip to main content
Coral saves the tool-calling code that an agent writes as a custom tool. The next agent calls that tool and does not write the code again. This page shows the effect on real tasks. The agent ran each task twice:
  • Without custom tools. The agent started cold, with only the provider tools (Slack, Linear, GitHub).
  • With custom tools. The agent could also call the custom tools that Coral saved from earlier runs.
Across the 34 tasks below, custom tools gave these results:

−37% tokens

12.2M → 7.7M

−31% cost

$9.86 → $6.79

−29% time

47 min → 33 min

34 / 34 correct

Up from 33 / 34

Tasks

Each cell shows the result without custom tools → the result with custom tools. The percentage is the change. Each task is a summary of the prompt. Every answer was correct in both runs, except where a task says otherwise.
TaskTokensCostTime

In benchmark mode, do running queries see newly planted flag rows, does a replant replace or add rows, and what does a corrupt flag file do?

github github.fileContentsAtRef github.pullRequestDetails github.searchPullRequests

346,520 → 355,907+3%$0.36 → $0.34−5%72 s → 72 s0%

A reviewer said a new states table gave the wrong rows. Was the table fixed or dropped, in which commit, and who gave the final approval?

github github.pullRequestReviews github.searchPullRequests

197,929 → 185,666−6%$0.26 → $0.20−23%45 s → 33 s−27%

A deploy script fails on a second run within one minute. The bug report was closed for another ticket. Which issue should I follow?

linear linear.issueRelations linear.searchIssues

314,996 → 286,839−9%$0.27 → $0.25−10%44 s → 42 s−3%

I reported that a deploy script fails when run twice in one minute, and it was closed for another ticket. Which issue should I follow?

linear linear.issueRelations linear.searchIssues

317,157 → 266,456−16%$0.28 → $0.22−19%42 s → 36 s−15%

A report that a deploy fails on a rerun within one minute was closed as a duplicate. Which issue is the canonical one?

linear linear.issueRelations linear.searchIssues

330,393 → 244,283−26%$0.31 → $0.24−22%50 s → 37 s−26%

A deploy bug (it fails on a rerun within one minute) was filed twice, apart from a similar CI bug. Which issue is the canonical one?

linear linear.issuePlanning linear.issueRelations linear.searchIssues

326,817 → 266,878−18%$0.28 → $0.29+1%69 s → 49 s−29%

An auth bug let a corrupt token scope give an agent unrestricted access. What was the issue ID, and which pull request fixed it?

github+linear github.searchPullRequests linear.issueComments linear.issuePlanning linear.searchIssues

595,074 → 240,775−60%$0.42 → $0.20−53%95 s → 61 s−36%

Benchmark runs failed when an image tag was re-pushed. Which issue stopped the build trusting mutable tags, and which PR fixed it?

github+linear github.searchPullRequests linear.issueComments linear.issuePlanning linear.searchIssues

325,055 → 283,273−13%$0.23 → $0.26+11%63 s → 60 s−5%

The nightly benchmark digest stopped comparing with only the previous run. What was the issue ID, and which baseline window did it use?

github+linear github.pullRequestReviews github.searchPullRequests linear.issueComments linear.issuePlanning linear.searchIssues

342,979 → 267,356−22%$0.31 → $0.25−19%50 s → 49 s−2%

The nightly benchmark digest got its link to the experiment run back. What was the issue ID, and which PR shipped it?

github+linear github.searchPullRequests linear.issuePlanning linear.searchIssues

482,900 → 246,864−49%$0.34 → $0.21−38%61 s → 45 s−26%

A cleanup ticket made the benchmark load its test cases once. Which issue was at the start of the chain of blockers that held it up?

linear linear.issueRelations linear.searchIssues

253,122 → 239,451−5%$0.23 → $0.20−14%46 s → 55 s+19%

In a thread about a benchmark regression, someone mentioned a graph bug. Which ticket tracked the fix, and which PR shipped it?

github+linear+slack github.searchPullRequests linear.issuePlanning linear.searchIssues slack.searchMessages slack.threadLinkedWork workspace.searchAll

602,256 → 351,791−42%$0.46 → $0.37−19%79 s → 78 s0%

Someone asked in the engineering channel why the cron alerts were noisy and got an answer. Who were the two people?

slack slack.searchMessages slack.threadReplies

270,707 → 154,637−43%$0.19 → $0.14−28%58 s → 38 s−34%

Who asked and who answered, in the engineering channel, how the cron alerts were set up and why they were noisy?

slack slack.searchMessages slack.threadReplies

376,155 → 155,076−59%$0.30 → $0.14−53%53 s → 40 s−25%

Someone asked if anyone objected to turning off some database metrics to cut monitoring cost. Who were the two people in that exchange?

slack slack.searchMessages slack.threadReplies

260,764 → 200,611−23%$0.19 → $0.16−19%60 s → 75 s+23%

Who were the two people who discussed the cloud account for desktop app code signing in the engineering channel?

slack slack.searchMessages slack.threadReplies

315,449 → 182,634−42%$0.22 → $0.14−36%64 s → 41 s−37%

One of our team and one customer worked to connect the customer’s dashboards for the first time. Who were the two people?

slack slack.searchMessages slack.threadReplies

310,920 → 189,488−39%$0.28 → $0.18−37%44 s → 75 s+70%

When we added rating buttons to investigations, what did we decide to rate when an investigation has several follow-up responses?

slack slack.searchMessages slack.threadReplies

368,112 → 295,645−20%$0.30 → $0.24−21%54 s → 101 s+87%

A thread proposed to move prompts out of the codebase because deploys were slow. Which option did we choose, and how did it end?

github+slack github.searchPullRequests slack.searchMessages slack.threadLinkedWork workspace.searchAll

259,713 → 216,735−17%$0.22 → $0.19−16%50 s → 52 s+2%

A thread compared two infrastructure-as-code tools before a cloud resource import. What did we decide, and when would we revisit it?

slack slack.searchMessages slack.threadReplies workspace.searchAll

265,311 → 330,849+25%$0.21 → $0.29+35%50 s → 118 s+133%

A thread complained about the cost of our infrastructure-as-code tool. What did we decide, and which other options came up?

slack slack.searchMessages slack.threadReplies workspace.searchAll

274,821 → 192,666−30%$0.22 → $0.18−16%71 s → 72 s0%

A thread listed places to keep prompts outside the codebase to avoid slow deploys. Which option was tried, and what was the final call?

slack slack.searchMessages slack.threadLinkedWork

207,450 → 161,123−22%$0.21 → $0.15−28%41 s → 48 s+17%

After a complaint about our infrastructure-as-code tool’s yearly cost, what did we decide, and which other option did we not choose?

linear+slack linear.searchIssues slack.searchMessages slack.threadReplies

398,130 → 340,896−14%$0.34 → $0.26−22%100 s → 112 s+12%

Someone asked about self-hosted docs for a customer. Which PR took the work, and did we publish the docs at once or wait?

slack slack.searchMessages slack.threadLinkedWork slack.threadReplies

321,103 → 173,440−46%$0.23 → $0.18−24%56 s → 43 s−23%

Someone asked for a backend engineer to fix a slow endpoint that blocked mentions. Who took it, and what was their draft PR number?

github+slack github.searchPullRequests slack.searchMessages slack.threadLinkedWork

248,045 → 255,342+3%$0.22 → $0.19−16%37 s → 87 s+133%

Someone reported a failed release workflow after a change to its push steps. Which pull request fixed it?

slack slack.searchMessages slack.threadLinkedWork

423,710 → 173,032−59%$0.36 → $0.13−65%78 s → 34 s−56%

An editor migration needed backend changes to convert saved documents to markdown. Which pull request shipped that fix?

github+slack github.searchPullRequests slack.searchMessages slack.threadLinkedWork

506,996 → 243,002−52%$0.43 → $0.19−55%91 s → 83 s−9%

The release workflow started to fail during desktop packaging work, and a fix was promised. Which pull request fixed it?

github+slack github.searchPullRequests slack.searchMessages slack.threadLinkedWork

323,958 → 203,697−37%$0.23 → $0.15−36%49 s → 48 s−1%

In which channel did someone share cheap flights for the team that went to the offsite?

slack slack.searchMessages slack.threadReplies

268,707 → 188,696−30%$0.26 → $0.17−36%46 s → 89 s+96%

In which channel did someone complain that a vendor merged its desktop chat app into its coding agent?

slack slack.messagesByUser slack.searchMessages

309,061 → 170,667−45%$0.26 → $0.18−29%141 s → 61 s−57%

In which channel did someone offer to clean up suppressed lint warnings?

slack slack.messagesByUser

238,641 → 124,227−48%$0.22 → $0.10−51%130 s → 34 s−74%

In which channel did someone share an AI agent’s diagnosis of why our release automation failed?

Answer: wrong without custom tools, correct with them

slack slack.messagesByUser

1,178,574 → 127,995−89%$0.73 → $0.11−84%516 s → 27 s−95%

In which channel did someone ask for help with a list of prompts to track if AI assistants recommend us?

slack slack.messagesByUser slack.searchMessages slack.threadReplies

355,140 → 233,702−34%$0.23 → $0.17−27%158 s → 83 s−48%

In which channel did someone complain that our public example agent repo needed a dependency update?

slack slack.messagesByUser

251,346 → 135,489−46%$0.25 → $0.13−46%149 s → 31 s−79%