Lessons from our agentic engineering spend benchmarking

A few weeks ago, I shared a benchmark with our portfolio. It looked at what they spend on AI engineering tools and the associated productivity gains. The conclusion cut against the recent media headlines about reining in escalating AI costs.

You've probably seen the same stories. Engineering teams are encouraged to adopt agentic coding tools and then the new token-hungry models eat up the budgeted spend far faster than expected with middling or incremental results. The lesson seems to be that these tools cost more than they're worth.

Our data points the other way. Among the companies furthest along in their agentic engineering adoption, engineers are producing around 2.5x the output per head they were a year ago. The tools that drive that cost a few thousand dollars per engineer each year, which works out at a return of roughly 27 times the spend.

This is not a portfolio average, and it would be wrong to read it as one. This kind of return has been achieved by our leading portfolio companies. It shows what good looks like, and where the rest of the portfolio can get to.

It is also specific to engineering, where the tools map directly onto how the work gets done. While we’re also seeing sizable AI gains in other areas, such as GTM or customer support, the returns aren't yet at the same level.

In short, my message to the portfolio was as follows:

Rebuild how engineering operates around the tools. This is what unlocks the real productivity gains.

Treat this spend as an investment that changes what the business can do, not a cost to control.

Only then, once that’s underway, you can start to tune for efficiency.

I’ll take each in turn.

Rebuild the factory

Simply giving everyone a Claude Code or Devin subscription does not produce a 27x return. We’ve seen companies with broad adoption struggle to achieve these gains because the tools were laid over the same workflows. The model was doing the old job slightly faster, rather than the workflows being rebuilt entirely. This is AI theatre and it is the most expensive way to adopt these tools.

There is a useful parallel in how factories moved from steam to electric power. The first firms to electrify simply swapped the steam engine for an electric motor, kept the rest of the factory floor as it was, and saw very little gain. The factories that transformed were those that redesigned the whole operation around what electric power made possible, with machines laid out by workflow rather than around a central driveshaft.

Agentic engineering tools are at the same point. Buying them, and getting people to use them, is swapping the engine. The companies seeing the full return have done the harder thing. They have reimagined how their engineering teams operate around the tools.

The core principle is straightforward to state, but not always easy to do. You need lots of agents running independently and in parallel, with engineers directing the work rather than watching it.

What makes that possible is less exotic than it sounds. The goal is for the agent to complete each task independently and unattended 95% of the time. A task that fails half the time has to be watched, and that’s what kills the gains. A task the agent can reliably get right, and reliably check, can be left to run.

Getting to that 95% is mostly a matter of engineering fundamentals, and the striking thing is how few of them are new. Slicing work into small, well-defined pieces. Fast, reliable test suites the agent can run to tell whether it has succeeded. Tight feedback loops and basic deploy hygiene. None of this is a novel agentic technique. It is the same discipline good engineering teams have always been told to invest in.

What's changed is the cost of skipping it. For a person, these practices were close to optional. Take away the fast test suite and a good engineer is slowed a little and made a little less accurate, but they push through, holding the missing context in their head and catching their own mistakes. An agent can't do that. Take away the same test suite and the agent is slowed a lot and made a lot less accurate, because the feedback it needs to check its own work simply isn't there. The practices that were a modest tax on humans turn out to be critical for agents. Agents don't fix the gaps in how an organization builds software. They expose them.

This changes what the engineers themselves do. Once the work is sliced and running in parallel, the constraint is no longer how fast a person writes code. It is how well they specify the work, direct the agents, and check what comes back. The agents do the volume. The engineers hold the specification, the judgement, and the accountability. That is a different job from the one most engineering teams were built around.

This is not an overhead to trim

With such a large ROI on offer, the tooling cost barely matters. The risk worth worrying about is underspending to look disciplined and starving a transformation of the investment it needs. The companies seeing the biggest gains are the ones leaning in hardest. This spend fundamentally changes what the business is capable of. It is not an overhead to trim.

That said, this is not an advert for ‘tokenmaxxing’. We don’t advocate for spending without thinking or making usage a goal in itself. There is real inefficiency to cut out, and the best teams do. But that is the wrong goal to start from. A company that opens with "how do we keep this down" optimizes away the investment before it has seen the return. Watch the spend, but don't be governed by it.

Now you can think about efficiency

Once the rollout is well underway, you can start to think about those inefficiencies. There are some low-hanging fruit that remove waste without slowing anyone down:

Right-size the model.

Route simple, mechanical tasks to a smaller, cheaper model, and reserve the strongest frontier models for the work that needs them.

Keep context lean.

A lot of runaway spend comes from bloated context, not from real work.

Wire in the deterministic work.

Some tasks have a single right answer every time, running the tests, checking the types, formatting the code. There is no point having a probabilistic model reason its way through them, slowly and at cost, when a tool can simply do it.

Again, these are follow-on optimizations, not preconditions. Don't let “we need to control the spend” become the reason a rollout stalls. If you spend half as much but end up with AI theatre, you end up spending a lot of money to get incremental gains, while your competitors are getting multiples.

Measurement is also a prerequisite before you scale, because it tells you whether the rollout is working. Set a baseline, then track not just how many engineers use the tools but how deeply. How much of the code is actually being generated rather than nudged, how much real work the agents are completing, and how often their output is good enough to keep. Read together, these tell you what is really happening. High adoption with low generation, for instance, usually means the tools are being used as a clever assistant rather than doing the work, which feels like progress but is not where the return comes from.

We also told companies to budget for this spend to rise over the next 12 to 18 months. As the tools become more capable and pricing evolves, per-engineer spend is likely to grow. That is not a problem to solve. It is a cost to plan for, because the return easily supports it.

Build-time versus run-time spend

There is a distinction that's easy to blur. Everything above is about build-time spend, which describes the cost of the tools engineers use to build the product. That is a productivity investment, with the returns we have described.

The cost of running AI inside the product a company sells is a different thing altogether. That sits in gross margin, scales with every customer, and deserves exactly the kind of discipline that build-time spend does not. ‘AI spend’ can easily cover both, but keeping them apart, especially in a board conversation, stops the logic for one being applied to the other.

The power of the portfolio

The tools themselves are broadly available. What's still scarce is the knowledge of how to rebuild an engineering organization to get this kind of return from them. That is the harder thing, and it is what compounds when around 60 software companies are doing it at the same time, with an AI team carrying what works in one across to the next.

In our Jump Start program, we work intensively with companies to make that change by embedding directly with an engineering team. The companies that have been through the program are seeing individual teams pass 3x, with the rest of the organization following within months. We also run Agentic Engineering Academies where we train senior engineers and first line engineering managers in the skills needed to properly leverage the tools. We've trained more than 60 engineers from 20 portfolio companies so far this year and expect to add at least 80 more in the coming months. Each one then goes on to share their learnings across the engineering org.

We’re still early, too. The leading companies are at the front of the portfolio, not the limit of what is possible. The gains are strikingly uneven, between companies and within them. A single business might have one team at 1.5x productivity and another above 4x. That unevenness isn't a ceiling. It is a map of how much further there is to go.

This benchmark is one output of that work, and we're now sharing it more widely. The learning is simple. Buying the tools and rolling them out gets you to the start line. Rebuilding the factory around them is what gets you the return.