Nobody is conducting your agents
Models got better fast, but some of the biggest jumps in AI coding came from the tooling built around them. The next one has to come from the tooling around the agents, and most teams don't have it yet.
In August we at Dualboot Partners built a software factory for a client in a week. Agents take every story from technical design to a feature that is deployed and checked on staging, and people step in only where the board asks for them. In the ten weeks since, agents have opened 1,363 of the project’s 1,575 pull requests and got 1,306 of them merged, in 21,515 agent sessions. A finished story costs $36 at the median and $63 on average, at API prices. It all runs on Aixle Flow, the open-source layer we built for running agents as a team.
The models didn’t make the difference; everyone has the same ones. The layer around the agents did. Three years ago nobody needed it: the agent itself was the problem.
In October 2023 the best AI system fixed 1.96% of the 2,294 real issues in SWE-bench. On 22 September 2026 Scale AI opened SWE-bench Pro V2, a cleaned-up, harder-to-game successor, and by the end of launch day the top score was 99.4%.
At Dualboot we’ve used coding agents since the first tools shipped: Cursor in 2023, then Claude Code from February 2025. The models were dumb enough that nobody believed writing code would stop being our job within a few years, and two engineers sharing one agent felt like an incredible speed-up. It was also chaos: everyone had their own setup, nobody knew who was running what, prices were opaque, there were no logs. The first developer to burn more than $1,000 in a week seemed like an outlier. He was on my project.
Andrej Karpathy described the feeling last December:
Clearly some powerful alien tool was handed around except it comes with no manual and everyone has to figure out how to hold it and operate it…
The players are ready
Every coding benchmark goes the same way: agents eat it, and it gets fixed, replaced or retired, each time faster. HumanEval took about twenty months to get a bare-model score above 50%, and SWE-bench Verified eleven weeks. Terminal-Bench 2.0 opened at about 50% and passed 80% within five months.
In January Karpathy put it this way:
The intelligence part suddenly feels quite a bit ahead of all the rest of it - integrations (tools, knowledge), the necessity for new organizational workflows, processes, diffusion more generally.
The loop mattered as much as the model
New models did a lot of that work. But some of the biggest jumps came with no new model at all, from the loop someone built around one. In March 2023 the Reflexion paper put GPT-4 in a loop: write the code, run the tests, read what failed, try again. The same model went from 80% to 91% on HumanEval. A year later Andrew Ng showed a loop beating a whole model generation.
It still happens: LangChain moved its agent from the top 30 to the top 5 on Terminal-Bench 2.0 and summed it up: “We only changed the harness.” The people who build agents have stopped prompting them by hand:
My job is to write loops.
And yet the work isn’t getting done
That first layer, the one that turns a model into an agent, got built. The work should be flowing. It isn’t. Ilya Sutskever, on how well models do on evals:
They are doing so well. But the economic impact seems to be dramatically behind.
On single tasks, the best model now wins or ties against industry experts 85% of the time on OpenAI’s GDPval. On whole jobs, real freelance projects someone paid for on the Remote Labor Index, the best agent delivers acceptable work 21% of the time.
Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, “due to escalating costs, unclear business value or inadequate risk controls”. A model that isn’t smart enough is not on that list.
Ten strong dogs, no plan.
Nobody built the second layer
The first layer sits around a model. The second sits around the agents: what they work on and in what order, who checks the result, what happens when a run dies halfway, what it all costs. Most of it is still missing, and the companies closest to the models say so. OpenAI, launching its Codex app:
The core challenge has shifted from what agents can do to how people can direct, supervise, and collaborate with them at scale—existing IDEs and terminal-based tools are not built to support this way of working.
Greg Brockman, OpenAI’s president, says engineers who haven’t handed their code to Codex are held back by “factors besides the capability of the model”, and names the infrastructure around the tools: “observability, tracking not just the committed code but the agent trajectories that led to them, and central management of the tools that agents are able to use.” That is the second layer.
What a conductor does
An orchestra of excellent musicians still needs someone to decide who comes in when, and to stop the piece when the brass goes wrong. A team running agents has the same work, and on most teams nobody owns it. On the factory we learned it splits into four jobs: the score (plan the work), listening (check the result), holding together (keep it running) and paying the players (count the cost).
The score
The work, written down: the steps, their order, what goes into each and what comes out, who does which part. McKinsey titled the first lesson from a year of agent deployments “It’s not about the agent; it’s about the workflow”.
Writing it down starts with the requirements, and those are still on people. Agents got great at code, but finding out what the client needs, clarifying it and iterating on it is still our job. Agents help there too, we use the BMAD method’s brainstorming workflow and Matt Pocock’s grill-me skill, but without a clear spec from people a big project is very hard to build.
On the factory up to fifteen stories are in flight at once, too many for anyone to switch between. So we wrote the plan down before any code, following the BMAD method: a PRD, epics and stories, and a large graph of which story depends on which. That graph lets a planner work in a loop. Every two hours it releases the next stories whose dependencies are done and whose files don’t overlap with anything in flight, so they don’t collide on merge. Fifteen is a cap we set, not a record: each story runs in its own containers, and people watch the Blocked column and a summary in Slack, not fifteen stories.
Not everything starts from a card. The planner runs on a schedule, a message in Slack starts its own workflow, and a request that turns out to be a whole feature goes to a separate step that slices it into stories first. Our first workflows weren’t perfect and missed a lot; we fixed them quickly with a personal MCP server, which deserves a post of its own.
Listening
Checks before anyone trusts the output. OpenAI’s GDPval paper has the sharpest number on this: on its own, GPT-5 looks “90x faster, 474x cheaper” than the experts, but once an expert reviews its work and redoes the tasks it got wrong, that shrinks to 1.12x faster and 1.18x cheaper.
Code generation is no longer the bottleneck — verification is.
A person becomes the bottleneck wherever you put them: make someone the reviewer, and within a couple of days they are drowning in a hundred pull requests. So on the factory most of the checking is done by agents, in layers. One reads the CI run and sends a red branch back. One reviews the diff against the acceptance criteria. The last opens the real app on staging, sets up its own data through the UI, walks every criterion in a browser with a screenshot for each, and checks every changed screen at three widths, in a second language and in the dark theme. Any of them can send the card back.
People keep the requirements, acceptance with the client, and the setup of skills, MCP servers, workflows and agents. Two things made this workable. A blocking protocol: an agent that can’t go on parks its card in a Blocked column, with a comment saying what is missing, what it tried and who it needs it from, and posts the same in Slack. And a daily Slack summary of what got done and which questions wait on the client.
Holding together
Runs that survive a restart, retries, clean-up after a cancel, and every agent at its own stand, with its own container and credentials. Failures are inevitable: a session dies, a deploy fails, a credential is missing. On the factory a card that bounces back from CI more than twice gets its own machine to reproduce the failure, and the planner re-triggers stuck cards. And an agent that can only break its own container can be trusted with more: even new environments are created by agents now, with a written list of what they must not do.
Temporal, which keeps long-running workflows alive through crashes, raised $300M in February and $550M more in September.
Agentic AI doesn’t fail because the models aren’t good enough. It fails because the systems around them can’t handle real-world execution.
He sells this layer, so he would say that. $850M in seven months says investors agree.
Paying the players
A price for every run, every workflow and every person. A Gartner analyst told The Register that AI coding bills were leaping to $2,000–5,000 per developer per month; in Gartner’s peer research, 6% of organisations already pay more than $2,000. Uber caps every employee at $1,500 a month in tokens per coding tool. Yet in KPMG’s June survey of US companies with revenue of $1 billion or more, only 26% reported “full, real-time visibility into what their AI systems cost to operate.”
Once you see the bill, you look much harder at extra context, because it decides how much money gets wasted. On the factory, implementation took $54,000 of the $96,000. Code review took 11%, the browser checks on staging 8.5%, and the planner 4%. One lesson from ours: don’t give a smart model a browser. Have the expensive model write the plan, and let a cheaper one click through the result.
Where this argument is weak
The benchmarks overstate, so read the ones above with care. In February OpenAI stopped reporting SWE-bench Verified after finding material issues in 59.4% of the 138 problems its o3 model couldn’t reliably solve.
The strongest objection is that better models need less harness. Noam Brown at OpenAI calls harnesses “like a crutch that eventually we’re going to be able to move beyond”. In July Boris Cherny’s team deleted 80% of Claude Code’s system prompt because Opus 5 “just does it”. But of the code left in the Claude Code harness, he said, “almost all of it is about safety and permissions and static analysis”.Noam Brown on Latent Space, 19 June 2025. Boris Cherny at Y Combinator, 27 July 2026: the prompt at 4:14, the harness at 6:24, from auto-captions. The model eats the instructions, not the checks. Permissions, isolation and the cost of every run don’t depend on how smart the model is, and a smarter agent that does more needs them more, not less.
Another objection: this month Ethan Mollick, who wrote in January that running agents is “basically reinventing management”, changed his mind:
It turns out that organizing work is just one more thing AI can learn to do.
He still leaves people one job: in his words, “people decided where to point them.”
What still breaks for us is rarely one agent not being smart enough. Things break at the seams: one agent built a design, another implemented it, a third changed the design again, and the checker rejected it, rightly. We don’t fully trust agents with security, so we run periodic security reviews and penetration tests. Nor with architecture: big tasks still start with a design session with a team lead. And rules overshoot. Our rules for automated tests left us with far too many tests, and a gate that demanded written proof for every finished story failed so many pull requests that the project lead retired it in September.
What we built
We built Aixle Flow to be that second layer, for three reasons. Everyone on the team used agents their own way, and we wanted to reuse each other’s experience and tools. We wanted to know what AI costs, per project and per task. And I was tired of closing my laptop and having the agent fall asleep with it. Agents should work at night and on weekends.
Plenty of tools run Claude Code, Codex and the rest side by side, on your laptop or in the vendor’s cloud.On your machine: Claude Squad, Sculptor, Gas Town. The vendor’s own agent in the vendor’s cloud, by default: Devin, Codex cloud, Jules. As of October 2026; this market changes every month. The closest to what we wanted answer one question differently: who runs the agents’ work?
| Who runs the work | What it is | |
|---|---|---|
| Conductor | You, mostly by hand | Claude Code, Codex, Cursor and OpenCode in parallel, on your Mac or in Conductor’s cloud. You assign, review and merge. |
| OpenHands | You, through code and settings | Open source: its own agent and the popular CLIs, an SDK, and automations on a schedule or a webhook. You build on top. |
| Paperclip | The agents, by hierarchy | Open source: a company of agents with roles and budgets that delegate to each other, with you as the board. Not only for code. |
| Flow | A process written down in advance | Workflows you defined start from a card on the board, a schedule, a Slack message, a webhook or a tracker. Each step is an agent session in its own container. |
Each is stronger somewhere: Conductor needs no setup, OpenHands gives you an SDK to build your own agent, and Paperclip installs with one command and stops an agent when it hits its budget. Flow takes more to install. We wanted the agents we already pay for, on our own servers, running a process we wrote down, while the laptop is closed.
Flow runs the agent CLIs a company already pays for, Claude Code, Codex, Cursor and four others, as steps of workflows. Steps wait for CI or a person where they should. Runs live on Temporal and survive restarts, and every session gets its own container and its own price.
A card goes into an automated column. Four steps run as separate agent sessions. The last screen is the bill.
The factory is an Aixle Factory project. Flow is the engine inside Aixle, our AI operating layer, and it’s available on its own. From 1 August to 12 October the factory used about 205 billion tokens and cost $96,500 at API prices; a finished story took about thirteen workflow runs, and one in ten cost more than $147.
Flow is open source under Apache 2.0 and runs on your own servers. See how it works, or clone it from GitHub; all it needs is Docker and git. If you’d rather have it wired into your own systems, with the workflows designed for you, talk to us.
Next in this series: Flow’s personal MCP server and how we fix workflows with it, how we built Flow, the business workflows we run on it, OpenTelemetry for what every agent costs, and a tour of the product. Tell me which one you want first. Write to artem.petrov@dualbootpartners.com, or find me on GitHub, LinkedIn and partos.me.