Share

TL;DR: When AI-assisted development moved from autocomplete to agentic workflows, the bottleneck for our engineering velocity stopped being the model and started being the vendor. Owning our own orchestration layer, running the same upstream agent stack across multiple developer surfaces, wrapping every change in a version-controlled spec, and treating the agent calls themselves as first-class telemetry is what made spec-driven development practical for us at NodeSource.

The last two years made one thing very clear to anyone running an engineering organization: AI-assisted development stopped being optional. The shape of the bottleneck, however, moved. It used to sit inside the model—what could the model do, how good was the reasoning, and how big was the context window. Those are still concerns, but they are not the ones that slowed us down at NodeSource. By the time we had integrated agentic workflows into day-to-day engineering, the friction had migrated up the stack into the tooling that wrapped around the model: the IDE plugin, the IDE fork, the chat client, the harness driving the agent, and the configuration surface that held everything together. That tooling was almost entirely rented. Every monthly invoice reminded us.

We did not start with the goal of building our own stack. We arrived there by running into the same walls other engineering organizations are running into right now and discovering that the walls were not technical; they were commercial. This is the story of how we got from early GitHub Copilot experiments to owning the orchestration layer that now drives our agentic SDLC, the architectural decisions that made it possible, and the principles we extracted along the way.

From autocomplete to agentic workflows

Our exploration began where most of the industry did, with GitHub Copilot. In the early days, Copilot felt like magic; it sped up boilerplate generation, suggested function signatures, and saved keystrokes that added up across a sprint. It also had a hard ceiling: it was a single-line autocomplete, period. As our systems grew, we quickly reached the limits of in-line suggestions. We needed tools that could understand intent, navigate a directory tree, propose multi-file changes, and execute commands. The gap between what an autocomplete does and what a real engineering workflow needs is enormous, and we needed the latter.

That led us to become early adopters of Windsurf. Transitioning to Windsurf's agentic workflows was a massive leap forward. Instead of suggesting the next line of code, the AI could now navigate the codebase, propose changes that touched multiple files at once, and run terminal tasks to verify them. It shifted our engineers' focus from writing code line-by-line to guiding an agent through specific technical objectives. This was the moment we knew the SDLC was going to change shape permanently.

Adding depth with MCPs and specialized skills

As we became more comfortable with agentic workflows, we realized that general-purpose agents still suffered from a lack of deep, domain-specific context. A model with no awareness of our internal conventions, our dependency policies, or our test pyramid would generate plausible-looking code that did not pass our standards, and the cost of catching those misses after the fact was much higher than preventing them up front. To address that, we began adding structural layers on top of the agent harness itself.

The first layer was Model Context Protocol servers. We started by wiring up the MCPs that gave the agent access to the systems our developers were already living in: a GitHub MCP for issues, pull requests, and repository context; a Zenhub MCP for our project management workspaces; and a Context7 MCP for up-to-date third-party documentation. These were the context bridges that let the agent operate over our real workflows without drowning the model's context window in raw source.

Once the context bridges were in place, the next problem was that the context itself was expensive. We added two optimization MCPs on top of the initial layer: codegraph, a tree-sitter-based semantic code intelligence service that runs locally, answers symbol queries, callers, callees, and impact analyses in sub-millisecond lookups, and in our measurements produces approximately 35% cheaper agent runs and around 70% fewer tool calls by keeping the agent out of the expensive 'read every file' mode; and context-mode, which indexes CLI output into a searchable local knowledge base and keeps raw command output out of the context window entirely. The ordering matters: the initial MCPs solved context access; the optimization layer solved context cost.

The second layer was specialized skills. We began codifying our engineering standards into structured, deterministic instructions—what we call "skills"—that the agent loads on demand. Some of those skills started as public work by Adi Osmani, Matt Pocock, and Matteo Collina, and we adapted and extended them; others are our own. Each skill is a small, focused body of instructions: how to do test-driven development here, how to run a security review, how to profile a performance regression, how to deprecate an API safely. Today the toolkit ships twenty skills across six categories—Build, Review, Verify, Ship, Productivity, and OpenSpec—and the difference is visible in every code review. Our agents are no longer writing generic JavaScript; they are writing NodeSource-standard JavaScript, with the same conventions a senior engineer would apply.

The commercial scale-up and the vendor bottleneck

To power these capabilities, we scaled up our usage of cutting-edge commercial models, utilizing systems like Codex and other closed-source endpoints. The capability was there; the operational story was not. As our developer activity surged, we ran into three severe challenges that threatened our velocity, and they are the same three challenges every organization using these tools at scale is going to hit.

• Obscure token costs. LLM billing quickly became a black box. A single developer run could inadvertently balloon the context window, and a single bloated context could multiply into a six-figure invoice by the end of the month. We could not predict our infrastructure costs from one sprint to the next, and predictability is what allows an engineering organization to plan.
• API rate limiting. During high-throughput periods our CI/CD pipelines and interactive developer sessions would frequently hit strict vendor rate limits, halting development loops entirely. A failed agent run in the middle of a code review is not just an inconvenience; it is lost context, lost state, and lost trust.
• Vendor lock-in, in the form of model-tier churn. The model we used last quarter is rarely the model we want next quarter. Every time a provider releases a new tier, it offers a real capability bump, but it also brings a new rate-limit regime and a new price curve, and our existing agent runs have to be re-tuned against the new envelope. Constant policy changes on the provider side compound this. We were paying for capability, but we were also paying for the instability of having the layer that owned our velocity change shape without our input.

We realized that to scale AI-native product engineering safely, we could not build our future on rented, locked-down infrastructure. We needed complete transparency, predictable performance, and total control over our tooling stack.

Reclaiming control with an in-house orchestrator

To break free from vendor lock-in and eliminate black-box costs, we developed an internal orchestration utility—a TypeScript CLI that installs, configures, profiles, and manages an open-source-aligned agentic stack locally on every engineer's machine. It is built on top of OpenCode, the open-source terminal-based AI coding agent, and it manages six configuration manifests—base settings, provider authentication, npm-package dependencies, external CLI binaries, skill definitions, and MCP server configurations—through a clean layered design with an entry point to command routing to handlers to library modules, with no leaking abstractions between layers.

The orchestrator resolves our scaling challenges through five core innovations.

• Local semantic code intelligence, treated as a default. The optimization MCPs previously introduced are wired in by the orchestrator during setup, so every agent run has sub-millisecond symbol queries, callers, callees, and impact analyses available without the engineer having to think about it. The orchestrator's expectation is that agent runs will not pay for redundant tool calls; the optimization layer that makes that possible is just there.
• Token compression, transparently wrapped around the shell. rtk—a Rust binary—sits between the agent and the shell for every command the agent runs. The orchestrator wires it in during setup, so neither the engineer nor the agent has to invoke it explicitly; the compression is just part of how the shell behaves inside an agent run. In our measurements this reduces LLM token consumption by sixty to ninety percent on the commands that agents run most often.
• Profile-aware provider independence. The orchestrator manages multiple OpenCode configurations as named profiles, and an engineer can switch between them with a single command. This allows our team to use state-of-the-art commercial models for high-level system design work and lightweight secure open-source models for continuous tasks like linting, testing, and debugging, all behind the same CLI surface and without rebuilding configuration from scratch each time. The result is that the choice of model becomes a configuration decision, not a workflow decision.
• Hybrid model routing through OpenRouter. The orchestrator standardizes on OpenRouter as the model gateway, which means the choice of provider becomes a routing decision rather than a wiring decision. Our current bias is toward open-source models that rank very high on independent coding benchmarks—see, for example, the best-AI-for-coding leaderboard— because the cost per task on a strong OSS model is now within a small multiple of a frontier commercial model, and the OSS path removes the vendor-tier-churn problem entirely. We still use commercial frontier models when a task genuinely earns the cost—complex architecture work, security review of high-stakes changes, and performance regression analysis—but those are exceptions, not the default. The orchestrator's model-curator plugin selects the appropriate tier per task class so the engineer never has to choose manually.
• Continuously synchronized tooling. The orchestrator's CLI is not just for first-time setup. It pulls updates and pushes configuration changes across the team, so when a new MCP or skill is added or a tool is deprecated, every engineer stays aligned with the current state without having to read a release note. The delta engine computes what changed since the last install and applies only the necessary changes. This is the operational expression of the ownership principle: the version of every tool in the stack is something we control, not something that drifts.

One stack, four developer experiences

From the moment we started rolling the agentic stack out across the team, we knew our developers did not all want to interact with the same kind of interface. Some of our engineers have built their careers in the terminal, and they want a TUI. Others came from an IDE background, and they want a sidebar, a file tree, clickable links to line numbers, and visual diff review. We prepared for that split in advance because we had been through enough tool-rollouts to know that forcing a uniform interface onto a team that disagrees about interfaces is its own form of lock-in and a worse one because it is self-imposed.

The answer we landed on is OpenChamber, an open-source companion to OpenCode that exposes the same underlying agent over four distinct front-ends. There is the OpenCode TUI for engineers who prefer the terminal. There is a desktop application built on Tauri for macOS, a web/PWA interface accessible from any browser, and a VS Code extension that embeds the experience directly in the editor. All four surfaces talk to the same OpenCode server over HTTP and server-sent events through the open OpenCode SDK, which means that a session started in the terminal can be picked up on a tablet, reviewed on a phone, and returned to the desktop, all without losing context.

The architectural lesson here is that the agent runtime should be separate from the agent interface. Once we made that split, the question of which UI to ship stopped being a strategic decision and became a personal preference. Engineers pick the surface that fits their workflow, and the orchestrator underneath does not care which one they chose.

Layering OpenSpec on top: spec-driven development

By this point we owned the runtime, the configuration, the front-end surfaces, the model routing, and the integration with our project tracker. The remaining problem was the workflow itself. Agents were still acting on unstructured conversational threads—useful for one-off questions, fragile as the canonical record of a change. A pull request could land without anyone being able to point at the proposal it was implementing, and a task list lived entirely in chat context until the chat scrolled away.

Before OpenSpec, we had been using Plannotator for review. Plannotator is a browser-based review UI that supports inline comments, PR-style diffs, and structured feedback, and it was the first place we got serious about letting humans and agents collaborate on a plan before any code was written. We still use Plannotator today, but it is the review surface; it is not what drives the planning. The thing that drives the planning is OpenSpec.

OpenSpec is a structured proposal system, distributed as @fission-ai/openspec and wired into the orchestrator via the dependencies manifest. It generates four artifacts for every change before any code is written:

• A proposal that frames the problem and the proposed solution.
• A design that lays out the architecture and the trade-offs.
• Scoped specifications that pin down behavior and interfaces.
• A task list that drives the implementation from start to merge.

Engineers and agents collaborate on those artifacts in Plannotator, where reviewers leave inline comments that flow back to the agent. Once a change is approved, the implementation runs as a sequence of tasks that an agent can pick up, work through, and verify locally against isolated test suites. Completed tasks sync directly to ZenHub, our project management tracker, where they become epics and issues automatically.

The result is that every change, regardless of who or what produced it, has a paper trail that begins as a proposal and ends as a merged artifact with a corresponding ticket in the tracker. Plans are verified locally using isolated test suites, and the artifacts the agent operates against are explicit, not implicit. That single shift—from 'the agent just does things' to 'the agent operates against an explicit, version-controlled spec'—is what made spec-driven development practical for us and what we believe will make it practical for any organization willing to make the same investment in their own tooling layer.

Observability: tagging every agent call

Owning the routing layer made the cost question answerable in principle, but it did not, on its own, make the cost answerable in practice. To know what we were actually paying for, we had to attach telemetry to every agent run, and that meant deciding where the tagging happens. We do it at the orchestrator's call sites. Every time the orchestrator wraps a call to the model, it pushes a trace to PostHog annotated with the context that turn matters: which skill was in play, which OpenSpec change was being implemented, which task within that change was currently running, which profile and model were selected, and which developer initiated the run.

Tagging at the call sites, rather than at the OpenRouter boundary, means the trace payload carries the why-it-happened dimensions that a raw provider log never sees. An OpenRouter trace can tell us that we spent thirty dollars on a particular model run; an orchestrator trace tells us that we spent thirty dollars on the security review of the auth refactor and on the implementation of the rate-limiter task while running the open-source model under the cost-default profile and that the developer who initiated it is on the backend team. That context is what makes the dashboards actionable.

We built custom dashboards in PostHog on top of those traces, and the slicing is what we actually look at. Cost per feature area tells product engineering where the AI spend is going relative to the value the feature delivers. Cost per project lets program managers catch overruns before they become invoice shocks. Cost per team surfaces whether one team is over-relying on the agent for work that another team has automated or does manually. Cost per developer is the most politically sensitive cut of the four, but it is also the one that catches patterns—a single developer who is consistently running larger contexts than their peers, or who is using the frontier model for tasks the OSS model would have handled at a tenth of the cost.

The principle behind this layer is the same one that runs through the rest of the stack: the closer a layer is to the work, the more important it is that we own its visibility. We did not want to wait on a vendor's dashboard to understand our own spend. We wanted the data to live in a system we control, shaped around the questions we actually ask, and queryable in the same way we query every other operational signal at NodeSource.

What we learned

It is essential to have observability in place. Still, most of the time, it comes with a very high price. The same lesson applies to the agentic tooling we depend on: the closer it is to the work, the more important it is that we own it. We learned this by paying for it, in unpredictable invoices, in stalled CI runs, and in deprecation notices we had no control over. The MCP context layer, the optimization layer on top of it, the orchestrator, the multi-surface front-ends, the hybrid model routing, the spec-driven workflow, and the observability layer on top of all of that were each one decision made in response to a wall we had run into, not a plan executed from scratch.

The decision to own the orchestration layer was not a bet against any specific vendor; it was a decision to stop renting the layer that had become load-bearing for our engineering velocity. We kept the parts that work — the open-source OpenCode runtime, the Rust compression binary, the upstream MCP servers, the OpenSpec workflow — and we replaced the parts that did not, namely, the dependency on a single commercial provider and the configuration drift that came with it.

If you are running into the same walls we were, the recommendation is straightforward: identify the layer in your agentic stack that, if it disappeared tomorrow, would halt your development loop, and start there. For us, that layer was the orchestration utility, the configuration manager, the multi-surface front-end, the model router, the spec-to-tracker bridge, and the telemetry that ties all of them together. For you it might be different, but the principle is the same. Reclaim the layer that matters most, and the rest of the stack will follow. At NodeSource, the same discipline is what shapes how we approach our broader product surface, from N|Solid down to the smallest dependency in a customer's application: own the layer that owns your velocity.