Tracker Grew Up: Two Months of Making Agent Pipelines Boring
← All WritingWhen we introduced Tracker, it was a pipeline engine with a nice dashboard in your terminal. You write a workflow as a graph of steps in a .dip file, and Tracker figures out the order from the graph and runs independent steps at the same time. A step can run an LLM agent, run a shell command, or stop and wait for a decision. You can make that decision yourself, or hand it to an LLM playing a persona you choose. If a run stops for any reason, you pick it back up from where it left off. That’s still the core.
The releases since early summer, v0.38 through v0.56, have been about a different question: what would Tracker need before we could trust it to run unattended against production code and paid model APIs?
Four things changed.
1. Tracker became a library you can build on
The biggest change here is invisible if you only use the command line. Tracker’s engine can now be embedded in other software.
There’s a public event stream in NDJSON, which is newline-delimited JSON, one record per line. It carries the same data as Tracker’s own internal log: costs, which path each run takes, when approval gates open and close, and how many tokens each provider and model uses. An automated test keeps the public stream and the internal log in sync, so one decoder reads both. You can route an approval gate out to your own system, like Slack or email or a web app, instead of only answering it in the terminal. And you can watch a live feed of the model’s calls as they happen.
There’s even a conformance suite for front-ends. It’s a set of tests that spells out what a correct front-end has to do, so you can build one and check it without reading the engine’s source code.
2. Running out of provider credits no longer fails a run
A run that uses up your provider credits used to just fail. Now it enters a state called paused_billing, which you can resume. Top up your account, run tracker -r <run-id>, and it picks up from its last checkpoint. This holds even when a run has fanned out into several parallel branches. If one branch hits a quota, the whole run pauses cleanly, instead of letting a successful branch mask the failure.
Before a run, tracker estimate gives you a projection of what it will cost and how big it will get. During the run, every cost is tagged to the step that ran it and priced by provider and model. If you run a model our price list doesn’t know, you get a clear unpriced flag instead of a silent $0.00. The price list itself now records where each price came from, and a check at build time flags it if the published rates have drifted. That makes projected costs auditable before a pipeline goes into production.
3. Safety now depends on the situation
Tool safety used to be a simple allow-or-deny list, as if the risk lived in the tool itself. But the risk comes from three things together: the tool, its arguments, and the context in which it is called. So Tracker now runs a safety check before every tool call, and that check can see all three. It fails closed: when the check itself goes wrong, it denies the call rather than allowing it. Even if the check crashes outright, it blocks just that one call and returns the reason as the tool result, so the agent can choose another action.
When a step runs an agent, it now writes inside a jail on the file system, so it can only touch the files you allow. We enforce that with Landlock, a Linux feature for sandboxing file access, and we review any change to the jail against a nine-point checklist. A separate check watches for real edits to your files, so a step that’s spinning its wheels without making progress gets caught. And if a tool call comes through cut off partway, Tracker fails it instead of running half a command.
4. Reliability fixes
Some of the less visible changes address failure modes that only appear in long-running or resumed workflows. One involved the interaction between prompt caching and auto-compaction. Prompt caching, which reuses parts of a prompt to save money, was quietly switching off auto-compaction on the default settings. Auto-compaction is the safety valve that trims a run’s context before it grows too big to handle. To know when to trim, Tracker measured how full the context was. But that measurement ignored anything that had been cached, and caching covers most of it. So a context that was nearly full looked almost empty. Nothing got trimmed, the context kept growing, and eventually it hit a hard limit and the run crashed. Tracker counts the cached part now.
The v0.56 batch alone fixed seven bugs in how runs resume. After a crash, a resumed run could follow an old decision about which way to go instead of the current one. A check that had already passed could be re-run and wrongly come back as failed. And to save time, Tracker sometimes reuses a step’s earlier result instead of running it again, but it was reusing that result even after the inputs changed, so the answer came back stale. For each of these, we added a regression test and verified that it reproduces the old failure. We also run the Go race detector, a tool that catches bugs where two tasks touch the same memory at once, as a gate on every release.
We write all of it up in the changelog, and it runs long. Because Tracker can incur provider costs while running unattended, we’d rather you read about our bugs than trip over them.
Where this is heading
Taken together, these releases make Tracker easier to embed and safer to run unattended. Tracker is becoming an embeddable orchestration engine. It now tracks costs, runs a safety check before it acts, and can stop and pick back up. The next post is the first thing we built on top of it.
v0.56 is on the Homebrew tap. Run brew install 2389-research/tap/tracker, or use go install if you prefer. If an early limitation kept you from using Tracker, many of those limitations have been addressed, and tracker doctor can check your current setup and report remaining problems.
