The thesis: tokens are cheap, verified judgment is scarce

I’ve spent about twenty-five years building ML systems, mostly recommendation and personalization for media companies, and every one of them promised to learn from its users. I meant it every time. The loop was real. It just ran quarterly. Feedback came in, somebody triaged it, it got prioritized into a roadmap and scheduled into a sprint and shipped a release or two later, and by then the person who gave the feedback had moved on and half of what they said didn’t apply anymore. For a long time I assumed the bottleneck was the model, or the data. It wasn’t. It was engineering calendar time.

That bottleneck is mostly gone now. A coding agent can take a day’s worth of corrections and have a new version by morning, and the cost of that scales with dollars and how much feedback you have, not with how many engineers you can hire. So the loop that ran quarterly can run daily. That sounds like a speed improvement. I think it’s actually a change in what the scarce input is, and that’s the thing I keep coming back to. Everything I’m doing with Signal Foundry comes out of it.

Tokens are cheap

AI will generate as much work as you want. Drafts, matches, flags, classifications, candidate answers of whatever kind. Want ten times more attempts? Buy them. The models don’t get tired and the price keeps dropping.

What you can’t buy is someone saying which of those attempts are right. For work that has to be right, an attempt without a verdict isn’t worth anything. Somebody who knows the domain has to look at it and say yes, or no, or “close, but here’s the fix.” And that capacity doesn’t scale with dollars. It’s a small number of people, they’re hard to grow, and every hour of their time already has three other claims on it.

So the constraint on a serious AI system isn’t compute. It’s validated judgment. Once I started seeing it that way a lot of the standard playbook flipped for me. The question stops being how do we generate more and becomes how do we get the most out of every hour an expert spends looking at output.

The judgment is with the people doing the work

Related point. The judgment you need isn’t in the documentation and it isn’t in the manager’s description of the job. It’s in the people running the workflow today. The calls they make, the exceptions they carry around in their heads, the standard they hold the output to. Some of that they can articulate. A lot of it they can’t, until they’re looking at a wrong answer and saying “no, that’s not it, because…”

Which is why I’d rather start an engagement sitting next to those people than writing a spec. Requirements gathered from the people doing the work describe the job as it is. Requirements gathered anywhere else describe the job as somebody remembers it. I’ve done it the second way. It’s not great.

Ground truth

When an expert judges a proposal with one click, agree, fix, or reject, that’s data. Enough of those and you have ground truth: the only thing you can honestly measure the system’s accuracy against, and the bar every change has to clear before it ships.

The fixes are the valuable part. A fix tells you exactly where the proposal went wrong and what right looked like instead. That’s a lesson, not just a grade. And after a few weeks the record starts to describe the problem itself. Which item types are mechanical. Which ones have hidden context. Which ones two reasonable experts would disagree on. You can’t buy that map, because it doesn’t exist until your own people draw it. If you’re automating a workflow without it, you’re automating on faith. Plenty of people are, and I get why. It’s faster to skip. It just doesn’t tell you anything.

Judgment unlocks the spend

Here’s the part I think most AI projects have backwards. Compute is elastic. Confidence isn’t. What actually limits how much of a workflow you can hand to machines is whether anybody can prove, with evidence, that the machines do it to the standard the work requires. Each verified judgment extends that proof a bit further, so the machines can take on a bit more, so the experts spend their time on harder cases. Human judgment unlocks the compute spend. The compute spend makes the judgment that’s left worth more.

This is why I’d argue experienced people get more valuable as you automate, not less. They stop doing the repetitive 80% and become the judges of the hard 20%. The machine handles volume. They set the standard and check it against real numbers. “Promoted, not replaced” is a slogan, I know. The numbers are what make it more than that.

The plateau

Run the loop daily and improvement is fast at first, then slower, then it flattens out. The flattening is not the method failing. It’s the answer. The plateau shows how much of the workflow machines can take at current capability, and how much genuinely needs a person. Below the line you automate with evidence. Above it you staff with confidence, because you can show exactly why those items need a human.

I don’t know where that line is for any given workflow. Nobody does until it’s measured, and I’m suspicious of anyone who says otherwise. What I do know is that the line moves. When the models improve, the same ground truth re-measures the frontier in an afternoon. That’s the quiet payoff of treating judgment as a dataset. Every capability jump in the industry turns into a question you can answer about your own workflow, that day, with your own record.

So

If tokens are cheap and verified judgment is scarce, then what’s worth selling isn’t hours. It’s verified outcomes, with the audit trail that proves them, from a loop that treats expert attention as the budget. That’s what I’m building. The mechanics are on the how-it-works page and the shape of an engagement is on the services page. If your team has a workflow where somebody answers “is this right?” a hundred times a week, I’d like to hear about it: hello@signalfoundry.ai.