The thesis: tokens are cheap, verified judgment is scarce

I’ve spent about twenty-five years building ML systems, mostly recommendation and personalization for media companies, and every one of them promised to learn from its users. I meant it every time. The loop was real. It just ran quarterly. Feedback came in, somebody triaged it, it got prioritized into a roadmap and scheduled into a sprint and shipped a release or two later, and by then the person who gave the feedback had moved on and half of what they said didn’t apply anymore. For a long time I assumed the bottleneck was the model, or the data. It wasn’t. It was engineering calendar time.

That bottleneck is mostly gone now. A coding agent can take a day’s worth of corrections and have a new version by morning, and the cost of that scales with dollars and how much feedback you have, not with how many engineers you can hire. So the loop that ran quarterly can run daily. That sounds like a speed improvement. I think it’s actually a change in what the scarce input is, and that’s the thing I keep coming back to. Everything I’m doing with Signal Foundry comes out of it.

Tokens are cheap

AI will generate as much work as you want. Drafts, matches, flags, classifications, candidate answers of whatever kind. Want ten times more attempts? Buy them. The models don’t get tired and the price keeps dropping.

What you can’t buy is someone saying which of those attempts are right. For work that has to be right, an attempt without a verdict isn’t worth anything. Somebody who knows the domain has to look at it and say yes, or no, or “close, but here’s the fix.” And that capacity doesn’t scale with dollars. It’s a small number of people, they’re hard to grow, and every hour of their time already has three other claims on it.

So the constraint on a serious AI system isn’t compute. It’s validated judgment. Once I started seeing it that way a lot of the standard playbook flipped for me. The question stops being how do we generate more and becomes how do we get the most out of every hour an expert spends looking at output.

The judgment is with the people doing the work

Related point. The judgment you need isn’t in the documentation and it isn’t in the manager’s description of the job. It’s in the people running the workflow today. The calls they make, the exceptions they carry around in their heads, the standard they hold the output to. Some of that they can articulate. A lot of it they can’t, until they’re looking at a wrong answer and saying “no, that’s not it, because…”

Which is why I’d rather start an engagement sitting next to those people than writing a spec. Requirements gathered from the people doing the work describe the job as it is. Requirements gathered anywhere else describe the job as somebody remembers it. I’ve done it the second way. It’s not great.

Ground truth

When an expert judges a proposal with one click, agree, fix, or reject, that’s data. Enough of those and you have ground truth: the only thing you can honestly measure the system’s accuracy against, and the bar every change has to clear before it ships.

The fixes are the valuable part. A fix tells you exactly where the proposal went wrong and what right looked like instead. That’s a lesson, not just a grade. And after a few weeks the record starts to describe the problem itself. Which item types are mechanical. Which ones have hidden context. Which ones two reasonable experts would disagree on. You can’t buy that map, because it doesn’t exist until your own people draw it. If you’re automating a workflow without it, you’re automating on faith. Plenty of people are, and I get why. It’s faster to skip. It just doesn’t tell you anything.

The cost of a decision

Here’s the arithmetic I use, because “mostly right” isn’t a number. The full cost of a decision the system makes is three terms added up.

cost of a decision  =  inference cost
                    +  share escalated          × review cost
                    +  share wrong, acted on    × cost of an error

The first term is the one everybody talks about and the one that matters least. Tokens are cheap and getting cheaper; for most decisions it’s a rounding error. The second term is your expert’s hour, times how often you send them the case. The third is the stakes: what it costs when the machine is wrong and nobody catches it. For work that has to be right, the third term is the whole game, and it’s the one you cannot measure without a record of what right looked like.

The knob is the confidence threshold. Set it low and you escalate little and act on more wrong answers. Set it high and you escalate everything and the experts drown. Where to set it isn’t a judgment call. It’s a calibration table: for each confidence bucket, how often the model agreed with a person. The lowest bucket that clears your bar is the threshold that has earned automation, and everything under it is your escalation rate.

Run the same buckets through the formula and the trade-off is plain. Each row is a place to set the knob. The share wrong and acted on is the term you pay for in errors; the share escalated is the term you pay for in expert hours. Put your own review cost and your own cost of an error against them and the cheapest row is the answer, for your workflow, this week.

Without ground truth you don’t know the share wrong. You guess it, usually optimistically, and the guess is the thing that blows up. With it, the formula is just arithmetic, and it re-runs every night as the record grows.

Judgment unlocks the spend

Here’s the part I think most AI projects have backwards. Compute is elastic. Confidence isn’t. What actually limits how much of a workflow you can hand to machines is whether anybody can prove, with evidence, that the machines do it to the standard the work requires. Each verified judgment extends that proof a bit further, so the machines can take on a bit more, so the experts spend their time on harder cases. Human judgment unlocks the compute spend. The compute spend makes the judgment that’s left worth more.

This is why I’d argue experienced people get more valuable as you automate, not less. They stop doing the repetitive 80% and become the judges of the hard 20%. The machine handles volume. They set the standard and check it against real numbers. “Promoted, not replaced” is a slogan, I know. The numbers are what make it more than that.

The plateau

Run the loop daily and improvement is fast at first, then slower, then it flattens out. The flattening is not the method failing. It’s the answer. The plateau shows how much of the workflow machines can take at current capability, and how much genuinely needs a person. Below the line you automate with evidence. Above it you staff with confidence, because you can show exactly why those items need a human.

I don’t know where that line is for any given workflow. Nobody does until it’s measured, and I’m suspicious of anyone who says otherwise. What I do know is that the line moves. When the models improve, the same ground truth re-measures the frontier in an afternoon. That’s the quiet payoff of treating judgment as a dataset. Every capability jump in the industry turns into a question you can answer about your own workflow, that day, with your own record.

So

If tokens are cheap and verified judgment is scarce, then what’s worth selling isn’t hours. It’s verified outcomes, with the audit trail that proves them, from a loop that treats expert attention as the budget. That’s what I’m building. The mechanics are on the how-it-works page and the tool is at tenali.dev. If your team has a workflow where somebody answers “is this right?” a hundred times a week, I’d like to hear about it: hello@signalfoundry.ai.