Reversible or not

An assistant that answers a question and an agent that does something are different pieces of engineering, and the difference is not the model.

A wrong answer is embarrassing. A wrong action sends an email, issues a refund, or deletes a record, and no amount of apologising afterwards puts it back. Once a system can act on your behalf, the interesting engineering stops being about accuracy and starts being about permission.

The classification

Before any tool is written, every operation the agent might perform gets sorted into one of two groups: reversible, or not.

Reading a record, running a query, fetching a document, drafting something for review. All reversible, all able to run unattended, and any mistake shows up as wasted time rather than damage.

Sending, paying, deleting, publishing, submitting. None reversible. Each one stops and waits for a person, every time, regardless of how confident the model appears to be.

That is the whole rule, and it is unglamorous enough that people skip it. It also removes most of the risk in an agent that can act, because the catastrophic outcomes all live on one side of the line.

Confidence is not a signal

The tempting refinement is to let the agent proceed unattended when it is sure, and ask only when it is uncertain.

This does not work, because the confidence is not measuring what you want it to measure. A model that has misread a situation is frequently very confident about the misreading. The cases where it hesitates are usually genuinely ambiguous ones, which are not the same as the cases where it is wrong.

An irreversible operation either needs a person or it does not. That is a property of the operation, not of how the model feels about a particular instance of it.

Narrow tools, not convenient ones

The second decision that matters is what the tools look like.

It is quicker to give an agent one broad capability. A function that runs arbitrary queries against the database covers every case you might need and takes an afternoon to write. It also means the worst thing the agent can do is bounded by the database rather than by anything you chose.

Each capability gets its own function with a narrow, typed input contract instead. More work up front, and the payoff is that the set of possible actions is a list somebody wrote deliberately rather than a consequence of how much access was convenient at the time.

What happens when step three fails

Sequences fail partway. The agent updates the record, posts the notification, and the third call times out.

An agent that retries the whole sequence sends the notification twice. An agent that gives up leaves the work half done with no indication of where it stopped. Neither is acceptable and both are what you get by default.

So every run logs the full call sequence and its result, and session state persists. A halted agent resumes from where it stopped rather than starting again or abandoning the job. The logging is not for debugging, or not only for that. It is what makes resumption possible at all.

Who the log is for

Every run records the full call sequence and its result, and that log is not primarily for debugging.

It is what lets somebody answer the question that arrives three weeks later, when a record looks wrong and nobody can say whether a person did it or the agent did. Without that trail the honest answer is that you do not know, and the usual response to not knowing is to switch the agent off.

Handing back

The last boundary is the one that keeps the system honest. Where the next step needs judgement rather than execution, the agent stops and passes the case to a person with its reasoning attached.

Deciding what to do about the exception is a person’s job. Doing the ninety-five straightforward cases without being asked is the agent’s, and that division is where the value actually is.

Why this is written down

We build agents into systems people depend on, so the boundaries get agreed before anything is built rather than discovered afterwards.

The alternative is finding out where they should have been from an incident, and by then the conversation is not about design.