A language model is a prediction tool. It predicts the next token, then the next. That stream is the output: the diff, then a summary of what it just changed.
The question is not whether that output can look right. It can. The question is this:
How can we possibly predict the consequences of that output, given that the thing producing it is a prediction tool?
Consequences are what happen after you accept the output. Other code still wired to the previous shape. Data the model was never given. A failure that only exists once the change is live. Those facts are not required in order to predict the next token.
So the model can produce the output and still have no prediction of what the output does.
That is the gap. Two jobs. Predicting the next token is the model's job. Establishing the consequences of the output is not.
You are the developer watching this in the IDE. The agent writes. Then it tells you what it did, as if the task is finished. That close is the implied prediction of consequences. Nobody has to say "this is safe." Done is enough.
The usual answer, inside the loop, is another pass. You take the output and send it through the model again. You add a pre-prompt, evals, a rules file. You ask what the first pass missed. Agent writes. Agent reviews. Independent review is not empirical validation.
The role has changed. The model has not. It is still a prediction tool. The extra text conditions the next tokens. The verdict is another output: a prediction of what a review sounds like given the rubric. It is not an observation of consequences. Consequences still sit outside the model.
Evals and rules do not change what the model is doing. They change which continuation is likely. A strict rubric makes a strict-sounding paragraph more likely. It does not put production in front of the model. It does not put next week in front of the model. Recirculating the first output as the second prompt adds the first prediction to the context window. It does not add the world.
A test run, a compiler, staging, telemetry: those are not predictions. They are observations of what the output did, in a place the model was not given. This is not a catalog of them. The habit worth naming is treating a second prediction as if it were one of them.
A prediction model cannot establish the consequences of its own output merely by generating another prediction about that output. If the only evaluator of a prediction is another prediction, you never leave the loop. You can get a better-looking output. You cannot, from inside that loop, claim you predicted what the output will do.
That is the thought worth sitting with. Not "the model is wrong." Not "add more agents." The tool you are using to produce the change is the same class of tool you are using to decide what the change will cause. Until that is visible, a finished-sounding summary will keep passing for a prediction of consequences.