Resolution, not prompt engineering
We were convinced the prompt controlled stability in reference-to-video generation. Six controlled tests said otherwise, and the real lever was the setting nobody was looking at.
We had a reference-to-video pipeline producing output that was good about half the time. The other half drifted: the subject would shift, the motion would come apart partway through, or the last second would dissolve into something unrelated.
The obvious explanation was the prompt. It usually is. Most published advice on unstable generation is about prompt structure, and we had a list of things to try.
That list turned out to be irrelevant.
How we set the tests up
The pipeline had six or seven settings that could plausibly matter, and the temptation with a problem like this is to change three of them at once because you are in a hurry. Do that and a result tells you nothing. If output improves you cannot say which change did it, and if it gets worse you cannot say which change to undo.
So each run changed exactly one setting and held everything else fixed, including the seed. Same reference image, same prompt, same everything, one variable moved.
Six runs in, the picture was unambiguous. Prompt variations produced differences in content and almost none in stability. Resolution produced the opposite: the prompt barely mattered, and above a threshold the drift stopped.
Why this was easy to miss
Resolution does not feel like a stability setting. It feels like a quality setting, the thing you raise when you want a sharper result and lower when you want to save time. Nobody reaches for it when output is falling apart structurally.
It also interacts with the thing everyone does reach for. Lowering resolution to iterate faster on prompts is standard practice, and it means every prompt experiment runs in the regime where the model is least stable. You end up tuning the wrong variable using a method that guarantees the right one stays hidden.
We had done exactly that for a week before running the tests properly.
Run-to-run variance is why this is hard
Generation is stochastic. Change nothing at all, run the same job twice, and the two outputs differ. That is expected, and it is also what makes casual tuning unreliable, because any change you make will sometimes appear to help simply because the next run happened to be better.
Fixing the seed removes that. It turns the comparison into one between two configurations rather than between two rolls of a die. Most tuning we are shown has not fixed it, which is why it produces strong opinions and no numbers.
The other thing six runs turned up
These models read prompts more literally than people expect.
Ask for a camera move on a tripod and some of them will put a tripod in the shot. Not a hint of one, not a suggestion of camera equipment, an actual tripod standing in frame. The model has no concept that some words in a prompt describe how a scene was captured and others describe what is in it.
Shot terminology now stays abstract in our prompts for that reason. Describe the movement, not the equipment that would produce it. Which models do this and which do not is only discoverable by running them.
What the discipline actually buys
Roughly a week, in this case. We would have kept adjusting prompt structure, seen occasional improvement from run-to-run variance, read that as progress, and carried on.
More usefully, it produced a number rather than an instinct. There is a threshold, we know where it is on this pipeline, and it is written into a configuration that gets rerun rather than remembered. Six months from now the same input produces the same output, and when a new model version arrives it gets tested against the same task rather than judged on a release note.
The part worth generalising
Not the finding. The threshold is specific to one pipeline and probably does not transfer.
What transfers is the method, and the uncomfortable half of it: we started with a hypothesis, designed the tests to check it, and the tests said we were wrong. The finding only existed because the experiment was set up in a way that could contradict us.
Most pipeline tuning is not set up that way. It is set up to confirm what somebody already believes, three variables at a time, and it produces confidence rather than knowledge.