Stop being the check

What happens to a fix after you make it?

Here’s a story from one of my game jobs. Every morning I’d pull the latest code and assets, build the game and run it. One morning it crashed: out of memory. Somebody on the art team had checked in textures that weren’t scaled to the proper size, and they blew the memory budget. No big deal. I wrote an email to the tech artist, who wasn’t the one who checked them in but was responsible for art asset issues, and he fixed it fast. The game worked.

Two weeks later, it happened again, with a different texture. This time I added a check to the game itself, so an oversized texture wouldn’t crash it anymore. It would show a bad-texture overlay instead. I asked the art team to fix that one too, and they did.

Then it happened a third time. The game didn’t crash, thanks to my check. But there was the overlay, bold and hot pink (the most annoying colour I could find), for the entire team to see.

That’s when I went and found out why. The artists who submitted those assets never ran the game. They had their own tools, and pulling the latest code, building the game and testing their assets in it took too long to be part of their day. So the mistake was always made in a place where nobody could see it, and caught in a place where everybody could. I put the same check into their tool chain, and it flat out failed the submission if a texture was over budget. It didn’t happen to the game again. I’m sure it still caught bad textures, but it caught them before they ever reached anyone else.

Every one of those three fixes was right. The email fixed one texture. The check in the game fixed the crash. Only the third one fixed the place the mistake was made.

In October I caught myself back on round one, for two whole chapters of MoVox, together with Claude, on a project with exactly one person on it.

How it was supposed to work.

The plan for the picture side of MoVox goes like this. Local models read the novel, write the prompt for each picture, check their own work, and render. I approve each character’s portrait and spot-check the result. Claude, the AI I build MoVox with, writes the code and the tests.

That’s the plan. Here’s what chapters 37 and 38 actually looked like.

I reviewed every keyframe, every photo and every clip in the review tool, by eye. When something was wrong, I flagged it, and Claude fixed it. Not the code. The result. An override that says “in this scene, draw these people, doing this,” a direct edit to the database, a reroll with a new seed. Each one was the fast choice in the moment. A flag came in, a fix went out, the picture was right.

Here’s a typical one. In chapter 37, 刘备 rides past farmers singing in a field, and their song starts 苍天如圆盖，陆地似棋局: the sky is like a round lid, and the land is like a chessboard. The model wrote a prompt for the song’s pictures instead of the people singing it, so the frame came back as a chessboard under a dome sky. I asked for a redraw. It came back a chessboard again. It survived every regenerate, because a regenerate re-rolls the same prompt, and the prompt asked for a chessboard. So the fix was an override: draw the farmers hoeing and singing.

That fix was right. The picture is right. And it went into a file the model that wrote the prompt never reads.

136 hand fixes.

By October 24, that override file had 136 entries in it, and 62 of them were for chapters 37 and 38 alone. Each one records what the model wrote, what it should have written, and why. That’s a pretty good description of a lesson. Nobody was learning it.

The chessboard is one class of mistake: a poem scene draws the poem’s metaphors. There were more than twenty classes. Someone who’s only talked about gets drawn standing in the room. A word that isn’t a person gets cast as one: 荆榛 (thorn bushes), 当日 (“that day”). An action that’s really two things in a row (ride the donkey over the bridge, then get off) gets crammed into one picture. And chapter 39 would make every one of those mistakes again, because nothing upstream had changed.

So the honest diagnosis was three things, and none of them was “the models aren’t smart enough.”

First, my review wasn’t captured as data. A flag in the review tool said “photo,” not why. When we went back to score things later, 0 of the 22 photo flags had a note. The reasons were real, but they lived in Claude Code chat sessions, in commit messages, in the issue tracker, and in Claude’s own notes. None of that reaches a local model.

Second, the local models were never asked the easy questions. Most of my fixes weren’t really about pictures. They were judgments about text, made before any GPU work: is this person present in this scene, or just mentioned? Is this one moment, or a sequence? Is this word a person at all? A 35B model can answer those from the source sentence. No step in the pipeline asked it to. The only reviewer in the loop looked at finished frames with open-ended questions, and I’ve already written about how well that went.

Third, and this is the one I want to be honest about: our roles drifted. Claude was supposed to write the checks, and it had become the check. I was supposed to spot-check, and I was reviewing everything. Every fix was quick, so it never felt like a decision. It just felt like getting the chapter done.

Where does a fix live?

This isn’t an AI problem. It’s the texture story again.

Every hand fix we made was round one: an email to the tech artist. It worked, today, and it would have to be sent again tomorrow. So the question to ask every time you catch something is the one I got to on round three: where should this catch live, so it runs without me? And the best answer is usually as close as you can get to where the mistake is made.

That’s why most of the checks we built never look at a picture. The chessboard wasn’t a drawing mistake. It was a prompt mistake, made by a text model reading the poem, before any GPU work started. Checking the finished picture is checking inside the game. Checking the prompt is checking in the artist’s tool.

For MoVox it came out as three places, in order of preference:

1. A rule in code. Deterministic, tested, never tired. If the action contains “then” or “after that,” it’s a sequence, not one moment.
2. A narrow question to a local model. Not “is this picture good?” but a yes/no with the source sentence quoted: is this person present, or only mentioned?
3. A test case. Every one of those 136 overrides is a labelled example: here’s the scene, here’s what the model wrote, here’s the right answer. A new check has to catch the known cases in its class before it ships, and it can’t fire on scenes I’d already marked OK.

And one rule on top of all three, which is the one that changed how we worked: a lesson that stays only in prose doesn’t count as moved. If it’s in Claude’s notes but not in a rule, a prompt or a test, the pipeline hasn’t learned it.

The flags changed too. Every flag in the review tool now asks for a reason, one click from a list of the mistake classes plus free text, so my review turns into test data instead of evaporating. And Claude’s job changed back: when something gets flagged, fix the rule, prompt or check that missed it, then re-run the pipeline on that scene. An override is the last resort, and it counts as a failure of the pipeline, not a fix.

What the checks actually caught.

We built the first round of checks over one weekend, before chapter 38’s photos were made, and scored every one against what I’d actually done by hand. The results were mixed, and I think the mix is the useful part.

Some checks just didn’t work. Asking the model “is the setting where the sentences say it is?” caught 2 of 16 known cases, so it got dropped. Asking a vision model whole-picture questions about the photos was right 14% of the time it raised a flag. Simpler comparisons did better: same person, keyframe against photo, does the robe colour match? The photo checks we shipped are right about 61% of the time they fire and catch about half of what I caught. That’s a filter, not a reviewer, so they stay opt-in.

And one check taught us something about checks. The question “does each person’s headwear appear concretely?” fired on 27 of 30 scenes that were supposed to be fine. That looks like a broken check. It wasn’t. Seventeen character descriptions upstream had an “or” in them. 刘备’s said 头戴逍遥巾或裹巾: he wears a scholar’s cap, or a head wrap. So every single prompt for him carried an either-or, and every picture picked one. The check was right. The data was wrong. When a check fires on almost everything, look upstream first, because it may have found a defect at the source.

Where it ran out.

In the last post I said chapter 37 took weeks and chapter 38 took days. That’s true, and the retro is a big part of why. But I don’t want to leave you thinking the checks took over, because they didn’t.

After the retro, 12 more hand overrides went into the file, 10 of them for the second half of chapter 38. A lot of chapter 38 was still me looking and Claude fixing. The checks made the review faster and pointed at the right things more often. They didn’t replace it.

And my own half didn’t fully hold either. The review tool asks for a reason with every flag now, and I didn’t always pick one. Sometimes I was too rushed, sometimes I just forgot, and sometimes I couldn’t decide which category a mistake belonged in, so I skipped it rather than guess. Ten of chapter 38’s photo flags have no reason on them. That’s the art team’s problem again, with me as the art team: a step that takes effort at the exact moment you’re trying to get something through is the step that gets skipped. A list of categories you can’t choose between is friction, and the fix there is probably a better list, or letting me skip the category and keep the note.

The real test is the one written into the retro doc: run chapter 39 end to end with no hand fixes, then I review and we count. That hasn’t happened yet. The next work went to the voices instead. So I can tell you the method works on the cases we already had. I can’t yet tell you how much of a new chapter it carries.

The other half of the job.

授人以鱼，不如授人以渔: give someone a fish and you feed them for a day; teach them to fish and you feed them for a lifetime. Every hand fix in that file was a fish. A good one, delivered fast. But we had a pipeline whose whole point is to fish for itself, and we kept handing it fish.

So here’s the lesson I’d take into your own work, whether the reviewer is you, a lead on your team, or an AI agent. When you catch a mistake, you’ve done half the job. The other half is putting the catch somewhere it runs without you. And if you notice the same fix going in for the third time, stop and ask who’s supposed to be the check, because it probably isn’t you.