My Local Coding Model Wrote the Tests Too. I Couldn’t Use All of Them.
My Web build passed on November 11. The game loaded, all five starting characters were selectable, and the basic journey checks passed. I still couldn’t say that all five new abilities worked.
Some of the generated checks weren’t usable. My Qwen-assisted workflow had produced mismatched field names, missing timer calls, and type or call-order errors across implementation and verification work. The successful build didn’t turn those incomplete checks into evidence.
That’s the failure I want to keep visible. I’d already paid for the extra review and regeneration in work, even though I don’t have a reliable time total for it.
What had actually passed
I was adding distinct active and passive abilities to five starting characters. Their shared flow was familiar: choose a character, enter combat, select upgrades, retry, and reconnect. The ability-specific behavior was new.
The November 11 record separates those layers. Desktop and mobile browser emulation covered the shared flow. Basic attacks had seven detailed conditions checked. One character’s new ability had six conditions checked. The remaining abilities still needed detailed verification.
| Evidence on November 11 | Recorded result | Claim it supported |
|---|---|---|
| Web build | Success, zero build errors | That revision compiled |
| Existing journey rules | 146 checks passed | Existing rule checks passed |
| Basic attack checks | Seven conditions passed | Those attack conditions worked |
| One character’s ability | Six conditions passed | Those ability conditions worked |
| Remaining detailed ability checks | Incomplete; flawed checks excluded | No full ability-coverage claim |
I didn’t add these numbers together into a coverage percentage. They count different things. A rule assertion isn’t equivalent to an end-to-end ability scenario.
I also didn’t treat choosing a character as proof that its timer, shield, or cleanup behavior worked. I’d verified the doorway into the feature. That wasn’t the whole feature.
A test can be wrong before its assertion runs
A check using a field that doesn’t exist can’t establish behavior. A check that doesn’t advance the relevant timer can’t answer a timing question. A call-order mistake can make the test exercise a state the real flow never reaches.
My notes record these categories, but they don’t give a clean defect count for each one. I won’t invent a table of model error rates from a work log.
The important decision was exclusion. Checks still carrying errors weren’t integrated into the game. I didn’t lower the acceptance criteria just to get a green result. Product corrections went back through smaller generation requests, review, and verification.
I use a local model to help write code. I don’t get an independent judge merely by asking that same workflow to write another file called a test. Independence has to come from the acceptance conditions and the observed result.
What changed in the later record
The work didn’t stay frozen at that partial state. On November 12, the final verification record identified a later build and matched 657 source files with no listed mismatches. Web and Windows builds exited successfully.
Its recorded browser checks included desktop and mobile execution, ability regression, specialization regression, and shared-trait regression. Those results belong to that later revision. They don’t retroactively make the November 11 checks complete.
I keep the revision beside the result because otherwise a status summary can quietly combine incompatible moments. “The build passed” from one revision and “the ability passed” from another aren’t automatically evidence for the package I’m about to publish.
The final record still separates real phones, native Windows play, and human audio evaluation from browser-based verification. I haven’t repeated those development tests for this article.
Don’t make the checker agree with the implementation
When a check fails, I need to identify whether the product, the check, or the setup is wrong. I can’t decide that from the red result alone. I also can’t change the expected behavior merely because the generated code chose something else.
My practical review starts with a plain-language contract: what should happen, what must not happen, and when the effect must end. Then I compare the test’s setup and observations with that contract.
For a timed ability, a visible effect isn’t enough. I’d want the start, duration, expiration, and interruption paths represented. That’s a review framework, not a claim that every boundary was covered in this incident.
My earlier mobile-menu test failure concerned a checker that accepted broken behavior. This case stopped earlier: some generated checks weren’t valid evidence at all. Both taught me to inspect what the green label actually means.
I’ve also described splitting an unfinished local-model request. Smaller patches helped review there. They aren’t a substitute for deciding, independently, what a correct result must do.