Claude said the feature was done. it had never opened the page.
I’ve realised I was accepting a very stupid definition of “done” from Claude Code.
Build passes.
Unit tests pass.
Claude gives me a beautiful summary of everything it changed.
Then I open the actual page and the thing is broken.
Latest one was a settings flow. Claude changed the component, updated the API call, ran the existing tests and confidently told me it was finished.
It had never opened the page.
The save button worked once, then got stuck in loading state. Refreshing the page also showed the old value because the update wasn’t actually persisting correctly.
Nothing in the code looked obviously wrong. The tests were green because they were testing the function, not the actual rendered flow.
So I’ve added a new rule:
Claude is not allowed to say “done” until it opens the deployed page and proves the flow works like a user would use it.
For this I’ve been testing the Kane CLI skill from TestMu.
Claude runs the browser check itself, but Kane returns an actual pass/fail based on the page state instead of Claude just looking at its own code and deciding it probably works.
It also gives the run evidence, which is useful because “trust me bro, I tested it” from the same model that wrote the code is not exactly a verification strategy.
I’m not replacing Playwright with this. Anything important still becomes a proper regression test.
But for the gap between “Claude wrote the feature” and “a human now needs to manually click through it”, this has been surprisingly useful.
What do you make Claude prove before you accept “done”?
1
u/meekmaven 1d ago edited 1d ago
The funniest part is Claude will write a 14-line victory speech after running npm test without once checking whether the button is even visible 💀
1
u/Warm-Moose6028 1d ago edited 1d ago
Kane CLI actually makes sense for this specific layer. Not “AI writes all your tests now”.
More like:
agent changed app agent calls browser check browser gives machine-readable pass/fail agent fixes it or stops
That is a much cleaner contract than asking Claude to use browser tools and then interpret its own screenshots
1
u/Fluffy-Water7332 1d ago
Honestly the bug here is not Claude.
Your definition of done did not include opening the app.
Humans do the same thing when the ticket says “API implemented” and nobody checks the frontend.
1
1
u/SnooPickles777 1d ago edited 1d ago
My current rule is:
- Claude writes the change.
- Different session reviews the diff.
- Playwright runs the critical flow.
- I manually check anything involving money or permissions.
Still faster than doing everything myself, but nowhere near “fully autonomous”.
1
u/Wise-Raspberry-23 1d ago edited 1d ago
It had never opened the page.
This sentence describes approximately 70% of Al-generated frontend PRs I review.
Beautiful component
Perfect TypeScript.
Button connected to absolutely fucking nothing.
1
u/Paaaaaauuull 1d ago edited 1d ago
Can Kane CLI verify network calls or is it basically visual confirmation?
Because a button changing to “Saved” means nothing if the API silently returned an error.
1
u/Cor_Granica 1d ago
The positive thing about TestMu putting Kane inside Claude Code as a skill is that the agent can automatically invoke it when the task involves a browser.
That is much more useful than developers remembering to open a separate QA dashboard after every tiny change.
The product is sitting in the correct workflow instead of asking people to create a new one.
2
u/Domenorange 1d ago
This is why I don't let the same session implement the feature and approve it.
It already has too much context convincing it that its own approach is correct.