Android screenshot goldens are order-sensitive: inserting a case moves its neighbour #235

New issue

Open

opened 2026-08-25 02:20:52 +00:00 by nalum · 0 comments

nalum commented

2026-08-25 02:20:52 +00:00

Owner

Found while adding one conformance case to render-job-sheet.json.

Adding a case in the middle of the array made an unrelated, untouched golden
fail — jobsheet-repeat-sheet-hearth-dark.png — with a bounded pixel difference
on the ⋯ button of the sheet behind the overlay. Establised by A/B, not
assumption:

remove the new case entirely → the repeat-sheet golden renders clean
keep the case but move it to the end of the array → the repeat-sheet golden is
clean again, and the new case's own freshly-recorded golden diverges instead

So the perturbation follows whichever case is adjacent. It is not the content of
any case.

Why

ScreenshotTest.shootHearth hosts every case from one fixture inside a single
persistent compose.setContent, flipping a current index and calling
captureRoboImage per shot. The subtree is wrapped in key(name), which should
tear down and rebuild it — but in practice a few dozen sub-pixels of the next
shot's stroke rendering depend on what was hosted before it.

The web equivalent has no such coupling: screens.spec.ts does a full
page.goto() per case, and the same reordering showed zero effect there.

Why it matters

The current workaround is a convention: append new cases, never insert. That
works and is what the make-it-part-of-another-job case does, but it is invisible,
unenforced, and one careless insertion away from a failure that looks like
machine drift — which is exactly how it was first misdiagnosed.

Worth fixing properly: give each shot its own setContent (or its own test
method) so a case's rendering cannot depend on its neighbour. Until then the
convention should be written into conformance/README.md and AGENTS.md's goldens
section, because a future author will hit this and reasonably conclude the
machine is at fault.

The trap that hid it

verifyRoborazziDebug without --rerun-tasks can return a cached BUILD
SUCCESSFUL that ran no tests at all — a passing result for a state that fails.
Any golden verification that matters needs --rerun-tasks.

Found while adding one conformance case to `render-job-sheet.json`. Adding a case in the middle of the array made an **unrelated, untouched** golden fail — `jobsheet-repeat-sheet-hearth-dark.png` — with a bounded pixel difference on the `⋯` button of the sheet behind the overlay. Establised by A/B, not assumption: - remove the new case entirely → the repeat-sheet golden renders clean - keep the case but move it to the end of the array → the repeat-sheet golden is clean again, and the **new** case's own freshly-recorded golden diverges instead So the perturbation follows whichever case is adjacent. It is not the content of any case. ## Why `ScreenshotTest.shootHearth` hosts every case from one fixture inside a single persistent `compose.setContent`, flipping a `current` index and calling `captureRoboImage` per shot. The subtree is wrapped in `key(name)`, which should tear down and rebuild it — but in practice a few dozen sub-pixels of the next shot's stroke rendering depend on what was hosted before it. The web equivalent has no such coupling: `screens.spec.ts` does a full `page.goto()` per case, and the same reordering showed zero effect there. ## Why it matters The current workaround is a convention: **append new cases, never insert**. That works and is what the make-it-part-of-another-job case does, but it is invisible, unenforced, and one careless insertion away from a failure that looks like machine drift — which is exactly how it was first misdiagnosed. Worth fixing properly: give each shot its own `setContent` (or its own test method) so a case's rendering cannot depend on its neighbour. Until then the convention should be written into `conformance/README.md` and AGENTS.md's goldens section, because a future author will hit this and reasonably conclude the machine is at fault. ## The trap that hid it `verifyRoborazziDebug` without `--rerun-tasks` can return a cached BUILD SUCCESSFUL that ran no tests at all — a passing result for a state that fails. Any golden verification that matters needs `--rerun-tasks`.

nalum added the

labels

2026-08-25 02:20:52 +00:00

nalum referenced this issue

2026-08-25 02:21:24 +00:00

test(jobs): pin the make-it-part-of-another-job sheet #237

nalum referenced this issue

2026-08-26 18:47:56 +00:00