The problem
Every build is a chance to regress — and most game-testing tooling answers a weaker question than the one you're asking. Screenshot diffing says two frames resemble each other within a tolerance. Record/replay tools prove inputs can be fed back, not that outputs stayed the same. A green suite exit code says nothing about where behavior drifted. And "close enough" is exactly where the bugs that matter hide.
A run isn't verified because it looked right. It's verified because every observed value in a declared domain matched the golden sample — and the harness can point at the exact tick, pixel, or sample where it first didn't.
The brief: regression QA an agent can drive end-to-end over JSON, and a human can inspect afterward — same run IDs, same evidence.
What it does differently
GameQA treats a game as a deterministic machine under observation, not a video feed.
- Exact comparison channels, not similarity. Indexed pixels with their palettes, structured state fields with declared width/endianness, ordered device events, and digital audio samples — each under an explicit equivalence rule. A different palette index that renders the same color is not a divergence; a one-pixel mutation is.
- Immutable input timelines. The canonical timeline hash is the run's identity — timebase, same-tick ordering, device mappings, initial held state. Dropped, duplicated, late, or reordered inputs block an agreement claim instead of silently passing.
- Provenance-bound manifests. Every run binds adapter and executable hashes, config, media/resource hashes, and seed state. Mutated artifacts are detected; unknowns stay unknown. Imports can be analyzed but never masquerade as executions.
- Capability-honest targets. Builds declare launch/reset/stop, input injection, stepping, observation streams, and clock domains individually. No silent fallback from deterministic stepping to wall-clock. Exclusive leases stop two jobs from resetting the same target — material for adapters with global singleton state.
- First-divergence reports. The report names the stream, the tick, the field or pixel or sample, expected vs. actual, and hands you a replay command.
- Agent-native, human-readable. A versioned JSON CLI is the primary contract; MCP wraps the same service rather than duplicating it. Bounded pagination, idempotent submission, disconnect-recovery by run ID and cursor.
How it works
- Capture the golden sample once. Run a known-good build under a fixed input stream; GameQA records the golden run — frames, state fields, device events, audio — bound to a provenance manifest.
- Replay against every new build. The same input stream drives the candidate build through the identical timebase and ordering. Inputs are applied, not approximated.
- Compare declared domains exactly. Each observation channel is checked under its own equivalence rule — no tolerance laundering.
- Get a verdict you can act on. Agreement, first divergence with location and replay command, or insufficient evidence — never a silent pass.
How it compares
| Tool class | What it answers | Where GameQA differs |
|---|---|---|
| Screenshot diffing (Percy, Chromatic) | Do rendered frames look alike within tolerance? | Whether the machine did the same thing — exact pixels, palettes, state, events, audio; visual resemblance can't promote semantic coverage. |
| Record/replay & input players | Can the same inputs be fed back? | Whether they were applied identically — acknowledgements, ordering, and timing are compared, not just replayed. |
| Bespoke per-project harnesses | Does this one build behave? | A common job model across builds and targets via declared capabilities; nothing project-specific leaks into the contract. |
| CI test runners | Did the suite exit 0? | Clock/field/tick semantics, first-divergence forensics, insufficient-evidence as an explicit verdict. |
What it proved on real work
GameQA was exercised against a live project — a golden sample captured from a reference build, replayed against a reimplemented candidate. These are observed results, not marketing numbers:
- 5,511,697 states verified across a 522-field observation window; all 26 periodic clock/register/memory-hash observations identical between golden and candidate.
- 23,281 audio events matched — and a deliberately planted +1 timer error was caught at event 1.
- First divergence located at event 40,824 — 24 ticks early — where screenshot equality alone would have reported agreement.
- 64,000-pixel snapshot comparison: full equality on the positive control; 18,519 differences individually located on the negative.
- Audio forensics, not vibes: packet-level continuity reporting exposed three unflagged gaps (960/1920/480 samples) summing exactly to the observed correlation-offset shift.
None of these passes claim whole-game parity — partial coverage is reported as such. That honesty is the point.
Under the hood
- Rust core, FlatBuffers contracts — the schema is the cross-language authority.
- Versioned JSON CLI — the agent entry point; structured errors, terminal job states, resumable polling.
- Host-capture worker — Windows.Graphics.Capture video + process-loopback audio with packet-continuity diagnostics and endpoint gain metadata.
- Operational UI — jobs, evidence, and comparison reports inspectable by humans through the same run IDs agents use.