Prabal Tripathi

Record, Replay, and the Lie of a Stable World

Capturing real traffic to use as tests is a wonderful idea with one hard assumption underneath.

The pitch for record-and-replay testing is irresistible. Instead of writing test cases by hand, record what your service actually did — the request, the response, every call it made to its dependencies. Later, replay the request, serve the recorded dependency responses back, and compare.

You get tests from real usage. No fixtures to invent. Coverage of paths you'd never have thought to write.

It genuinely works. It's also built on an assumption that quietly fails, and understanding which assumption is most of the skill.

The assumption

That the recorded world is still a description of the current world.

A recording is a photograph of a conversation. Replaying it asserts that if you say the same thing again, everyone else says the same thing back.

That's true right up until a dependency changes a field, a token expires, a clock moves, or an ID gets regenerated.

The three failures, which look identical and aren't

When a replayed case fails you get a diff. The hard part is that three very different situations produce the same-looking diff:

A real regression. Your code changed and now does the wrong thing. This is what you wanted. This is the product working.

Drift. Nothing is wrong with your code. The recording is stale — the dependency legitimately changed, the response is correct, the recording is describing a world that no longer exists. The fix is to re-record, and "re-record" is not a defeat, it's maintenance.

Noise. Timestamps, UUIDs, durations, anything that is supposed to be different each run. If you haven't told the comparator to ignore these, every case fails forever and people stop reading the results.

Noise is the one that kills adoption. A suite that cries wolf on every run gets ignored within two weeks, and the regression it later catches goes unread with everything else.

What this actually demands

The thing nobody tells you: record-and-replay isn't a way to avoid thinking about your tests. It relocates the thinking.

You don't write assertions any more. Instead you decide what counts as different — which fields are semantic and which are incidental. That's the same judgment, moved. If you skip it, you have a very sophisticated machine for generating false alarms.

Where it shines

Refactors. You're changing how something is built, not what it does. You have hundreds of real interactions and a cheap way to assert the behaviour is unchanged.

That's the case where hand-written tests are weakest and recordings are strongest, and it's worth the maintenance all on its own.