Prabal Tripathi

The Test That Only Fails in Someone Else's PR

The most valuable test in a codebase is the one that breaks for a stranger.

Here's a way to rank the tests in a codebase that I've found more useful than coverage: who does this test fail for?

Three tiers

Fails for you, immediately. A unit test you run while writing the function. Genuinely useful — it's a fast feedback loop — but it's really a development tool. You'd have caught most of these anyway, a bit slower.

Fails for you, later. A regression test that catches you breaking something you wrote six months ago. More valuable, because six-months-ago you is effectively a different person with worse documentation.

Fails for someone else. A test that goes red in a pull request written by a person who has never seen this file, in a different repository, who is not thinking about your feature at all.

That third one is doing something no code review and no amount of care can replicate. It's catching an interaction nobody held in their head.

Why this reframes a lot of arguments

Once you rank tests this way, some long-running debates get shorter.

Is this test worth the maintenance? Depends entirely on tier. A tier-three test that's awkward and slow can still be worth real pain, because the thing it prevents is a regression that ships silently. A tier-one test that's awkward is usually just awkward.

Should we mock this dependency? If you mock it, you've demoted the test. It can no longer fail for the person who changes the dependency, which was the whole point.

Why is this suite so slow? Because tier three requires real things. Real database, real auth, real network. The slowness isn't incidental — it's the mechanism.

The uncomfortable implication

Most suites I've seen are overwhelmingly tier one, with a thin layer of tier three that people are quietly trying to delete because it's flaky and slow.

That's backwards, and it's an understandable mistake, because tier one is pleasant and tier three is a nuisance until the exact moment it saves you.

The flakiness is a real problem. The answer is to fix the flakiness, not to demote the test.