"That test is flaky" is one of the most expensive sentences in software, because it sounds like an explanation and functions as a decision to stop investigating.

Nothing is random. The test passes sometimes and fails sometimes because something differs between those runs. You just haven't found it.

The usual suspects

Almost every "flaky" test I've chased has been one of these:

A race you're usually winning. The test asserts after an async thing it doesn't wait for. On a fast machine it's there in time. On a loaded CI runner it isn't. Not random — load-dependent.

Order dependence. The test needs state another test creates, or is broken by state another test leaves. Change the order, or run in parallel, and it moves.

Shared mutable state. Two tests using the same record, the same account, the same name. Fine serially. A coin flip in parallel.

Time. Tests that break at midnight, at month boundaries, in a different timezone, during a leap second. These look mystical until you plot failures against a clock.

Accumulation. Passes on a fresh environment, fails once enough junk has built up. This one can take a year to show up and will not correlate with any change.

The cost of the label

Once a test is labelled flaky, three things happen, all bad.

People retry it. A retry that passes is read as "see, flaky," which actively confirms the wrong diagnosis.

It gets quarantined. Now it's a test nobody looks at, providing no protection, but still costing CI minutes and appearing in the count.

And worst: other failures in that test get ignored. When a genuine regression breaks it, the response is a retry.

The one thing that actually helps

Run it a hundred times and record pass/fail with the conditions.

If it fails 50/100 under parallelism and 0/100 serially, you have order or shared state. If it fails only on CI and never locally, you have timing or environment. If it fails in a burst and then stops, something upstream changed.

That's an afternoon. It replaces a label with a cause, and causes can be fixed.

The reframe

"Flaky" describes your knowledge, not the test. It means: this fails non-deterministically and I don't know why.

Said that way it stops sounding like a conclusion, which is the point.