A pipeline in our repo had been failing on every merge for five weeks. Not flaking. Failing. Every single time.

The thing I keep coming back to isn't the fix. It's that five weeks passed.

How a red build becomes furniture

Nobody decided to ignore it. What happened is smaller and more human than that. Someone looked once, saw a wall of logs, didn't find anything in ten minutes, and moved on. The next person saw it was already red before their change, concluded "not mine," and moved on. Repeat for a month and the red light stops being information. It's just part of the room.

This is the real cost of a long-broken build, and it's not the broken thing. It's that you lose the signal. A pipeline that is always red tells you nothing on the day it breaks for a second, worse reason.

What it actually was

Three tests that could not pass. Not flaky, not slow — structurally impossible in the environment they were running in. They needed a fixture the shared backend could never have.

No product code was wrong. The fix was five lines of config.

The part I got wrong

I spent two days on a theory I built from a local reproduction. It was coherent. It explained the symptom. It was also completely wrong, and I'd never once opened the actual log from the actual failing run.

When I finally read it, the answer was in the first screen.

I've written the lesson on a sticky note now, which tells you how much I trust myself to remember it: read the log before you build the theory. A local repro is a hypothesis wearing a lab coat. It looks like evidence. It isn't.

The rule I'd keep

If a build has been red for more than a day, that is its own incident, separate from whatever broke it. Someone owns it by name. Otherwise you don't have a failing test — you have a light nobody looks at.