A service started getting killed by the kernel. Exit code 137 — out of memory. Restarted, served traffic happily, got killed again a few hours later.

No code had changed in weeks.

The shape of the problem

When memory grows until something dies, there are broadly two causes, and they need different fixes.

A leak: you're holding references to things you're done with. Memory climbs and never comes down, independent of load.

An unbounded working set: you're correctly releasing everything, but you load too much at once. Memory spikes with a particular request and returns to normal after.

The graph tells you which. A leak is a staircase. An unbounded set is a mountain range — up, down, up higher.

Ours was mountains. That rules out a whole class of investigation immediately, which is why looking at the shape first is worth the two minutes.

The actual line

A function fetched results and materialised them all into a slice. Every row, into memory, at once.

No limit. No pagination. No streaming.

This was completely fine when written — a few hundred rows. Then the table grew, and one caller fanned the function out concurrently across every item in a set. So: unbounded result set, times concurrency, in one process.

Nothing was wrong with the code. The code's assumption about the world had expired.

Why "add more memory" is the wrong instinct

It's the fastest fix and it buys you an amount of time you cannot calculate, because you don't know the growth rate of the thing you didn't bound.

Worse, in our case the process runs in someone else's cluster. Asking for more memory isn't a config change — it's asking a customer to give up capacity to cover our inefficiency. That reframes it completely: the working set has to be inherently small, not accommodated.

The fixes, in order of durability

1. Bound the query. A LIMIT with explicit pagination. The caller handles pages.
2. Stream instead of materialising. Process a row, release it, move on. Memory becomes independent of result size.
3. Don't fetch what you don't need. We were loading full records to count them.

The third one was most of the win, and it's the one that would never have shown up in a profiler as a bug — just as a function doing exactly what it said.

The habit

Any time I write something that returns a collection, the question now is: what bounds this? If the answer is "the data is small," that's an assumption with a shelf life. Write it down, or enforce it.