📉 The One Metric That Predicts Test Suite Rot

The Canary in the Coal Mine

If I could only track one metric for a test suite's health, it would be flaky test rate.

A flaky test is one that passes and fails on the same code. No changes, different results. It's the test equivalent of a car that sometimes starts and sometimes doesn't.

Why Flakiness Compounds

Here's the insidious pattern I've seen play out dozens of times:

1. A test becomes flaky
2. The team ignores it ("it'll pass on retry")
3. More tests become flaky
4. Developers start ignoring failures ("probably just flaky")
5. A real bug slips through because the failure was dismissed
6. Trust in the suite collapses

Once developers stop trusting test results, you don't have a test suite anymore. You have a CI tax that slows everyone down while catching nothing.

The Fix Is Simple (Not Easy)

Track your flaky rate weekly. Here's a baseline:

• Under 1%: Healthy. Address flakes as they appear.
• 1-5%: Warning zone. Dedicate time each sprint to fixing or deleting.
• Over 5%: Emergency. Stop adding new tests until you fix existing ones.

When you find a flaky test, you have two options:

1. Fix it - Usually timing issues, test isolation, or environment dependencies
2. Delete it - A flaky test that doesn't get fixed is worse than no test

Yes, deleting tests feels wrong. But a test that cries wolf trains your team to ignore all wolves.

One Action

Set up a dashboard that shows flaky test rate over time. Review it weekly. Treat a rising trend like you'd treat a production incident—because eventually, it will cause one.

The best test suites aren't the ones with the most tests. They're the ones where every test result means something.