Blog Details

blog
about

Stale feature flags are production code without an owner

Stale feature flags are production code without an owner

On the morning of August 1, 2012, Knight Capital pushed new trading code to eight production servers and missed one. The new code needed an activation switch, so the engineers reused the name of a flag that had once switched on Power Peg, a piece of order routing the firm had retired nine years earlier. On seven servers the flag turned the new feature on. On the eighth it did what it had always done: it woke up the old routine, which was still compiled into the binary and still listening for that key. When the market opened, that server started sending orders nobody had authorized. About 45 minutes later the firm had lost $440 million. It was acquired before the end of the year.

No vulnerability was involved. Nobody phished a credential or found a zero-day. One flag outlived its purpose, one name got reused, and one box was running code that everyone assumed was dead.

Feature flags are standard practice now, and most companies of any size carry hundreds of them. The failure mode rarely looks like Knight. It looks like a flag that has been on for 100 percent of traffic since early 2025, whose author has left, whose second code path still compiles, and whose only remaining function is to make the next configuration change a little more dangerous than it needs to be.

The number that matters is how many flags you have and how old they are. Very few teams can answer either question without opening a dashboard and guessing.

What a flag actually is

A feature flag is a branch at runtime on a value that lives outside your build. The code ships either way. The flag decides which path executes. That property is why flags make deploys safe, and it is also why they make codebases heavier: the branch you are not using today is still inside the artifact you deployed, and it is still reachable.

Four kinds of flags end up in the same system, and they do not share a lifespan.

  • Release flags gate a change until it is fully rolled out. These should live for days or weeks, not quarters. Once the change is at 100 percent for a full release cycle, the flag has no job left.
  • Experiment flags split traffic to answer a question. When the readout is written, the flag should close in the same week. An experiment flag that outlives its experiment is an untested branch with an audience.
  • Kill switches are deliberately permanent. They exist to turn a feature off during an incident, and a kill switch with an expiry date is a kill switch you will not have when you need it.
  • Entitlement flags encode what a customer paid for. They belong to billing or product, they change with contracts, and they should be driven by data rather than by whoever is on call at the time.

Most teams treat all four as "flags" and apply one cleanup rule to the lot. The policy cannot work that way. You cannot expire a kill switch on a 30-day clock, and you cannot leave a release flag open indefinitely. It has to be written per type and enforced by something other than memory.

How the debt piles up

Uber published the clearest numbers available on this. Their Piranha tool generated cleanup diffs for 1,381 flags between December 2017 and May 2019, which was about 17 percent of the flags in the codebases they studied. Two thirds of those diffs landed without a human editing them, and more than 85 percent compiled and passed tests on the first submit.

Uber has since removed roughly two thousand stale flags. That took dedicated tooling and an engineering-wide push, at a company that already cared about this. Most of the industry waits until something breaks.

Creating a flag costs almost nothing. A dashboard form takes 20 seconds and the key is live in every environment. Nothing in that flow asks for an owner, a type, or a date by which the flag should be gone.

Visibility is the second problem. A developer who suspects a flag is dead still has to prove that no service, no scheduled job, and no installed mobile client reads it. When that proof is expensive, the flag stays. Each individual decision is rational. The aggregate is a codebase where nobody can enumerate the states.

The third is that the flag is often the rollback plan. Deleting it feels like removing the emergency exit. The honest replacement is a revert you have actually tested, plus a deploy pipeline fast enough to use it. If you cannot roll back in ten minutes, you will keep every flag forever, and you will be right to.

The costs that never appear in an incident report

Testing is the easiest one to explain. Every live flag doubles the number of states the application can be in. Twelve boolean flags give you 4,096 combinations. Nobody tests 4,096 combinations. In practice teams test the path that is switched on, ship it, and find out about the other path when a customer account has a legacy entitlement or when an incident forces a kill switch into a state nobody has exercised in a year.

Debugging gets harder in a specific way. Two users on the same build, same browser, and same region behaving differently is a class of bug that costs support time before it costs engineering time. A flag evaluated in three places (application server, edge, and client) with three different cache windows produces exactly that. Support cannot reproduce it, engineering cannot reproduce it, and the flag dashboard shows the value as correct because the user's client cached the old value.

Security is the part that gets skipped. A flag that bypasses an authorization check is an access control change shipped outside your access control review. The same is true of a flag whose SDK default is true on the client and false on the server: a network failure or an expired cache resolves to the permissive side, and nobody wrote a test for it because the flag was labelled temporary. Stale flags also make a convenient hiding place. A key like internal_admin_tools that stayed enabled for staff long after the tool moved elsewhere is a standing exception that no longer has a reason to exist, and it will not appear in a permissions review that reads roles.

Then there is the reading cost, which is real even when it pages nobody. A new engineer opening a checkout module with nine flags has to reconstruct which combination production actually runs. That takes a day and is wrong maybe a fifth of the time. Multiply it across onboarding and across every incident where someone has to reason about a path they have never executed.

Flags change data, not just code

The cleanup conversation usually stops at code, and that is a mistake, because the hardest flags to remove are the ones that changed what got written to the database.

A flag that moves checkout from one pricing service to another is easy. A flag that writes orders into a new column, or a new table with a different shape, is not. Once that flag has been on for months, your data exists in two shapes, and deleting the flag means deleting the code that could still read the old one. The backfill that was supposed to run "after launch" is now the actual blocker.

This is where flag debt and schema debt become the same problem. The rule that works: a flag which changes persisted data cannot be deleted until you have verified that every row is in the new shape, that no reader still needs the old shape, and that the backfill is idempotent in case you have to run it again. Until then it is not a stale flag. It is a migration that has not finished, and it deserves a ticket rather than a cleanup sprint.

An audit you can finish this week

Start with the list, not the code. Every flag provider exposes an API for its flag set, and most expose evaluation counts per environment. Pull the full list with creation dates and last-modified dates into a table you can query. If you keep flags in a config file, git log -p on that file gives you the same data with worse formatting.

Then join the list against the codebase. For each key, count references with ripgrep. A flag with no references in the repository but evaluations in production is usually read by a client, a template, or another service, so check those before calling it dead. A flag with references but no evaluations in 90 days is a stronger deletion candidate than one with no references at all, because at least you know what it was doing.

Classify what you find into the four types above. This step decides the policy, and it is worth an hour with whoever owns the product surface.

Then answer five questions in writing. Which release flags have been at 100 percent for more than 30 days? Which experiments ended without a readout? Which flags have no named owner in the last year? Which flags are evaluated in more than one service with different default values? Which flags touch authorization or billing code? The first three are cleanup work. The last two are security and correctness findings, and they belong in the same queue as anything else of that severity.

A lifecycle policy that survives contact with reality

Every flag gets four fields at creation: owner, type, tracking ticket, and review date. If the dashboard will not enforce them, put the check in CI on the pull request that introduces the key. A YAML file in the repo listing flags and their review dates is enough to start, and it gives you a diff trail that a dashboard does not.

Release flags close 30 days after full rollout. Automate the reminder, then automate the removal pull request. Uber's numbers suggest most of those diffs will land as generated, which means the cost per flag trends toward zero once the pipeline exists. That is the whole argument for building it: cleanup is expensive once and cheap forever.

Experiment flags close with the experiment. Write the readout, delete the flag, and keep the key in the analytics export so the history survives without the code.

Kill switches get a documented trigger and an annual drill. Turn one off in staging during a game day and confirm the system does what the runbook says. A kill switch that has never been flipped is a hypothesis, not a control.

Entitlement flags belong to billing, get tested in the billing suite, and are excluded from the release-flag cleanup count so that the metric means something.

One rule matters more than the rest: never reuse a flag name. New behaviour gets a new key. That single rule would have prevented Knight Capital's failure without any other control in place, and it costs a developer nothing.

Deleting a flag without breaking production

The order of operations is what teams get wrong. Confirm the flag is at 100 percent with no targeting rules and no per-user overrides. Confirm no client is still requesting it. That second check matters most for mobile: an app version from eight months ago will keep asking for a key that no longer exists, and the SDK will answer with the default value. Make the default match the behaviour you intend to keep before you delete anything.

Remove the targeting rules first and leave the flag returning a constant for one release cycle. Traffic still resolves the key, logs stay readable, and you have a clean rollback if the constant turns out to be wrong. After that cycle, delete the losing branch, delete the tests that covered it, and remove the key. Archive the flag definition read-only for a quarter rather than deleting the record, so old logs and incident timelines still resolve when someone asks what was live in March.

Remove the losing branch as well as the guard that selected it. A removal that leaves the old path behind a TODO or a commented block has not reduced the state space. The next person will find it and treat it as live.

What to measure

Five numbers, on the same dashboard as deploy frequency: total flags, median age of release flags, share of flags at 100 percent for more than 30 days, count with no owner, and median days from full rollout to code removal. The last one tells you whether any of this is real. If it sits flat at 400 days, you have a document and not a process.

Knight Capital did not fail because someone used a feature flag. It failed because the code behind an old flag was never removed and somebody trusted a name. Deletion is the control that keeps a flag system safe, and no dashboard will do it for you.

If your team inherited a codebase with a few hundred flags and no inventory, this is the kind of work we do as part of platform and application audits. Talk to Tech Hub Services and we will start with the list.

Send Us a Message