Ship behind a flag
decide on real numbers, not on who argued longer

What it is

A feature flag wraps a piece of code in a condition. Show the new checkout flow to this percentage of users, or to users with this attribute, and the old flow to everyone else. A switch controls it, and that switch doesn’t need a deploy to change. Experiment infrastructure builds on top of this. It assigns users consistently to a test group, and measures a defined metric for each group. The result is a statistically grounded answer to whether the change actually helped.

When you need this (and when you don’t)

You need this once a change is risky enough that rolling it out to everyone at once is a real gamble. A new pricing page, a redesigned onboarding flow, a changed algorithm, all qualify, anywhere being wrong costs real revenue or real users. It’s also the right build once “ship it and see” has produced a few disagreements about whether a past change actually helped. That usually means nobody measured it properly at the time.

You don’t need this for low-risk changes, a copy tweak, a color change, a bug fix. The cost of being wrong there is trivial, and a flag just adds process without adding value. The tell that you need it: someone in the room says “I think this will help,” someone else disagrees, and neither has data to settle it.

How we build it

For most clients already on PostHog for product analytics, we use its built-in feature flags. That keeps assignment, tracking and results in one tool instead of stitching two systems together. Rollout rules support simple percentage-based assignment and attribute-based targeting alike. Think a specific plan tier or region, for cases where a blanket percentage isn’t the right test.

Every flag has an instant kill switch. Flipping it off takes effect immediately, with no deploy. That’s the entire point: a bad change should be reversible in seconds, not after an incident review and a hotfix. For experiments specifically, we help define the success metric and minimum sample size before launch, not after results come in. A metric chosen after the fact tends to be the one that makes the result look good, not the one that actually answers the question.

What to watch

The real risk with feature flags isn’t technical. It’s organizational drift. Flags meant to be temporary for a rollout stay in the codebase for years, and eventually nobody remembers which combination of flags is actually live for which users. The code itself turns into a maze of conditionals. We recommend a flag cleanup review on a schedule, and build flags to be removed once a rollout completes, not left in indefinitely by default.

On the experimentation side, calling a result significant before the sample size is reached is the most common statistical mistake teams make on their own. We build a minimum sample size check into the reporting, so a test doesn’t get called early just because the numbers looked good on day two.

Sample ratio mismatch is a subtler failure worth watching for. Say your assignment logic is even slightly biased, excluding users on an older app version from one group but not the other. The groups are then no longer comparable. The result isn’t trustworthy no matter how significant it looks. We check for this before reading any result as final. It’s a common, easy-to-miss way an otherwise correct-looking experiment produces a wrong answer, and nobody catches it until a decision is already made.

Price and timeline

What this pairs with

Built as part of custom development and analytics. Builds directly on product analytics setup for measuring results. See the engagement mechanics in an AI coach and gamification fitness app. Tell us what change you want to test before committing to it: get in touch.