Self-Host GrowthBook: Feature Flags for AI Rollouts

The first time we shipped an AI summary feature, it went sideways at the worst possible hour. A model update started returning confidently wrong summaries for a chunk of users, and the only way to turn it off was a code change, a CI run, and a deploy. That is a long twenty minutes when the fix is conceptually just "stop doing the thing." What I wanted was a switch on the wall. What I had was a redeploy.

Feature flags are that switch, and GrowthBook is a very good open-source one. It is a feature flagging and experimentation platform you can self-host, and it happens to fit AI features unusually well, because AI features are exactly the ones you want to roll out slowly and kill instantly.

What you actually get

GrowthBook is mostly MIT-licensed (a few enterprise directories aside) and ships as a Docker deployment: a frontend on port 3000, an API on port 3100, and MongoDB as its datastore. That is the whole hard dependency. Your flags, targeting rules, and experiment configs live in Mongo.

On top of that it gives you 24 SDKs (JavaScript, React, Python, Go, Ruby, PHP, Java, iOS, Android, and more), a warehouse-native stats engine, and, handy for this crowd, a built-in MCP server so an agent can manage flags too.

How the SDK decides

Here is the part worth understanding before you trust it in production. GrowthBook does not phone home on every flag check. Your app's SDK pulls the full set of feature definitions once, using a read-only client key, and then evaluates flags locally, in memory. An isOn("ai-summaries") call is just a function running against cached rules, so it costs microseconds and keeps working even if the GrowthBook API blips.

When you flip a flag in the dashboard, GrowthBook pushes the new definitions to connected SDKs over a streaming connection (server-sent events), so clients update within a second or two without a redeploy and without polling. That combination, local evaluation plus streaming updates, is what makes the kill switch feel instant instead of eventually-consistent.

Gating an AI feature

Wrapping a feature takes a few lines. Here is the JavaScript SDK guarding an AI code path:

import { GrowthBook } from "@growthbook/growthbook";

const gb = new GrowthBook({
  apiHost: "https://your-growthbook.elestio.app",
  clientKey: "sdk-abc123",
});

// Fetch feature definitions and keep them live over streaming
await gb.init({ streaming: true });

// Describe the current user so targeting rules can match
gb.setAttributes({
  id: user.id,
  plan: user.plan,
  betaTester: user.betaTester,
});

if (gb.isOn("ai-summaries")) {
  return generateAISummary(text);
}
return classicSummary(text);

You create the ai-summaries flag once in the GrowthBook dashboard (until it exists, the SDK just returns the default), and from then on the interesting rules live there, not in the code. You can ship ai-summaries to 10 percent of traffic, or only to users where betaTester is true, or to everyone on the pro plan. When the model misbehaves, you set the flag to off and every SDK follows within seconds. No branch, no deploy, no 2 a.m. heroics.

Testing two prompts, honestly

The same machinery runs experiments. Instead of a simple on or off, a feature can carry an experiment rule that splits traffic between, say, prompt A and prompt B, or two different models. GrowthBook assigns users consistently, and its stats engine (Bayesian and frequentist, with CUPED, sequential testing, and sample-ratio-mismatch checks) reads results straight from your own data warehouse. It queries 11 sources including BigQuery, Snowflake, ClickHouse, and plain Postgres, so your event data never leaves your infrastructure. You get a real answer about which prompt converted better, not a vibe.

Running it on Elestio

GrowthBook plus MongoDB is a two-service stack you could wire up yourself, but you would also own the database, TLS, and backups. Managed GrowthBook on Elestio starts at $11/month with MongoDB, SSL, automated backups, and updates already set up. Deploy it from the GrowthBook service page, then create an SDK connection to get the client key you saw above.

Troubleshooting

• Flags do not update when I toggle them: you are almost certainly on the default polling cache, not streaming. Pass { streaming: true } to gb.init() and confirm your reverse proxy is not buffering server-sent events.
• isOn() always returns false: the SDK is falling back to defaults, usually because the client key is for the wrong environment (production versus development have separate keys) or the app cannot reach apiHost.
• Targeting rules never match: attributes are case-sensitive and must be set before you read the flag. Log gb.getAttributes() once to see what the SDK actually has.
• The app will not start after deploy: check MONGODB_URI. A missing or wrong Mongo connection string is the most common first-boot failure, since every config read hits the database.

Feature flags will not make a bad AI feature good. But they turn "we have to redeploy to stop the bleeding" into "I clicked the toggle," and they let you find out whether version B is actually better before you bet the product on it. For anything with a language model behind it, that safety net is worth the eleven dollars.

Thanks for reading ❤️ See you in the next one 👋