Automated PostHog Feature Flag Rollout Alerts
Every hour, WebRun opens PostHog, checks the error rate and conversion for a flagged rollout cohort against the control group and your threshold, posts a Slack alert on any breach, and escalates to Telegram if the gap keeps widening.
How do I catch a bad PostHog feature flag rollout before it affects more users?
WebRun checks a PostHog feature-flag rollout every hour, comparing the flagged cohort's error rate and conversion against the control group and your threshold. A breach posts a Slack alert with the gap and rollout percentage, and a worsening trend across two checks escalates to Telegram with a recommendation to roll back.
- Bad rollouts get caught within the hour, before wider exposure
- Only genuinely worsening trends escalate, not single-hour noise
- Every alert names the exact gap from the control group
Built for product engineering teams · platform engineers · growth engineers · site reliability engineers
What does WebRun do on every run?
The exact actions WebRun takes, in order - in plain language, so you can adjust anything.
-
WebRun signs in and gets to work
Opens
app.posthog.comin a real browser with your saved login - no setup, no API keys. -
1
PostHog - check the rollout's error rate and conversion
- Open PostHog and check the error rate and conversion for the flagged cohort
- Compare it to the control group and your rollout threshold
- Note the size of the gap if the threshold is breached
Done when The flagged rollout has been checked against its threshold this hour.
-
2
Slack - alert on a threshold breach
slack.com
WebRun posts to Slack to alert on a threshold breach. - Post an alert to the engineering channel naming the flag and the gap from control
- Include the current rollout percentage
Done when The team has a Slack alert for every threshold breach.
-
3
Telegram - escalate if it worsens
telegram.org
WebRun sends a Telegram message to escalate if it worsens. - Send a Telegram message to the on-call engineer if the gap has widened for two checks in a row
- Recommend rolling the flag back to 0 percent for review
Done when The on-call engineer has a Telegram escalation for any worsening rollout.
How is each run configured?
Secure by default
Connect once, stays signed in
WebRun signs in once and keeps each session in a persistent environment, so every run picks up right where it left off.
Every action is checked against this policy before it runs.
Questions, answered
Will it roll back the feature flag on its own?
No. WebRun only alerts and recommends. Rolling a flag back to 0 percent is a call left to the on-call engineer.
How does it decide the rollout is bad?
It compares the flagged cohort's error rate and conversion against the control group and the threshold you set, so the alert reflects a real regression, not normal noise.
Why does it wait for two checks before escalating?
A single hour's dip can be noise. Escalating only after the gap has widened for two checks in a row keeps the Telegram channel for rollouts that are genuinely getting worse.
Related templates
Put this on autopilot.
Turn it on in minutes - or have our team set it up for you.