Automated StreamSets Overnight Pipeline Reconciliation
Every morning, WebRun opens StreamSets, checks the run history for every scheduled pipeline from overnight, logs any failure to Airtable with the pipeline name, stage, and error, and drafts a summary email in Gmail for the data engineering lead ranking failures by pipeline criticality, so broken pipelines get fixed before the day's dashboards and reports rely on them.
How do I confirm my overnight StreamSets pipelines actually ran successfully?
WebRun checks every scheduled StreamSets pipeline from overnight each morning, logs any failure to Airtable with the failing stage and error, and drafts a ranked summary email in Gmail for the data engineering lead. Broken pipelines get logged and surfaced the same morning, so downstream data does not quietly run on incomplete results.
- Failed pipelines get logged and surfaced the same morning
- The data engineering lead reviews a ranked summary instead of digging through logs
- Resolved pipelines are marked automatically once they run clean again
Built for Data engineering teams · Data platform teams · Analytics engineers · DataOps teams
What does WebRun do on every run?
The exact actions WebRun takes, in order - in plain language, so you can adjust anything.
-
WebRun signs in and gets to work
Opens
streamsets.comin a real browser with your saved login - no setup, no API keys. -
1
StreamSets - check overnight pipeline runs
streamsets.com
WebRun opens StreamSets to check overnight pipeline runs. - Open StreamSets and check the run history for every scheduled pipeline from overnight
- Find any run that ended in a failure
- Capture the pipeline name, the failing stage, and error message
Done when Every failed overnight run has been captured with its detail.
-
2
Airtable - log the failures
airtable.com
WebRun opens Airtable to log the failures. - Create a record for each failed pipeline on the data health base
- Include the failing stage, error, and how long it has been failing
- Mark a record resolved once its pipeline runs successfully again
Done when Every currently failing pipeline has a logged Airtable record.
-
3
Gmail - draft the summary email
gmail.com
WebRun opens Gmail to draft the summary email. - Draft a summary email ranking failed pipelines by criticality
- List the pipeline name, stage, and error for each
- Leave the email as a draft. Never send it without review
Done when A summary email is waiting in Gmail drafts for the data engineering lead.
How is each run configured?
Secure by default
Connect once, stays signed in
WebRun signs in once and keeps each session in a persistent environment, so every run picks up right where it left off.
Every action is checked against this policy before it runs.
Questions, answered
Will the summary email go out without anyone reviewing it?
No. WebRun only drafts it in Gmail. Nothing reaches the data engineering lead's inbox until it is reviewed and sent.
Does WebRun restart a failed pipeline?
No. It only detects and reports the failure. Restarting a pipeline or fixing the failing stage stays with your data engineering team.
How does it decide which pipelines are critical?
You mark which pipelines are critical in StreamSets. Those are ranked first in the summary email so the most important issues get seen first.
Related templates
Put this on autopilot.
Turn it on in minutes - or have our team set it up for you.