Automated LangSmith Trace Error Alerts
Every hour, WebRun opens LangSmith, checks the latest traces for error rate or latency above your threshold, logs each incident to a Notion log with the run details, and exports the hour's trace metrics to Google Sheets so engineering has a running record.
How do I get alerted when LangSmith traces show errors or latency spikes?
WebRun checks your LangSmith traces every hour for error rates or latency above the threshold you set. It logs each flagged run to a Notion incident log with the run ID and error, then exports the hour's totals to Google Sheets, so engineering has a live record without watching dashboards all day.
- Error and latency spikes get logged within the hour instead of at standup
- Engineering keeps a running metrics record without opening a dashboard
- Repeat errors from the same chain are easy to spot in the log
Built for AI engineering teams · LLM app developers · ML platform teams · AI product teams
What does WebRun do on every run?
The exact actions WebRun takes, in order - in plain language, so you can adjust anything.
-
WebRun signs in and gets to work
Opens
smith.langchain.comin a real browser with your saved login - no setup, no API keys. -
1
LangSmith - check traces for errors and latency
smith.langchain.com
WebRun opens LangSmith to check traces for errors and latency. - Open LangSmith and filter the latest traces by project
- Flag any run with an error or latency above your set threshold
- Capture the run ID, error message, and latency for each flagged trace
Done when Every flagged trace this hour has been captured with its details.
-
2
Notion - log each incident
notion.so
WebRun opens Notion to log each incident. - Open the incident log in Notion
- Add a row for each flagged trace with the run ID, error, and time
- Mark repeat errors from the same chain so patterns are easy to spot
Done when Every flagged trace is logged in Notion.
-
3
Google Sheets - export the trace metrics
google.com
WebRun opens Google Sheets to export the trace metrics. - Open the trace metrics sheet
- Append the hour's totals for runs, errors, and average latency
- Keep a running daily total at the top of the sheet
Done when The hour's trace metrics are recorded in Google Sheets.
How is each run configured?
Secure by default
Connect once, stays signed in
WebRun signs in once and keeps each session in a persistent environment, so every run picks up right where it left off.
Every action is checked against this policy before it runs.
Questions, answered
Will it post errors somewhere customers can see?
No. Everything stays internal. The incident log lives in Notion and the metrics sheet in Google Sheets, both private to your team.
What counts as an error worth flagging?
Any trace with a run error or latency above the threshold you set. Normal runs are left out so the log only shows what needs a look.
Does it try to fix the underlying issue?
No. It only detects, logs, and reports. Your engineers decide what to change, WebRun just makes sure nothing gets missed.
Related templates
Put this on autopilot.
Turn it on in minutes - or have our team set it up for you.