Skip to main content
When a metric crosses its threshold, a LangSmith alert rule can POST a notification to a webhook URL. Flashduty parses these notifications directly, and each LangSmith alert rule maps to one Flashduty alert. LangSmith’s documentation describes no “resolved” notification: once the metric falls back within the threshold, nothing more is sent. Every notification Flashduty receives triggers (or updates) an alert, so close alerts manually or turn on the auto-resolve timeout as described below.

​
In Flashduty On-call

You can get the integration push URL in either of two ways.

​
Use a dedicated integration

  1. In the Flashduty console, select Channels and open a channel
  2. Select Settings → Integrations → Dedicated integrations, then click Add an integration
  3. Select LangSmith and click Save
  4. Open the generated integration card and copy the push URL

​
Use a shared integration

  1. In the Flashduty console, select Integration Center → Alert Events
  2. Select LangSmith and enter an integration name
  3. Configure the default route and pick a channel; you can add more rules under Routes after creation
  4. Click Save and copy the generated push URL

​
Configure in LangSmith

1

Create an alert rule with a webhook

  1. Sign in to LangSmith, open the tracing project to monitor, go to Monitoring → Alerts (or the project’s monitoring dashboard), click Alert, and create an alert rule: pick the metric (Run Count, Cost, Errors, Feedback Score or Latency), the threshold and the time window (1 to 60 minutes)
  2. Under Notification Settings, choose Webhook and fill in:
    • URL: the full push URL of the Flashduty integration
    • Headers: leave empty
    • Body: leave empty. LangSmith merges the fields listed under “Payload” below into the request body as top-level keys; it does no template substitution
  3. Save the alert rule
2

Check connectivity

Edit the Webhook action in the alert rule’s notification settings and click Send Test Notification. LangSmith does not validate the receiver’s response, so the UI reports success even if Flashduty returns an error; check the Flashduty console instead. The test payload carries the all-zero UUID 00000000-0000-0000-0000-000000000000 as alert_rule_id, and Flashduty returns success without creating an alert.
3

Turn on the auto-resolve timeout

A LangSmith webhook has no recovery notification: after the metric falls back, nothing more is sent and the Flashduty alert stays triggered. In the channel that receives these alerts, turn on the auto-resolve timeout with a duration of 1 hour and the timer start set to Stop merging new alerts. If LangSmith sends again while the metric stays above the threshold, the alert stays open; an alert with no new notification within the timeout is closed.

​
Payload

LangSmith POSTs these fields as application/json, and Flashduty parses them directly: The alert title is the rule name; if it is empty, LangSmith alert <alert_rule_id> is used.

​
Alert Key

Flashduty uses alert_rule_id as the Alert Key. LangSmith documents it as the UUID identifying the alert, and every firing of one rule carries the same alert_rule_id, so repeated notifications while the metric stays above the threshold merge into one alert and different rules stay separate. Changing the rule name, threshold or metric value does not change the Alert Key. A request without alert_rule_id, or with the all-zero UUID that LangSmith uses for test notifications, is treated as a test notification: Flashduty returns success and creates no alert.

​
Status and severity

LangSmith notifications carry no severity, so Flashduty treats every one as Warning. A threshold breach does not mean the service is down, so the default is not Critical.

​
FAQ

The LangSmith webhook only fires when the metric crosses the threshold and sends nothing when it falls back. Turn on the channel’s auto-resolve timeout, or close the alert manually in Flashduty.
LangSmith does not validate the receiver’s response. Flashduty drops a test payload (alert_rule_id missing or the all-zero UUID), which is expected; real alerts are sent only when the metric actually crosses the threshold.
Yes. The LangSmith documentation requires Helm chart 0.10.3 or later for alerts on self-hosted deployments, and the deployment must be able to reach the Flashduty push URL.
For more on the fields, see the LangSmith documentation: Alerts webhook and Alerts.