Skip to main content
When a metric crosses its threshold, a LangSmith alert rule can POST a notification to a webhook URL. Flashduty parses these notifications directly, and each LangSmith alert rule maps to one Flashduty alert. LangSmith’s documentation describes no “resolved” notification: once the metric falls back within the threshold, nothing more is sent. Every notification Flashduty receives triggers (or updates) an alert, so close alerts manually or turn on the auto-resolve timeout as described below.

In Flashduty On-call

You can get the integration push URL in either of two ways.

Use a dedicated integration

  1. In the Flashduty console, select Channels and open a channel
  2. Select Settings → Integrations → Dedicated integrations, then click Add integration
  3. Select LangSmith and click Save
  4. Open the generated integration card and copy the Push URL

Use a shared integration

  1. In the Flashduty console, go to Integration Center → Alert Events
  2. Select LangSmith and enter an integration name
  3. Configure the default route and select a channel. You can add more rules under Routing after creation
  4. Click Save and copy the generated Push URL

Configure in LangSmith

1

Create an alert rule with a webhook in LangSmith

Where: LangSmith › the tracing project to monitor › Alerts (the Alerts icon at the top right of the page, under Monitoring) › + AlertPick the metric (Run Count, Cost, Errors, Feedback Score or Latency), the threshold and the time window (LangSmith’s current documentation lists 5 or 15 minutes). Then, under Notification Settings, choose Webhook and fill in:Save the alert rule.
LangSmith’s Send Test Alert button only proves the request can be sent: LangSmith does not validate the receiver’s response, so the UI reports success even if Flashduty returns an error; check the Flashduty console instead. The test payload carries the all-zero UUID 00000000-0000-0000-0000-000000000000 as alert_rule_id, and Flashduty returns success without creating an alert.
2

Turn on the auto-resolve timeout in Flashduty

A LangSmith webhook has no recovery notification: after the metric falls back, nothing more is sent and the Flashduty alert stays triggered. In the channel that receives these alerts, turn on the auto-resolve timeout with a duration of 1 hour and the timer start set to Stop merging new alerts. If LangSmith sends again while the metric stays above the threshold, the alert stays open; an alert with no new notification within the timeout is closed.
3

Verify with a real alert

Let the metric actually cross the threshold (for example, temporarily lower the threshold). After a few seconds, go back to Flashduty. If the integration shows a Latest event time, the setup works. Restore the threshold afterwards.

Payload

LangSmith POSTs these fields as application/json, and Flashduty parses them directly: The alert title is the rule name; if it is empty, LangSmith alert <alert_rule_id> is used.

Alert key

Flashduty uses alert_rule_id as the Alert Key. LangSmith documents it as the UUID identifying the alert, but does not state that the ID stays the same on every firing of one rule. Flashduty merges by rule ID: repeated notifications for one rule merge into one alert and different rules stay separate. Changing the rule name, threshold or metric value does not change the Alert Key. A request without alert_rule_id, or with the all-zero UUID that LangSmith uses for test notifications, is treated as a test notification: Flashduty returns success and creates no alert.

Status and severity

LangSmith notifications carry no severity, so Flashduty treats every one as Warning. A threshold breach does not mean the service is down, so the default is not Critical.

FAQ

The LangSmith webhook only fires when the metric crosses the threshold and sends nothing when it falls back. Turn on the channel’s auto-resolve timeout, or close the alert manually in Flashduty.
LangSmith does not validate the receiver’s response. Flashduty drops a test payload (alert_rule_id missing or the all-zero UUID), which is expected; real alerts are sent only when the metric actually crosses the threshold.
Yes. The LangSmith documentation requires Helm chart 0.10.3 or later for alerts on self-hosted deployments, and the deployment must be able to reach the Flashduty push URL.
For more on the fields, see the LangSmith documentation: Alerts webhook and Alerts.