English
Japanese Tieng Viet
AITracer
AITracer / Documentation / Alert Configuration

Alert Configuration

Detect error rates and cost overruns, and receive instant notifications

Table of Contents

Alert Feature Overview

AITracer's alert feature allows you to instantly detect anomalies in your LLM applications and receive notifications.

Error Detection

Get notified when error rate exceeds threshold

Cost Monitoring

Get warned before budget overruns

Latency Monitoring

Detect response time anomalies

Usage Monitoring

Detect request count anomalies

Types of Alerts

1. Error Rate Alert

Triggers when the error rate exceeds the threshold within a specified time window.

Setting Description Example
Threshold Error rate percentage 5%
Time Window Evaluation period 5min, 15min, 1hr
Minimum Requests Minimum requests needed for evaluation 10 requests

2. Cost Alert

Triggers when daily/weekly/monthly costs exceed the threshold or are projected to exceed.

Setting Description Example
Budget Amount (USD) $100
Period Daily/Weekly/Monthly Monthly
Warning Threshold Percentage of budget for warning 80%

3. Latency Alert

Triggers when P50/P95/P99 latency exceeds the threshold.

Setting Description Example
Percentile P50, P95, P99 P95
Threshold Milliseconds 3000ms
Time Window Evaluation period 15min

4. Request Count Alert

Triggers when request count spikes or drops suddenly.

Setting Description Example
Condition Above/Below Above
Threshold Request count 1000/hour
Time Window Evaluation period 1 hour

Creating Alert Rules

Create from Dashboard

  1. Log in to the Dashboard
  2. Open the "Alerts" menu
  3. Click "+ New Rule" button
  4. Select alert type
  5. Configure conditions and thresholds
  6. Select notification channels
  7. Click "Save"

Filter Conditions

You can set the following filters for alert rules:

  • Project: Target specific projects only
  • Model: Target specific models (gpt-4, claude-3, etc.) only
  • Provider: Target specific providers like OpenAI, Anthropic only
  • Metadata: Filter by custom metadata

Notification Channels

Email Notifications

Available on all plans. Sends notifications to team members' email addresses.

  • Instant notification when alert triggers
  • Can send to multiple email addresses
  • Notification frequency control (minimum interval setting)

Slack Integration

Available on Pro plan and above. Sends real-time notifications to Slack channels.

  1. Go to "Settings" -> "Integrations" in the dashboard
  2. Click "Connect to Slack"
  3. Authenticate your Slack workspace
  4. Select the notification channel
Slack Notification Contents
Includes alert name, trigger conditions, current value, and affected project/model.

Webhook

Available on Starter and above. Sends POST requests to any endpoint.

// Webhook payload example
{
  "alert_id": "alert_abc123",
  "alert_name": "High Error Rate",
  "type": "error_rate",
  "severity": "critical",
  "triggered_at": "2025-01-15T10:30:00Z",
  "condition": {
    "threshold": 5,
    "current_value": 8.5,
    "time_window": "5m"
  },
  "filters": {
    "project": "my-chatbot",
    "model": "gpt-4"
  }
}

Configuration Examples

Production Error Monitoring

Configuration:

  • Alert Type: Error Rate
  • Threshold: 5%
  • Time Window: 5 minutes
  • Minimum Requests: 20
  • Filter: Project = production
  • Notification: Slack #alerts channel

Monthly Cost Budget Monitoring

Configuration:

  • Alert Type: Cost
  • Budget: $500/month
  • Warning Threshold: 80% ($400)
  • Critical Threshold: 100% ($500)
  • Notification: Email + Slack

Latency Anomaly Detection

Configuration:

  • Alert Type: Latency
  • Percentile: P95
  • Threshold: 5000ms
  • Time Window: 15 minutes
  • Filter: Model = gpt-4
  • Notification: Webhook (PagerDuty)

Plan Limits

Feature Free Starter Pro Enterprise
Alert Rules 3 10 50 Unlimited
Email Notifications 1 address 5 addresses Unlimited Unlimited
Slack Integration - - 1 workspace Unlimited
Webhooks - 2 endpoints 10 endpoints Unlimited
Evaluation Interval 15 min 5 min 1 min Real-time
Alert Firing Limits
Alerts for the same rule fire at minimum 5-minute intervals. This prevents notification spam.