Alertmanager (and Prometheus) — Ingest Setup

This guide shows how to route alerts from Prometheus Alertmanager straight to Culprit with no forwarder in between.

Prometheus users: Alertmanager is the component that sends webhooks on behalf of Prometheus. This recipe covers Prometheus-sourced alerts.

Endpoint

POST https://ingest.theculprit.ai/ingest/<tenant_id>/<service_name>/alertmanager

Replace <tenant_id> and <service_name> with the values shown on your service's settings page.

Authentication

Requests must carry an ingest token in the Authorization header:

Authorization: Bearer <ingest_token>

Getting your ingest token:

  1. Open the Culprit dashboard → Services → select the service.
  2. In the Ingest token section, click Issue.
  3. Copy the token shown — it is displayed once. After you navigate away, only the token prefix is shown.

To rotate a compromised token, click Rotate. The previous token remains valid for the grace period shown, giving you time to update Alertmanager without an interruption.

Alertmanager configuration

Add a receiver to your alertmanager.yml and update your route to point to it:

receivers:
  - name: culprit
    webhook_configs:
      - url: 'https://ingest.theculprit.ai/ingest/<tenant_id>/<service_name>/alertmanager'
        send_resolved: true
        http_config:
          authorization:
            type: Bearer
            credentials: '<ingest_token>'

route:
  receiver: culprit          # set as default, or nest under a sub-route
  # group_by, repeat_interval, etc. — your existing settings

Set send_resolved: true so Culprit receives the resolution signal when an alert clears. Resolved alerts are ingested as events and factor into correlation and root-cause analysis.

How Culprit maps Alertmanager alerts

Alertmanager sends batches of alerts in a single POST:

{
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "HighMemoryUsage",
        "severity": "critical",
        "instance": "web-01.internal"
      },
      "annotations": {
        "summary": "Memory usage above 90% on web-01.internal",
        "description": "Memory has been above 90% for 10 minutes."
      },
      "startsAt": "2026-05-29T01:00:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://prometheus.internal/graph?..."
    }
  ]
}

Each element of the alerts array becomes one Culprit event. The mapping is:

| Alertmanager field | Culprit event field | Notes | |---|---|---| | labels.alertname | event_type | Falls back to alertmanager.alert if absent | | labels.severity | severity | critical, warning, info, low are recognized; firing with no label → warning; resolved with no label → low | | annotations.summary | message | Falls back to annotations.description, then alertname | | status, labels, annotations, startsAt, endsAt, generatorURL | context | All preserved; original alert object also stored in context.raw |

Resolved alerts are ingested. When Alertmanager sends "status": "resolved", Culprit ingests it as a low-severity event (unless a severity label is present). Resolution signal improves correlation accuracy and RCA quality.

PII in labels or annotations (IP addresses, hostnames, email addresses, API keys) is detected and tokenized before any storage or notification. The original values are preserved encrypted and are visible only in the incident detail view after authentication.

Worked example

Alertmanager sends:

{
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "PodCrashLooping",
        "severity": "warning",
        "namespace": "production",
        "pod": "api-deployment-7d4b9c-xkp2q"
      },
      "annotations": {
        "summary": "Pod api-deployment-7d4b9c-xkp2q is crash-looping"
      },
      "startsAt": "2026-05-29T01:15:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://prometheus.internal/graph?g0.expr=..."
    }
  ]
}

Culprit event (before tokenization):

{
  "event_type": "PodCrashLooping",
  "severity": "warning",
  "source": "alertmanager",
  "message": "Pod api-deployment-7d4b9c-xkp2q is crash-looping",
  "context": {
    "status": "firing",
    "labels": {
      "alertname": "PodCrashLooping",
      "severity": "warning",
      "namespace": "production",
      "pod": "api-deployment-7d4b9c-xkp2q"
    },
    "annotations": {
      "summary": "Pod api-deployment-7d4b9c-xkp2q is crash-looping"
    },
    "startsAt": "2026-05-29T01:15:00Z",
    "endsAt": "0001-01-01T00:00:00Z",
    "generatorURL": "http://prometheus.internal/graph?g0.expr=...",
    "raw": { "...original alert object..." }
  }
}

This event flows through the standard pipeline: encrypted vault storage → PII tokenization → storm check → embedding → correlation → root-cause analysis.

Successful response

{ "accepted": true, "ids": ["<vault_id>"] }

For a batch of N alerts, ids contains N vault IDs.

Error responses

| Status | Body | Meaning | |---|---|---| | 401 | { "error": "missing_auth" } | No Authorization header and no ?token= parameter | | 401 | { "error": "invalid_ingest_token" } | Token is incorrect or has been rotated/revoked | | 400 | { "error": "unrecognized_payload" } | Body is not valid JSON or does not contain an alerts array | | 403 | { "error": "service_disabled" } | The service exists but monitoring is paused | | 404 | { "error": "service_deleted_or_not_found" } | The service does not exist | | 429 | { "error": "rate_limited" } | Storm protection active (sustained high volume) |