Grafana — Ingest Setup

This guide shows how to route Grafana alerts straight to Culprit using Grafana's built-in webhook contact point. Both Grafana unified alerting (the default since Grafana 9) and classic (legacy) alerting are supported.

Endpoint

POST https://ingest.theculprit.ai/ingest/<tenant_id>/<service_name>/grafana

Replace <tenant_id> and <service_name> with the values shown on your service's settings page.

Authentication

Requests must carry an ingest token in the Authorization header:

Authorization: Bearer <ingest_token>

Getting your ingest token:

  1. Open the Culprit dashboard → Services → select the service.
  2. In the Ingest token section, click Issue.
  3. Copy the token shown — it is displayed once. After you navigate away, only the token prefix is shown.

To rotate a compromised token, click Rotate. The previous token remains valid for the grace period shown, giving you time to update Grafana without an interruption.

Grafana unified alerting (recommended)

Grafana unified alerting uses the same Alertmanager-compatible webhook format. Add a contact point of type Webhook in your Grafana instance:

  1. Open Alerting → Contact points → Add contact point.
  2. Set Type to Webhook.
  3. Set URL to your Culprit endpoint.
  4. Under Optional settings → HTTP headers, add:
    • Header name: Authorization
    • Header value: Bearer <ingest_token>
  5. Save the contact point and assign it to a notification policy.

Grafana will POST each alert batch to Culprit as it fires and resolves.

Grafana classic alerting

For dashboards using the legacy alert engine (Grafana 8 and below, or classic alerting enabled on a later version):

  1. Open the panel → Alert tab → Notifications.
  2. Add a Webhook notification channel pointing to your Culprit endpoint.
  3. Grafana classic webhooks do not support custom headers — instead, pass the token as a URL parameter:
https://ingest.theculprit.ai/ingest/<tenant_id>/<service_name>/grafana?token=<ingest_token>

How Culprit maps Grafana alerts

Unified alerting (Alertmanager-compatible)

Grafana unified alerting sends an alerts array in the same format as Prometheus Alertmanager. Each element of the alerts array becomes one Culprit event:

| Grafana field | Culprit event field | Notes | |---|---|---| | labels.alertname | event_type | Falls back to alertmanager.alert if absent | | labels.severity | severity | critical, warning, info are recognized; firing with no label → warning; resolved with no label → low | | annotations.summary | message | Falls back to annotations.description, then alertname | | status, labels, annotations | context | All preserved for correlation and RCA |

Classic alerting

Classic Grafana webhooks carry a flat payload with the following mapping:

| Grafana field | Culprit event field | Notes | |---|---|---| | ruleName | event_type | Falls back to grafana.alert if absent | | state | severity | alerting → high; no_data → medium; ok → low | | message | message | Falls back to title, then ruleName | | state, ruleName, evalMatches | context | Preserved for correlation and RCA |

Resolved alerts are ingested. When Grafana sends state: "ok", Culprit ingests it as a low-severity event. Resolution signal improves correlation accuracy and root-cause analysis.

PII in labels, annotations, or rule names (IP addresses, hostnames, email addresses, API keys) is detected and tokenized before any storage or notification. The original values are preserved encrypted and are visible only in the incident detail view after authentication.

Worked example — unified alerting

Grafana sends:

{
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "HighLatency",
        "severity": "warning",
        "instance": "api-3.acme.internal"
      },
      "annotations": {
        "summary": "p99 latency above 2s on api-3.acme.internal"
      },
      "startsAt": "2026-05-29T01:00:00Z",
      "endsAt": "0001-01-01T00:00:00Z"
    }
  ]
}

Culprit event (before tokenization):

{
  "event_type": "HighLatency",
  "severity": "warning",
  "source": "grafana",
  "message": "p99 latency above 2s on api-3.acme.internal",
  "context": {
    "status": "firing",
    "labels": { "alertname": "HighLatency", "severity": "warning", "instance": "api-3.acme.internal" },
    "annotations": { "summary": "p99 latency above 2s on api-3.acme.internal" },
    "startsAt": "2026-05-29T01:00:00Z"
  }
}

Worked example — classic alerting

Grafana sends:

{
  "title": "[Alerting] High latency",
  "ruleName": "High latency",
  "state": "alerting",
  "message": "p99 4200ms on api-3.acme.internal",
  "evalMatches": [{ "metric": "p99", "value": 4200 }]
}

Culprit event (before tokenization):

{
  "event_type": "High latency",
  "severity": "high",
  "source": "grafana",
  "message": "p99 4200ms on api-3.acme.internal",
  "context": {
    "state": "alerting",
    "ruleName": "High latency",
    "evalMatches": [{ "metric": "p99", "value": 4200 }]
  }
}

This event flows through the standard pipeline: encrypted vault storage → PII tokenization → storm check → embedding → correlation → root-cause analysis.

Successful response

{ "accepted": true, "ids": ["<vault_id>"] }

For a batch of N alerts (unified alerting), ids contains N vault IDs.

Error responses

| Status | Body | Meaning | |---|---|---| | 401 | { "error": "missing_auth" } | No Authorization header and no ?token= parameter | | 401 | { "error": "invalid_ingest_token" } | Token is incorrect or has been rotated/revoked | | 400 | { "error": "unrecognized_payload" } | Body is not valid JSON or does not match either Grafana shape | | 403 | { "error": "service_disabled" } | The service exists but monitoring is paused | | 404 | { "error": "service_deleted_or_not_found" } | The service does not exist | | 429 | { "error": "rate_limited" } | Storm protection active (sustained high volume) |