Grafana alerts to any webhook, held and retried
Short answer: In Grafana, create a contact point with the Webhook integration and put your HoldAlarms webhook URL in URL. HoldAlarms answers Grafana, queues the alert and retries delivery to your endpoint (Grafana: webhook notifier).
Grafana sends a JSON body with an alerts list. Max Alerts limits how many alerts go into one notification; alerts above the limit are dropped, and 0 means no limit.
Why alerts get lost
- Your endpoint is down or restarting. Grafana gets an error and the step fails; your endpoint never gets that alert.
- Slow replies. Grafana waits on your endpoint, and a timeout fails the step.
- Nobody notices. A failed step shows in Grafana's history, but often no one looks until something is missing.
Set it up with HoldAlarms
- Start a trial and copy the webhook URL from your dashboard. It looks like
https://api.holdalarms.com/hook/<your token>. - Add your own endpoint as a destination in HoldAlarms.
- In Grafana, go to Alerts & IRM → Alerting → Notification configuration → Contact points, add a contact point with the Webhook integration, paste the URL, leave HTTP Method on POST and save.
- Send one test and check Recent alerts on your dashboard.
Full reference: Docs: Zapier, Make, n8n and Grafana.
What HoldAlarms does
We answer Grafana with HTTP 200 once the alert is queued, then deliver it to your endpoint in the background. Your endpoint gets 10 seconds per attempt. If it fails, we retry up to 5 times, 2 to 60 seconds apart (permanent errors such as 401 or 403 aren't retried). Retries finish within about 3 minutes, or up to about 6 if your server asks for longer waits with Retry-After. After that the alert is parked in a dead-letter queue and you get an email.
Two honest limits: we add a hop, so HoldAlarms doesn't cut latency; and a delivered alert means your endpoint accepted it, not that your app finished processing it. Starter is €19/mo and Ops €49/mo.
Questions
Can my endpoint spot a repeated Grafana alert?
Yes. Each alert in the body has a fingerprint, and alerts with the same labels have the same fingerprint. Store fingerprint plus startsAt and ignore repeats.
Is the Grafana webhook URL secret?
No. Grafana stores the URL in plain text and shows it when the contact point is read back, so anyone who can view the contact point can see your token. Limit who can view contact points.
Can I keep the token out of the Grafana URL?
Yes, if your Grafana version lets you add custom HTTP headers to the webhook contact point. Use https://api.holdalarms.com/ingest as the URL and send your token in an X-Hook-Token header. See Docs: Zapier, Make, n8n and Grafana.
Is the Grafana HMAC signature passed on?
No. HoldAlarms does not forward custom headers, including the signature header Grafana can add.
Can I use PUT instead of POST?
No. HoldAlarms accepts POST only on the webhook URL.
Are my custom headers passed on?
No. Custom headers you add are not forwarded to your destinations. Put anything your endpoint needs into the body.
Does HoldAlarms reduce latency?
No. HoldAlarms adds a hop, so each alert reaches your endpoint slightly later than it would directly. What it gives you is an answer to Grafana as soon as the alert is queued, and 5 retries to your endpoint.
More guides
- Zapier webhook alerts
- Make webhook alerts
- n8n webhook alerts
- Webhook alert delivery: slow or down endpoints
- All guides
- All HoldAlarms questions
HoldAlarms is not affiliated with Zapier, Make, n8n or Grafana Labs. Their names are trademarks of their owners.