Matthias Friedrich's Blog

Testing Alertmanager

When a Prometheus alerting rule fires, the Prometheus server sends a notification to the Alertmanager, which is then responsible for processing that alert further, i.e. by routing it to an appropriate alerting channel (e-mail, Slack, ...). In order to test the Alertmanager configuration, it is useful to trigger alerts directly via Alertmanager's API. That API is not documented on the Prometheus website, but it's easy enough to figure out how it works.

To see what's going on, I created a simple Prometheus alerting rule that checks for a server that isn't there, triggering an alert:

groups:
- name: example
  rules:
  - alert: InstanceDown
    expr: up{job="node"} == 0
    labels:
      severity: critical
    annotations:
      summary: Instance is down

Instead of an Alertmanager instance, we run netcat to see the API request:

$ nc -l 9093
POST /api/v1/alerts HTTP/1.1
Host: localhost:9093
User-Agent: Prometheus/2.21.0
Content-Length: 330
Content-Type: application/json

[{"labels":{"alertname":"InstanceDown","instance":"localhost:8080","job":"node","severity":"critical"},"annotations":{"summary":"Instance is down"},"startsAt":"2020-09-13T12:27:02.153716049Z","endsAt":"2020-09-13T12:31:02.153716049Z","generatorURL":"http://sia:9090/graph?g0.expr=up%7Bjob%3D%22node%22%7D+%3D%3D+0\u0026g0.tab=1"}]
$

That seems simple enough. Let's format it and put it into a shell script:

#! /usr/bin/env sh

URL="http://localhost:9093/api/v1/alerts"

curl -si -X POST -H "Content-Type: application/json" "$URL" -d '
[
  {
    "labels": {
      "alertname": "InstanceDown",
      "instance": "localhost:8080",
      "job": "node",
      "severity": "critical"
    },
    "annotations": {
      "summary": "Instance is down"
    },
    "generatorURL": "http://localhost:9090/graph"
  }
]
'

I have removed the startsAt and endsAt attributes, which are optional. You can see the annotations and labels from the alerting rule plus some additional labels that Prometheus created for us.

Now you can use the script to send the alert to Alertmanager directly.

Other articles

Quick Tip #6: Triggering Actions on File Close

Published: Mon 30 December 2013
By mafr

In shell.

tags: linux quick tips scripting

Sometimes it's useful to trigger an action after a file is closed. Suppose you started a lengthy download on your notebook and you want to suspend it as soon as the download is done. There are several ways to achieve this.

If you started the download using a web browser …
read more
Quick Tip #5: Navigating in Source Trees

Published: Sun 14 April 2013
By mafr

In shell.

tags: linux quick tips

I often move around in checked out Subversion or Git source trees on a Bash command line. Directory structures tend to get quite deep (especially in Java projects), so one problem keeps coming up frequently: Changing back to the base directory of the source tree. Typing "cd .." ten times is …
read more
Bourne Shell Parameter Expansion

Published: Sun 20 January 2013
By mafr

In shell.

tags: scripting

Classic Bourne Shell has a handy feature that many people don't seem to know: Parameter Expansion. Stephen Bourne describes it in An Introduction to the UNIX Shell, which is part of the original Unix V7 manuals from 1978. Among other things, parameter expansion provides a notation for making sure that …
read more
Scripts That Just Don't Work

Published: Sun 22 July 2012
By mafr

In shell.

tags: linux scripting

Sometimes people ask for my help when their shell scripts don't work. I've seen all kinds of problems, ranging from trivial to really, really weird. The one I'm describing in this article is definitely from the weird category, but I've seen it quite a few times.

The story goes like …
read more
Quick Tip #4: Sorting Large Files

Published: Sun 23 May 2010
By mafr

In shell.

tags: linux quick tips scripting tools

With traditional Unix sort(1), the size of the files you can sort is limited by the amount of available main memory. As soon as the file gets larger and your system has to swap, performance degrades significantly. Even GNU sort which uses temporary files to get around this limitation …
read more
Quick Tip #2: Set Operations Using Shell Tools

Published: Wed 19 August 2009
By mafr

In shell.

tags: linux quick tips tools

Everybody knows that Unix shell utilities are powerful. Even though they're text-based, you can build a lot of useful things outside of the text domain. Today I'll show you how to implement set operations. All we need are sorted files as input, with each file representing a set.

Let's create …
read more
Quick Tip #1: Implicit FIFOs in Bash

Published: Sat 05 July 2008
By mafr

In shell.

tags: linux quick tips tools

One could assume that I have run out of material and retreat to safer ground, but far from it. From now on I'll just throw in a few quick shell hacks hoping some of you don't know them yet. So, here it goes ...

Unix pipes are cool, but it's quite …
read more
Parsing Command Line Options in Shell Scripts

Published: Sun 05 August 2007
By mafr

In shell.

tags: linux scripting

In programs written in C, command line argument parsing has always been done using the getopt(3) library function. This function has set the standards Linux/Unix users have come to expect from command line interfaces. Fortunately, there's a getopt(3) equivalent for almost every programming language and the shell …
read more
Changing Directories Quickly
Published: Sun 11 February 2007
By mafr

In shell.

tags: linux

When you work with the interactive shell, there's a common pattern that is used a lot: Copying or moving files to a directory and then changing to the target directory using cd. In these cases, you find yourself typing the name twice:
```
$ cp file1 file2 file3 /very/long/path/name …
```
read more

Other articles

social