Testing Alertmanager

Posted on September 13, 2020 by

When a Prometheus alerting rule fires, the Prometheus server sends a notification to the Alertmanager, which is then responsible for processing that alert further, i.e. by routing it to an appropriate alerting channel (e-mail, Slack, …). In order to test the Alertmanager configuration, it is useful to trigger alerts directly via Alertmanager’s API. That API is not documented on the Prometheus website, but it’s easy enough to figure out how it works.

Continue reading →
Posted in shell | Tagged , | Leave a comment

Playing with WebAuthn

Posted on September 8, 2019 by

On the internet today, our online accounts are exposed to a lot of risks. Leaked passwords from break-ins at major Internet properties or phishing attacks have lead to many account breaches. In order to protect my high-value accounts, I have either enabled two-factor authentication or I use federated sign-in, delegating to an account that has 2FA enabled (usually Google).

Recently, the FIDO industry association and the W3C have come up with WebAuthn, an open standard to promote passwordless sign-in using hardware tokens. If you run Android 7 or newer, you can use your phone as a token, and I have additionally bought a hardware Security Key from Yubikey to give this a try.

Continue reading →

Posted in best practices | Tagged , | 1 Comment

Using Kubectl Printers and Plugins

Posted on April 22, 2019 by

Even though we have plenty of metrics and dashboards at work, I use the kubectl command line tool a lot for looking at resources and for troubleshooting. Because the defaults don’t always display the information that I need, I often use the kubectl printer mechanisms. In many cases, this is flexible enough so that I don’t have to write a custom script against the Kubernetes API.

Continue reading →

Posted in tools | Tagged , | Leave a comment

Monitoring Log Statements in Go

Posted on March 3, 2019 by

Good monitoring and alerting are essential for running services in production. As a Java developer, I’m a bit spoiled by my platform of choice, Spring Boot 2. Thanks to Micrometer it provides out of the box instrumentation for HTTP requests, data sources, caches, memory, threads, logging, and many more. When playing with Go, I found the experience less than ideal, to say the least.

Continue reading →

Posted in go, Uncategorized | Tagged , , | Leave a comment

Additional Workspaces in Gnome

Posted on February 24, 2019 by

I use Gnome on Ubuntu 18.04 with static workspaces and switch between them using keyboard shortcuts – for example, ALT+1 takes me to workspace one. Unfortunately, using the graphical settings editor, I was only able to assign keyboard shortcuts to the first four workspaces. After a bit of research and playing with gconf-editor, I found the command line to achieve what I need.

Continue reading →

Posted in misc | Tagged | Leave a comment

Retries Can Kill You

Posted on July 30, 2018 by

In a large-scale distributed system, it’s inevitable that some requests will fail. Even if your collaborating systems work perfectly, sooner or later you will experience temporary network issues and other intermittent errors. That’s why a lot of people try to paint over this issue by implementing retries in their applications. Unfortunately, if it’s not done properly, this can cause serious stability problems.

Continue reading →

Posted in best practices | Tagged , , | Leave a comment

Prometheus and Spring Boot Health Checks

Posted on June 24, 2018 by

When trying to set up alerting for Spring Boot services with Prometheus, I discovered the synthetic “up” time series which is great for checking whether the monitoring system can reach my service instances. While this is a great thing, I also wanted to alert on the health status of my instances, as reported by /actuator/health. Unfortunately, there is nothing in Spring Boot’s /actuator/prometheus endpoint that I could use.

Continue reading →

Posted in java | Tagged | 5 Comments

DNS Over HTTPS

Posted on April 8, 2018 by

With much of the web switching to HTTPS and at least some people becoming more concerned about privacy, DNS has recently come into the spotlight because it provides an ISP with the ability to monitor which websites a user visits. Multiple mitigations have been proposed – Android for example is going to support DNS Over TLS, which tunnels good old DNS payload over TLS. And now there is a working group at the IETF developing DNS Over HTTPS (DoH), which layers DNS on top of HTTP/2.

Continue reading →

Posted in misc | Tagged , , | 2 Comments