Good monitoring and alerting are essential for running services in production. As a Java developer, I'm a bit spoiled by my platform of choice, Spring Boot 2. Thanks to Micrometer it provides out of the box instrumentation for HTTP requests, data sources, caches, memory, threads, logging, and many more. When playing with Go, I found the experience less than ideal, to say the least.

One of the most useful things to alert on is the number of warnings or errors that an application logs because that catches a lot of unexpected issues that are not monitored by other means. With the logrus logging framework and Prometheus this is easy: we implement the Hook interface and register our code with the framework (if you prefer zap, they have pretty much the same concept).

Have a look at this example:

package monitoring

import (
    log "github.com/sirupsen/logrus"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

func NewPrometheusHook() *PrometheusHook {
    counter := promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "log_statements_total",
            Help: "Number of log statements, differentiated by log level.",
        },
        []string{"level"},
    )

    return &PrometheusHook{
        counter: counter,
    }
}

type PrometheusHook struct {
    counter *prometheus.CounterVec
}

func (h *PrometheusHook) Levels() []log.Level {
    return log.AllLevels
}

func (h *PrometheusHook) Fire(e *log.Entry) error {
    h.counter.WithLabelValues(e.Level.String()).Inc()
    return nil
}

We create a Prometheus counter that is incremented for each log statement on each log level. All that's left is register it from your main function:

log.AddHook(monitoring.NewPrometheusHook())

In my toy example, this is what I get when querying the Prometheus metrics endpoint:

log_statements_total{level="error"} 3
log_statements_total{level="info"} 19
log_statements_total{level="warning"} 1

Very easy to implement, no log parsing required.