Monitoring Log Statements in Go

Good monitoring and alerting are essential for running services in production. As a Java developer, I’m a bit spoiled by my platform of choice, Spring Boot 2. Thanks to Micrometer it provides out of the box instrumentation for HTTP requests, data sources, caches, memory, threads, logging, and many more. When playing with Go, I found the experience less than ideal, to say the least.

One of the most useful things to alert on is the number of warnings or errors that an application logs because that catches a lot of unexpected issues that are not monitored by other means. With the logrus logging framework and Prometheus this is easy: we implement the Hook interface and register our code with the framework (if you prefer zap, they have pretty much the same concept).

Have a look at this example:

package monitoring

import (
    log "github.com/sirupsen/logrus"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

func NewPrometheusHook() *PrometheusHook {
    counter := promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "log_statements_total",
            Help: "Number of log statements, differentiated by log level.",
        },
        []string{"level"},
    )

    return &PrometheusHook{
        counter: counter,
    }
}

type PrometheusHook struct {
    counter *prometheus.CounterVec
}

func (h *PrometheusHook) Levels() []log.Level {
    return log.AllLevels
}

func (h *PrometheusHook) Fire(e *log.Entry) error {
    h.counter.WithLabelValues(e.Level.String()).Inc()
    return nil
}

We create a Prometheus counter that is incremented for each log statement on each log level. All that’s left is register it from your main function:

 log.AddHook(monitoring.NewPrometheusHook())

In my toy example, this is what I get when querying the Prometheus metrics endpoint:

log_statements_total{level="error"} 3
log_statements_total{level="info"} 19
log_statements_total{level="warning"} 1

Very easy to implement, no log parsing required.

Advertisements
This entry was posted in go, Uncategorized and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s