Before You Begin

  • Docker & Docker Compose (v2.20+)
  • Go 1.22+ (for custom metrics instrumentation)
  • Prometheus, Grafana, AlertManager (we'll use official Docker images)
  • Basic familiarity with YAML, Go, and HTTP endpoints

Architecture: The WildBlog Engine

We'll run a multi-service blog engine with three animal-themed services:

  • Fox – post service (port 8081)
  • Owl – comment service (port 8082)
  • Badger – auth service (port 8083)

Each service exposes a /metrics endpoint built with Prometheus Go client library. A single Prometheus instance scrapes all three, and AlertManager handles routing. Grafana visualizes everything with dashboards and alert annotations.

1. Instrumenting Services with Custom Metrics

A common pattern seen in production Go repositories on GitHub is to create a dedicated metrics.go file per service. We'll use the official Prometheus Go client (v1.20.0).

// fox/metrics.go
package main

import (
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
)

var (
	foxRequestsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
		Name: "fox_requests_total",
		Help: "Total number of HTTP requests to Fox service",
	}, []string{"method", "endpoint", "status"})

	foxRequestDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
		Name:    "fox_request_duration_seconds",
		Help:    "Latency of HTTP requests",
		Buckets: prometheus.DefBuckets,
	}, []string{"method", "endpoint"})

	foxErrorsTotal = promauto.NewCounter(prometheus.CounterOpts{
		Name: "fox_errors_total",
		Help: "Total number of 5xx errors",
	})
)

Register the metrics handler in main.go:

http.Handle("/metrics", promhttp.Handler())

Each service follows the same pattern, replacing fox with owl and badger. This ensures label consistency across services for aggregated dashboards.

Key best practice: Use promauto to avoid boilerplate registration. Many developers on StackOverflow encounter the "duplicate metrics collector" error when manually registering – promauto handles that.

2. Prometheus Configuration with Intelligent Relabeling

We'll use Docker Compose to run all services. Prometheus needs to discover each service's metrics endpoint. The official docs recommend using dnssdconfigs or static_configs for composed environments.

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'blog-engine'
    static_configs:
      - targets:
          - 'fox:8081'
          - 'owl:8082'
          - 'badger:8083'
        labels:
          environment: 'production'
          team: 'wildblog'
    relabel_configs:
      - source_labels: [__address__]
        regex:  '(.*):.*'
        target_label: 'service'
      - source_labels: [__address__]
        regex:  '.*:(\d+)'
        target_label: 'port'
      - source_labels: [__meta_docker_container_name]
        target_label: 'container_name'

Add recording rules for SLOs:

# prometheus-rules.yml
groups:
  - name: slo_metrics
    interval: 30s
    rules:
      - record: job:fox_error_rate_5m
        expr: rate(fox_errors_total[5m]) / rate(fox_requests_total[5m])
      - record: job:all_services_p99_latency
        expr: histogram_quantile(0.99, sum(rate(fox_request_duration_seconds_bucket[5m])) by (le, service))

3. Grafana Dashboards with Alert Annotations

Create a dashboard that uses alert state annotations from AlertManager. Grafana's built-in alert annotation datasource can pull from AlertManager API.

Add a variable $service with query: label_values(up, service)

Panels:

  • Request Rate (QPS) per service: rate(foxrequeststotal[$_rateinterval]) – use $service filter.
  • Error Rate (5xx): rate(foxerrorstotal[5m]) / rate(foxrequeststotal[5m])
  • Latency Heatmap: histogram_quantile(0.99, ...)

Enable alert annotations: Go to Dashboard settings → Annotations → Add annotation query → Type: AlertManager, URL: http://alertmanager:9093.

This gives you visual markers on the graph when alerts fire.

4. AlertManager: Inhibition, Routing, and Grouping

AlertManager configuration for realistic multi-receiver routing:

# alertmanager.yml
route:
  receiver: 'default'
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  group_by: ['alertname', 'service']
  routes:
    - match:
        severity: 'critical'
      receiver: 'slack_ops'
      continue: true
    - match:
        severity: 'warning'
      receiver: 'email_devs'

receivers:
- name: 'default'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/T...'
    channel: '#alerts-default'

- name: 'slack_ops'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/T...'
    channel: '#ops-critical'
    title: '{{ .GroupLabels.alertname }} - {{ .CommonLabels.service }}'

- name: 'email_devs'
  email_configs:
  - to: 'dev-team@wildblog.com'
    from: 'alertmanager@wildblog.com'
    smarthost: 'smtp.example.com:587'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['service', 'alertname']

This inhibits warnings for the same service/alert when a critical fires – reducing noise. Engineering blogs from companies like SoundCloud advocate for this pattern to combat alert fatigue.

Add alerting rules in Prometheus:

- alert: HighErrorRate
  expr: job:fox_error_rate_5m > 0.05
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Fox service error rate > 5%"

5. Multi-Region Federation with Thanos

If your blog engine runs in two data centers (e.g., us-east, eu-west), you need hierarchical aggregation. Use Thanos sidecar to expose Prometheus TSDB and write to a central Thanos querier.

Docker Compose snippet for Thanos sidecar:

thanos-sidecar:
  image: quay.io/thanos/thanos:v0.34.0
  command:
    - sidecar
    - --tsdb.path=/prometheus
    - --prometheus.url=http://prometheus:9090
    - --grpc-address=0.0.0.0:10901

Then a central Thanos querier can query both regions. For alerting, each region has its own AlertManager, but you can also set up a global AlertManager federation via alertmanagers in Prometheus config.

6. Dockerfile Best Practices for the Services

Each service Dockerfile follows the multi-stage pattern from the standard:

# fox/Dockerfile
FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o fox .

FROM alpine:3.19
RUN adduser -D -u 1001 wildblog
USER wildblog
COPY --from=builder /app/fox /fox
EXPOSE 8081
CMD ["/fox"]

Include HEALTHCHECK instruction:

HEALTHCHECK --interval=30s --timeout=3s CMD wget -qO- http://localhost:8081/health || exit 1

7. Kubernetes Manifest (Optional Production Target)

If deploying to Kubernetes, the deployment must include resource requests/limits, probes, and ConfigMap for Prometheus config. Example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fox-service
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: app
        image: wildblog/fox:latest
        ports:
        - containerPort: 8081
        env:
        - name: METRICS_PORT
          value: "8081"
        resources:
          requests:
            memory: "64Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
            cpu: "100m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8081
          initialDelaySeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8081

Common Issues and Solutions

  1. High cardinality labels – Adding user_id as a label can explode metric series. Use a separate counter per user or limit to top N. The Prometheus team recommends careful label selection.
  1. Scrape timeouts – If your services are slow, increase scrapetimeout and use scrapeinterval accordingly. Also check for blocking metrics endpoint.
  1. AlertManager not receiving alerts – Verify alerting block in Prometheus config points to correct AlertManager URL. Use amtool to test.
  1. Duplicate metrics – Ensure each service uses unique metric name prefixes. The promauto package helps but still possible if you register same metric twice.
Sponsored Deal

Next Steps

  • Add Thanos Query for long-term storage (S3/GCS)
  • Implement SLOs with sloth or pyrra
  • Use Grafana Loki for log aggregation alongside metrics
  • Explore OpenTelemetry for distributed tracing