Building Production-Grade Monitoring with Prometheus

Prometheus has become the de facto standard for monitoring cloud-native applications. Its powerful data model, flexible query language, and ecosystem make it ideal for production environments. Let's explore how to build a comprehensive monitoring solution with Prometheus.

Why Prometheus?

Prometheus offers several advantages that make it perfect for modern monitoring:

Pull-based monitoring: Prometheus scrapes targets, pushing complexity to the edge. This makes it more reliable and easier to manage.

Multi-dimensional data model: Metrics can have multiple labels, enabling powerful querying and aggregation.

Powerful query language: PromQL allows complex queries and aggregations.

Service discovery: Native integration with Kubernetes and other platforms.

Open source: Large community and no vendor lock-in.

Prometheus Architecture

Understanding Prometheus's architecture is crucial for effective implementation:

Prometheus Server: The core that scrapes and stores time-series data.

Exporters: Programs that expose metrics for Prometheus to scrape.

Alertmanager: Handles alert routing and deduplication.

Pushgateway: For short-lived jobs (use sparingly).

Getting Started with Prometheus

Installation

Start with Docker for development:

yaml

# docker-compose.yml
version: '3'
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana_data:/var/lib/grafana

volumes:
  prometheus_data:
  grafana_data:

Basic Configuration

yaml

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'production'
    environment: 'us-west-2'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'application'
    static_configs:
      - targets: ['localhost:8080']
    metrics_path: '/metrics'

Instrumenting Your Application

Using the Prometheus Client

First, install the client library:

bash

npm install prom-client

Basic Metrics Collection

typescript

import promClient from 'prom-client';

// Create a Registry
const register = new promClient.Registry();

// Add default metrics (CPU, memory, etc.)
promClient.collectDefaultMetrics({ register });

// Define a counter
const httpRequestsTotal = new promClient.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status_code']
});

// Define a histogram
const httpRequestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.1, 0.5, 1, 2, 5]
});

// Define a gauge
const activeConnections = new promClient.Gauge({
  name: 'active_connections',
  help: 'Number of active connections'
});

// Express middleware
export function promMiddleware(req: any, res: any, next: any) {
  const start = Date.now();

  activeConnections.inc();

  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    const route = req.route?.path || req.path;

    httpRequestsTotal.inc({
      method: req.method,
      route,
      status_code: res.statusCode
    });

    httpRequestDuration.observe({
      method: req.method,
      route,
      status_code: res.statusCode
    }, duration);

    activeConnections.dec();
  });

  next();
}

// Metrics endpoint
export async function metricsEndpoint(req: any, res: any) {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
}

Custom Business Metrics

Track metrics specific to your business:

typescript

// Business metrics
const orderTotal = new promClient.Counter({
  name: 'orders_total',
  help: 'Total number of orders',
  labelNames: ['status', 'payment_method']
});

const orderValue = new promClient.Histogram({
  name: 'order_value_usd',
  help: 'Order value in USD',
  buckets: [10, 50, 100, 200, 500, 1000]
});

const inventoryLevel = new promClient.Gauge({
  name: 'inventory_level',
  help: 'Current inventory level',
  labelNames: ['product_id', 'warehouse']
});

// Usage in your application
function createOrder(order: Order) {
  try {
    const result = processOrder(order);

    orderTotal.inc({
      status: 'success',
      payment_method: order.paymentMethod
    });

    orderValue.observe(order.total);

    return result;
  } catch (error) {
    orderTotal.inc({
      status: 'failed',
      payment_method: order.paymentMethod
    });
    throw error;
  }
}

Advanced Prometheus Features

Labels: The Secret Weapon

Labels make Prometheus powerful. Use them wisely:

typescript

// Good: Specific labels
const apiRequests = new promClient.Counter({
  name: 'api_requests_total',
  help: 'Total API requests',
  labelNames: ['endpoint', 'version', 'client_type']
});

// Bad: Too many labels (cardinality explosion)
const badCounter = new promClient.Counter({
  name: 'user_requests_total',
  help: 'Total user requests',
  labelNames: ['user_id'] // Don't use high-cardinality labels!
});

Summary Metrics

Use summaries for pre-calculated quantiles:

typescript

const apiLatency = new promClient.Summary({
  name: 'api_latency_seconds',
  help: 'API latency',
  percentiles: [0.5, 0.9, 0.99],
  maxAgeSeconds: 300,
  ageBuckets: 5
});

Metric Families

Group related metrics:

typescript

const databaseMetrics = {
  connections: new promClient.Gauge({
    name: 'db_connections',
    help: 'Database connections',
    labelNames: ['database', 'state']
  }),
  queryDuration: new promClient.Histogram({
    name: 'db_query_duration_seconds',
    help: 'Database query duration',
    labelNames: ['database', 'operation', 'table']
  }),
  queryErrors: new promClient.Counter({
    name: 'db_query_errors_total',
    help: 'Database query errors',
    labelNames: ['database', 'operation']
  })
};

PromQL: Query Language

Prometheus's query language is powerful and expressive.

Basic Queries

promql

# Get current value
http_requests_total

# Get rate over time
rate(http_requests_total[5m])

# Get increase over time
increase(http_requests_total[1h])

# Filter by label
http_requests_total{status_code="500"}

# Regex match
http_requests_total{route=~"/api/.*"}

Aggregation Operators

promql

# Sum by label
sum(http_requests_total) by (status_code)

# Average
avg(http_request_duration_seconds)

# Max value
max(memory_usage)

# Count
count(up == 0)

Advanced Queries

promql

# 95th percentile
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Percentage of errors
sum(rate(http_requests_total{status_code=~"5.."}[5m])) /
sum(rate(http_requests_total[5m])) * 100

# SLO calculation
sum(rate(http_requests_total{status_code!~"5.."}[5m])) /
sum(rate(http_requests_total[5m]))

Alerting with Prometheus

Alert Rules

Define alert rules in a separate file:

yaml

# alerts.yml
groups:
  - name: api_alerts
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status_code=~"5.."}[5m])) /
          sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }}"

      - alert: HighLatency
        expr: |
          histogram_quantile(0.95,
            rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High P95 latency"
          description: "P95 latency is {{ $value }}s"

      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service is down"
          description: "{{ $labels.instance }} has been down for more than 1 minute"

Configuring Alertmanager

yaml

# alertmanager.yml
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'critical'
      continue: true

    - match:
        severity: warning
      receiver: 'warnings'

receivers:
  - name: 'default'
    webhook_configs:
      - url: 'http://slack:3000/hooks/prometheus'

  - name: 'critical'
    pagerduty_configs:
      - service_key: 'YOUR_KEY'
    slack_configs:
      - api_url: 'YOUR_WEBHOOK_URL'
        channel: '#alerts-critical'

  - name: 'warnings'
    slack_configs:
      - api_url: 'YOUR_WEBHOOK_URL'
        channel: '#alerts-warning'

Production Best Practices

Storage Optimization

yaml

# prometheus.yml
global:
  # Reduce retention if storage is limited
  # retention for metrics
  retention.time: 15d

  # Sample data to reduce storage
  retention.size: 10GB

Performance Tuning

yaml

# prometheus.yml
global:
  # Adjust scrape intervals
  scrape_interval: 15s
  evaluation_interval: 15s

# Reduce number of samples
scrape_configs:
  - job_name: 'application'
    scrape_interval: 30s
    scrape_timeout: 10s
    sample_limit: 10000

High Availability

Run multiple Prometheus servers:

yaml

# Server 1
global:
  external_labels:
    replica: 'a'

# Server 2
global:
  external_labels:
    replica: 'b'

Federation and Remote Write

Remote Write to Long-term Storage

yaml

# prometheus.yml
remote_write:
  - url: "http://thanos-receiver:19291/api/v1/receive"
    queue_config:
      capacity: 10000
      max_shards: 200
      min_shards: 1
      max_samples_per_send: 5000
      batch_send_deadline: 5s
      min_backoff: 30ms
      max_backoff: 100ms

Federating Across Clusters

yaml

scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'
    static_configs:
      - targets:
        - 'prometheus-1:9090'
        - 'prometheus-2:9090'

Monitoring Prometheus Itself

Prometheus exposes its own metrics:

promql

# Prometheus target health
up{job="prometheus"}

# Scraping errors
rate(prometheus_target_scrapes_pool_exceeded_total[5m])

# Storage performance
rate(prometheus_tsdb_compaction_duration_seconds_sum[5m])

# Alert evaluation
prometheus_rule_evaluation_duration_seconds

Complete Example

Here's a complete production-ready setup:

typescript

import express from 'express';
import promClient from 'prom-client';

const app = express();
const register = new promClient.Registry();

promClient.collectDefaultMetrics({ register });

const httpRequestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route', 'status'],
  buckets: [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5]
});

const httpRequestsTotal = new promClient.Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status']
});

register.registerMetric(httpRequestDuration);
register.registerMetric(httpRequestsTotal);

app.use((req, res, next) => {
  const start = Date.now();

  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    const route = req.route?.path || req.path;

    httpRequestDuration.observe({
      method: req.method,
      route,
      status: res.statusCode
    }, duration);

    httpRequestsTotal.inc({
      method: req.method,
      route,
      status: res.statusCode
    });
  });

  next();
});

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

app.get('/api/users', (req, res) => {
  res.json({ users: [] });
});

app.listen(8080, () => {
  console.log('Server running on port 8080');
});

Conclusion

Prometheus provides a powerful, scalable foundation for monitoring modern applications. Its pull-based model, multi-dimensional metrics, and query language make it ideal for production environments.

Start with basic metrics, add labels thoughtfully, create meaningful alerts, and continuously iterate on your monitoring strategy. With Prometheus, you'll have the visibility you need to build reliable systems.

Remember: Good monitoring is an iterative process. Start simple, measure what matters, and refine based on what you learn.

Building Production-Grade Monitoring with Prometheus

Why Prometheus?

Prometheus Architecture

Getting Started with Prometheus

Installation

Basic Configuration

Instrumenting Your Application

Using the Prometheus Client

Basic Metrics Collection

Custom Business Metrics

Advanced Prometheus Features

Labels: The Secret Weapon

Summary Metrics

Metric Families

PromQL: Query Language

Basic Queries

Aggregation Operators

Advanced Queries

Alerting with Prometheus

Alert Rules

Configuring Alertmanager

Production Best Practices

Storage Optimization

Performance Tuning

High Availability

Federation and Remote Write

Remote Write to Long-term Storage

Federating Across Clusters

Monitoring Prometheus Itself

Complete Example

Conclusion

Related essays

Modern CLI Frameworks: Oclif, Yargs, and Cac

Building CLI Tools with Node.js: Complete Guide

Git LFS: Managing Large Files in Git Repositories