Monitoring is the backbone of modern application reliability and performance. In today's distributed systems, understanding what's happening in your application is not just nice-to-have—it's essential. Let's dive into the fundamentals of monitoring and how to implement it effectively.
What is Monitoring?
Monitoring is the practice of collecting, analyzing, and acting on data about your application's behavior and performance. It provides visibility into your system's health, helps you detect issues before they become outages, and enables data-driven decision-making.
The Three Pillars of Observability
Before diving deep, it's important to understand that monitoring is part of a broader concept called observability. Observability has three main pillars:
- 1Metrics: Numerical time-series data about your system
- 2Logs: Discrete events recorded by your application
- 3Traces: Records of requests as they travel through distributed systems
These three pillars work together to give you a complete picture of your system's behavior.
Metrics: The Foundation
Metrics are numerical measurements collected over time. They're the foundation of most monitoring systems because they're efficient to store and query.
Types of Metrics
Counters: These only increase (or decrease). They're great for counting events.
// Example: Request counter
let requestCount = 0;
app.use((req, res, next) => {
requestCount++;
next();
});Gauges: These can go up or down. They're perfect for current values like memory usage or active connections.
// Example: Active connections gauge
let activeConnections = 0;
app.on('connection', () => {
activeConnections++;
});
app.on('disconnect', () => {
activeConnections--;
});Histograms: These track the distribution of values. They're essential for understanding percentiles.
// Example: Request duration histogram
const requestDurations = [];
app.use((req, res, next) => {
const start = Date.now();
res.on('finish', () => {
const duration = Date.now() - start;
requestDurations.push(duration);
});
next();
});Key Metrics to Track
Every application should track these fundamental metrics:
Request Rate: How many requests per second your system handles.
Error Rate: Percentage of requests that fail.
Latency: How long requests take to complete (include percentiles like p50, p95, p99).
Saturation: How close your system is to its limits (CPU, memory, disk, network).
Availability: Whether your system is up and responding.
Implementing Metrics Collection
Let's build a simple metrics collection system:
class MetricsCollector {
private counters: Map<string, number> = new Map();
private gauges: Map<string, number> = new Map();
private histograms: Map<string, number[]> = new Map();
incrementCounter(name: string, value: number = 1): void {
const current = this.counters.get(name) || 0;
this.counters.set(name, current + value);
}
setGauge(name: string, value: number): void {
this.gauges.set(name, value);
}
recordHistogram(name: string, value: number): void {
const values = this.histograms.get(name) || [];
values.push(value);
this.histograms.set(name, values);
}
getPercentile(name: string, percentile: number): number {
const values = this.histograms.get(name) || [];
if (values.length === 0) return 0;
const sorted = values.sort((a, b) => a - b);
const index = Math.ceil((percentile / 100) * sorted.length) - 1;
return sorted[index];
}
getMetrics() {
return {
counters: Object.fromEntries(this.counters),
gauges: Object.fromEntries(this.gauges),
histograms: Object.fromEntries(this.histograms)
};
}
}
// Usage
const metrics = new MetricsCollector();
app.use((req, res, next) => {
const start = Date.now();
metrics.incrementCounter('requests.total');
res.on('finish', () => {
const duration = Date.now() - start;
metrics.recordHistogram('request.duration', duration);
if (res.statusCode >= 400) {
metrics.incrementCounter('requests.errors');
}
});
next();
});Logging: Capturing Events
Logs are discrete events recorded by your application. They provide context about what happened and when.
Structured Logging
Structured logging uses a consistent format (usually JSON) that makes logs searchable and analyzable.
interface LogEntry {
timestamp: string;
level: 'info' | 'warn' | 'error' | 'debug';
message: string;
context?: Record<string, any>;
}
class Logger {
log(level: LogEntry['level'], message: string, context?: Record<string, any>) {
const entry: LogEntry = {
timestamp: new Date().toISOString(),
level,
message,
context
};
console.log(JSON.stringify(entry));
}
info(message: string, context?: Record<string, any>) {
this.log('info', message, context);
}
error(message: string, context?: Record<string, any>) {
this.log('error', message, context);
}
}
// Usage
const logger = new Logger();
logger.info('User logged in', {
userId: '123',
ip: '192.168.1.1'
});
logger.error('Database connection failed', {
error: 'Connection timeout',
retries: 3
});Log Levels
Use appropriate log levels to filter and prioritize logs:
DEBUG: Detailed information for debugging.
INFO: General informational messages.
WARN: Warning messages for potentially harmful situations.
ERROR: Error events that might still allow the application to continue.
FATAL: Critical errors that require immediate attention.
Best Practices
- Include context in your logs (userId, requestId, etc.)
- Use structured formats (JSON)
- Don't log sensitive information (passwords, tokens)
- Use appropriate log levels
- Implement log rotation to prevent disk space issues
Alerting: Knowing When to Act
Collecting metrics and logs is useless if you don't know when something needs attention. Alerting bridges this gap.
Setting Up Alerts
Good alerts are actionable, specific, and not noisy.
interface AlertRule {
name: string;
condition: () => boolean;
severity: 'low' | 'medium' | 'high' | 'critical';
message: string;
}
class AlertManager {
private rules: AlertRule[] = [];
addRule(rule: AlertRule) {
this.rules.push(rule);
}
checkRules() {
this.rules.forEach(rule => {
if (rule.condition()) {
this.sendAlert(rule);
}
});
}
sendAlert(rule: AlertRule) {
// Send to your alerting system (PagerDuty, Slack, etc.)
console.log(`ALERT [${rule.severity}]: ${rule.message}`);
}
}
// Usage
const alertManager = new AlertManager();
alertManager.addRule({
name: 'High Error Rate',
condition: () => {
const errorRate = calculateErrorRate();
return errorRate > 0.05; // 5% error rate
},
severity: 'high',
message: 'Error rate is above 5%'
});
alertManager.addRule({
name: 'High Memory Usage',
condition: () => {
const memUsage = process.memoryUsage().heapUsed / process.memoryUsage().heapTotal;
return memUsage > 0.9; // 90% memory usage
},
severity: 'critical',
message: 'Memory usage is above 90%'
});Alerting Best Practices
- Set thresholds based on baseline data, not guesses
- Include context in alerts (what, where, impact)
- Avoid alert fatigue by eliminating noise
- Have different severity levels with different response times
- Regularly review and tune alert thresholds
Dashboards: Visualizing Your Data
Dashboards give you an at-a-glance view of your system's health.
Key Dashboard Elements
Current Status: Show the current state of key metrics.
Historical Trends: Display metrics over time to identify patterns.
Service Health: Show which services are up or down.
Active Incidents: Highlight current problems and their status.
Creating Effective Dashboards
- 1Start with the most critical metrics
- 2Group related metrics together
- 3Use appropriate visualizations (lines for trends, numbers for current values)
- 4Keep it simple—don't overcrowd
- 5Make it actionable
Monitoring Architecture
A good monitoring architecture has several layers:
Application-Level Monitoring
Monitor your application code directly:
- Business metrics (signups, conversions)
- Application performance
- Custom metrics specific to your domain
Infrastructure Monitoring
Monitor the underlying infrastructure:
- CPU, memory, disk, network usage
- System health
- Service availability
Synthetic Monitoring
Active monitoring from external locations:
- Uptime monitoring
- API endpoint checks
- User journey monitoring
Real User Monitoring (RUM)
Passive monitoring of actual user experiences:
- Page load times
- JavaScript errors
- User interactions
Common Monitoring Tools
Prometheus: Open-source metrics collection and alerting
Grafana: Visualization and dashboarding
ELK Stack: Elasticsearch, Logstash, Kibana for logs
DataDog: All-in-one monitoring platform
New Relic: Application performance monitoring
CloudWatch: AWS monitoring service
Implementing a Complete Monitoring System
Let's put it all together:
class MonitoringSystem {
private metrics: MetricsCollector;
private logger: Logger;
private alerts: AlertManager;
constructor() {
this.metrics = new MetricsCollector();
this.logger = new Logger();
this.alerts = new AlertManager();
this.setupAlerts();
}
private setupAlerts() {
this.alerts.addRule({
name: 'High Error Rate',
condition: () => this.getErrorRate() > 0.05,
severity: 'high',
message: 'Error rate threshold exceeded'
});
this.alerts.addRule({
name: 'High Latency',
condition: () => this.getPercentile('request.duration', 95) > 1000,
severity: 'medium',
message: 'P95 latency above 1 second'
});
}
private getErrorRate(): number {
const total = this.metrics.getMetrics().counters['requests.total'] || 0;
const errors = this.metrics.getMetrics().counters['requests.errors'] || 0;
return total > 0 ? errors / total : 0;
}
private getPercentile(metric: string, percentile: number): number {
return this.metrics.getPercentile(metric, percentile);
}
middleware() {
return (req: any, res: any, next: any) => {
const start = Date.now();
const requestId = generateRequestId();
res.setHeader('X-Request-ID', requestId);
res.on('finish', () => {
const duration = Date.now() - start;
this.metrics.incrementCounter('requests.total');
this.metrics.recordHistogram('request.duration', duration);
if (res.statusCode >= 400) {
this.metrics.incrementCounter('requests.errors');
this.logger.warn('Request failed', {
requestId,
statusCode: res.statusCode,
path: req.path,
duration
});
}
this.logger.info('Request completed', {
requestId,
statusCode: res.statusCode,
path: req.path,
method: req.method,
duration
});
});
next();
};
}
startHealthCheck(interval: number = 60000) {
setInterval(() => {
this.alerts.checkRules();
this.metrics.setGauge('memory.usage', process.memoryUsage().heapUsed);
this.metrics.setGauge('cpu.usage', process.cpuUsage().user);
}, interval);
}
}
// Usage
const monitoring = new MonitoringSystem();
app.use(monitoring.middleware());
monitoring.startHealthCheck();Monitoring Best Practices
1. Start Simple
Begin with basic metrics and gradually add more sophistication. You can't monitor everything at once.
2. Measure What Matters
Focus on metrics that directly impact your users and business. Vanity metrics are misleading.
3. Set Realistic Thresholds
Base your alert thresholds on actual data, not arbitrary numbers. Understand your baseline.
4. Monitor the Monitoring
Ensure your monitoring system itself is reliable. If it fails, you lose visibility.
5. Make Metrics Actionable
Every metric should have a corresponding action you can take when it goes outside expected bounds.
6. Keep Context
Logs and metrics without context are hard to interpret. Always include relevant context.
7. Regular Review
Monitoring needs evolve with your application. Regularly review and update your monitoring strategy.
Conclusion
Effective monitoring is fundamental to building reliable applications. Start with the basics—metrics, logs, and alerts—and gradually add sophistication as your needs grow.
Remember that monitoring is not a set-it-and-forget-it practice. It requires ongoing attention and refinement. As your application evolves, so should your monitoring.
The investment you make in monitoring pays dividends in faster incident response, better user experiences, and data-driven improvements. Start monitoring today, and continuously improve based on what you learn.
Your future self (and your users) will thank you.