Apache Kafka is a distributed event streaming platform capable of handling trillions of events a day. It's designed for high-throughput, fault-tolerant, and scalable real-time data streaming.
Kafka Architecture
Broker: Kafka server
Topic: Category for messages
Partition: Topic division for parallelism
Producer: Sends messages to topics
Consumer: Reads messages from topics
Core Concepts
Topics and Partitions
Topics are divided into partitions for scalability:
Topic: user-events
├── Partition 0
├── Partition 1
└── Partition 2Message Structure
Each message has:
- Key (optional)
- Value
- Timestamp
- Headers
Producer Implementation
typescript
import { Kafka } from 'kafkajs';
const kafka = new Kafka({
clientId: 'my-app',
brokers: ['localhost:9092']
});
const producer = kafka.producer();
async function produceMessage(topic: string, key: string, value: any) {
await producer.connect();
await producer.send({
topic,
messages: [{
key,
value: JSON.stringify(value),
timestamp: Date.now().toString()
}]
});
await producer.disconnect();
}Consumer Implementation
typescript
const consumer = kafka.consumer({ groupId: 'my-group' });
async function consumeMessages(topic: string) {
await consumer.connect();
await consumer.subscribe({ topic, fromBeginning: false });
await consumer.run({
eachMessage: async ({ topic, partition, message }) => {
const value = JSON.parse(message.value!.toString());
console.log(`Received: ${value}`);
// Process message
await processMessage(value);
}
});
}Consumer Groups
Consumer groups enable parallel processing:
typescript
// Multiple consumers in same group
const consumers = [
kafka.consumer({ groupId: 'processing-group' }),
kafka.consumer({ groupId: 'processing-group' }),
kafka.consumer({ groupId: 'processing-group' })
];
// Each partition assigned to one consumerExactly-Once Semantics
typescript
const transaction = producer.transaction();
try {
await transaction.send({
topic: 'events',
messages: [{ key: '1', value: JSON.stringify(event) }]
});
await transaction.commit();
} catch (error) {
await transaction.abort();
}Conclusion
Kafka excels at real-time data streaming and event-driven architectures.