Build Your Own Kafka
Apache Kafka is the backbone of modern data streaming. In this challenge, you will build a distributed log-structured message broker. You'll handle TCP connections, implement a binary protocol, manage partition logs on disk, and coordinate consumer groups.
Implementation Roadmap
01The TCP Server
Spin up a TCP server that can handle concurrent connections and decode a custom binary frame format.
02The Write-Ahead Log
Implement an append-only log structure on disk with offset indexing for fast lookups.
03Topics & Partitions
Abstract the log into Topics and Partitions, allowing logical separation of data streams.
04Producers & Consumers
Implement the API for sending messages (Produce) and reading them (Consume) starting from an offset.
05Consumer Groups
Add coordination logic to track consumption state across multiple consumers.
Module Status: Upcoming
The interactive modules for this challenge are currently being finalized.
(Detailed coursework and verification systems are in development)