Building real-time data flows with Kafka on Mesos [Mesosphere HackWeek]

This installment of the Mesosphere HackWeek series covers the exciting work being done to integrate Apache Kafka and Apache Mesos.

Kafka is a high-throughput, low-latency publish-subscribe messaging system. Originally developed at LinkedIn, Kafka was open-sourced in 2011 and has been adopted widely across the industry at companies such as Cisco, Goldman Sachs, Netflix, and many more. It forms the backbone of critical real-time data flows in these companies, processing hundreds of billions of messages per day and is typically used in tandem with stream-processing systems such as Storm, Spark or Samza.

During the HackWeek, we worked with our partner Confluent (founded by the team that developed Kafka at LinkedIn) on a scheduler that runs on Marathon and allows users to launch any version of Kafka. The scheduler takes care of operational automation, manages broker configurations, exposes a REST API, and provides for a command line interface that enables elastic scaling and smart partitioning.

To learn more about simplifying the operation of Kafka with Mesos and the Mesosphere Datacenter Operating System, see: