For this Kafka Meetup, we welcome Ricardo De Cillo, Dmitriy Sorokin, Andrey Dyachkov, Max Schultze, Daniel Truemper, Ian Duffy, and Conor Clifford with a series of short talks highlighting tales from the trenches. Topics will be focused on operating large, critical Kafka clusters in the cloud - and hope to foster discussion and share best practices for running Kafka without losing sleep.
*Please bring your ID to guarantee entrance.
17:30 - Doors Open - Drinks + Snacks 18:20 - Welcome and Intro 18:30 - Operating Kafka and Zookeeper on AWS - Ricardo De Cillo 19:45 - Bubuku - A Supervisor to Run Kafka on AWS - Dmitriy Sorokin 19:00 - Kafka and EBS - Andrey Dyachkov 19:15 - The Tricky Thing about Offsets - Max Schultze 19:30 - Break - Drinks, snacks, and discussions 19:45 - Kafka-powered Microservice - Daniel Truemper 20:00 - Lessons learned Building and Operating a new event driven Platform - Conor Clifford 20:15 - Backing up Kafka to S3 - Ian Duffy 20:30 - Networking + Drinks 21:45 - Event ends
For more details on topics and speakers, please read below:
Operating Kafka and Zookeeper on AWS
Speaker: Ricardo De Cillo Deploying and operating Kafka and Zookeeper reliably on AWS is a non trivial task. Most teams learn it the hard way. In this talk we are going to share our experience growing a Kafka cluster to ingest 5 TB of data per day with 99,999% availability.
You can expect to hear about: Deployment, Monitoring, Recovering from various failure scenarios, Scaling, Hardware choices, Important configurations
There is no single solution to this challenge, so let's talk. What choices did you make regarding operations and why?
Speaker: Dmitriy Sorokin
Operating a Kafka cluster on a cloud platform such as AWS comes with its challenges - finding exhibitor nodes, updating the configurations, reacting to lost instances without downtime, rebalancing data across brokers, and many others). Daily operations on large clusters, with thousands of topics, are not trivial, and can take up a lot of time.
To simplify operations, we have built Bubuku, a supervisor for Kafka. In this talk, we will explore some of Bubuku's most useful features, such as rebalances and migration of data, updates of Zookeeper, and rolling restarts of Kafka brokers. We will share our experience in running a large cluster in production using Bubuku, and discuss future plans for the project.
Speaker: Andrey Dyachkov (github)
Kafka cluster is able to grow to huge amount of data stored on the disks. Hosting Kafka, requires support of instance termination (on purpose or just because
cloud provider decided to terminate the instance), which in our case introduces a node with no data, the rebalance of the whole cluster has to accomplished in order to evenly distribute the data among the nodes, taking hours of data copying. In this talk, we will present how we avoid rebalance after node termination in Kafka cluster hosted on AWS.
Speaker: Max Schultze (twitter)
The Data Lake at Zalando is all about archiving data and making it accessible efficiently. For that, we consume a lot of data from Zalando's internal event bus Nakadi, which is based on Kafka. Additionally, we run another internal Kafka that data flows through first before it gets archived. This talk will be about presenting our current infrastructure, as well as talking about the concept you can mess up the most with during manual operations: offsets.
Speaker: Daniel Truemper (twitter, github)
Microservices have fueled an industry-wide transition toward massively distributed architectures with many more people working on the same systems. They also allow for greater individual ownership, freedom of deployment, and diverse technology choices.
In this distributed landscape, easily creating and maintaining consistency among various views of business-critical data, decoupling reads from writes, decoupling microservice go-lives, while adhering strictly to incrementality emerge as important topics.
This presentation will demonstrate how Kafka can function as a backbone for the microservice world thanks to log compaction, which makes it unique compared to other event brokers. We’ll show how log compaction is essential in delivering a consistent user experience, and how this can empower teams developing their own autonomy. We will walk through a migration path from a hypothetical monolithic shop to a set of microservices.
Speaker: Conor Clifford (twitter)
In this talk, we present several lessons learned by our team, Buffalo, while building, and operating, the new Smart Product Platform - an event driven architecture built on top of Kafka and Nakadi.
Backing up Kafka to S3 Speaker: Ian Duffy (website, github, twitter)
In this talk, we demonstrate how we used Kafka Connect to backup topics to Amazon S3. We show how we can get a backup up and running in minutes, and how to recover data when everything goes wrong.