28. Scio: Moving Big Data to Google Cloud, a Spotify Story

Agenda 
• 17.45: drink, socialize
• 18.00: first talk: Scio: Moving Big Data to Google Cloud, a Spotify Story
Abstract: We will talk about Spotify’s story of migrating our big data infrastructure to Google Cloud. Over the past year or so we moved away from maintaining our own 2500+ node Hadoop cluster to managed services in the cloud. We replaced two key components in our data processing stack, Hive and Scalding, with BigQuery and Scio and are able to iterate at a much faster speed. We will focus the technical aspect of Scio, a Scala API for Apache Beam and Google Cloud Dataflow and how it changed the way we process data.
Speaker: Neville Li - Neville is a software engineer at Spotify working on data and machine learning infrastructure. In the past few years he has been driving the adoption of Scala and new tools for data processing, including Scalding, Spark, Storm and Parquet. Before that he worked on search quality at Yahoo! and old school distributed systems like MPI.
• 18.45: eat, drink, socialize  (more) 
• 19.00: TBA
Looking for a second speaker. Reach out to us if you are interested in holding a talk.
• 19.45: drink, socialize  (even more)