Open to new job opportunities? Check out the positions from Toughbyte.

Managing data pipelines with Luigi

Teemu Kurppa Teemu Kurppa
Mon
30
May
Metrics Monday at Custobar Metrics Monday Metrics Monday

Luigi is a Python library from Spotify that helps you build and manage complex pipelines for data processing.

The talk introduces the concept of workflow and data pipeline management and why almost everybody working with data will benefit from using such tools.

I'll describe how we use Luigi to help us to build and manage ETL (extract-transform-load) tasks at Custobar (http://custobar.com).