Considering freelancing? Check out the contract positions from Toughbyte.

Managing data pipelines with Luigi

Luigi is a Python library from Spotify that helps you build and manage complex pipelines for data processing.

The talk introduces the concept of workflow and data pipeline management and why almost everybody working with data will benefit from using such tools.

I'll describe how we use Luigi to help us to build and manage ETL (extract-transform-load) tasks at Custobar (