Scaling Your Data Pipelines with Prefect and Coiled

In our October webinar, Matt Rocklin, CEO of Coiled, and Chris White, CTO of Prefect, shared how data teams can drive higher pipeline performance at scale with less engineering overhead using Prefect and Coiled together. They also discussed trends in the Python space and shared some personal learnings and advice.

Prefect was founded with the aim to empower its users. They create frameworks that govern automation and deliver confidence. Prefect focuses on design because they know that sophisticated products can still be easy to use.

Many of the people who work at Prefect are reformed data scientists and engineers who have been given the means to create the tools they wished they had in previous careers. 

In this webinar, Chris and Matthew cover:

  • What Prefect and Coiled do, and how they fit into the data engineering landscape
  • Challenges data engineers face when needing to scale and when the right time is to consider a scaled solution
  • How to scale with Prefect and Coiled, with hands-on examples, from a single process to an auto-scaling cluster
  • Matt and Chris’s views on where the modern data stack is going and trends to follow

Level up your Dask using Coiled

Coiled makes it easy to scale Dask maturely in the cloud