The cloud is wonderful but expensive. Spot/preemptible instances offer dramatic cost savings, but using them well requires considerable nuance which we explore in this post. 

Address performance issues for large-scale data visualizations by making smart choices about cluster memory, data types and data partitioning.

This article will go into a few things that I think are great ideas, but are not yet major bottlenecks and why. Then it will go into a few things that I think are less-than-exciting-but-really-important improvements for common workloads.

Coiled can often save money for an organization running Dask. This article goes through the most common ways in which we see that happen. 

Anecdotally the Matplotlib maintainers were told "About 15% of arXiv papers use Matplotlib" arXiv is the preeminent repository for scholarly preprint articles. It stores millions of journal articles used across science. It's also public access, and so we can just scrape the entire thing given enough compute power.

Kubernetes is great if you need to organize many always-on services and have in-house expertise, but can add an extra burden and abstraction when deploying a single bursty service like Dask, especially in a user environment with quickly changing needs.‍

Over the last couple of years, Coiled has made a cloud-SaaS application that runs Dask for folks smoothly and securely in the cloud. We thought you would like to hear a bit about the choices we made and why.

Along with the Cloud SaaS product, Coiled sells enterprise support for Dask. This article will be a more informal description of how we operate and why.

This article discusses the problems users looking for a Spark/Databricks replacement face, the relative strengths of Dask/Coiled for large-scale ETL processing, and also the current shortcomings.