The cloud is wonderful but expensive. Spot/preemptible instances offer dramatic cost savings, but using them well requires considerable nuance which we explore in this post. 

Address performance issues for large-scale data visualizations by making smart choices about cluster memory, data types and data partitioning.

The latest version of Dask (2022.11.0) can significantly reduce your memory usage. Here's how we did it.

This article will go into a few things that I think are great ideas, but are not yet major bottlenecks and why. Then it will go into a few things that I think are less-than-exciting-but-really-important improvements for common workloads.

Anecdotally the Matplotlib maintainers were told "About 15% of arXiv papers use Matplotlib" arXiv is the preeminent repository for scholarly preprint articles. It stores millions of journal articles used across science. It's also public access, and so we can just scrape the entire thing given enough compute power.

Coiled can often save money for an organization running Dask. This article goes through the most common ways in which we see that happen. 

Kubernetes is great if you need to organize many always-on services and have in-house expertise, but can add an extra burden and abstraction when deploying a single bursty service like Dask, especially in a user environment with quickly changing needs.‍

Having Dask workers die from memory overuse is common, so we thought that we’d investigate this further.