Filtered blogs

This post explains how to convert from a Dask DataFrame to a pandas DataFrame and when it’s a good idea to perform this operation.

With Dask’s map_partitions(), you can work on each partition of your Dask DataFrame, which is a pandas DataFrame, while leveraging parallelism for various custom workflows.

This blog post explains how to read one or multiple CSV files into a Dask DataFrame with read_csv.

This post demonstrates how to speed up a pandas query to run 10 times faster with Dask using six performance optimizations.

Alex Egg, Senior Data Scientist at Grubhub, joins Matt Rocklin and Hugo Bowne-Anderson to talk and code about how Dask and distributed compute are used throughout the user intent classification pipeline at Grubhub!

In this article, we discuss an interesting use case of Dask and Coiled: Accelerating Volumetric X-ray Microstructural Analytics using distributed and high-performance computing.

Unmanaged memory is RAM that the Dask scheduler is not directly aware of and which can cause workers to run out of memory and cause computations to hang and crash.

Pandas is a very powerful Python library for manipulating and analyzing structured data, but it has some limitations. Dask can help.

You can use Coiled to convert large JSON data into a tabular DataFrame stored as Parquet in a cloud object store. Iterate locally first to build and test your pipeline, then transfer the same workflow to Coiled with only minimal code changes.

This post explains how to filter Dask DataFrames based on the DataFrame index and on column values using loc.

This article discusses the problems users looking for a Spark/Databricks replacement face, the relative strengths of Dask/Coiled for large-scale ETL processing, and also the current shortcomings.

This post explains how to create disk partitioned Parquet lakes with Dask using partition_on. It also explains how to read disk partitioned lakes with read_parquet and how this can improve query speeds.

This post outlines the Coiled team's recent experimentation with a new approach to DataFrame shuffling in Dask.

This post gives an overview of DataFrame datatypes (dtypes), explains how to set dtypes when reading data, and shows how to change column types.

Dask release 2021.10.0 introduces the first piece of a new modular system called Active Memory Manager, which aims to alleviate memory issues.

This post demonstrates how to change a DataFrame index with sebt_index and explains when you should perform this operation.

This post demonstrates how to merge Dask DataFrames and discusses important considerations when making large joins.

This blog post explains how to write Parquet files with Dask using the to_parquet method.

Having Dask workers die from memory overuse is common, so we thought that we’d investigate this further.

This article discusses why and how to use both together, and dives into the challenges of bulk parallel reads and writes into data warehouses...

This analysis tracks the growth of Matplotlib on the preprint server arXiv beginning in 2002 with 1% up to 2022 with 17% of all papers using Matplotlib...

This blog post explains how to read Parquet files into Dask DataFrames with read_parquet...

This post will demonstrate how running Github Actions on Coiled can be a useful way to schedule automated data-processing jobs...

Coiled is solely focused on building a Dask service and has a dedicated engineering team where thousands of customers can share costs...

The latest version of Dask (2022.11.0) can significantly reduce your memory usage. Here's how we did it.

Dask developers care about performance, we’ve always taken a pragmatic rather than exciting approach to the problem...

Address performance issues for large-scale data visualizations by making smart choices about cluster memory, data types and data partitioning.

Dask is a flexible Python library for parallel and distributed computing. There are a number of ways you can create Dask clusters, each with their own benefits. In this article, we explore how Coiled provides a managed cloud infrastructure solution for Dask users.