Running PyData libraries on an Apple M1 machine requires you to use an ARM64-tailored version of conda-forge. This article provides a step-by-step guide of how to set that up on your machine using mambaforge.
Note: this article was first published in August 2021. Some of the issues with conda described below have since been resolved. I still find that mambaforge builds environments and installs packages faster and recommend trying it out.
The Apple M1 silicon chip is a fantastic innovation: it’s low-power, high-efficiency, has great battery life, and is inexpensive. However, if you’re running the PyData Stack with default builds, you may encounter some strange behaviour.
When you install a library written for Intel CPU architecture, the code will be run through the Rosetta emulator. This often works great but may cause unexpected problems for specific PyData libraries. For example, I personally encountered this with Dask’s memory usage blowing up when importing a <100MB dataset.
Since the internals of Rosetta is proprietary, there’s no way to know exactly why this is happening.
But fear not, there is a way forward. We can resolve much of this strange behaviour by installing ARM64-native versions of the PyData libraries. While we typically recommend Anaconda/miniconda for most installations, currently, the mambaforge deployment seems to have the smoothest ARM installation process.
Below is a step-by-step guide to installing this version of conda and using it to install your favorite PyData libraries.
1. Check your version of conda
2. Download the mambaforge installer
3. Run the installer.
After restarting your terminal, you should now be able to use conda as you always have. You can, for example, begin by rebuilding your software environments using conda env create -f <path/to/environment.yml>.
Below are some further notes on issues I encountered while trying to run a workload using Dask and XGBoost on Coiled:
from dask import Client
import dask.dataframe as dd
cluster = coiled.Cluster()
client = Client(cluster)
ddf = dd.read_parquet("<path/to/100MB/file/on/s3>")
ddf = ddf.persist()//]]>
I encountered three issues and would like to share my solutions for them as I can imagine others might be running into this as well:
//distributed.nanny — WARNING — Worker exceeded 95% memory budget.
The first issue was resolved by switching over to mambaforge and installing all the required PyData libraries from there, as outlined in the step-by-step guide above.
The second issue was due to a conflict caused by the python-blosc library. For some unknown reason, kernels crash on Apple M1 machines when importing python-blosc (e.g. as part of import coiled) with a segmentation fault. To avoid this, make sure that python-blosc is not installed in your current software environment. If it is, run conda uninstall python-blosc to remove it.
Finally, If you’re looking to run XGBoost on your Apple M1 machine, you’ll need to jump through some extra hoops as there is currently no arm64-compatible version of XGBoost available on conda-forge. This is not a problem because XGBoost itself can actually run on your Apple M1 machine; the issue is with the dependencies that XGBoost installs, specifically numpy, scipy, and scikit-learn. These dependencies need to be arm64-compatible, which we’ll ensure by completing the following steps:
//conda create -n boost
conda activate boost
conda install python=3.8.8 numpy scipy scikit-learn//]]>
2. Install the necessary libraries to compile XGBoost
//conda install cmake llvm-openmp compilers//]]>
3. Now that we have the right dependencies in place, we can install XGBoost from pip. This is fine since we have all the right arm-64 dependencies installed already.
//pip install xgboost//]]>
4. Take it for a spin!
As with any technological breakthrough, there are going to be some birthing pains as the industry recalibrates and adjusts to ripple-waves caused by Apple’s switch to M1 silicon. We're excited to see companies like Anaconda and QuanStack and the Conda-Forge community continue to improve support for ARM hardware and stoked for the powerful future of PyData this promises!
We hope this post helps get your PyData workflows running smoothly and effectively on your Apple M1 machine. If you have any feedback or suggestions for future material, please do not hesitate to reach out to us on Twitter or our Coiled Community Slack channel.
If you’re interested in trying out Coiled, which provides hosted Dask clusters, docker-less managed software, and one-click deployments, you can do so for free today when you click below.