Coiled Product Detail

Observable

Sometimes things break. We can tell you why.

Logs

What is it?
Cloud storage of Dask worker and scheduler logs, with easy access via the Coiled web app.

Why it matters.
Easy access to your logs is critical to help debug issues. Coiled’s log storage integrates easy with your existing IT logging system and provides historical provenance of runs.

Our Design.
We leverage existing, well-trusted log stores like Amazon CloudWatch and Google Cloud Logging. These are stored in your cloud account. We direct logs straight from your Dask workers to log storage, all while keeping logs in your account. We use sensible retention policies and provide automated setup and creation of the infrastructure for log storage.Coiled makes it easy for you to access your logs via a convenient web UI. Of course, any of your existing log management systems will just work because these logs already exist in your standard infrastructure.

Teams

What is it?
Team members can share resources, track usage, and consolidate billing with anyone else on the same account.

Why it matters.
Administrators can set up a Coiled account once for all team members to use, so your teammates don't need their own separate AWS credentials to use Coiled once you’ve set it up. You can also limit your teammates’ cluster sizes to keep costs under control.

Cluster Infrastructure Observability

What is it?
See every component in your cluster and its history.

Why it matters.
You can see what’s working and efficiently track down and solve problems. Coiled enables you to respond quickly to a variety of cluster errors, whether you’ve hit an AWS quota limit or your cluster needs some help booting up.

Our Design.
We track the full history of every component of your cluster and break it down for you in our web app, highlighting anything we think you’ll find important or may need to address. We want to avoid showing you a dreaded ‘unknown error’ message at all costs, which has influenced our architecture from the bottom to the top.

Dask Observability

What is it?
Like the Dashboard, but across time and across all users in your team.

Why it matters.
You can track code submitted by everyone in your team to identify common errors, highly failed jobs, and highlight cost centers.

Our Design.
Every Dask scheduler checks into Coiled and downloads a SchedulerPlugin. This plugin tracks the same kind of information tracked in the real-time Dask dashboard, aggregates that data, and then regularly ships it to the Coiled Control Plane. We track user code, task groups, profile information, critical Dask failures, and more, indexed by user.We then present views of this data, both in our web app and through a Python API for programmatic access. The control plane can also be used by non-Coiled clusters (feel free to use it with on-prem systems that you may be transitioning from) and provides some lightweight features like auto-shutdown even in non-Coiled-enhanced environments.

Level up your Dask using Coiled

Coiled makes it easy to scale Dask maturely in the cloud