dask

Name: dask
Author: K-Dense-AI

864installs3.1kforks

How to install dask?

k-dense-ai/scientific-agent-skills/dask

$npx -y skills add K-Dense-AI/scientific-agent-skills --skill dask

Installs into the current project.

Summary

Distributed computing for larger-than-RAM pandas/NumPy workflows. Use when you need to scale existing pandas/NumPy code beyond memory or across clusters. Best for parallel file processing, distributed ML, integration with existing pandas code. For out-of-core analytics on single

Files · 1

View on GitHub

SKILL.md

1	# Dask
2
3	## Overview
4
5	Dask is a Python library for parallel and distributed computing that enables three critical capabilities:
6	- Larger-than-memory execution on single machines for data exceeding available RAM
7	- Parallel processing for improved computational speed across multiple cores
8	- Distributed computation supporting terabyte-scale datasets across multiple machines
9
10	Dask scales from laptops (processing ~100 GiB) to clusters (processing ~100 TiB) while maintaining familiar Python APIs.
11
12	Current upstream: dask 2026.3.0 (PyPI, March 2026). Docs: [docs.dask.org](https://docs.dask.org/en/stable/). Since 2025.1.0, the expression-based DataFrame API with query planning is the only implementation — do not install `dask-expr` separately or set `dataframe.query-planning: False`.
13
14	## Quick Start
15
16	### Installation
17
18	```bash
19	uv pip install "dask>=2025.1"
20	```
21
22	For a typical pandas/NumPy workflow with the distributed scheduler and dashboard:
23
24	```bash
25	uv pip install "dask[complete]"
26	```
27
28	Remote object storage (S3, GCS, Azure):
29
30	```bash
31	uv pip install s3fs # s3:// paths
32	uv pip install gcsfs # gs:// paths
33	```
34
35	Requires Python 3.10+ (3.9 support dropped in 2024.12). DataFrame I/O requires PyArrow 16+ (as of dask 2026.1.2).
36
37	## When to Use This Skill
38
39	This skill should be used when:
40	- Process datasets that exceed available RAM
41	- Scale pandas or NumPy operations to larger datasets
42	- Parallelize computations for performance improvements
43	- Process multiple files efficiently (CSVs, Parquet, JSON, text logs)
44	- Build custom parallel workflows with task dependencies
45	- Distribute workloads across multiple cores or machines
46
47	## Core Capabilities
48
49	Dask provides five main components, each suited to different use cases:
50
51	### 1. DataFrames - Parallel Pandas Operations
52
53	Purpose: Scale pandas operations to larger datasets through parallel processing.
54
55	When to Use:
56	- Tabular data exceeds available RAM
57	- Need to process multiple CSV/Parquet files together
58	- Pandas operations are slow and need parallelization
59	- Scaling from pandas prototype to production
60
61	Reference Documentation: For comprehensive guidance on Dask DataFrames, refer to `references/dataframes.md` which includes:
62	- Reading data (single files, multiple files, glob patterns)
63	- Common operations (filtering, groupby, joins, aggregations)
64	- Custom operations with `map_partitions`
65	- Performance optimization tips
66	- Common patterns (ETL, time series, multi-file processing)
67
68	Quick Example:
69	```python
70	import dask.dataframe as dd
71
72	# Read multiple files as single DataFrame
73	ddf = dd.read_csv('data/2024-*.csv')
74
75	# Operations are lazy until compute()
76	filtered = ddf[ddf['value'] > 100]
77	result = filtered.groupby('category').mean().compute()
78	```
79
80	Key Points:
81	- Operations are lazy (build task graph) until `.compute()` called
82	- Use `map_partitions` for efficient custom operations
83	- Convert to DataFrame early when working with structured data from other sources
84
85	### 2. Arrays - Parallel NumPy Operations
86
87	Purpose: Extend NumPy capabilities to datasets larger than memory using blocked algorithms.
88
89	When to Use:
90	- Arrays exceed available RAM
91	- NumPy operations need parallelization
92	- Working with scientific datasets (HDF5, Zarr, NetCDF)
93	- Need parallel linear algebra or array operations
94
95	Reference Documentation: For comprehensive guidance on Dask Arrays, refer to `references/arrays.md` which includes:
96	- Creating arrays (from NumPy, random, from disk)
97	- Chunking strategies and optimization
98	- Common operations (arithmetic, reductions, linear algebra)
99	- Custom operations with `map_blocks`
100	- Integration with HDF5, Zarr, and XArray
101
102	Quick Example:
103	```python
104	import dask.array as da
105
106	# Create large array with chunks
107	x = da.random.random((100000, 100000), chunks=(10000, 10000))
108
109	# Operations are lazy
110	y = x + 100
111	z = y.mean(axis=0)
112
113	# Compute result
114	result = z.compute()
115	```
116
117	Key Points:
118	- Chunk size is critical (aim for ~100 MB per chunk)
119	- Operations work on chunks in parallel
120	- Rechunk data when needed for efficient operations
121	- Use `map_blocks` for operations not available in Dask
122
123	### 3. Bags - Parallel Processing of Unstructured Data
124
125	Purpose: Process unstructured or semi-structured data (text, JSON, logs) with functional o

People who liked this also viewed

6 picks

Type

dask

864installs3.1kforks

View on GitHub

How to install dask?

k-dense-ai/scientific-agent-skills/dask

$npx -y skills add K-Dense-AI/scientific-agent-skills --skill dask

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Dask
2
3	## Overview
4
5	Dask is a Python library for parallel and distributed computing that enables three critical capabilities:
6	- Larger-than-memory execution on single machines for data exceeding available RAM
7	- Parallel processing for improved computational speed across multiple cores
8	- Distributed computation supporting terabyte-scale datasets across multiple machines
9
10	Dask scales from laptops (processing ~100 GiB) to clusters (processing ~100 TiB) while maintaining familiar Python APIs.
11
12	Current upstream: dask 2026.3.0 (PyPI, March 2026). Docs: [docs.dask.org](https://docs.dask.org/en/stable/). Since 2025.1.0, the expression-based DataFrame API with query planning is the only implementation — do not install `dask-expr` separately or set `dataframe.query-planning: False`.
13
14	## Quick Start
15
16	### Installation
17
18	```bash
19	uv pip install "dask>=2025.1"
20	```
21
22	For a typical pandas/NumPy workflow with the distributed scheduler and dashboard:
23
24	```bash
25	uv pip install "dask[complete]"
26	```
27
28	Remote object storage (S3, GCS, Azure):
29
30	```bash
31	uv pip install s3fs # s3:// paths
32	uv pip install gcsfs # gs:// paths
33	```
34
35	Requires Python 3.10+ (3.9 support dropped in 2024.12). DataFrame I/O requires PyArrow 16+ (as of dask 2026.1.2).
36
37	## When to Use This Skill
38
39	This skill should be used when:
40	- Process datasets that exceed available RAM
41	- Scale pandas or NumPy operations to larger datasets
42	- Parallelize computations for performance improvements
43	- Process multiple files efficiently (CSVs, Parquet, JSON, text logs)
44	- Build custom parallel workflows with task dependencies
45	- Distribute workloads across multiple cores or machines
46
47	## Core Capabilities
48
49	Dask provides five main components, each suited to different use cases:
50
51	### 1. DataFrames - Parallel Pandas Operations
52
53	Purpose: Scale pandas operations to larger datasets through parallel processing.
54
55	When to Use:
56	- Tabular data exceeds available RAM
57	- Need to process multiple CSV/Parquet files together
58	- Pandas operations are slow and need parallelization
59	- Scaling from pandas prototype to production
60
61	Reference Documentation: For comprehensive guidance on Dask DataFrames, refer to `references/dataframes.md` which includes:
62	- Reading data (single files, multiple files, glob patterns)
63	- Common operations (filtering, groupby, joins, aggregations)
64	- Custom operations with `map_partitions`
65	- Performance optimization tips
66	- Common patterns (ETL, time series, multi-file processing)
67
68	Quick Example:
69	```python
70	import dask.dataframe as dd
71
72	# Read multiple files as single DataFrame
73	ddf = dd.read_csv('data/2024-*.csv')
74
75	# Operations are lazy until compute()
76	filtered = ddf[ddf['value'] > 100]
77	result = filtered.groupby('category').mean().compute()
78	```
79
80	Key Points:
81	- Operations are lazy (build task graph) until `.compute()` called
82	- Use `map_partitions` for efficient custom operations
83	- Convert to DataFrame early when working with structured data from other sources
84
85	### 2. Arrays - Parallel NumPy Operations
86
87	Purpose: Extend NumPy capabilities to datasets larger than memory using blocked algorithms.
88
89	When to Use:
90	- Arrays exceed available RAM
91	- NumPy operations need parallelization
92	- Working with scientific datasets (HDF5, Zarr, NetCDF)
93	- Need parallel linear algebra or array operations
94
95	Reference Documentation: For comprehensive guidance on Dask Arrays, refer to `references/arrays.md` which includes:
96	- Creating arrays (from NumPy, random, from disk)
97	- Chunking strategies and optimization
98	- Common operations (arithmetic, reductions, linear algebra)
99	- Custom operations with `map_blocks`
100	- Integration with HDF5, Zarr, and XArray
101
102	Quick Example:
103	```python
104	import dask.array as da
105
106	# Create large array with chunks
107	x = da.random.random((100000, 100000), chunks=(10000, 10000))
108
109	# Operations are lazy
110	y = x + 100
111	z = y.mean(axis=0)
112
113	# Compute result
114	result = z.compute()
115	```
116
117	Key Points:
118	- Chunk size is critical (aim for ~100 MB per chunk)
119	- Operations work on chunks in parallel
120	- Rechunk data when needed for efficient operations
121	- Use `map_blocks` for operations not available in Dask
122
123	### 3. Bags - Parallel Processing of Unstructured Data
124
125	Purpose: Process unstructured or semi-structured data (text, JSON, logs) with functional o

People who liked this also viewed

6 picks

Type