zarr-python

Name: zarr-python
Author: LeonChaoX

698stars61forks

How to install zarr-python?

leonchaox/qinyan-academic-skills/zarr-python

$npx -y skills add LeonChaoX/qinyan-academic-skills --skill zarr-python

Installs into the current project.

Summary

Chunked N-D arrays for cloud storage. Compressed arrays, parallel I/O, S3/GCS integration, NumPy/Dask/Xarray compatible, for large-scale scientific computing pipelines.

Files · 1

View on GitHub

SKILL.md

1	# Zarr Python
2
3	## Overview
4
5	Zarr is a Python library for storing large N-dimensional arrays with chunking and compression. Apply this skill for efficient parallel I/O, cloud-native workflows, and seamless integration with NumPy, Dask, and Xarray.
6
7	## Quick Start
8
9	### Installation
10
11	```bash
12	uv pip install zarr
13	```
14
15	Requires Python 3.11+. For cloud storage support, install additional packages:
16	```python
17	uv pip install s3fs # For S3
18	uv pip install gcsfs # For Google Cloud Storage
19	```
20
21	### Basic Array Creation
22
23	```python
24	import zarr
25	import numpy as np
26
27	# Create a 2D array with chunking and compression
28	z = zarr.create_array(
29	store="data/my_array.zarr",
30	shape=(10000, 10000),
31	chunks=(1000, 1000),
32	dtype="f4"
33	)
34
35	# Write data using NumPy-style indexing
36	z[:, :] = np.random.random((10000, 10000))
37
38	# Read data
39	data = z[0:100, 0:100] # Returns NumPy array
40	```
41
42	## Core Operations
43
44	### Creating Arrays
45
46	Zarr provides multiple convenience functions for array creation:
47
48	```python
49	# Create empty array
50	z = zarr.zeros(shape=(10000, 10000), chunks=(1000, 1000), dtype='f4',
51	store='data.zarr')
52
53	# Create filled arrays
54	z = zarr.ones((5000, 5000), chunks=(500, 500))
55	z = zarr.full((1000, 1000), fill_value=42, chunks=(100, 100))
56
57	# Create from existing data
58	data = np.arange(10000).reshape(100, 100)
59	z = zarr.array(data, chunks=(10, 10), store='data.zarr')
60
61	# Create like another array
62	z2 = zarr.zeros_like(z) # Matches shape, chunks, dtype of z
63	```
64
65	### Opening Existing Arrays
66
67	```python
68	# Open array (read/write mode by default)
69	z = zarr.open_array('data.zarr', mode='r+')
70
71	# Read-only mode
72	z = zarr.open_array('data.zarr', mode='r')
73
74	# The open() function auto-detects arrays vs groups
75	z = zarr.open('data.zarr') # Returns Array or Group
76	```
77
78	### Reading and Writing Data
79
80	Zarr arrays support NumPy-like indexing:
81
82	```python
83	# Write entire array
84	z[:] = 42
85
86	# Write slices
87	z[0, :] = np.arange(100)
88	z[10:20, 50:60] = np.random.random((10, 10))
89
90	# Read data (returns NumPy array)
91	data = z[0:100, 0:100]
92	row = z[5, :]
93
94	# Advanced indexing
95	z.vindex[[0, 5, 10], [2, 8, 15]] # Coordinate indexing
96	z.oindex[0:10, [5, 10, 15]] # Orthogonal indexing
97	z.blocks[0, 0] # Block/chunk indexing
98	```
99
100	### Resizing and Appending
101
102	```python
103	# Resize array
104	z.resize(15000, 15000) # Expands or shrinks dimensions
105
106	# Append data along an axis
107	z.append(np.random.random((1000, 10000)), axis=0) # Adds rows
108	```
109
110	## Chunking Strategies
111
112	Chunking is critical for performance. Choose chunk sizes and shapes based on access patterns.
113
114	### Chunk Size Guidelines
115
116	- Minimum chunk size: 1 MB recommended for optimal performance
117	- Balance: Larger chunks = fewer metadata operations; smaller chunks = better parallel access
118	- Memory consideration: Entire chunks must fit in memory during compression
119
120	```python
121	# Configure chunk size (aim for ~1MB per chunk)
122	# For float32 data: 1MB = 262,144 elements = 512×512 array
123	z = zarr.zeros(
124	shape=(10000, 10000),
125	chunks=(512, 512), # ~1MB chunks
126	dtype='f4'
127	)
128	```
129
130	### Aligning Chunks with Access Patterns
131
132	Critical: Chunk shape dramatically affects performance based on how data is accessed.
133
134	```python
135	# If accessing rows frequently (first dimension)
136	z = zarr.zeros((10000, 10000), chunks=(10, 10000)) # Chunk spans columns
137
138	# If accessing columns frequently (second dimension)
139	z = zarr.zeros((10000, 10000), chunks=(10000, 10)) # Chunk spans rows
140
141	# For mixed access patterns (balanced approach)
142	z = zarr.zeros((10000, 10000), chunks=(1000, 1000)) # Square chunks
143	```
144
145	Performance example: For a (200, 200, 200) array, reading along the first dimension:
146	- Using chunks (1, 200, 200): ~107ms
147	- Using chunks (200, 200, 1): ~1.65ms (65× faster!)
148
149	### Sharding for Large-Scale Storage
150
151	When arrays have millions of small chunks, use sharding to group chunks into larger storage objects:
152
153	```python
154	from zarr.codecs import ShardingCodec, BytesCodec
155	from zarr.codecs.blosc import BloscCodec
156
157	# Create array with sharding
158	z = zarr.create_array(
159	store='data.zarr',
160	shape=(100000, 100000),
161	chunks=(100, 100), # Small chunks for access
162	shards=(1000, 1000), # Groups 100 chunks per shard
163	dtype='f4'
164	)
165	```
166
167	Benefits:
168	- Reduces file system overhead from millions of small files
169	- Improves cloud storage performance (fewer object requests)
170	- Prevents filesystem block size waste
171
172	Important: Entire shards must fit in memory before writing.
173
174	## Compression
175
176	Zarr applies compression per chunk to reduce storage while maintaining fast access.
177
178	### Configuring Compression
179
180	```python
181	from zarr.codecs.blosc import BloscCodec
182	from zarr.codecs import GzipCodec, ZstdCodec
183
184	# Default: Blosc with Zstandard
185	z = zarr

People who liked this also viewed

6 picks

Type

find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 20262.6M27k
grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026633k182k
vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026573k29k
improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026522k182k
setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026454k182k
triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026424k182k

zarr-python

698stars61forks

View on GitHub

How to install zarr-python?

leonchaox/qinyan-academic-skills/zarr-python

$npx -y skills add LeonChaoX/qinyan-academic-skills --skill zarr-python

Installs into the current project.

Summary

Chunked N-D arrays for cloud storage. Compressed arrays, parallel I/O, S3/GCS integration, NumPy/Dask/Xarray compatible, for large-scale scientific computing pipelines.

Files · 1

View on GitHub

SKILL.md

1	# Zarr Python
2
3	## Overview
4
5	Zarr is a Python library for storing large N-dimensional arrays with chunking and compression. Apply this skill for efficient parallel I/O, cloud-native workflows, and seamless integration with NumPy, Dask, and Xarray.
6
7	## Quick Start
8
9	### Installation
10
11	```bash
12	uv pip install zarr
13	```
14
15	Requires Python 3.11+. For cloud storage support, install additional packages:
16	```python
17	uv pip install s3fs # For S3
18	uv pip install gcsfs # For Google Cloud Storage
19	```
20
21	### Basic Array Creation
22
23	```python
24	import zarr
25	import numpy as np
26
27	# Create a 2D array with chunking and compression
28	z = zarr.create_array(
29	store="data/my_array.zarr",
30	shape=(10000, 10000),
31	chunks=(1000, 1000),
32	dtype="f4"
33	)
34
35	# Write data using NumPy-style indexing
36	z[:, :] = np.random.random((10000, 10000))
37
38	# Read data
39	data = z[0:100, 0:100] # Returns NumPy array
40	```
41
42	## Core Operations
43
44	### Creating Arrays
45
46	Zarr provides multiple convenience functions for array creation:
47
48	```python
49	# Create empty array
50	z = zarr.zeros(shape=(10000, 10000), chunks=(1000, 1000), dtype='f4',
51	store='data.zarr')
52
53	# Create filled arrays
54	z = zarr.ones((5000, 5000), chunks=(500, 500))
55	z = zarr.full((1000, 1000), fill_value=42, chunks=(100, 100))
56
57	# Create from existing data
58	data = np.arange(10000).reshape(100, 100)
59	z = zarr.array(data, chunks=(10, 10), store='data.zarr')
60
61	# Create like another array
62	z2 = zarr.zeros_like(z) # Matches shape, chunks, dtype of z
63	```
64
65	### Opening Existing Arrays
66
67	```python
68	# Open array (read/write mode by default)
69	z = zarr.open_array('data.zarr', mode='r+')
70
71	# Read-only mode
72	z = zarr.open_array('data.zarr', mode='r')
73
74	# The open() function auto-detects arrays vs groups
75	z = zarr.open('data.zarr') # Returns Array or Group
76	```
77
78	### Reading and Writing Data
79
80	Zarr arrays support NumPy-like indexing:
81
82	```python
83	# Write entire array
84	z[:] = 42
85
86	# Write slices
87	z[0, :] = np.arange(100)
88	z[10:20, 50:60] = np.random.random((10, 10))
89
90	# Read data (returns NumPy array)
91	data = z[0:100, 0:100]
92	row = z[5, :]
93
94	# Advanced indexing
95	z.vindex[[0, 5, 10], [2, 8, 15]] # Coordinate indexing
96	z.oindex[0:10, [5, 10, 15]] # Orthogonal indexing
97	z.blocks[0, 0] # Block/chunk indexing
98	```
99
100	### Resizing and Appending
101
102	```python
103	# Resize array
104	z.resize(15000, 15000) # Expands or shrinks dimensions
105
106	# Append data along an axis
107	z.append(np.random.random((1000, 10000)), axis=0) # Adds rows
108	```
109
110	## Chunking Strategies
111
112	Chunking is critical for performance. Choose chunk sizes and shapes based on access patterns.
113
114	### Chunk Size Guidelines
115
116	- Minimum chunk size: 1 MB recommended for optimal performance
117	- Balance: Larger chunks = fewer metadata operations; smaller chunks = better parallel access
118	- Memory consideration: Entire chunks must fit in memory during compression
119
120	```python
121	# Configure chunk size (aim for ~1MB per chunk)
122	# For float32 data: 1MB = 262,144 elements = 512×512 array
123	z = zarr.zeros(
124	shape=(10000, 10000),
125	chunks=(512, 512), # ~1MB chunks
126	dtype='f4'
127	)
128	```
129
130	### Aligning Chunks with Access Patterns
131
132	Critical: Chunk shape dramatically affects performance based on how data is accessed.
133
134	```python
135	# If accessing rows frequently (first dimension)
136	z = zarr.zeros((10000, 10000), chunks=(10, 10000)) # Chunk spans columns
137
138	# If accessing columns frequently (second dimension)
139	z = zarr.zeros((10000, 10000), chunks=(10000, 10)) # Chunk spans rows
140
141	# For mixed access patterns (balanced approach)
142	z = zarr.zeros((10000, 10000), chunks=(1000, 1000)) # Square chunks
143	```
144
145	Performance example: For a (200, 200, 200) array, reading along the first dimension:
146	- Using chunks (1, 200, 200): ~107ms
147	- Using chunks (200, 200, 1): ~1.65ms (65× faster!)
148
149	### Sharding for Large-Scale Storage
150
151	When arrays have millions of small chunks, use sharding to group chunks into larger storage objects:
152
153	```python
154	from zarr.codecs import ShardingCodec, BytesCodec
155	from zarr.codecs.blosc import BloscCodec
156
157	# Create array with sharding
158	z = zarr.create_array(
159	store='data.zarr',
160	shape=(100000, 100000),
161	chunks=(100, 100), # Small chunks for access
162	shards=(1000, 1000), # Groups 100 chunks per shard
163	dtype='f4'
164	)
165	```
166
167	Benefits:
168	- Reduces file system overhead from millions of small files
169	- Improves cloud storage performance (fewer object requests)
170	- Prevents filesystem block size waste
171
172	Important: Entire shards must fit in memory before writing.
173
174	## Compression
175
176	Zarr applies compression per chunk to reduce storage while maintaining fast access.
177
178	### Configuring Compression
179
180	```python
181	from zarr.codecs.blosc import BloscCodec
182	from zarr.codecs import GzipCodec, ZstdCodec
183
184	# Default: Blosc with Zstandard
185	z = zarr

People who liked this also viewed

6 picks

Type