matlab-choose-bigdata-solution

Name: matlab-choose-bigdata-solution
Author: matlab

815stars105forks

How to install matlab-choose-bigdata-solution?

matlab/matlab-agentic-toolkit/matlab-choose-bigdata-solution

$npx -y skills add matlab/matlab-agentic-toolkit --skill matlab-choose-bigdata-solution

Installs into the current project.

Summary

Guide users or agents to the correct MATLAB tool for processing large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF) that may not fit in memory. Use when a user or agent mentions large files, big data, out-of-memory errors, OOM, scaling up, t

Files · 1

View on GitHub

SKILL.md

1	# Choose Big Data Solution
2
3	Help users and agents select the right MATLAB tool for large **tabular data in
4	file-based formats** (CSV, Parquet, delimited text, spreadsheets, MDF). The
5	skill encodes a decision flowchart — recommend one clear path, not a menu of
6	options.
7
8	## When to Use
9
10	- User or agent has large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF)
11	- User or agent says data is "large", "big", or "huge" (even without a specific size)
12	- User or agent hits an out-of-memory error with `readtable`, `parquetread`, or similar
13	- User or agent has multiple tabular files to process as a single dataset
14	- User or agent has multiple tabular files to process independently (per-file)
15	- User or agent asks how to scale up an existing workflow on tabular file data
16	- User or agent asks about datastores, tall arrays, or mapreduce
17
18	## When NOT to Use
19
20	- Single file that fits in memory (file-size-to-RAM ratio < 0.5) — see Pre-Flight Check
21	- User or agent is working with MAT files — `matfile` provides partial I/O for large `.mat` files, different workflow
22	- User or agent is working with databases (SQL, ODBC) — use Database Toolbox. See the `matlab-read-database` and `matlab-use-duckdb` skills in the `reporting-and-database-access` category
23	- User or agent needs GPU acceleration — different domain
24	- User or agent is building a custom datastore class — see `doc matlab.io.Datastore`
25	- User or agent has large XML, JSON, HTML, or Word document files — `readtable` supports these formats but the datastores covered by this skill do not. Scaling these requires a custom datastore (see "Unsupported Formats" section below)
26	- Deep learning training workflows — datastore is correct for data loading, but use `minibatchqueue` for batching (not tall or transform)
27
28	## Pre-Flight Check
29
30	ALWAYS run this check BEFORE recommending datastore or tall array patterns.
31	Estimate the file-size-to-available-RAM ratio (accounting for in-memory
32	expansion of the file format).
33
34	\| Condition \| Route \| Rationale \|
35	\|-----------\|-------\|-----------\|
36	\| Single file, ratio < 0.5 \| Native MATLAB I/O (`readtable`, `parquetread`) \| Fits in memory; datastore/tall adds unnecessary complexity \|
37	\| Single file, ratio ≥ 0.5, OR user/agent reports OOM \| Continue to Decision Flowchart below \| Data may not fit in memory \|
38	\| Multiple files \| Continue to Decision Flowchart below \| Datastore patterns provide unified multi-file access \|
39	\| Size unknown and user/agent describes data as "large", "big", or "huge" \| Continue to Decision Flowchart below \| Assume large until proven otherwise \|
40
41	If the data fits in memory (single file, ratio < 0.5): recommend native
42	I/O and stop. Mention that datastore and tall array patterns exist if the
43	data grows beyond memory in the future, but do not implement them now.
44
45	## Decision Flowchart
46
47	Follow this flowchart strictly. Present ONE recommended path, not multiple
48	alternatives. Only mention alternatives if the situation is ambiguous.
49
50	```
51	Is the data described as "large" or causing OOM?
52	│
53	├── YES
54	│ │
55	│ ├── Is the goal to process all data as ONE continuous dataset?
56	│ │ │
57	│ │ └── YES → Datastore + Tall Arrays
58	│ │ Choose datastore by format:
59	│ │ CSV/delimited text → tabularTextDatastore
60	│ │ Parquet → parquetDatastore
61	│ │ Excel (.xlsx/.xls) → spreadsheetDatastore
62	│ │ MDF (.mf4/.mdf) → mdfDatastore (requires Vehicle Network Toolbox)
63	│ │
64	│ └── Is the goal to process each unit INDEPENDENTLY?
65	│ │
66	│ ├── One read = one FILE
67	│ │ CSV/delimited text → tabularTextDatastore
68	│ │ Excel (.xlsx/.xls) → spreadsheetDatastore
69	│ │
70	│ └── One read = one ROW GROUP
71	│ Parquet → parquetDatastore
72	│
73	└── NO / UNCLEAR
74	└── STOP. Ask: (1) what is the file format? (2) do you need to process
75	all data as one dataset, or each file/unit independently?
76	Do NOT show code until these are answered.
77
78	Optional (requires Parallel Computing Toolbox):
79	├── Speed up tall arrays locally → Open a parallel pool
80	├── Speed up readall on transforms → UseParallel
81	└── Speed up tall arrays on Hadoop/S

People who liked this also viewed

6 picks

Type

matlab-choose-bigdata-solution

815stars105forks

View on GitHub

How to install matlab-choose-bigdata-solution?

matlab/matlab-agentic-toolkit/matlab-choose-bigdata-solution

$npx -y skills add matlab/matlab-agentic-toolkit --skill matlab-choose-bigdata-solution

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Choose Big Data Solution
2
3	Help users and agents select the right MATLAB tool for large **tabular data in
4	file-based formats** (CSV, Parquet, delimited text, spreadsheets, MDF). The
5	skill encodes a decision flowchart — recommend one clear path, not a menu of
6	options.
7
8	## When to Use
9
10	- User or agent has large tabular data in file-based formats (CSV, Parquet, delimited text, spreadsheets, MDF)
11	- User or agent says data is "large", "big", or "huge" (even without a specific size)
12	- User or agent hits an out-of-memory error with `readtable`, `parquetread`, or similar
13	- User or agent has multiple tabular files to process as a single dataset
14	- User or agent has multiple tabular files to process independently (per-file)
15	- User or agent asks how to scale up an existing workflow on tabular file data
16	- User or agent asks about datastores, tall arrays, or mapreduce
17
18	## When NOT to Use
19
20	- Single file that fits in memory (file-size-to-RAM ratio < 0.5) — see Pre-Flight Check
21	- User or agent is working with MAT files — `matfile` provides partial I/O for large `.mat` files, different workflow
22	- User or agent is working with databases (SQL, ODBC) — use Database Toolbox. See the `matlab-read-database` and `matlab-use-duckdb` skills in the `reporting-and-database-access` category
23	- User or agent needs GPU acceleration — different domain
24	- User or agent is building a custom datastore class — see `doc matlab.io.Datastore`
25	- User or agent has large XML, JSON, HTML, or Word document files — `readtable` supports these formats but the datastores covered by this skill do not. Scaling these requires a custom datastore (see "Unsupported Formats" section below)
26	- Deep learning training workflows — datastore is correct for data loading, but use `minibatchqueue` for batching (not tall or transform)
27
28	## Pre-Flight Check
29
30	ALWAYS run this check BEFORE recommending datastore or tall array patterns.
31	Estimate the file-size-to-available-RAM ratio (accounting for in-memory
32	expansion of the file format).
33
34	\| Condition \| Route \| Rationale \|
35	\|-----------\|-------\|-----------\|
36	\| Single file, ratio < 0.5 \| Native MATLAB I/O (`readtable`, `parquetread`) \| Fits in memory; datastore/tall adds unnecessary complexity \|
37	\| Single file, ratio ≥ 0.5, OR user/agent reports OOM \| Continue to Decision Flowchart below \| Data may not fit in memory \|
38	\| Multiple files \| Continue to Decision Flowchart below \| Datastore patterns provide unified multi-file access \|
39	\| Size unknown and user/agent describes data as "large", "big", or "huge" \| Continue to Decision Flowchart below \| Assume large until proven otherwise \|
40
41	If the data fits in memory (single file, ratio < 0.5): recommend native
42	I/O and stop. Mention that datastore and tall array patterns exist if the
43	data grows beyond memory in the future, but do not implement them now.
44
45	## Decision Flowchart
46
47	Follow this flowchart strictly. Present ONE recommended path, not multiple
48	alternatives. Only mention alternatives if the situation is ambiguous.
49
50	```
51	Is the data described as "large" or causing OOM?
52	│
53	├── YES
54	│ │
55	│ ├── Is the goal to process all data as ONE continuous dataset?
56	│ │ │
57	│ │ └── YES → Datastore + Tall Arrays
58	│ │ Choose datastore by format:
59	│ │ CSV/delimited text → tabularTextDatastore
60	│ │ Parquet → parquetDatastore
61	│ │ Excel (.xlsx/.xls) → spreadsheetDatastore
62	│ │ MDF (.mf4/.mdf) → mdfDatastore (requires Vehicle Network Toolbox)
63	│ │
64	│ └── Is the goal to process each unit INDEPENDENTLY?
65	│ │
66	│ ├── One read = one FILE
67	│ │ CSV/delimited text → tabularTextDatastore
68	│ │ Excel (.xlsx/.xls) → spreadsheetDatastore
69	│ │
70	│ └── One read = one ROW GROUP
71	│ Parquet → parquetDatastore
72	│
73	└── NO / UNCLEAR
74	└── STOP. Ask: (1) what is the file format? (2) do you need to process
75	all data as one dataset, or each file/unit independently?
76	Do NOT show code until these are answered.
77
78	Optional (requires Parallel Computing Toolbox):
79	├── Speed up tall arrays locally → Open a parallel pool
80	├── Speed up readall on transforms → UseParallel
81	└── Speed up tall arrays on Hadoop/S

People who liked this also viewed

6 picks

Type