scala-data-engineering-on-jvm-runtimes

Name: scala-data-engineering-on-jvm-runtimes
Author: vaquarkhan

19stars4forks

How to install scala-data-engineering-on-jvm-runtimes?

vaquarkhan/data-engineering-agent-skills/scala-data-engineering-on-jvm-runtimes

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill scala-data-engineering-on-jvm-runtimes

Installs into the current project.

Summary

Guides agents through Scala-based data engineering on JVM runtimes. Use when building Spark, Flink, Kafka Streams, or other Scala data jobs that require explicit build, packaging, type, and runtime discipline.

Files · 1

View on GitHub

SKILL.md

1	# Scala Data Engineering On JVM Runtimes
2
3	## Overview
4
5	Use this skill when `Scala` is the implementation language for distributed or streaming data systems. It helps agents manage build compatibility, packaging, JVM dependency issues, typed data models, serialization behavior, and runtime assumptions that often cause production failures long after code compiles.
6
7	## When to Use
8
9	- building or modifying `Scala` `Spark` jobs
10	- implementing `Flink`, `Kafka Streams`, or JVM-native data processors in `Scala`
11	- managing `sbt` builds, shaded JARs, or runtime compatibility
12	- choosing typed datasets, encoders, or JVM serialization strategies
13	- debugging classpath, version, or packaging problems in distributed runtimes
14
15	Do not assume compile success means distributed runtime safety.
16
17	## Workflow
18
19	1. Define the runtime and compatibility surface.
20	Include:
21	- engine and version
22	- `Scala` version
23	- JVM level
24	- cluster or container runtime
25	- connector and library compatibility
26
27	2. Shape the data model intentionally.
28	Choose:
29	- typed case classes or schemas
30	- serialization strategy
31	- encoder behavior
32	- where UDFs are truly needed
33	- how nulls and optional fields are represented
34
35	3. Package for the real deployment target.
36	Decide:
37	- `sbt` or other build surface
38	- fat-jar or shaded-jar approach
39	- dependency conflict handling
40	- resource and config loading behavior
41	- how the job is launched and parameterized
42
43	4. Design for distributed execution, not driver-local convenience.
44	Check:
45	- partition behavior
46	- skew
47	- state growth
48	- checkpoint or savepoint needs
49	- accidental driver-side collection or closure capture
50
51	5. Validate with runtime realism.
52	Require:
53	- representative local or test-cluster execution
54	- packaging verification
55	- schema and compatibility checks
56	- observability and failure handling expectations
57
58	## Common Rationalizations
59
60	\| Rationalization \| Reality \|
61	\| --- \| --- \|
62	\| "It compiles, so the job is fine." \| Distributed classpath, serialization, and dependency issues often appear only at runtime. \|
63	\| "We can fix the JAR if deployment fails." \| Packaging problems discovered at deploy time slow delivery and often hide deeper compatibility issues. \|
64	\| "A quick UDF is simpler." \| Overusing UDFs can hide schema, optimizer, and performance problems in JVM data engines. \|
65
66	## Red Flags
67
68	- `Scala`, engine, and connector versions are not pinned together intentionally
69	- driver-local collections or closures leak into distributed execution
70	- shading or dependency conflicts are discovered only after deployment
71	- type or null behavior is unclear across schemas and case classes
72	- runtime config is embedded in code instead of explicit launch parameters
73
74	## Verification
75
76	- [ ] Runtime, `Scala`, and library compatibility is explicit
77	- [ ] Packaging strategy matches the target execution environment
78	- [ ] Data model and serialization behavior are intentional
79	- [ ] Distributed execution risks are reviewed beyond compile success

People who liked this also viewed

6 picks

Type

scala-data-engineering-on-jvm-runtimes

19stars4forks

View on GitHub

How to install scala-data-engineering-on-jvm-runtimes?

vaquarkhan/data-engineering-agent-skills/scala-data-engineering-on-jvm-runtimes

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill scala-data-engineering-on-jvm-runtimes

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Scala Data Engineering On JVM Runtimes
2
3	## Overview
4
5	Use this skill when `Scala` is the implementation language for distributed or streaming data systems. It helps agents manage build compatibility, packaging, JVM dependency issues, typed data models, serialization behavior, and runtime assumptions that often cause production failures long after code compiles.
6
7	## When to Use
8
9	- building or modifying `Scala` `Spark` jobs
10	- implementing `Flink`, `Kafka Streams`, or JVM-native data processors in `Scala`
11	- managing `sbt` builds, shaded JARs, or runtime compatibility
12	- choosing typed datasets, encoders, or JVM serialization strategies
13	- debugging classpath, version, or packaging problems in distributed runtimes
14
15	Do not assume compile success means distributed runtime safety.
16
17	## Workflow
18
19	1. Define the runtime and compatibility surface.
20	Include:
21	- engine and version
22	- `Scala` version
23	- JVM level
24	- cluster or container runtime
25	- connector and library compatibility
26
27	2. Shape the data model intentionally.
28	Choose:
29	- typed case classes or schemas
30	- serialization strategy
31	- encoder behavior
32	- where UDFs are truly needed
33	- how nulls and optional fields are represented
34
35	3. Package for the real deployment target.
36	Decide:
37	- `sbt` or other build surface
38	- fat-jar or shaded-jar approach
39	- dependency conflict handling
40	- resource and config loading behavior
41	- how the job is launched and parameterized
42
43	4. Design for distributed execution, not driver-local convenience.
44	Check:
45	- partition behavior
46	- skew
47	- state growth
48	- checkpoint or savepoint needs
49	- accidental driver-side collection or closure capture
50
51	5. Validate with runtime realism.
52	Require:
53	- representative local or test-cluster execution
54	- packaging verification
55	- schema and compatibility checks
56	- observability and failure handling expectations
57
58	## Common Rationalizations
59
60	\| Rationalization \| Reality \|
61	\| --- \| --- \|
62	\| "It compiles, so the job is fine." \| Distributed classpath, serialization, and dependency issues often appear only at runtime. \|
63	\| "We can fix the JAR if deployment fails." \| Packaging problems discovered at deploy time slow delivery and often hide deeper compatibility issues. \|
64	\| "A quick UDF is simpler." \| Overusing UDFs can hide schema, optimizer, and performance problems in JVM data engines. \|
65
66	## Red Flags
67
68	- `Scala`, engine, and connector versions are not pinned together intentionally
69	- driver-local collections or closures leak into distributed execution
70	- shading or dependency conflicts are discovered only after deployment
71	- type or null behavior is unclear across schemas and case classes
72	- runtime config is embedded in code instead of explicit launch parameters
73
74	## Verification
75
76	- [ ] Runtime, `Scala`, and library compatibility is explicit
77	- [ ] Packaging strategy matches the target execution environment
78	- [ ] Data model and serialization behavior are intentional
79	- [ ] Distributed execution risks are reviewed beyond compile success

People who liked this also viewed

6 picks

Type