apache-beam-unified-batch-and-stream

Name: apache-beam-unified-batch-and-stream
Author: vaquarkhan

19stars4forks

How to install apache-beam-unified-batch-and-stream?

vaquarkhan/data-engineering-agent-skills/apache-beam-unified-batch-and-stream

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill apache-beam-unified-batch-and-stream

Installs into the current project.

Summary

Guides agents through Apache Beam pipelines that unify batch and streaming logic. Use when designing Beam transforms, windowing, runners, replay behavior, or portability across execution backends.

Files · 1

View on GitHub

SKILL.md

1	# Apache Beam Unified Batch And Stream
2
3	## Overview
4
5	Use this skill when `Apache Beam` is the abstraction layer for both batch and streaming data processing. It helps agents preserve portability without hiding time semantics or runner-specific constraints.
6
7	## When to Use
8
9	- building `Apache Beam` pipelines for batch, streaming, or unified workloads
10	- targeting multiple runners such as `Dataflow`, `Flink`, `Spark`, or `Direct Runner`
11	- sharing transform logic across batch and streaming modes
12	- managing windowing, watermarks, triggers, and late data handling
13	- designing portable pipelines that must run across environments
14
15	Do not use this when the workload is locked to a single runner and Beam portability is not a goal.
16
17	## Workflow
18
19	1. Define the pipeline contract and time semantics.
20	Include:
21	- input PCollections and their bounded or unbounded nature
22	- event-time versus processing-time expectations
23	- output schema, grain, and freshness requirements
24	- delivery guarantees expected by downstream consumers
25
26	2. Separate portable pipeline logic from runner-specific deployment.
27	- keep transforms, DoFns, and combiners runner-agnostic
28	- isolate runner configuration (parallelism, autoscaling, resource hints) into pipeline options
29	- document which runners are supported and tested
30	- avoid runner-specific APIs unless portability is explicitly sacrificed
31
32	3. Design windowing and trigger strategy explicitly.
33	Account for:
34	- fixed, sliding, session, or global windows
35	- trigger behavior: when to emit, accumulate, or discard
36	- allowed lateness and late data routing
37	- watermark advancement assumptions per source
38
39	4. Handle state and side inputs carefully.
40	- stateful DoFns bind to a specific key space — document key cardinality
41	- side inputs can become bottlenecks at scale — prefer bounded and small
42	- timers must account for watermark-driven versus processing-time semantics
43	- state cleanup must be explicit for unbounded pipelines
44
45	5. Make testing and local validation part of the workflow.
46	- use `DirectRunner` for correctness tests
47	- validate windowing behavior with synthetic watermark progression
48	- test exactly-once semantics through pipeline drains and restarts
49	- confirm output idempotency for at-least-once runners
50
51	6. Plan deployment, scaling, and operational observability.
52	- define autoscaling boundaries and cost limits per runner
53	- expose transform-level metrics for latency, throughput, and backlog
54	- plan drain and update strategies for streaming pipelines
55	- document rollback: can the pipeline be redeployed at an earlier version safely?
56
57	## Common Rationalizations
58
59	\| Rationalization \| Reality \|
60	\| --- \| --- \|
61	\| "Beam is portable so we don't need to think about runners." \| Runner differences in autoscaling, state backends, shuffle behavior, and fusion affect correctness and cost. Testing on a single runner is not proof of portability. \|
62	\| "We can use the same pipeline for batch and streaming without changes." \| Bounded and unbounded PCollections behave differently with respect to windowing, triggers, and watermarks. Unified does not mean identical. \|
63	\| "DirectRunner tests are enough." \| DirectRunner does not expose parallelism, shuffle, or scaling issues. Integration tests on the target runner are required for production confidence. \|
64	\| "Late data can be handled later." \| Late data handling must be part of the initial window and trigger design. Retrofitting allowed lateness after consumers depend on outputs is breaking. \|
65
66	## Red Flags
67
68	- pipeline runs on DirectRunner only and has never been tested on the production runner
69	- windowing strategy is undefined or uses global windows for unbounded sources
70	- no documentation of which runners are supported or tested
71	- stateful DoFns have no key cardinality analysis or state cleanup plan
72	- pipeline updates require full reprocessing because no drain or savepoint strategy exists
73	- side inputs are unbounded or grow without limit
74	- no metrics or observability beyond runner-provided defaults
75
76	## Verification
77
78	- [ ] Pipeline contract defines inputs, outputs, time semantics, and delivery guarantees
79	- [ ] Windowing, triggers, and allowed lateness are explicitly designed and tested
80	- [ ] Runner-specific configuration is isolated from portable transform logic
81	- [ ] State and side inputs are bounded and have cleanup or eviction plans
82	- [ ] Pipeline has been tested on the target production runner, not just DirectRunner
83	- [ ] Drain, update, and rollback strategies are documented
84	- [ ] Operational metrics cover latency, throughput, backlog, and error rates per transform

People who liked this also viewed

6 picks

Type

apache-beam-unified-batch-and-stream

19stars4forks

View on GitHub

How to install apache-beam-unified-batch-and-stream?

vaquarkhan/data-engineering-agent-skills/apache-beam-unified-batch-and-stream

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill apache-beam-unified-batch-and-stream

Installs into the current project.

Summary

Guides agents through Apache Beam pipelines that unify batch and streaming logic. Use when designing Beam transforms, windowing, runners, replay behavior, or portability across execution backends.

Files · 1

View on GitHub

SKILL.md

1	# Apache Beam Unified Batch And Stream
2
3	## Overview
4
5	Use this skill when `Apache Beam` is the abstraction layer for both batch and streaming data processing. It helps agents preserve portability without hiding time semantics or runner-specific constraints.
6
7	## When to Use
8
9	- building `Apache Beam` pipelines for batch, streaming, or unified workloads
10	- targeting multiple runners such as `Dataflow`, `Flink`, `Spark`, or `Direct Runner`
11	- sharing transform logic across batch and streaming modes
12	- managing windowing, watermarks, triggers, and late data handling
13	- designing portable pipelines that must run across environments
14
15	Do not use this when the workload is locked to a single runner and Beam portability is not a goal.
16
17	## Workflow
18
19	1. Define the pipeline contract and time semantics.
20	Include:
21	- input PCollections and their bounded or unbounded nature
22	- event-time versus processing-time expectations
23	- output schema, grain, and freshness requirements
24	- delivery guarantees expected by downstream consumers
25
26	2. Separate portable pipeline logic from runner-specific deployment.
27	- keep transforms, DoFns, and combiners runner-agnostic
28	- isolate runner configuration (parallelism, autoscaling, resource hints) into pipeline options
29	- document which runners are supported and tested
30	- avoid runner-specific APIs unless portability is explicitly sacrificed
31
32	3. Design windowing and trigger strategy explicitly.
33	Account for:
34	- fixed, sliding, session, or global windows
35	- trigger behavior: when to emit, accumulate, or discard
36	- allowed lateness and late data routing
37	- watermark advancement assumptions per source
38
39	4. Handle state and side inputs carefully.
40	- stateful DoFns bind to a specific key space — document key cardinality
41	- side inputs can become bottlenecks at scale — prefer bounded and small
42	- timers must account for watermark-driven versus processing-time semantics
43	- state cleanup must be explicit for unbounded pipelines
44
45	5. Make testing and local validation part of the workflow.
46	- use `DirectRunner` for correctness tests
47	- validate windowing behavior with synthetic watermark progression
48	- test exactly-once semantics through pipeline drains and restarts
49	- confirm output idempotency for at-least-once runners
50
51	6. Plan deployment, scaling, and operational observability.
52	- define autoscaling boundaries and cost limits per runner
53	- expose transform-level metrics for latency, throughput, and backlog
54	- plan drain and update strategies for streaming pipelines
55	- document rollback: can the pipeline be redeployed at an earlier version safely?
56
57	## Common Rationalizations
58
59	\| Rationalization \| Reality \|
60	\| --- \| --- \|
61	\| "Beam is portable so we don't need to think about runners." \| Runner differences in autoscaling, state backends, shuffle behavior, and fusion affect correctness and cost. Testing on a single runner is not proof of portability. \|
62	\| "We can use the same pipeline for batch and streaming without changes." \| Bounded and unbounded PCollections behave differently with respect to windowing, triggers, and watermarks. Unified does not mean identical. \|
63	\| "DirectRunner tests are enough." \| DirectRunner does not expose parallelism, shuffle, or scaling issues. Integration tests on the target runner are required for production confidence. \|
64	\| "Late data can be handled later." \| Late data handling must be part of the initial window and trigger design. Retrofitting allowed lateness after consumers depend on outputs is breaking. \|
65
66	## Red Flags
67
68	- pipeline runs on DirectRunner only and has never been tested on the production runner
69	- windowing strategy is undefined or uses global windows for unbounded sources
70	- no documentation of which runners are supported or tested
71	- stateful DoFns have no key cardinality analysis or state cleanup plan
72	- pipeline updates require full reprocessing because no drain or savepoint strategy exists
73	- side inputs are unbounded or grow without limit
74	- no metrics or observability beyond runner-provided defaults
75
76	## Verification
77
78	- [ ] Pipeline contract defines inputs, outputs, time semantics, and delivery guarantees
79	- [ ] Windowing, triggers, and allowed lateness are explicitly designed and tested
80	- [ ] Runner-specific configuration is isolated from portable transform logic
81	- [ ] State and side inputs are bounded and have cleanup or eviction plans
82	- [ ] Pipeline has been tested on the target production runner, not just DirectRunner
83	- [ ] Drain, update, and rollback strategies are documented
84	- [ ] Operational metrics cover latency, throughput, backlog, and error rates per transform

People who liked this also viewed

6 picks

Type