configuring-airflow-language-sdks

Name: configuring-airflow-language-sdks
Author: astronomer

26installs55forks

How to install configuring-airflow-language-sdks?

astronomer/agents/configuring-airflow-language-sdks

$npx -y skills add astronomer/agents --skill configuring-airflow-language-sdks

Installs into the current project.

Summary

Configures Airflow to run language SDK tasks (Java, Go, and future native SDKs) — register a coordinator, map a queue to it, ensure the runtime/artifact on workers, and tune coordinator options. Use when the user wants Airflow to route a queue to a native-language coordinator, as

Files · 1

View on GitHub

SKILL.md

1	# Configuring Airflow for Language SDKs
2
3	To run language SDK tasks, Airflow needs to know two things: which coordinator launches the native subprocess, and which queue routes to that coordinator. The mechanism is identical across every language SDK — only each coordinator's `classpath` and `kwargs` differ. This skill documents the shared wiring once, then the per-coordinator options. It is platform-neutral: the same settings apply on open-source Airflow and on managed platforms like Astro.
4
5	> Experimental. The language SDKs are in preview; configuration keys may change.
6
7	> For the task code, see authoring-language-sdk-tasks (and the per-language authoring skill, e.g. authoring-java-sdk-tasks, authoring-go-sdk-tasks). For building and shipping the artifact, see the per-language deploy skill (e.g. deploying-java-sdk-bundles, deploying-go-sdk-bundles).
8
9	---
10
11	## Prerequisites on the worker
12
13	- The runtime or artifact the SDK needs must be present on the worker nodes, because the coordinator spawns a native subprocess per task instance. The exact requirement is per-SDK — see [Per-coordinator options](#per-coordinator-options) (the Java SDK needs a JRE 17+; the Go SDK needs no language runtime — the bundle is a self-contained native executable, but it must be built for the worker's OS/arch).
14	- The compiled/native artifact(s) must be reachable on the worker. See the per-language deploy skill.
15	- The coordinators ship with the Airflow Task SDK (`apache-airflow-task-sdk`, installed with Airflow). No extra Python package is required.
16
17	---
18
19	## The two settings
20
21	Both live in the `[sdk]` configuration section and apply to every language SDK:
22
23	1. `coordinators` — a JSON object mapping a coordinator name you choose to its implementation (`classpath`) and constructor `kwargs`.
24	2. `queue_to_coordinator` — a JSON object mapping a task queue to a coordinator name.
25
26	A task whose stub sets `queue="..."` is handed to the named coordinator, which launches the native subprocess. The coordinator name is arbitrary — it just has to be the same string in both settings. The queue name must match the `queue=` set on the Python `@task.stub`.
27
28	### Option A: `airflow.cfg`
29
30	```ini
31	[sdk]
32	coordinators = {
33	"java-jdk17": {
34	"classpath": "airflow.sdk.coordinators.java.JavaCoordinator",
35	"kwargs": {"jars_root": ["/opt/airflow/jars"]}
36	},
37	"go": {
38	"classpath": "airflow.sdk.coordinators.executable.ExecutableCoordinator",
39	"kwargs": {"executables_root": ["/opt/airflow/executable-bundles"]}
40	}
41	}
42	queue_to_coordinator = {"java": "java-jdk17", "golang": "go"}
43	```
44
45	### Option B: environment variables
46
47	Each value must be valid one-line JSON. This form is convenient for containers, `.env` files, Docker Compose, and Helm.
48
49	```bash
50	export AIRFLOW__SDK__COORDINATORS='{"java-jdk17": {"classpath": "airflow.sdk.coordinators.java.JavaCoordinator", "kwargs": {"jars_root": ["/opt/airflow/jars"]}}, "go": {"classpath": "airflow.sdk.coordinators.executable.ExecutableCoordinator", "kwargs": {"executables_root": ["/opt/airflow/executable-bundles"]}}}'
51	export AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"java": "java-jdk17", "golang": "go"}'
52	```
53
54	The examples above register multiple coordinators at once (one per language) and map a different queue to each — register only the ones you use.
55
56	---
57
58	## Per-coordinator options
59
60	The `classpath` and `kwargs` are specific to each coordinator. Add a subsection here as new language SDKs land.
61
62	### JavaCoordinator
63
64	- `classpath`: `airflow.sdk.coordinators.java.JavaCoordinator`
65	- Worker runtime: JRE 17+ (`java` on `PATH`, or set `java_executable`).
66
67	\| Parameter \| Default \| Description \|
68	\|-----------\|---------\|-------------\|
69	\| `jars_root` \| (required) \| One or more directories scanned recursively for `.jar` files. Accepts a string or a list of strings/paths. The classpath is assembled automatically. \|
70	\| `java_executable` \| `"java"` \| Path to the `java` binary. Defaults to `java` on `$PATH`. \|
71	\| `jvm_args` \| `[]` \| Extra JVM arguments, e.g. `["-Xmx1g", "-Dsome.property=value"]`. \|
72	\| `main_class` \| (auto-detect) \| Explicit entry-point class. If omitted, the coordinator scans `jars_root` for a JAR whose manifest declares `Main-Class`. **Set this explicitly if multiple ex

People who liked this also viewed

6 picks

Type

configuring-airflow-language-sdks

26installs55forks

View on GitHub

How to install configuring-airflow-language-sdks?

astronomer/agents/configuring-airflow-language-sdks

$npx -y skills add astronomer/agents --skill configuring-airflow-language-sdks

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Configuring Airflow for Language SDKs
2
3	To run language SDK tasks, Airflow needs to know two things: which coordinator launches the native subprocess, and which queue routes to that coordinator. The mechanism is identical across every language SDK — only each coordinator's `classpath` and `kwargs` differ. This skill documents the shared wiring once, then the per-coordinator options. It is platform-neutral: the same settings apply on open-source Airflow and on managed platforms like Astro.
4
5	> Experimental. The language SDKs are in preview; configuration keys may change.
6
7	> For the task code, see authoring-language-sdk-tasks (and the per-language authoring skill, e.g. authoring-java-sdk-tasks, authoring-go-sdk-tasks). For building and shipping the artifact, see the per-language deploy skill (e.g. deploying-java-sdk-bundles, deploying-go-sdk-bundles).
8
9	---
10
11	## Prerequisites on the worker
12
13	- The runtime or artifact the SDK needs must be present on the worker nodes, because the coordinator spawns a native subprocess per task instance. The exact requirement is per-SDK — see [Per-coordinator options](#per-coordinator-options) (the Java SDK needs a JRE 17+; the Go SDK needs no language runtime — the bundle is a self-contained native executable, but it must be built for the worker's OS/arch).
14	- The compiled/native artifact(s) must be reachable on the worker. See the per-language deploy skill.
15	- The coordinators ship with the Airflow Task SDK (`apache-airflow-task-sdk`, installed with Airflow). No extra Python package is required.
16
17	---
18
19	## The two settings
20
21	Both live in the `[sdk]` configuration section and apply to every language SDK:
22
23	1. `coordinators` — a JSON object mapping a coordinator name you choose to its implementation (`classpath`) and constructor `kwargs`.
24	2. `queue_to_coordinator` — a JSON object mapping a task queue to a coordinator name.
25
26	A task whose stub sets `queue="..."` is handed to the named coordinator, which launches the native subprocess. The coordinator name is arbitrary — it just has to be the same string in both settings. The queue name must match the `queue=` set on the Python `@task.stub`.
27
28	### Option A: `airflow.cfg`
29
30	```ini
31	[sdk]
32	coordinators = {
33	"java-jdk17": {
34	"classpath": "airflow.sdk.coordinators.java.JavaCoordinator",
35	"kwargs": {"jars_root": ["/opt/airflow/jars"]}
36	},
37	"go": {
38	"classpath": "airflow.sdk.coordinators.executable.ExecutableCoordinator",
39	"kwargs": {"executables_root": ["/opt/airflow/executable-bundles"]}
40	}
41	}
42	queue_to_coordinator = {"java": "java-jdk17", "golang": "go"}
43	```
44
45	### Option B: environment variables
46
47	Each value must be valid one-line JSON. This form is convenient for containers, `.env` files, Docker Compose, and Helm.
48
49	```bash
50	export AIRFLOW__SDK__COORDINATORS='{"java-jdk17": {"classpath": "airflow.sdk.coordinators.java.JavaCoordinator", "kwargs": {"jars_root": ["/opt/airflow/jars"]}}, "go": {"classpath": "airflow.sdk.coordinators.executable.ExecutableCoordinator", "kwargs": {"executables_root": ["/opt/airflow/executable-bundles"]}}}'
51	export AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"java": "java-jdk17", "golang": "go"}'
52	```
53
54	The examples above register multiple coordinators at once (one per language) and map a different queue to each — register only the ones you use.
55
56	---
57
58	## Per-coordinator options
59
60	The `classpath` and `kwargs` are specific to each coordinator. Add a subsection here as new language SDKs land.
61
62	### JavaCoordinator
63
64	- `classpath`: `airflow.sdk.coordinators.java.JavaCoordinator`
65	- Worker runtime: JRE 17+ (`java` on `PATH`, or set `java_executable`).
66
67	\| Parameter \| Default \| Description \|
68	\|-----------\|---------\|-------------\|
69	\| `jars_root` \| (required) \| One or more directories scanned recursively for `.jar` files. Accepts a string or a list of strings/paths. The classpath is assembled automatically. \|
70	\| `java_executable` \| `"java"` \| Path to the `java` binary. Defaults to `java` on `$PATH`. \|
71	\| `jvm_args` \| `[]` \| Extra JVM arguments, e.g. `["-Xmx1g", "-Dsome.property=value"]`. \|
72	\| `main_class` \| (auto-detect) \| Explicit entry-point class. If omitted, the coordinator scans `jars_root` for a JAR whose manifest declares `Main-Class`. **Set this explicitly if multiple ex

People who liked this also viewed

6 picks

Type