kafka

Name: kafka
Author: mjunaidca

19installs11forks

How to install kafka?

mjunaidca/mjs-agent-skills/kafka

$npx -y skills add mjunaidca/mjs-agent-skills --skill kafka

Installs into the current project.

Summary

Apache Kafka on Kubernetes with Strimzi (KRaft mode, no ZooKeeper). This skill should be used when users ask to deploy Kafka clusters, build producers/consumers, implement event-driven patterns, or debug Kafka issues. Includes tested manifests and Makefile for one-command deploym

Files · 1

View on GitHub

SKILL.md

1	# Apache Kafka
2
3	Event streaming for Kubernetes. Strimzi operator, KRaft mode, no ZooKeeper.
4
5	## Quick Start (Tested)
6
7	```bash
8	make install # Deploy Strimzi + Kafka
9	make test # Verify everything works
10	make status # Show resources
11	make uninstall # Clean up
12	```
13
14	Requirements: Kubernetes cluster, Helm 3+
15
16	Versions: Strimzi 0.49+, Kafka 4.1.1
17
18	---
19
20	## Resource Detection & Adaptation
21
22	Before generating manifests, detect the target environment:
23
24	```bash
25	# Detect machine memory
26	sysctl -n hw.memsize 2>/dev/null \| awk '{print $0/1024/1024/1024 " GB"}' \|\| \
27	grep MemTotal /proc/meminfo \| awk '{print $2/1024/1024 " GB"}'
28
29	# Detect Docker Desktop allocation
30	docker info --format '{{.MemTotal}}' 2>/dev/null \| awk '{print $0/1024/1024/1024 " GB"}'
31
32	# Detect Kubernetes node capacity
33	kubectl get nodes -o jsonpath='{.items[0].status.capacity.memory}' 2>/dev/null
34	```
35
36	Adapt resource configuration based on detection:
37
38	\| Detected RAM \| Profile \| Kafka Memory \| Action \|
39	\|--------------\|---------\|--------------\|--------\|
40	\| < 12GB \| Minimal \| 512Mi-1Gi \| Warn user about constraints \|
41	\| 12-24GB \| Standard \| 1Gi-2Gi \| Default configuration \|
42	\| > 24GB \| Production \| 4Gi-8Gi \| Enable full features \|
43
44	### Adaptive Resource Templates
45
46	Minimal (detected < 12GB):
47	```yaml
48	resources:
49	requests:
50	memory: 512Mi
51	cpu: 200m
52	limits:
53	memory: 1Gi
54	cpu: 500m
55	```
56	⚠️ Agent should warn: "Limited resources detected. Kafka may be unstable under load."
57
58	Standard (detected 12-24GB):
59	```yaml
60	resources:
61	requests:
62	memory: 1Gi
63	cpu: 250m
64	limits:
65	memory: 2Gi
66	cpu: 1000m
67	```
68
69	Production (detected > 24GB or real cluster):
70	```yaml
71	resources:
72	requests:
73	memory: 4Gi
74	cpu: 1000m
75	limits:
76	memory: 8Gi
77	cpu: 4000m
78	```
79
80	### Agent Behavior
81
82	1. Always detect before generating manifests
83	2. Adapt resource configs to detected environment
84	3. Warn if resources are insufficient for requested workload
85	4. Suggest Docker Desktop settings if running locally
86
87	---
88
89	## What This Skill Does
90
91	\| Task \| How \|
92	\|------\|-----\|
93	\| Analyze coupling \| Identify temporal, availability, behavioral issues \|
94	\| Explain eventual consistency \| Consistency windows, read-your-writes patterns \|
95	\| Design events \| Domain events, CloudEvents, Avro schemas \|
96	\| Deploy Kafka \| Helm (Strimzi) + kubectl (manifests) \|
97	\| Create topics \| KafkaTopic CRD \|
98	\| Build producers \| confluent-kafka-python templates \|
99	\| Build consumers \| AIOConsumer for FastAPI \|
100	\| Debug issues \| Runbooks in references/ \|
101
102	## What This Skill Does NOT Do
103
104	- Deploy ZooKeeper (KRaft only)
105	- Manage Kafka Streams applications
106	- Configure multi-datacenter replication
107
108	---
109
110	## Deployment
111
112	### Install Strimzi Operator
113
114	```bash
115	helm repo add strimzi https://strimzi.io/charts
116	helm install strimzi-operator strimzi/strimzi-kafka-operator -n kafka --create-namespace --wait
117	```
118
119	### Deploy Kafka Cluster
120
121	```bash
122	kubectl apply -f manifests/kafka-cluster.yaml -n kafka
123	kubectl wait kafka/dev-cluster --for=condition=Ready --timeout=300s -n kafka
124	```
125
126	### Create Topic
127
128	```bash
129	kubectl apply -f manifests/kafka-topic.yaml -n kafka
130	```
131
132	### Verify
133
134	```bash
135	kubectl get kafka,kafkatopic,pods -n kafka
136	```
137
138	---
139
140	## Core Concepts
141
142	```
143	Topic = Named stream (like a database table)
144	Partition = Ordered log within topic (parallelism unit)
145	Consumer Group = Consumers sharing work (partition → one consumer)
146	Offset = Consumer position (commit to track progress)
147	Broker = Kafka server
148	Controller = Metadata manager (KRaft replaces ZooKeeper)
149	```
150
151	---
152
153	## Local Development
154
155	Connect from your host machine (no port-forward needed):
156
157	```python
158	# From your local machine (outside Kubernetes)
159	producer = Producer({'bootstrap.servers': 'localhost:30092'})
160	```
161
162	Connect from inside Kubernetes (pod-to-pod):
163
164	```python
165	# From another pod in the cluster
166	producer = Producer({'bootstrap.servers': 'dev-cluster-kafka-bootstrap.kafka:9092'})
167	```
168
169	\| Location \| Bootstrap Server \|
170	\|----------\|------------------\|
171	\| Local machine \| `localhost:30092` \|
172	\| Same namespace \| `dev-cluster-kafka-bootstrap:9092` \|
173	\| Different namespace \| `dev-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092` \|
174
175	---
176
177	## Producer/Consumer (Python)
178
179	```python
180	from confluent_kafka import Producer, Consumer
181
182	# Producer (production config)
183	producer = Producer({
184	'bootstrap.servers': 'localhost:30092', # Or K8s service for pods
185	'acks': 'all',
186	'enable.idempotence': True,
187	})
188	producer.produce('my-topic'

People who liked this also viewed

6 picks

Type

kafka

19installs11forks

View on GitHub

How to install kafka?

mjunaidca/mjs-agent-skills/kafka

$npx -y skills add mjunaidca/mjs-agent-skills --skill kafka

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Apache Kafka
2
3	Event streaming for Kubernetes. Strimzi operator, KRaft mode, no ZooKeeper.
4
5	## Quick Start (Tested)
6
7	```bash
8	make install # Deploy Strimzi + Kafka
9	make test # Verify everything works
10	make status # Show resources
11	make uninstall # Clean up
12	```
13
14	Requirements: Kubernetes cluster, Helm 3+
15
16	Versions: Strimzi 0.49+, Kafka 4.1.1
17
18	---
19
20	## Resource Detection & Adaptation
21
22	Before generating manifests, detect the target environment:
23
24	```bash
25	# Detect machine memory
26	sysctl -n hw.memsize 2>/dev/null \| awk '{print $0/1024/1024/1024 " GB"}' \|\| \
27	grep MemTotal /proc/meminfo \| awk '{print $2/1024/1024 " GB"}'
28
29	# Detect Docker Desktop allocation
30	docker info --format '{{.MemTotal}}' 2>/dev/null \| awk '{print $0/1024/1024/1024 " GB"}'
31
32	# Detect Kubernetes node capacity
33	kubectl get nodes -o jsonpath='{.items[0].status.capacity.memory}' 2>/dev/null
34	```
35
36	Adapt resource configuration based on detection:
37
38	\| Detected RAM \| Profile \| Kafka Memory \| Action \|
39	\|--------------\|---------\|--------------\|--------\|
40	\| < 12GB \| Minimal \| 512Mi-1Gi \| Warn user about constraints \|
41	\| 12-24GB \| Standard \| 1Gi-2Gi \| Default configuration \|
42	\| > 24GB \| Production \| 4Gi-8Gi \| Enable full features \|
43
44	### Adaptive Resource Templates
45
46	Minimal (detected < 12GB):
47	```yaml
48	resources:
49	requests:
50	memory: 512Mi
51	cpu: 200m
52	limits:
53	memory: 1Gi
54	cpu: 500m
55	```
56	⚠️ Agent should warn: "Limited resources detected. Kafka may be unstable under load."
57
58	Standard (detected 12-24GB):
59	```yaml
60	resources:
61	requests:
62	memory: 1Gi
63	cpu: 250m
64	limits:
65	memory: 2Gi
66	cpu: 1000m
67	```
68
69	Production (detected > 24GB or real cluster):
70	```yaml
71	resources:
72	requests:
73	memory: 4Gi
74	cpu: 1000m
75	limits:
76	memory: 8Gi
77	cpu: 4000m
78	```
79
80	### Agent Behavior
81
82	1. Always detect before generating manifests
83	2. Adapt resource configs to detected environment
84	3. Warn if resources are insufficient for requested workload
85	4. Suggest Docker Desktop settings if running locally
86
87	---
88
89	## What This Skill Does
90
91	\| Task \| How \|
92	\|------\|-----\|
93	\| Analyze coupling \| Identify temporal, availability, behavioral issues \|
94	\| Explain eventual consistency \| Consistency windows, read-your-writes patterns \|
95	\| Design events \| Domain events, CloudEvents, Avro schemas \|
96	\| Deploy Kafka \| Helm (Strimzi) + kubectl (manifests) \|
97	\| Create topics \| KafkaTopic CRD \|
98	\| Build producers \| confluent-kafka-python templates \|
99	\| Build consumers \| AIOConsumer for FastAPI \|
100	\| Debug issues \| Runbooks in references/ \|
101
102	## What This Skill Does NOT Do
103
104	- Deploy ZooKeeper (KRaft only)
105	- Manage Kafka Streams applications
106	- Configure multi-datacenter replication
107
108	---
109
110	## Deployment
111
112	### Install Strimzi Operator
113
114	```bash
115	helm repo add strimzi https://strimzi.io/charts
116	helm install strimzi-operator strimzi/strimzi-kafka-operator -n kafka --create-namespace --wait
117	```
118
119	### Deploy Kafka Cluster
120
121	```bash
122	kubectl apply -f manifests/kafka-cluster.yaml -n kafka
123	kubectl wait kafka/dev-cluster --for=condition=Ready --timeout=300s -n kafka
124	```
125
126	### Create Topic
127
128	```bash
129	kubectl apply -f manifests/kafka-topic.yaml -n kafka
130	```
131
132	### Verify
133
134	```bash
135	kubectl get kafka,kafkatopic,pods -n kafka
136	```
137
138	---
139
140	## Core Concepts
141
142	```
143	Topic = Named stream (like a database table)
144	Partition = Ordered log within topic (parallelism unit)
145	Consumer Group = Consumers sharing work (partition → one consumer)
146	Offset = Consumer position (commit to track progress)
147	Broker = Kafka server
148	Controller = Metadata manager (KRaft replaces ZooKeeper)
149	```
150
151	---
152
153	## Local Development
154
155	Connect from your host machine (no port-forward needed):
156
157	```python
158	# From your local machine (outside Kubernetes)
159	producer = Producer({'bootstrap.servers': 'localhost:30092'})
160	```
161
162	Connect from inside Kubernetes (pod-to-pod):
163
164	```python
165	# From another pod in the cluster
166	producer = Producer({'bootstrap.servers': 'dev-cluster-kafka-bootstrap.kafka:9092'})
167	```
168
169	\| Location \| Bootstrap Server \|
170	\|----------\|------------------\|
171	\| Local machine \| `localhost:30092` \|
172	\| Same namespace \| `dev-cluster-kafka-bootstrap:9092` \|
173	\| Different namespace \| `dev-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092` \|
174
175	---
176
177	## Producer/Consumer (Python)
178
179	```python
180	from confluent_kafka import Producer, Consumer
181
182	# Producer (production config)
183	producer = Producer({
184	'bootstrap.servers': 'localhost:30092', # Or K8s service for pods
185	'acks': 'all',
186	'enable.idempotence': True,
187	})
188	producer.produce('my-topic'

People who liked this also viewed

6 picks

Type