mcp-data-observability-integration

Name: mcp-data-observability-integration
Author: vaquarkhan

19stars4forks

How to install mcp-data-observability-integration?

vaquarkhan/data-engineering-agent-skills/mcp-data-observability-integration

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill mcp-data-observability-integration

Installs into the current project.

Summary

Guides agents to wire Model Context Protocol servers for live data platform observability including Spark execution plans, OOM diagnosis, Kafka consumer lag, and orchestration run state. Use when agents need cluster metrics, streaming lag, or job diagnostics instead of blind code

Files · 1

View on GitHub

SKILL.md

1	# MCP Data Observability Integration
2
3	## Overview
4
5	Agents that cannot see production signals guess at fixes — scaling clusters blindly, rerunning jobs that OOM for structural reasons, or changing Kafka consumers without checking lag. This skill pairs MCP server templates with safe read-only observability workflows so diagnosis precedes mutation.
6
7	## When to Use
8
9	- diagnosing Spark OOM, shuffle skew, or stage failures
10	- inspecting Kafka consumer group lag before replay or consumer changes
11	- reviewing Airflow DAG run state, task duration drift, or failed retries
12	- validating warehouse publish state before reopening downstream consumers
13	- setting up IDE-integrated observability for data engineering agents
14
15	Do not use MCP write capabilities for destructive fixes during incident triage unless explicitly approved and scoped.
16
17	## Workflow
18
19	1. Choose the smallest MCP surface for the question.
20	- Spark plans and stage metrics: `mcp/spark.mcp.json` or `mcp/databricks.mcp.json`
21	- Kafka lag and topic metadata: `mcp/kafka.mcp.json`
22	- Orchestration run state: `mcp/airflow.mcp.json`
23	- Warehouse inspection: `mcp/snowflake.mcp.json`, `mcp/bigquery.mcp.json`, or `mcp/postgres.mcp.json`
24	- Release and job metadata: `mcp/dbt-cloud.mcp.json`, `mcp/github.mcp.json`
25
26	2. Configure read-only first.
27	From `mcp/README.md`:
28	- replace placeholder `command` with the MCP server binary in use
29	- scope credentials to read-only inspection roles
30	- set allowlists (`topic`, catalog, schema) to minimum necessary scope
31	- validate connectivity outside the agent session before relying on tool output
32
33	3. Establish the diagnostic sequence.
34	Typical order:
35	- confirm symptom (lag, failure rate, freshness breach)
36	- pull live metadata (consumer lag, last successful run, stage skew)
37	- compare against baseline or SLA from `data-observability-and-sla-management`
38	- form hypothesis before proposing code or infra changes
39	- record findings in incident or backfill evidence templates when action follows
40
41	4. Apply platform-specific investigation patterns.
42	- Spark OOM or timeout: inspect stage breakdown, shuffle read/write bytes, spill metrics, and partition counts; prefer physical plan fixes over blind scale-up
43	- Kafka lag: inspect group lag per partition, DLQ rate, and retention headroom; pause replay until lag root cause is classified
44	- Orchestration failures: inspect task logs, retry history, and upstream sensors; do not trigger wide backfills from a single red task without plan gates
45
46	Load `references/mcp-data-observability-patterns.md` for tool-to-symptom mapping.
47
48	5. Pair observability with safety skills.
49	- replay or repair: `safe-backfill-and-replay-orchestration`
50	- streaming hardening: `kafka-resilience-and-schema-evolution`
51	- serverless Spark: `spark-serverless-reliability-and-state-management`
52	- incident response: `incident-triage-and-pipeline-recovery`
53
54	6. Close the loop after changes.
55	Re-query MCP signals to confirm lag recovery, run success, or freshness restoration before publish reopen.
56
57	## Common Rationalizations
58
59	\| Rationalization \| Reality \|
60	\| --- \| --- \|
61	\| "We can infer the problem from logs alone." \| Logs without lag, plan, and run metadata miss systemic causes like skew and consumer stall. \|
62	\| "MCP access should use admin credentials for speed." \| Over-scoped tokens increase blast radius when agents suggest broad changes. \|
63	\| "Observability is only for on-call humans." \| Agents without live signals replicate dangerous guesswork at machine speed. \|
64	\| "One MCP server is enough for every stack." \| Multi-platform estates need focused templates per system, not one mega-connection. \|
65
66	## Red Flags
67
68	- agents propose infra or code changes without querying live signals
69	- MCP templates use write-capable credentials by default
70	- no allowlists on Kafka topics or warehouse schemas
71	- lag or run failure is acknowledged but not tied to an owner or SLA
72	- observability findings are not linked to backfill or incident evidence
73
74	## Verification
75
76	- [ ] The correct MCP template is selected and validated read-only
77	- [ ] Live signals were queried before structural changes were proposed
78	- [ ] Findings map to a named safety skill when replay or publish is involved
79	- [ ] Post-change signals confirm recovery before publish reopen

People who liked this also viewed

6 picks

Type

mcp-data-observability-integration

19stars4forks

View on GitHub

How to install mcp-data-observability-integration?

vaquarkhan/data-engineering-agent-skills/mcp-data-observability-integration

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill mcp-data-observability-integration

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# MCP Data Observability Integration
2
3	## Overview
4
5	Agents that cannot see production signals guess at fixes — scaling clusters blindly, rerunning jobs that OOM for structural reasons, or changing Kafka consumers without checking lag. This skill pairs MCP server templates with safe read-only observability workflows so diagnosis precedes mutation.
6
7	## When to Use
8
9	- diagnosing Spark OOM, shuffle skew, or stage failures
10	- inspecting Kafka consumer group lag before replay or consumer changes
11	- reviewing Airflow DAG run state, task duration drift, or failed retries
12	- validating warehouse publish state before reopening downstream consumers
13	- setting up IDE-integrated observability for data engineering agents
14
15	Do not use MCP write capabilities for destructive fixes during incident triage unless explicitly approved and scoped.
16
17	## Workflow
18
19	1. Choose the smallest MCP surface for the question.
20	- Spark plans and stage metrics: `mcp/spark.mcp.json` or `mcp/databricks.mcp.json`
21	- Kafka lag and topic metadata: `mcp/kafka.mcp.json`
22	- Orchestration run state: `mcp/airflow.mcp.json`
23	- Warehouse inspection: `mcp/snowflake.mcp.json`, `mcp/bigquery.mcp.json`, or `mcp/postgres.mcp.json`
24	- Release and job metadata: `mcp/dbt-cloud.mcp.json`, `mcp/github.mcp.json`
25
26	2. Configure read-only first.
27	From `mcp/README.md`:
28	- replace placeholder `command` with the MCP server binary in use
29	- scope credentials to read-only inspection roles
30	- set allowlists (`topic`, catalog, schema) to minimum necessary scope
31	- validate connectivity outside the agent session before relying on tool output
32
33	3. Establish the diagnostic sequence.
34	Typical order:
35	- confirm symptom (lag, failure rate, freshness breach)
36	- pull live metadata (consumer lag, last successful run, stage skew)
37	- compare against baseline or SLA from `data-observability-and-sla-management`
38	- form hypothesis before proposing code or infra changes
39	- record findings in incident or backfill evidence templates when action follows
40
41	4. Apply platform-specific investigation patterns.
42	- Spark OOM or timeout: inspect stage breakdown, shuffle read/write bytes, spill metrics, and partition counts; prefer physical plan fixes over blind scale-up
43	- Kafka lag: inspect group lag per partition, DLQ rate, and retention headroom; pause replay until lag root cause is classified
44	- Orchestration failures: inspect task logs, retry history, and upstream sensors; do not trigger wide backfills from a single red task without plan gates
45
46	Load `references/mcp-data-observability-patterns.md` for tool-to-symptom mapping.
47
48	5. Pair observability with safety skills.
49	- replay or repair: `safe-backfill-and-replay-orchestration`
50	- streaming hardening: `kafka-resilience-and-schema-evolution`
51	- serverless Spark: `spark-serverless-reliability-and-state-management`
52	- incident response: `incident-triage-and-pipeline-recovery`
53
54	6. Close the loop after changes.
55	Re-query MCP signals to confirm lag recovery, run success, or freshness restoration before publish reopen.
56
57	## Common Rationalizations
58
59	\| Rationalization \| Reality \|
60	\| --- \| --- \|
61	\| "We can infer the problem from logs alone." \| Logs without lag, plan, and run metadata miss systemic causes like skew and consumer stall. \|
62	\| "MCP access should use admin credentials for speed." \| Over-scoped tokens increase blast radius when agents suggest broad changes. \|
63	\| "Observability is only for on-call humans." \| Agents without live signals replicate dangerous guesswork at machine speed. \|
64	\| "One MCP server is enough for every stack." \| Multi-platform estates need focused templates per system, not one mega-connection. \|
65
66	## Red Flags
67
68	- agents propose infra or code changes without querying live signals
69	- MCP templates use write-capable credentials by default
70	- no allowlists on Kafka topics or warehouse schemas
71	- lag or run failure is acknowledged but not tied to an owner or SLA
72	- observability findings are not linked to backfill or incident evidence
73
74	## Verification
75
76	- [ ] The correct MCP template is selected and validated read-only
77	- [ ] Live signals were queried before structural changes were proposed
78	- [ ] Findings map to a named safety skill when replay or publish is involved
79	- [ ] Post-change signals confirm recovery before publish reopen

People who liked this also viewed

6 picks

Type