mlops-observability

Name: mlops-observability
Author: MLOps-Courses

19stars3forks

How to install mlops-observability?

mlops-courses/mlops-coding-skills/mlops-observability

$npx -y skills add MLOps-Courses/mlops-coding-skills --skill mlops-observability

Installs into the current project.

Summary

Guide to implement full stack observability including reproducibility, lineage, monitoring, alerting, and explainability.

Files · 1

View on GitHub

SKILL.md

1	# MLOps Observability
2
3	## Goal
4
5	To implement a "Glass Box" system where every result is Reproducible, every asset has Lineage, and system health is Monitored, Alerted on, and Explained.
6
7	## Prerequisites
8
9	- Language: Python
10	- Context: Production monitoring and debugging.
11	- Platform Suggestion: MLflow, SHAP, Evidently, ...
12
13	## Instructions
14
15	### 1. Guarantee Reproducibility
16
17	Consistency is key. For instance:
18
19	1. Randomness: Set seeds for `random`, `numpy`, `torch`, `tensorflow`.
20	2. Environment: Use `docker` and locked dependencies (`uv.lock`).
21	3. Builds: Use `mise run build` (wrapping `uv build --build-constraint`) for deterministic wheels.
22	4. Code: Track git commit hash for every run.
23
24	### 2. Track Data Lineage
25
26	Know the origin of your data. For instance:
27
28	1. Datasets: Create MLflow Datasets with `mlflow.data.from_pandas`.
29	2. Logging: Log inputs to MLflow context with `mlflow.log_input`.
30	3. Versioning: Version data files (e.g., `data/v1.csv`) or use DVC.
31	4. Transformations: Log preprocessing parameters mapping data versions to model versions.
32
33	### 3. Monitoring & Drift Detection
34
35	Watch for silent failures. For instance:
36
37	1. Validation: Gate models against quality thresholds with `mlflow.validate_evaluation_results` (MLflow 3).
38	2. Drift: Use `evidently` to compare `reference` (training) vs `current` (production) data.
39	- Detect Data Drift (input distribution changes) and Concept Drift (relationship changes).
40	3. System: Enable MLflow System Metrics (`log_system_metrics=True`) for CPU/GPU.
41
42	### 4. Alerting
43
44	Don't stare at dashboards. For instance:
45
46	1. Local: Use `plyer` for desktop notifications during long training runs.
47	2. Production: Use `PagerDuty` (critical) or `Slack` (warnings).
48	3. Thresholds: Use Static (fixed value) or Dynamic (anomaly detection) rules.
49	4. Action: Alerts must link to a dashboard or playbook.
50
51	### 5. Explainability (XAI)
52
53	Trust but verify. For instance:
54
55	1. Global: Use Feature Importance (e.g., Random Forest) to understand overall logic.
56	2. Local: Use `SHAP` values to explain individual predictions.
57	3. Artifacts: Save explanations (plots/tables) as MLflow artifacts.
58
59	### 6. Infrastructure & Costs
60
61	Optimize resources. For instance:
62
63	1. Tags: Tag runs with `project`, `env`, `user`.
64	2. Costs: Log `run_time` and instance type to estimate ROI.
65
66	## Self-Correction Checklist
67
68	- [ ] Seeds: Are random seeds fixed?
69	- [ ] Inputs: Are input datasets logged to MLflow?
70	- [ ] System Metrics: Is `log_system_metrics` enabled?
71	- [ ] Explanations: Are SHAP values generated?
72	- [ ] Alerts: Are thresholds defined for failures?

People who liked this also viewed

6 picks

Type

find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 20262.6M27k
grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026633k182k
vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026573k29k
improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026522k182k
setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026454k182k
triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026424k182k

mlops-observability

19stars3forks

View on GitHub

How to install mlops-observability?

mlops-courses/mlops-coding-skills/mlops-observability

$npx -y skills add MLOps-Courses/mlops-coding-skills --skill mlops-observability

Installs into the current project.

Summary

Guide to implement full stack observability including reproducibility, lineage, monitoring, alerting, and explainability.

Files · 1

View on GitHub

SKILL.md

1	# MLOps Observability
2
3	## Goal
4
5	To implement a "Glass Box" system where every result is Reproducible, every asset has Lineage, and system health is Monitored, Alerted on, and Explained.
6
7	## Prerequisites
8
9	- Language: Python
10	- Context: Production monitoring and debugging.
11	- Platform Suggestion: MLflow, SHAP, Evidently, ...
12
13	## Instructions
14
15	### 1. Guarantee Reproducibility
16
17	Consistency is key. For instance:
18
19	1. Randomness: Set seeds for `random`, `numpy`, `torch`, `tensorflow`.
20	2. Environment: Use `docker` and locked dependencies (`uv.lock`).
21	3. Builds: Use `mise run build` (wrapping `uv build --build-constraint`) for deterministic wheels.
22	4. Code: Track git commit hash for every run.
23
24	### 2. Track Data Lineage
25
26	Know the origin of your data. For instance:
27
28	1. Datasets: Create MLflow Datasets with `mlflow.data.from_pandas`.
29	2. Logging: Log inputs to MLflow context with `mlflow.log_input`.
30	3. Versioning: Version data files (e.g., `data/v1.csv`) or use DVC.
31	4. Transformations: Log preprocessing parameters mapping data versions to model versions.
32
33	### 3. Monitoring & Drift Detection
34
35	Watch for silent failures. For instance:
36
37	1. Validation: Gate models against quality thresholds with `mlflow.validate_evaluation_results` (MLflow 3).
38	2. Drift: Use `evidently` to compare `reference` (training) vs `current` (production) data.
39	- Detect Data Drift (input distribution changes) and Concept Drift (relationship changes).
40	3. System: Enable MLflow System Metrics (`log_system_metrics=True`) for CPU/GPU.
41
42	### 4. Alerting
43
44	Don't stare at dashboards. For instance:
45
46	1. Local: Use `plyer` for desktop notifications during long training runs.
47	2. Production: Use `PagerDuty` (critical) or `Slack` (warnings).
48	3. Thresholds: Use Static (fixed value) or Dynamic (anomaly detection) rules.
49	4. Action: Alerts must link to a dashboard or playbook.
50
51	### 5. Explainability (XAI)
52
53	Trust but verify. For instance:
54
55	1. Global: Use Feature Importance (e.g., Random Forest) to understand overall logic.
56	2. Local: Use `SHAP` values to explain individual predictions.
57	3. Artifacts: Save explanations (plots/tables) as MLflow artifacts.
58
59	### 6. Infrastructure & Costs
60
61	Optimize resources. For instance:
62
63	1. Tags: Tag runs with `project`, `env`, `user`.
64	2. Costs: Log `run_time` and instance type to estimate ROI.
65
66	## Self-Correction Checklist
67
68	- [ ] Seeds: Are random seeds fixed?
69	- [ ] Inputs: Are input datasets logged to MLflow?
70	- [ ] System Metrics: Is `log_system_metrics` enabled?
71	- [ ] Explanations: Are SHAP values generated?
72	- [ ] Alerts: Are thresholds defined for failures?

People who liked this also viewed

6 picks

Type