datarobot-model-monitoring

Name: datarobot-model-monitoring
Author: datarobot-oss

22stars19forks

How to install datarobot-model-monitoring?

datarobot-oss/datarobot-agent-skills/datarobot-model-monitoring

$npx -y skills add datarobot-oss/datarobot-agent-skills --skill datarobot-model-monitoring

Installs into the current project.

Summary

Tools and guidance for monitoring model performance, tracking data drift, managing model health, and detecting prediction anomalies. Use when monitoring deployed models, tracking drift, or investigating prediction anomalies.

Files · 1

View on GitHub

SKILL.md

1	# DataRobot Model Monitoring Skill
2
3	This skill provides comprehensive guidance for monitoring deployed models, tracking performance metrics, detecting data drift, and managing model health.
4
5	## Quick Start
6
7	Most common use case: Check deployment health and data drift
8
9	1. Check service stats: `deployment.get_service_stats(...)` to review prediction volume/latency
10	2. Check drift: `deployment.get_feature_drift(...)` / `deployment.get_target_drift(...)`
11	3. Compare over time: Use `get_service_stats_over_time(...)` and drift periods to assess trends
12
13	Example: "Check the health of deployment abc123 and report any data drift issues"
14
15	## When to use this skill
16
17	Use this skill when you need to:
18	- Monitor model performance in production
19	- Track data drift and feature drift
20	- Detect prediction anomalies
21	- Monitor prediction accuracy over time
22	- Set up alerts for model degradation
23	- Analyze model health metrics
24	- Compare production performance to training performance
25
26	## Key capabilities
27
28	### 1. Performance Monitoring
29
30	- Track prediction accuracy and metrics over time
31	- Compare production metrics to training metrics
32	- Monitor prediction volume and latency
33	- Identify performance degradation trends
34
35	### 2. Data Drift Detection
36
37	- Detect changes in feature distributions
38	- Identify feature drift (statistical changes)
39	- Monitor target drift (if actuals available)
40	- Alert on significant drift events
41
42	### 3. Prediction Monitoring
43
44	- Monitor prediction distributions
45	- Detect prediction anomalies
46	- Track prediction confidence scores
47	- Identify unusual prediction patterns
48
49	### 4. Health Management
50
51	- Assess overall model health
52	- Generate monitoring reports
53	- Set up automated alerts
54	- Manage model retraining triggers
55
56	## Workflow examples
57
58	### Example 1: Check model health and drift
59
60	User request: "Check the health of deployment abc123 and report any data drift issues."
61
62	Agent workflow:
63	1. Get deployment monitoring status
64	2. Retrieve recent performance metrics
65	3. Check for data drift in key features
66	4. Compare current metrics to baseline (training)
67	5. Identify any significant drift or degradation
68	6. Report findings with recommendations
69
70	### Example 2: Set up drift monitoring alerts
71
72	User request: "Set up alerts for deployment xyz789 to notify when feature drift exceeds 0.2."
73
74	Agent workflow:
75	1. Get deployment configuration
76	2. Configure drift threshold (0.2)
77	3. Set up alert notifications
78	4. Specify which features to monitor
79	5. Test alert configuration
80	6. Confirm monitoring is active
81
82	## Using DataRobot SDK
83
84	This skill guides you to use the DataRobot Python SDK directly. Install the SDK if needed:
85
86	```bash
87	pip install datarobot
88	```
89
90	### Key SDK Operations
91
92	Use these DataRobot SDK and MLOps API methods for monitoring:
93
94	Deployment Monitoring:
95	- `deployment.get_service_stats(...)` - Get service statistics (latency, volume, etc.)
96	- `deployment.get_feature_drift(...)` - Get feature drift metrics (returns `FeatureDrift` objects)
97	- `deployment.get_target_drift(...)` - Get target drift metrics (returns `TargetDrift`)
98	- `deployment.get_prediction_results(...)` - Retrieve recorded prediction results (if enabled)
99
100	Model Performance:
101	- `model.get_metrics()` - Get model performance metrics
102	- `model.get_roc_curve()` - Get ROC curve for comparison
103
104	Note: Some monitoring features may require DataRobot MLOps API. See the [Common Patterns](#common-patterns) section below for examples.
105
106	## Best practices
107
108	1. Regular monitoring: Check model health regularly, not just when issues arise
109	2. Baseline comparison: Always compare production metrics to training baseline
110	3. Drift thresholds: Set appropriate drift thresholds based on your domain
111	4. Key features: Focus monitoring on high-importance features
112	5. Automated alerts: Set up alerts for critical issues
113	6. Historical analysis: Track trends over time, not just point-in-time metrics
114
115	## Common patterns
116
117	### Pattern 1: Health check
118	```python
119	import datarobot as dr
120	import os
121
122	# Initialize client
123	client = dr.Client(
124	token=os.getenv("DATAROBOT_API_TOKEN"),
125	endpoint=os.getenv("DATAROBOT_ENDPOINT")
126	)
127
128	# Get deployment
129	deployment = dr.Deployment.get("abc123")
130
131	# Get service stats (requires MLOps monitoring to be enabled)
132	stats = deployment.get_service_stats()
133	print(f"Prediction count: {stats.prediction_count}")
134	print(f"Mean response time (ms): {stats.mean_response_time}")
135
136	# Get recorded prediction results (if available / enabled)
137	try:
138	recent = deployment.get_prediction_results(limit=10)
139	print(f"Recent prediction results: {len(recent)}")
140	except Exception as e:
141	print(f"Prediction results not available: {e}")
142	```
143
144	### Pattern 2: Drift detection
145	```py

People who liked this also viewed

6 picks

Type

datarobot-model-monitoring

22stars19forks

View on GitHub

How to install datarobot-model-monitoring?

datarobot-oss/datarobot-agent-skills/datarobot-model-monitoring

$npx -y skills add datarobot-oss/datarobot-agent-skills --skill datarobot-model-monitoring

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# DataRobot Model Monitoring Skill
2
3	This skill provides comprehensive guidance for monitoring deployed models, tracking performance metrics, detecting data drift, and managing model health.
4
5	## Quick Start
6
7	Most common use case: Check deployment health and data drift
8
9	1. Check service stats: `deployment.get_service_stats(...)` to review prediction volume/latency
10	2. Check drift: `deployment.get_feature_drift(...)` / `deployment.get_target_drift(...)`
11	3. Compare over time: Use `get_service_stats_over_time(...)` and drift periods to assess trends
12
13	Example: "Check the health of deployment abc123 and report any data drift issues"
14
15	## When to use this skill
16
17	Use this skill when you need to:
18	- Monitor model performance in production
19	- Track data drift and feature drift
20	- Detect prediction anomalies
21	- Monitor prediction accuracy over time
22	- Set up alerts for model degradation
23	- Analyze model health metrics
24	- Compare production performance to training performance
25
26	## Key capabilities
27
28	### 1. Performance Monitoring
29
30	- Track prediction accuracy and metrics over time
31	- Compare production metrics to training metrics
32	- Monitor prediction volume and latency
33	- Identify performance degradation trends
34
35	### 2. Data Drift Detection
36
37	- Detect changes in feature distributions
38	- Identify feature drift (statistical changes)
39	- Monitor target drift (if actuals available)
40	- Alert on significant drift events
41
42	### 3. Prediction Monitoring
43
44	- Monitor prediction distributions
45	- Detect prediction anomalies
46	- Track prediction confidence scores
47	- Identify unusual prediction patterns
48
49	### 4. Health Management
50
51	- Assess overall model health
52	- Generate monitoring reports
53	- Set up automated alerts
54	- Manage model retraining triggers
55
56	## Workflow examples
57
58	### Example 1: Check model health and drift
59
60	User request: "Check the health of deployment abc123 and report any data drift issues."
61
62	Agent workflow:
63	1. Get deployment monitoring status
64	2. Retrieve recent performance metrics
65	3. Check for data drift in key features
66	4. Compare current metrics to baseline (training)
67	5. Identify any significant drift or degradation
68	6. Report findings with recommendations
69
70	### Example 2: Set up drift monitoring alerts
71
72	User request: "Set up alerts for deployment xyz789 to notify when feature drift exceeds 0.2."
73
74	Agent workflow:
75	1. Get deployment configuration
76	2. Configure drift threshold (0.2)
77	3. Set up alert notifications
78	4. Specify which features to monitor
79	5. Test alert configuration
80	6. Confirm monitoring is active
81
82	## Using DataRobot SDK
83
84	This skill guides you to use the DataRobot Python SDK directly. Install the SDK if needed:
85
86	```bash
87	pip install datarobot
88	```
89
90	### Key SDK Operations
91
92	Use these DataRobot SDK and MLOps API methods for monitoring:
93
94	Deployment Monitoring:
95	- `deployment.get_service_stats(...)` - Get service statistics (latency, volume, etc.)
96	- `deployment.get_feature_drift(...)` - Get feature drift metrics (returns `FeatureDrift` objects)
97	- `deployment.get_target_drift(...)` - Get target drift metrics (returns `TargetDrift`)
98	- `deployment.get_prediction_results(...)` - Retrieve recorded prediction results (if enabled)
99
100	Model Performance:
101	- `model.get_metrics()` - Get model performance metrics
102	- `model.get_roc_curve()` - Get ROC curve for comparison
103
104	Note: Some monitoring features may require DataRobot MLOps API. See the [Common Patterns](#common-patterns) section below for examples.
105
106	## Best practices
107
108	1. Regular monitoring: Check model health regularly, not just when issues arise
109	2. Baseline comparison: Always compare production metrics to training baseline
110	3. Drift thresholds: Set appropriate drift thresholds based on your domain
111	4. Key features: Focus monitoring on high-importance features
112	5. Automated alerts: Set up alerts for critical issues
113	6. Historical analysis: Track trends over time, not just point-in-time metrics
114
115	## Common patterns
116
117	### Pattern 1: Health check
118	```python
119	import datarobot as dr
120	import os
121
122	# Initialize client
123	client = dr.Client(
124	token=os.getenv("DATAROBOT_API_TOKEN"),
125	endpoint=os.getenv("DATAROBOT_ENDPOINT")
126	)
127
128	# Get deployment
129	deployment = dr.Deployment.get("abc123")
130
131	# Get service stats (requires MLOps monitoring to be enabled)
132	stats = deployment.get_service_stats()
133	print(f"Prediction count: {stats.prediction_count}")
134	print(f"Mean response time (ms): {stats.mean_response_time}")
135
136	# Get recorded prediction results (if available / enabled)
137	try:
138	recent = deployment.get_prediction_results(limit=10)
139	print(f"Recent prediction results: {len(recent)}")
140	except Exception as e:
141	print(f"Prediction results not available: {e}")
142	```
143
144	### Pattern 2: Drift detection
145	```py

People who liked this also viewed

6 picks

Type