contextjet-ai/awesome-llm-observability
26 skills
$npx skills add contextjet-ai/awesome-llm-observabilitySkillInstalls
add-llm-evalsUse this when adding evaluation to an LLM/agent app - measuring output quality (correctness, faithfulness, relevance, safety) rather than just watching traces.add-llm-fallbacksUse this to make an LLM app resilient to provider failures, rate limits, timeouts, and outages.add-llm-guardrailsUse this to add safety and security guardrails to an LLM/agent app - blocking prompt injection, PII leakage, jailbreaks, toxic output, off-topic responses, or…annotate-traces-for-reviewUse this to set up human review and annotation of LLM traces, so people (often domain experts) can label outputs, do error analysis, and build a trustworthy…build-eval-datasetUse this to build a good evaluation dataset for an LLM app, the part everyone underestimates.check-answer-consistencyUse this to get a cheap, reference-free signal that an LLM answer might be made up, by sampling the same prompt a few times and measuring agreement.choose-observability-stackUse this to recommend an LLM observability / evaluation tool or stack for a specific situation.collect-user-feedbackUse this to capture user feedback on LLM outputs (thumbs up/down, edits, corrections, implicit signals) and feed it back into observability and evals.compare-llm-modelsUse this to pick or switch the LLM behind a feature, based on evidence instead of hype or the newest release.debug-agent-from-tracesUse this to diagnose WHY an LLM agent or chain produced a wrong, empty, slow, or expensive result, by reading its observability trace.detect-hallucinationsUse this to detect when an LLM is making things up, so you can flag or block confident-but-wrong answers before users see them.estimate-llm-costUse this to estimate what an LLM call or feature will cost, and to compare models on price, before or after shipping.eval-driven-developmentUse this to build or change an LLM feature the reliable way, by writing evals first and iterating against them, instead of tweaking prompts by vibes.instrument-llm-observabilityUse this when adding tracing/observability to an LLM or AI-agent application - capturing prompts, tool calls, token usage, latency, and cost per step.measure-agent-task-successUse this to measure whether an AI agent actually completed its task end to end, not just whether individual LLM calls looked fine.monitor-rag-qualityUse this to measure and monitor the quality of a RAG (retrieval-augmented generation) pipeline - whether it retrieves the right context and answers faithfully.optimize-promptsUse this to improve a prompt systematically instead of hand-tweaking it by feel.red-team-llm-appUse this to adversarially test an LLM/agent app before attackers do - prompt injection, jailbreaks, data exfiltration, tool misuse, and unsafe output.redact-pii-for-tracingUse this when adding LLM observability to an app that handles sensitive data (finance, healthcare, PII) and you must NOT ship raw prompts/PII to a third-party…reduce-llm-costUse this to cut the cost of an LLM app using observability data.scrub-pii-from-textUse this to strip PII (emails, credit cards, SSNs, phone numbers, IPs, IBANs) out of text before it is logged to a tracing backend or sent to a third-party…set-up-ab-testingUse this to test an LLM change (new prompt, new model, new retrieval) on real traffic before rolling it out to everyone.set-up-drift-alertsUse this to catch an LLM app silently getting worse in production - quality dropping, cost creeping up, inputs shifting away from what you tested.trace-based-testingUse this to turn real production traces into a regression test suite, so bugs you saw once never come back.trace-multi-agent-systemUse this to add observability to a multi-agent or agentic system (multiple agents, sub-agents, tool loops, handoffs).validate-genai-spansUse this to check that your LLM tracing actually emits complete, spec-compliant spans, so cost/latency/model dashboards downstream are not full of holes.