$npx -y skills add briiirussell/cybersecurity-skills --skill ai-risk-managementApply the NIST AI Risk Management Framework (AI RMF 1.0) and adjacent guidance to AI / ML systems — model lifecycle governance, fairness and bias evaluation, robustness, transparency, accountability, third-party model risk, monitoring for drift, and AI incident response. Broader
| 1 | # AI Risk Management — Beyond Security, the Whole Model Lifecycle |
| 2 | |
| 3 | `prompt-injection` covers the AI security slice — attackers manipulating LLM inputs. This skill covers everything else risk-related about deploying AI / ML systems: governance, fairness, robustness, transparency, monitoring, incident response specific to AI failures, third-party model risk, and compliance with the emerging AI regulatory landscape. |
| 4 | |
| 5 | The framing is NIST AI RMF 1.0 (released 2023) — the most widely-adopted voluntary framework — plus the regulatory layer (EU AI Act, US executive orders, sector-specific guidance). Use this skill when you are deploying AI features beyond a chatbot wrapper, when a regulator asks "how do you govern your AI," or when something has gone wrong with an AI system in production. |
| 6 | |
| 7 | Cross-references: `prompt-injection` for prompt-injection / LLM-specific security attacks; `threat-modeling` for design-time AI risk modeling; `incident-triage` and `breach-patterns` for AI-related incident response patterns; `csf-mapping` for the broader governance frame that AI RMF sits within. |
| 8 | |
| 9 | ## The NIST AI RMF — four functions |
| 10 | |
| 11 | Just like the cybersecurity framework, the AI RMF organizes the work into functions. Same shape, different content. |
| 12 | |
| 13 | | Function | What it covers | |
| 14 | |---|---| |
| 15 | | **Govern (GOV)** | Policy, accountability, roles, risk appetite, AI principles, board oversight, governance structures | |
| 16 | | **Map (MAP)** | Context — what is the AI system, what does it do, who is impacted, what could go wrong, what are the legal / ethical constraints | |
| 17 | | **Measure (MEAS)** | Evaluate the system — fairness, robustness, accuracy, explainability, privacy, security; quantitative + qualitative metrics | |
| 18 | | **Manage (MAN)** | Treat the risks — mitigations, monitoring, incident response, decommissioning, ongoing review | |
| 19 | |
| 20 | The framework is voluntary but increasingly cited in contracts, RFPs, executive orders, and emerging regulations. Treat it as the lingua franca of AI risk. |
| 21 | |
| 22 | ## Workflow |
| 23 | |
| 24 | ### Step 1 — Inventory AI systems |
| 25 | |
| 26 | Before assessment, build the inventory. Most organizations underestimate how much AI they actually deploy. |
| 27 | |
| 28 | | Category | Examples | |
| 29 | |---|---| |
| 30 | | **First-party trained models** | Recommendation engines, fraud detection, churn prediction, internal ML pipelines | |
| 31 | | **First-party LLM use** | Customer support chat, content generation, summarization, code generation, embeddings for search | |
| 32 | | **Third-party AI features** | Stripe Radar (fraud), GitHub Copilot (code completion), Salesforce Einstein, Notion AI, Linear AI | |
| 33 | | **Embedded AI in products you ship** | Suggested responses, smart defaults, AI sorting / ranking | |
| 34 | | **AI in HR / hiring** | Resume screening, candidate matching, performance evaluation — high regulatory exposure | |
| 35 | | **AI in customer-facing decisions** | Pricing, eligibility, content moderation, ad targeting — high regulatory exposure | |
| 36 | |
| 37 | For each, record: vendor (if any), training data source, deployment context, who it affects, the decision it informs, how decisions are reviewed. |
| 38 | |
| 39 | ### Step 2 — MAP: assess the context per system |
| 40 | |
| 41 | For each AI system in the inventory, answer: |
| 42 | |
| 43 | - **Purpose** — what is this system's stated goal? Does the actual deployment match? |
| 44 | - **Stakeholders** — who interacts with it, who is affected by its decisions, who is in a position to challenge those decisions? |
| 45 | - **Legal / regulatory context** — is this in scope for a specific law? (EU AI Act high-risk categories, US HUD fair-housing rules, EEOC for employment AI, FTC for unfair / deceptive practices, sector laws) |
| 46 | - **Failure modes** — what does "broken" look like? (Wrong answer, biased answer, hallucinated answer, slow answer, expensive answer, refused-to-answer-something-it-should, answered-something-it-should-not) |
| 47 | - **Reversibility** — when this system makes a wrong call, can the decision be undone? (Mortgage denial: hard to undo. Spam filter: easy) |
| 48 | |
| 49 | ### Step 3 — MEASURE: evaluate the system |
| 50 | |
| 51 | The categories of evaluation, with the engineering hooks for each: |
| 52 | |
| 53 | #### Accuracy / performance |
| 54 | |
| 55 | - Test set evaluation — held-out data, not the training data |
| 56 | - Performance on slices of data, not just aggregate |