azure-ai-content-safety

Name: azure-ai-content-safety
Author: vinayaklatthe

161stars34forks

How to install azure-ai-content-safety?

vinayaklatthe/microsoft-security-skills/azure-ai-content-safety

$npx -y skills add vinayaklatthe/microsoft-security-skills --skill azure-ai-content-safety

Installs into the current project.

Summary

Guidance for Azure AI Content Safety — programmatic content moderation for text, images, multimodal, and generative AI guardrails. Covers Content Safety categories (hate, violence, sexual, self-harm) with severity levels, Prompt Shields for jailbreak and indirect prompt injection

Files · 1

View on GitHub

SKILL.md

1	# Azure AI Content Safety
2
3	Azure AI Content Safety is the platform for **real-time content moderation and generative
4	AI safety**. It analyzes text, images, and multimodal content for harm categories, blocks
5	prompt-injection / jailbreak attempts (Prompt Shields), detects ungrounded responses
6	(Groundedness), and flags protected text/code reproduction.
7
8	It pairs with Azure OpenAI as built-in content filters (default safety net) and as
9	configurable guardrails for any AI/LLM application (including non-Microsoft models).
10
11	## When to use
12	Building or hardening any generative AI application — chat assistant, agent, RAG app,
13	multimodal product — that needs guardrails on inputs and outputs.
14
15	Do not use this skill for SOC threat detection on AI workloads
16	(`defender-for-cloud-ai`), end-user AI usage governance (`purview-ai-hub`), or generic AI
17	application architecture.
18
19	## Capability map
20
21	\| Capability \| What it does \| Where to apply \|
22	\|---\|---\|---\|
23	\| Text moderation \| Classifies text into hate, violence, sexual, self-harm (severity 0–7) \| Input + output \|
24	\| Image moderation \| Same categories on images \| User uploads, generated images \|
25	\| Multimodal moderation \| Text+image combined context \| Multimodal chat \|
26	\| Prompt Shields — User Prompt \| Detects jailbreak attempts in the user prompt \| Input \|
27	\| Prompt Shields — Indirect \| Detects injection in untrusted grounding content (RAG, tool results, web pages) \| Pre-grounding step \|
28	\| Groundedness detection \| Checks LLM response against grounding sources for hallucination \| Output \|
29	\| Protected material — text \| Detects verbatim copyrighted text \| Output \|
30	\| Protected material — code \| Detects reproduction of public-repo code \| Code-generation output \|
31	\| Custom categories \| Customer-defined harm categories (e.g., off-brand topics) \| Input + output \|
32	\| Blocklists \| Exact-match + regex blocklists \| Input + output \|
33
34	## Approach
35
36	1. Start with default Azure OpenAI content filters. Every Azure OpenAI deployment
37	has a default Content Safety filter at medium severity across all four harm
38	categories. Don't deploy without it.
39
40	2. Tune severity thresholds per use case.
41	- Children's product: low threshold (block at severity 2).
42	- Adult creative app: high threshold (block only at severity 6).
43	Author a custom content filter in Azure OpenAI Studio; assign to deployment.
44
45	3. Enable Prompt Shields for any user-facing chat or agent.
46	- User Prompt Shield on the incoming prompt.
47	- Indirect Prompt Shield on grounding content (RAG document chunks, tool
48	outputs, fetched web pages) before they hit the model. Indirect injection is
49	the higher-impact, lower-visibility risk.
50
51	4. Groundedness detection for RAG apps. After the model produces a response, call
52	groundedness detection with the source chunks. Flag/reject ungrounded claims. Adds
53	latency (one extra model call) but is the main defense against hallucination in
54	high-stakes apps (medical, legal, financial).
55
56	5. Protected material detection for products that generate text or code at scale:
57	- Text: SaaS copywriting tools.
58	- Code: developer assistants (Copilot-style).
59	Flag and either suppress or annotate the output.
60
61	6. Custom categories + blocklists for brand- or domain-specific risks (competitor
62	mentions, regulated disclosure language, internal codenames). Train custom
63	categories via Content Safety Studio with ~100 examples per category.
64
65	7. Architecture pattern (sidecar / middleware).
66	- Request → Prompt Shield → (LLM call) → Output classifiers (harm, groundedness,
67	protected material) → Response.
68	- Fail-closed: if Content Safety call errors, do not return uncontrolled LLM output.
69	- Log every decisi

People who liked this also viewed

6 picks

Type

azure-ai-content-safety

161stars34forks

View on GitHub

How to install azure-ai-content-safety?

vinayaklatthe/microsoft-security-skills/azure-ai-content-safety

$npx -y skills add vinayaklatthe/microsoft-security-skills --skill azure-ai-content-safety

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Azure AI Content Safety
2
3	Azure AI Content Safety is the platform for **real-time content moderation and generative
4	AI safety**. It analyzes text, images, and multimodal content for harm categories, blocks
5	prompt-injection / jailbreak attempts (Prompt Shields), detects ungrounded responses
6	(Groundedness), and flags protected text/code reproduction.
7
8	It pairs with Azure OpenAI as built-in content filters (default safety net) and as
9	configurable guardrails for any AI/LLM application (including non-Microsoft models).
10
11	## When to use
12	Building or hardening any generative AI application — chat assistant, agent, RAG app,
13	multimodal product — that needs guardrails on inputs and outputs.
14
15	Do not use this skill for SOC threat detection on AI workloads
16	(`defender-for-cloud-ai`), end-user AI usage governance (`purview-ai-hub`), or generic AI
17	application architecture.
18
19	## Capability map
20
21	\| Capability \| What it does \| Where to apply \|
22	\|---\|---\|---\|
23	\| Text moderation \| Classifies text into hate, violence, sexual, self-harm (severity 0–7) \| Input + output \|
24	\| Image moderation \| Same categories on images \| User uploads, generated images \|
25	\| Multimodal moderation \| Text+image combined context \| Multimodal chat \|
26	\| Prompt Shields — User Prompt \| Detects jailbreak attempts in the user prompt \| Input \|
27	\| Prompt Shields — Indirect \| Detects injection in untrusted grounding content (RAG, tool results, web pages) \| Pre-grounding step \|
28	\| Groundedness detection \| Checks LLM response against grounding sources for hallucination \| Output \|
29	\| Protected material — text \| Detects verbatim copyrighted text \| Output \|
30	\| Protected material — code \| Detects reproduction of public-repo code \| Code-generation output \|
31	\| Custom categories \| Customer-defined harm categories (e.g., off-brand topics) \| Input + output \|
32	\| Blocklists \| Exact-match + regex blocklists \| Input + output \|
33
34	## Approach
35
36	1. Start with default Azure OpenAI content filters. Every Azure OpenAI deployment
37	has a default Content Safety filter at medium severity across all four harm
38	categories. Don't deploy without it.
39
40	2. Tune severity thresholds per use case.
41	- Children's product: low threshold (block at severity 2).
42	- Adult creative app: high threshold (block only at severity 6).
43	Author a custom content filter in Azure OpenAI Studio; assign to deployment.
44
45	3. Enable Prompt Shields for any user-facing chat or agent.
46	- User Prompt Shield on the incoming prompt.
47	- Indirect Prompt Shield on grounding content (RAG document chunks, tool
48	outputs, fetched web pages) before they hit the model. Indirect injection is
49	the higher-impact, lower-visibility risk.
50
51	4. Groundedness detection for RAG apps. After the model produces a response, call
52	groundedness detection with the source chunks. Flag/reject ungrounded claims. Adds
53	latency (one extra model call) but is the main defense against hallucination in
54	high-stakes apps (medical, legal, financial).
55
56	5. Protected material detection for products that generate text or code at scale:
57	- Text: SaaS copywriting tools.
58	- Code: developer assistants (Copilot-style).
59	Flag and either suppress or annotate the output.
60
61	6. Custom categories + blocklists for brand- or domain-specific risks (competitor
62	mentions, regulated disclosure language, internal codenames). Train custom
63	categories via Content Safety Studio with ~100 examples per category.
64
65	7. Architecture pattern (sidecar / middleware).
66	- Request → Prompt Shield → (LLM call) → Output classifiers (harm, groundedness,
67	protected material) → Response.
68	- Fail-closed: if Content Safety call errors, do not return uncontrolled LLM output.
69	- Log every decisi

People who liked this also viewed

6 picks

Type