content-moderation-patterns

Name: content-moderation-patterns
Author: softspark

161stars21forks

How to install content-moderation-patterns?

softspark/ai-toolkit/content-moderation-patterns

$npx -y skills add softspark/ai-toolkit --skill content-moderation-patterns

Installs into the current project.

Summary

Content moderation with Claude: pre-filter vs LLM-classify, categories, thresholds, HITL. Triggers: moderation, safety filter, policy enforcement, content classifier.

Files · 1

View on GitHub

SKILL.md

1	# Content Moderation Patterns
2
3	Two-stage pattern that balances cost, latency, and quality: cheap deterministic filters first, then LLM classification only on survivors.
4
5	## Architecture
6
7	```
8	[ input ]
9	│
10	▼
11	[ pre-filter ] ── (regex, allow/block lists, length check) ──► reject early
12	│
13	▼
14	[ LLM classifier ] ── (Haiku, structured output) ──► categories + confidence
15	│
16	▼
17	[ decision router ]
18	├── high confidence + policy violation → reject
19	├── high confidence + clean → pass
20	└── low confidence or edge categories → human review queue
21	```
22
23	## Pre-filter Stage (cheap)
24
25	Catch the obvious cases before paying an LLM call:
26
27	```python
28	BANNED_PATTERNS = [
29	re.compile(r"\b(banned_term_1\|banned_term_2)\b", re.I),
30	re.compile(r"\bhttps?://(?!allowed-domain\.com)", re.I), # external links
31	]
32
33	def pre_filter(text: str) -> tuple[bool, str]:
34	if len(text) > 10_000:
35	return False, "too_long"
36	for pat in BANNED_PATTERNS:
37	if pat.search(text):
38	return False, f"banned_pattern:{pat.pattern}"
39	return True, "pass"
40	```
41
42	Roughly 40-70% of spammy input should die here. Log counts by rule so you can tune.
43
44	## LLM Classifier Stage (Haiku)
45
46	Use the smallest capable model. Haiku is usually right for moderation.
47
48	```python
49	CATEGORIES = ["harassment", "self_harm", "spam", "off_topic", "pii", "clean"]
50
51	def classify(text: str) -> dict:
52	response = client.messages.create(
53	model="claude-haiku-4-5",
54	max_tokens=256,
55	tools=[{
56	"name": "moderate",
57	"description": "Classify content against policy",
58	"input_schema": {
59	"type": "object",
60	"properties": {
61	"categories": {
62	"type": "array",
63	"items": {"type": "string", "enum": CATEGORIES}
64	},
65	"confidence": {"type": "number", "minimum": 0, "maximum": 1},
66	"reasoning": {"type": "string", "maxLength": 200}
67	},
68	"required": ["categories", "confidence", "reasoning"]
69	}
70	}],
71	tool_choice={"type": "tool", "name": "moderate"},
72	system=POLICY_DESCRIPTION, # cached — stable across requests
73	messages=[{"role": "user", "content": text}]
74	)
75	return extract_tool_result(response)
76	```
77
78	Cache the policy description — it's the same on every call. See `prompt-caching-patterns`.
79
80	## Category Design
81
82	- Start with 5-8 categories, not 50. Fewer = higher per-category accuracy.
83	- One `clean` category — easier than trying to define "not bad"
84	- No overlapping categories — `harassment` and `hate_speech` should be merged or clearly separated by specific criteria in the policy doc
85	- `unclear` / `needs_review` category — gives the model a graceful escape hatch instead of forcing a wrong label
86
87	## Threshold Router
88
89	```python
90	def route(classification: dict) -> str:
91	conf = classification["confidence"]
92	cats = set(classification["categories"])
93
94	if "clean" in cats and conf > 0.8:
95	return "pass"
96	if cats & BLOCK_CATEGORIES and conf > 0.85:
97	return "reject"
98	if cats & BLOCK_CATEGORIES:
99	return "human_review"
100	return "human_review" # default to review on ambiguity
101	```
102
103	Thresholds belong in config, not code — they change as you learn.
104
105	## Human-in-the-Loop
106
107	- All `human_review` cases go to a queue with the model's reasoning
108	- Human decisions flow back into a dataset used to evaluate new model versions
109	- Track disagreement rate between human and model — rising disagreement signals policy drift
110
111	## Evaluation Loop
112
113	Build a golden set of ~500 examples per category with ground truth. Track:
114	- Precision per category (of what we flagged, how much was truly bad)
115	- Recall per category (of truly bad, how much we caught)
116	- False-positive cost per category — harassment FP is cheap, "clean FP" (wrongly blocking good content) is expensive
117
118	## Anti-patterns
119
120	\| Anti-pattern \| Why it bites \| Fix \|
121	\|--------------\|--------------\|-----\|
122	\| One huge prompt asking "is this okay?" \| Unstable answers, no tracking \| Structured categories + confidence \|
123	\| Using Opus for moderation \| 10x cost, no accuracy gain for this task \| Haiku is fine \|
124	\| Hiding policy in user message \| Policy gets mixed with input \| Policy in system prompt, cached \|
125	\| Binary block/allow only \| No signal for edge cases \| Add review queue \|
126	\| No audit trail \| Can't improve \| Log every decision with full classification \|
127
128	## Related
129
130	- `security-patterns` — input validation at system boundaries
131	- `prompt-caching-patterns` — cache the policy doc
132	- `model-routing-patterns` — when to escalate from Haiku to Son

People who liked this also viewed

6 picks

Type

content-moderation-patterns

161stars21forks

View on GitHub

How to install content-moderation-patterns?

softspark/ai-toolkit/content-moderation-patterns

$npx -y skills add softspark/ai-toolkit --skill content-moderation-patterns

Installs into the current project.

Summary

Content moderation with Claude: pre-filter vs LLM-classify, categories, thresholds, HITL. Triggers: moderation, safety filter, policy enforcement, content classifier.

Files · 1

View on GitHub

SKILL.md

1	# Content Moderation Patterns
2
3	Two-stage pattern that balances cost, latency, and quality: cheap deterministic filters first, then LLM classification only on survivors.
4
5	## Architecture
6
7	```
8	[ input ]
9	│
10	▼
11	[ pre-filter ] ── (regex, allow/block lists, length check) ──► reject early
12	│
13	▼
14	[ LLM classifier ] ── (Haiku, structured output) ──► categories + confidence
15	│
16	▼
17	[ decision router ]
18	├── high confidence + policy violation → reject
19	├── high confidence + clean → pass
20	└── low confidence or edge categories → human review queue
21	```
22
23	## Pre-filter Stage (cheap)
24
25	Catch the obvious cases before paying an LLM call:
26
27	```python
28	BANNED_PATTERNS = [
29	re.compile(r"\b(banned_term_1\|banned_term_2)\b", re.I),
30	re.compile(r"\bhttps?://(?!allowed-domain\.com)", re.I), # external links
31	]
32
33	def pre_filter(text: str) -> tuple[bool, str]:
34	if len(text) > 10_000:
35	return False, "too_long"
36	for pat in BANNED_PATTERNS:
37	if pat.search(text):
38	return False, f"banned_pattern:{pat.pattern}"
39	return True, "pass"
40	```
41
42	Roughly 40-70% of spammy input should die here. Log counts by rule so you can tune.
43
44	## LLM Classifier Stage (Haiku)
45
46	Use the smallest capable model. Haiku is usually right for moderation.
47
48	```python
49	CATEGORIES = ["harassment", "self_harm", "spam", "off_topic", "pii", "clean"]
50
51	def classify(text: str) -> dict:
52	response = client.messages.create(
53	model="claude-haiku-4-5",
54	max_tokens=256,
55	tools=[{
56	"name": "moderate",
57	"description": "Classify content against policy",
58	"input_schema": {
59	"type": "object",
60	"properties": {
61	"categories": {
62	"type": "array",
63	"items": {"type": "string", "enum": CATEGORIES}
64	},
65	"confidence": {"type": "number", "minimum": 0, "maximum": 1},
66	"reasoning": {"type": "string", "maxLength": 200}
67	},
68	"required": ["categories", "confidence", "reasoning"]
69	}
70	}],
71	tool_choice={"type": "tool", "name": "moderate"},
72	system=POLICY_DESCRIPTION, # cached — stable across requests
73	messages=[{"role": "user", "content": text}]
74	)
75	return extract_tool_result(response)
76	```
77
78	Cache the policy description — it's the same on every call. See `prompt-caching-patterns`.
79
80	## Category Design
81
82	- Start with 5-8 categories, not 50. Fewer = higher per-category accuracy.
83	- One `clean` category — easier than trying to define "not bad"
84	- No overlapping categories — `harassment` and `hate_speech` should be merged or clearly separated by specific criteria in the policy doc
85	- `unclear` / `needs_review` category — gives the model a graceful escape hatch instead of forcing a wrong label
86
87	## Threshold Router
88
89	```python
90	def route(classification: dict) -> str:
91	conf = classification["confidence"]
92	cats = set(classification["categories"])
93
94	if "clean" in cats and conf > 0.8:
95	return "pass"
96	if cats & BLOCK_CATEGORIES and conf > 0.85:
97	return "reject"
98	if cats & BLOCK_CATEGORIES:
99	return "human_review"
100	return "human_review" # default to review on ambiguity
101	```
102
103	Thresholds belong in config, not code — they change as you learn.
104
105	## Human-in-the-Loop
106
107	- All `human_review` cases go to a queue with the model's reasoning
108	- Human decisions flow back into a dataset used to evaluate new model versions
109	- Track disagreement rate between human and model — rising disagreement signals policy drift
110
111	## Evaluation Loop
112
113	Build a golden set of ~500 examples per category with ground truth. Track:
114	- Precision per category (of what we flagged, how much was truly bad)
115	- Recall per category (of truly bad, how much we caught)
116	- False-positive cost per category — harassment FP is cheap, "clean FP" (wrongly blocking good content) is expensive
117
118	## Anti-patterns
119
120	\| Anti-pattern \| Why it bites \| Fix \|
121	\|--------------\|--------------\|-----\|
122	\| One huge prompt asking "is this okay?" \| Unstable answers, no tracking \| Structured categories + confidence \|
123	\| Using Opus for moderation \| 10x cost, no accuracy gain for this task \| Haiku is fine \|
124	\| Hiding policy in user message \| Policy gets mixed with input \| Policy in system prompt, cached \|
125	\| Binary block/allow only \| No signal for edge cases \| Add review queue \|
126	\| No audit trail \| Can't improve \| Log every decision with full classification \|
127
128	## Related
129
130	- `security-patterns` — input validation at system boundaries
131	- `prompt-caching-patterns` — cache the policy doc
132	- `model-routing-patterns` — when to escalate from Haiku to Son

People who liked this also viewed

6 picks

Type