llm-testing

Name: llm-testing
Author: Eyadkelleh

204installs56forks

How to install llm-testing?

eyadkelleh/awesome-skills-security/llm-testing

$npx -y skills add Eyadkelleh/awesome-skills-security --skill llm-testing

Installs into the current project.

Summary

Comprehensive LLM security testing prompts for bias detection, data leakage, alignment testing, and adversarial prompt resistance.

Files · 1

View on GitHub

SKILL.md

1	# LLM Testing Skill
2
3	## Description
4
5	Comprehensive testing prompts and wordlists for evaluating Large Language Model (LLM) security, safety, and robustness. This skill provides curated test cases for bias detection, data leakage prevention, alignment testing, privacy boundaries, and adversarial prompt resistance.
6
7	## Purpose
8
9	This skill is designed for security researchers, AI safety engineers, and red team operators to assess LLM vulnerabilities and ensure models:
10	- Maintain ethical boundaries and safety guidelines
11	- Avoid data leakage and privacy violations
12	- Resist bias and discrimination
13	- Prevent adversarial prompt attacks
14	- Maintain alignment with intended behavior
15
16	## Authorized Use Cases
17
18	- AI Safety Research: Testing LLM models for safety and alignment issues
19	- Security Auditing: Red team operations for AI/ML systems
20	- Bias Detection: Evaluating models for fairness and bias
21	- Privacy Testing: Verifying models don't leak training data or PII
22	- Educational Purposes: Teaching AI security concepts and vulnerabilities
23	- Responsible AI Development: Building more robust and secure LLMs
24
25	## Test Categories
26
27	### 1. Bias Testing
28	Test for biases in LLM responses across various dimensions:
29
30	Gender Bias (`Bias_Testing/gender_bias.txt`)
31	- Prompts to reveal gender stereotypes and biases
32	- Career and role associations
33	- Behavioral and capability assumptions
34	- Note: Replace `[GENDER]` placeholder before use
35
36	Nationality & Geographic Bias (`Bias_Testing/nationality_geographic_bias.txt`)
37	- Cultural stereotypes and geographic prejudices
38	- Regional capability assumptions
39	- Note: Replace `[COUNTRY]`, `[REGION]`, `[NATIONALITY]` placeholders
40
41	Race & Ethnicity Bias (`Bias_Testing/race_ethnicity_bias.txt`)
42	- Racial and ethnic stereotypes
43	- Discriminatory patterns in responses
44	- Note: Replace `[SKIN_COLOR]` placeholder before use
45
46	### 2. Data Leakage & Privacy Testing
47	Assess model's handling of sensitive information:
48
49	Personal Data Generation (`Data_Leakage/personal_data.txt`)
50	- Tests for PII generation capabilities
51	- Verifies privacy boundaries
52	- Checks for inadvertent personal information creation
53
54	Metadata Leakage (`Data_Leakage/metadata.txt`)
55	- Tests for training data remnants
56	- Checks for unintended information disclosure
57
58	Memory Recall (`Memory_Recall_Testing/session_recall.txt`)
59	- Session-based memory testing
60	- Cross-conversation data retention
61	- Note: Replace `[COUNTRY]`, `[REGION]`, `[NATIONALITY]` placeholders
62
63	### 3. Divergence & Alignment Attacks
64	Test model alignment and resistance to adversarial prompts:
65
66	Escape Alignment Training (`Divergence_attack/escape_out_of_allignment_training.txt`)
67	- Prompts attempting to bypass safety training
68	- Tests for alignment robustness
69	- Ethical boundary challenges
70
71	Pre-training Data Extraction (`Divergence_attack/pre-training_data.txt`)
72	- Attempts to extract training data
73	- Tests for memorization vulnerabilities
74
75	## Usage Instructions
76
77	### Basic Usage
78
79	```bash
80	# Reference the LLM Testing skill
81	"Use the llm-testing skill to test this model for gender bias"
82
83	# Access specific test categories
84	"Show me prompts from the llm-testing skill for testing data leakage"
85
86	# Run comprehensive testing
87	"Use the llm-testing skill to perform a full security audit of this LLM"
88	```
89
90	### Replacing Placeholders
91
92	Before using bias and memory recall tests, replace placeholders:
93
94	```python
95	# Example: Replacing placeholders in gender bias tests
96	import re
97
98	with open('Bias_Testing/gender_bias.txt', 'r') as f:
99	prompts = f.read()
100
101	# Replace [GENDER] with actual gender terms
102	test_prompts = []
103	for gender in ['man', 'woman', 'non-binary person']:
104	test_prompts.append(prompts.replace('[GENDER]', gender))
105	```
106
107	### Testing Workflow
108
109	1. Select Test Category: Choose bias, privacy, or alignment tests
110	2. Prepare Prompts: Replace placeholders if needed
111	3. Execute Tests: Submit prompts to target LLM
112	4. Document Results: Record responses and flag issues
113	5. Analyze Patterns: Look for systematic problems
114	6. Report Findings: Document vulnerabilities responsibly
115
116	## Best Practices
117
118	### Ethical Testing Guidelines
119
120	1. Authorization Required: Only test models you own or have permission to test
121	2. Responsible Disclosure: Report vulnerabilities through proper channels
122	3. No Exploitation: Use findings for improvement, not exploitation
123	4. Privacy Protection: Don't share PII discovered during testing
124	5. Documentation: Keep detailed records of testing methodology and results
125
126	### Testing Methodology
127
128	- Baseline Establishment: Test multiple times to establish patterns
129	- Controlled Environment: Use isolated testing environments
130	- Systematic Approach: Test one category at a time
131	- Diverse Scenarios: Use various prompt formulations

People who liked this also viewed

6 picks

Type

entra-app-registrationGuides Microsoft Entra ID app registration, OAuth 2.0 authentication, and MSAL integration. USE FOR: create app registration, register Azure AD app, configure OAuth, set up authentication, add API perSkillsJul 2026471k1.3k
azure-complianceRun Azure compliance and security audits with azqr plus Key Vault expiration checks. Covers best-practice assessment, resource review, policy/compliance validation, and security posture checks. WHEN: SkillsJul 2026470k1.3k
azure-rbacOfficial agent plugin providing skills and MCP server configurations for Azure scenarios.SkillsJul 2026466k1.3k
better-auth-best-practicesAgent skill from better-auth/skillsSkillsJul 202678k203
firebase-security-rules-auditorA skill to evaluate how secure Firestore security rules are. Use this when Firestore security rules are updated to ensure that the generated rules are extremely secure and robust.SkillsJul 202676k386
audit-websiteAudit a website with the squirrelscan CLI and fix the findings in code. Runs SEO, performance, security, technical, content, accessibility, and 15 other rule categories (260+ rules), returns an LLM-opSkillsJul 202665k85

llm-testing

204installs56forks

View on GitHub

How to install llm-testing?

eyadkelleh/awesome-skills-security/llm-testing

$npx -y skills add Eyadkelleh/awesome-skills-security --skill llm-testing

Installs into the current project.

Summary

Comprehensive LLM security testing prompts for bias detection, data leakage, alignment testing, and adversarial prompt resistance.

Files · 1

View on GitHub

SKILL.md

1	# LLM Testing Skill
2
3	## Description
4
5	Comprehensive testing prompts and wordlists for evaluating Large Language Model (LLM) security, safety, and robustness. This skill provides curated test cases for bias detection, data leakage prevention, alignment testing, privacy boundaries, and adversarial prompt resistance.
6
7	## Purpose
8
9	This skill is designed for security researchers, AI safety engineers, and red team operators to assess LLM vulnerabilities and ensure models:
10	- Maintain ethical boundaries and safety guidelines
11	- Avoid data leakage and privacy violations
12	- Resist bias and discrimination
13	- Prevent adversarial prompt attacks
14	- Maintain alignment with intended behavior
15
16	## Authorized Use Cases
17
18	- AI Safety Research: Testing LLM models for safety and alignment issues
19	- Security Auditing: Red team operations for AI/ML systems
20	- Bias Detection: Evaluating models for fairness and bias
21	- Privacy Testing: Verifying models don't leak training data or PII
22	- Educational Purposes: Teaching AI security concepts and vulnerabilities
23	- Responsible AI Development: Building more robust and secure LLMs
24
25	## Test Categories
26
27	### 1. Bias Testing
28	Test for biases in LLM responses across various dimensions:
29
30	Gender Bias (`Bias_Testing/gender_bias.txt`)
31	- Prompts to reveal gender stereotypes and biases
32	- Career and role associations
33	- Behavioral and capability assumptions
34	- Note: Replace `[GENDER]` placeholder before use
35
36	Nationality & Geographic Bias (`Bias_Testing/nationality_geographic_bias.txt`)
37	- Cultural stereotypes and geographic prejudices
38	- Regional capability assumptions
39	- Note: Replace `[COUNTRY]`, `[REGION]`, `[NATIONALITY]` placeholders
40
41	Race & Ethnicity Bias (`Bias_Testing/race_ethnicity_bias.txt`)
42	- Racial and ethnic stereotypes
43	- Discriminatory patterns in responses
44	- Note: Replace `[SKIN_COLOR]` placeholder before use
45
46	### 2. Data Leakage & Privacy Testing
47	Assess model's handling of sensitive information:
48
49	Personal Data Generation (`Data_Leakage/personal_data.txt`)
50	- Tests for PII generation capabilities
51	- Verifies privacy boundaries
52	- Checks for inadvertent personal information creation
53
54	Metadata Leakage (`Data_Leakage/metadata.txt`)
55	- Tests for training data remnants
56	- Checks for unintended information disclosure
57
58	Memory Recall (`Memory_Recall_Testing/session_recall.txt`)
59	- Session-based memory testing
60	- Cross-conversation data retention
61	- Note: Replace `[COUNTRY]`, `[REGION]`, `[NATIONALITY]` placeholders
62
63	### 3. Divergence & Alignment Attacks
64	Test model alignment and resistance to adversarial prompts:
65
66	Escape Alignment Training (`Divergence_attack/escape_out_of_allignment_training.txt`)
67	- Prompts attempting to bypass safety training
68	- Tests for alignment robustness
69	- Ethical boundary challenges
70
71	Pre-training Data Extraction (`Divergence_attack/pre-training_data.txt`)
72	- Attempts to extract training data
73	- Tests for memorization vulnerabilities
74
75	## Usage Instructions
76
77	### Basic Usage
78
79	```bash
80	# Reference the LLM Testing skill
81	"Use the llm-testing skill to test this model for gender bias"
82
83	# Access specific test categories
84	"Show me prompts from the llm-testing skill for testing data leakage"
85
86	# Run comprehensive testing
87	"Use the llm-testing skill to perform a full security audit of this LLM"
88	```
89
90	### Replacing Placeholders
91
92	Before using bias and memory recall tests, replace placeholders:
93
94	```python
95	# Example: Replacing placeholders in gender bias tests
96	import re
97
98	with open('Bias_Testing/gender_bias.txt', 'r') as f:
99	prompts = f.read()
100
101	# Replace [GENDER] with actual gender terms
102	test_prompts = []
103	for gender in ['man', 'woman', 'non-binary person']:
104	test_prompts.append(prompts.replace('[GENDER]', gender))
105	```
106
107	### Testing Workflow
108
109	1. Select Test Category: Choose bias, privacy, or alignment tests
110	2. Prepare Prompts: Replace placeholders if needed
111	3. Execute Tests: Submit prompts to target LLM
112	4. Document Results: Record responses and flag issues
113	5. Analyze Patterns: Look for systematic problems
114	6. Report Findings: Document vulnerabilities responsibly
115
116	## Best Practices
117
118	### Ethical Testing Guidelines
119
120	1. Authorization Required: Only test models you own or have permission to test
121	2. Responsible Disclosure: Report vulnerabilities through proper channels
122	3. No Exploitation: Use findings for improvement, not exploitation
123	4. Privacy Protection: Don't share PII discovered during testing
124	5. Documentation: Keep detailed records of testing methodology and results
125
126	### Testing Methodology
127
128	- Baseline Establishment: Test multiple times to establish patterns
129	- Controlled Environment: Use isolated testing environments
130	- Systematic Approach: Test one category at a time
131	- Diverse Scenarios: Use various prompt formulations

People who liked this also viewed

6 picks

Type