book-sft-pipeline

Name: book-sft-pipeline
Author: muratcankoylan

25installs1.4kforks

How to install book-sft-pipeline?

muratcankoylan/agent-skills-for-context-engineering/book-sft-pipeline

$npx -y skills add muratcankoylan/Agent-Skills-for-Context-Engineering --skill book-sft-pipeline

Installs into the current project.

Summary

This skill should be used for book-to-SFT pipelines: ePub extraction, literary segmentation, author-voice dataset construction, style-transfer training, LoRA workflows, and model evaluation for voice replication.

Files · 1

View on GitHub

SKILL.md

1	# Book SFT Pipeline
2
3	A complete system for converting books into SFT datasets and training style-transfer models. This skill teaches the pipeline from raw ePub to a model that writes in any author's voice.
4
5	## When to Activate
6
7	Activate this skill when:
8	- Building fine-tuning datasets from literary works
9	- Creating author-voice or style-transfer models
10	- Preparing training data for Tinker or similar SFT platforms
11	- Designing text segmentation pipelines for long-form content
12	- Training small models (8B or less) on limited data
13
14	## Core Concepts
15
16	### The Three Pillars of Book SFT
17
18	1. Intelligent Segmentation
19	Text chunks must be semantically coherent. Breaking mid-sentence teaches the model to produce fragmented output. Target: 150-400 words per chunk, always at natural boundaries.
20
21	2. Diverse Instruction Generation
22	Use multiple prompt templates and system prompts to prevent overfitting. A single prompt style leads to memorization. Use 15+ prompt templates with 5+ system prompts.
23
24	3. Style Over Content
25	The goal is learning the author's rhythm and vocabulary patterns, not memorizing plots. Synthetic instructions describe what happens without quoting the text.
26
27	## Pipeline Architecture
28
29	```
30	┌─────────────────────────────────────────────────────────────────┐
31	│ ORCHESTRATOR AGENT │
32	│ Coordinates pipeline phases, manages state, handles failures │
33	└──────────────────────┬──────────────────────────────────────────┘
34	│
35	┌───────────────┼───────────────┬───────────────┐
36	▼ ▼ ▼ ▼
37	┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
38	│ EXTRACTION │ │ SEGMENTATION │ │ INSTRUCTION │ │ DATASET │
39	│ AGENT │ │ AGENT │ │ AGENT │ │ BUILDER │
40	│ ePub → Text │ │ Text → Chunks│ │ Chunks → │ │ Pairs → │
41	│ │ │ 150-400 words│ │ Prompts │ │ JSONL │
42	└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
43	│
44	┌───────────────┴───────────────┐
45	▼ ▼
46	┌──────────────┐ ┌──────────────┐
47	│ TRAINING │ │ VALIDATION │
48	│ AGENT │ │ AGENT │
49	│ LoRA on │ │ AI detector │
50	│ Tinker │ │ Originality │
51	└──────────────┘ └──────────────┘
52	```
53
54	## Phase 1: Text Extraction
55
56	### Critical Rules
57	1. Always source ePub over PDF - OCR errors become learned patterns
58	2. Use paragraph-level extraction - Extract from `<p>` tags to preserve breaks
59	3. Remove front/back matter - Copyright and TOC pollute the dataset
60
61	```python
62	# Extract text from ePub paragraphs
63	from epub2 import EPub
64	from bs4 import BeautifulSoup
65
66	def extract_epub(path):
67	book = EPub(path)
68	chapters = []
69	for item in book.flow:
70	html = book.get_chapter(item.id)
71	soup = BeautifulSoup(html, 'html.parser')
72	paragraphs = [p.get_text().strip() for p in soup.find_all('p')]
73	chapters.append('\n\n'.join(p for p in paragraphs if p))
74	return '\n\n'.join(chapters)
75	```
76
77	## Phase 2: Intelligent Segmentation
78
79	### Smaller Chunks + Overlap
80
81	Smaller chunks (150-400 words) produce more training examples and better style transfer than larger chunks (250-650).
82
83	```python
84	def segment(text, min_words=150, max_words=400):
85	paragraphs = text.split('\n\n')
86	chunks, buffer, buffer_words = [], [], 0
87
88	for para in paragraphs:
89	words = len(para.split())
90	if buffer_words + words > max_words and buffer_words >= min_words:
91	chunks.append('\n\n'.join(buffer))
92	# Keep last paragraph for overlap
93	buffer = [buffer[-1], para] if buffer else [para]
94	buffer_words = sum(len(p.split()) for p in buffer)
95	else:
96	buffer.append(para)
97	buffer_words += words
98
99	if buffer:
100	chunks.append('\n\n'.join(buffer))
101	return chunks
102	```
103
104	### Expected Results
105
106	For an 86,000-word book:
107	- Old method (250-650 words): ~150 chunks
108	- New method (150-400 + overlap): ~300 chunks
109	- With 2 variants per chunk: 600+ training examples
110
111	## Phase 3: Diverse Instruction Generation
112
113	### The Key Insight
114
115	Using a single prompt template causes memorization. Diverse templates teach the underlying style.
116
117	```python
118	SYSTEM_PROMPTS = [
119	"You are an expert creative writer capable of emulating specific literary styles.",
120	"You are a literary writer with deep knowledge of classic prose styles.",
121	"You are a creative writer skilled at emulating distinctive authorial voices.",
122	"You write prose that captures the essence of modernist literature.

People who liked this also viewed

6 picks

Type

book-sft-pipeline

25installs1.4kforks

View on GitHub

How to install book-sft-pipeline?

muratcankoylan/agent-skills-for-context-engineering/book-sft-pipeline

$npx -y skills add muratcankoylan/Agent-Skills-for-Context-Engineering --skill book-sft-pipeline

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Book SFT Pipeline
2
3	A complete system for converting books into SFT datasets and training style-transfer models. This skill teaches the pipeline from raw ePub to a model that writes in any author's voice.
4
5	## When to Activate
6
7	Activate this skill when:
8	- Building fine-tuning datasets from literary works
9	- Creating author-voice or style-transfer models
10	- Preparing training data for Tinker or similar SFT platforms
11	- Designing text segmentation pipelines for long-form content
12	- Training small models (8B or less) on limited data
13
14	## Core Concepts
15
16	### The Three Pillars of Book SFT
17
18	1. Intelligent Segmentation
19	Text chunks must be semantically coherent. Breaking mid-sentence teaches the model to produce fragmented output. Target: 150-400 words per chunk, always at natural boundaries.
20
21	2. Diverse Instruction Generation
22	Use multiple prompt templates and system prompts to prevent overfitting. A single prompt style leads to memorization. Use 15+ prompt templates with 5+ system prompts.
23
24	3. Style Over Content
25	The goal is learning the author's rhythm and vocabulary patterns, not memorizing plots. Synthetic instructions describe what happens without quoting the text.
26
27	## Pipeline Architecture
28
29	```
30	┌─────────────────────────────────────────────────────────────────┐
31	│ ORCHESTRATOR AGENT │
32	│ Coordinates pipeline phases, manages state, handles failures │
33	└──────────────────────┬──────────────────────────────────────────┘
34	│
35	┌───────────────┼───────────────┬───────────────┐
36	▼ ▼ ▼ ▼
37	┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
38	│ EXTRACTION │ │ SEGMENTATION │ │ INSTRUCTION │ │ DATASET │
39	│ AGENT │ │ AGENT │ │ AGENT │ │ BUILDER │
40	│ ePub → Text │ │ Text → Chunks│ │ Chunks → │ │ Pairs → │
41	│ │ │ 150-400 words│ │ Prompts │ │ JSONL │
42	└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
43	│
44	┌───────────────┴───────────────┐
45	▼ ▼
46	┌──────────────┐ ┌──────────────┐
47	│ TRAINING │ │ VALIDATION │
48	│ AGENT │ │ AGENT │
49	│ LoRA on │ │ AI detector │
50	│ Tinker │ │ Originality │
51	└──────────────┘ └──────────────┘
52	```
53
54	## Phase 1: Text Extraction
55
56	### Critical Rules
57	1. Always source ePub over PDF - OCR errors become learned patterns
58	2. Use paragraph-level extraction - Extract from `<p>` tags to preserve breaks
59	3. Remove front/back matter - Copyright and TOC pollute the dataset
60
61	```python
62	# Extract text from ePub paragraphs
63	from epub2 import EPub
64	from bs4 import BeautifulSoup
65
66	def extract_epub(path):
67	book = EPub(path)
68	chapters = []
69	for item in book.flow:
70	html = book.get_chapter(item.id)
71	soup = BeautifulSoup(html, 'html.parser')
72	paragraphs = [p.get_text().strip() for p in soup.find_all('p')]
73	chapters.append('\n\n'.join(p for p in paragraphs if p))
74	return '\n\n'.join(chapters)
75	```
76
77	## Phase 2: Intelligent Segmentation
78
79	### Smaller Chunks + Overlap
80
81	Smaller chunks (150-400 words) produce more training examples and better style transfer than larger chunks (250-650).
82
83	```python
84	def segment(text, min_words=150, max_words=400):
85	paragraphs = text.split('\n\n')
86	chunks, buffer, buffer_words = [], [], 0
87
88	for para in paragraphs:
89	words = len(para.split())
90	if buffer_words + words > max_words and buffer_words >= min_words:
91	chunks.append('\n\n'.join(buffer))
92	# Keep last paragraph for overlap
93	buffer = [buffer[-1], para] if buffer else [para]
94	buffer_words = sum(len(p.split()) for p in buffer)
95	else:
96	buffer.append(para)
97	buffer_words += words
98
99	if buffer:
100	chunks.append('\n\n'.join(buffer))
101	return chunks
102	```
103
104	### Expected Results
105
106	For an 86,000-word book:
107	- Old method (250-650 words): ~150 chunks
108	- New method (150-400 + overlap): ~300 chunks
109	- With 2 variants per chunk: 600+ training examples
110
111	## Phase 3: Diverse Instruction Generation
112
113	### The Key Insight
114
115	Using a single prompt template causes memorization. Diverse templates teach the underlying style.
116
117	```python
118	SYSTEM_PROMPTS = [
119	"You are an expert creative writer capable of emulating specific literary styles.",
120	"You are a literary writer with deep knowledge of classic prose styles.",
121	"You are a creative writer skilled at emulating distinctive authorial voices.",
122	"You write prose that captures the essence of modernist literature.

People who liked this also viewed

6 picks

Type