token-counter

Name: token-counter
Author: girijashankarj

27stars3forks

How to install token-counter?

girijashankarj/cursor-handbook/token-counter

$npx -y skills add girijashankarj/cursor-handbook --skill token-counter

Installs into the current project.

Summary

Estimate LLM token counts for files or directories using a word-based heuristic. Use when the user asks how many tokens a file uses, wants to check context size, or needs to estimate token cost.

Files · 1

View on GitHub

SKILL.md

1	# Skill: Token Counter
2
3	Estimate token consumption for files and directories to help manage LLM context budgets.
4
5	## Trigger
6	When the user asks to count tokens, estimate context size, check how much a file/directory costs in tokens, or wants to optimize token usage.
7
8	## Prerequisites
9	- [ ] Target file or directory exists
10	- [ ] `wc` command available (standard on macOS/Linux)
11
12	## Usage
13
14	Run the bundled script:
15
16	```bash
17	scripts/count-tokens.sh <file-or-directory>
18	```
19
20	### Examples
21
22	Count tokens for a single file:
23	```bash
24	scripts/count-tokens.sh .cursor/rules/main-rules.mdc
25	```
26
27	Count tokens for an entire directory:
28	```bash
29	scripts/count-tokens.sh .cursor/rules/
30	```
31
32	Count tokens for the whole project:
33	```bash
34	scripts/count-tokens.sh .
35	```
36
37	## Steps
38
39	### Step 1: Identify Target
40	- [ ] Determine what the user wants to measure (single file, directory, or set of files)
41	- [ ] Verify the target path exists
42
43	### Step 2: Run Token Count
44	- [ ] Execute `scripts/count-tokens.sh <target>` using the Shell tool
45	- [ ] Capture output
46
47	### Step 3: Interpret Results
48	- [ ] Present the token estimates in a clear table or list
49	- [ ] Flag any files that exceed context layer budgets:
50	- Immediate context: ~2000 tokens
51	- Relevant context: ~3000 tokens
52	- Extended context: ~3000 tokens
53	- [ ] Suggest optimizations if totals are high (split files, trim comments, use references)
54
55	### Step 4: Recommendations
56	- [ ] If a single file exceeds 2000 tokens, suggest splitting or using progressive loading
57	- [ ] If a directory exceeds 10,000 tokens, recommend selective inclusion
58	- [ ] Reference the token efficiency rules for budget guidance
59
60	## How It Works
61
62	The script uses a simple heuristic: `word_count × 1.3 ≈ token_count`. This approximation works well for English text and code. For precise counts, use `tiktoken` or the OpenAI tokenizer.
63
64	Supported file types: `.md`, `.mdc`, `.ts`, `.tsx`, `.py`
65
66	## Rules
67	- The heuristic is an approximation — actual token counts vary by model and tokenizer
68	- For precise counts, recommend `tiktoken` (Python) or equivalent
69	- Use this for quick estimates and budget planning, not exact billing
70
71	## Completion
72	Token estimates displayed with optimization suggestions if any files exceed recommended budgets.
73
74	## If a Step Fails
75	- File not found: Verify the path and suggest alternatives
76	- Empty output: Check if the directory contains supported file types
77	- Inaccurate results: Suggest installing `tiktoken` for precise counts

People who liked this also viewed

6 picks

Type

token-counter

27stars3forks

View on GitHub

How to install token-counter?

girijashankarj/cursor-handbook/token-counter

$npx -y skills add girijashankarj/cursor-handbook --skill token-counter

Installs into the current project.

Summary

Estimate LLM token counts for files or directories using a word-based heuristic. Use when the user asks how many tokens a file uses, wants to check context size, or needs to estimate token cost.

Files · 1

View on GitHub

SKILL.md

1	# Skill: Token Counter
2
3	Estimate token consumption for files and directories to help manage LLM context budgets.
4
5	## Trigger
6	When the user asks to count tokens, estimate context size, check how much a file/directory costs in tokens, or wants to optimize token usage.
7
8	## Prerequisites
9	- [ ] Target file or directory exists
10	- [ ] `wc` command available (standard on macOS/Linux)
11
12	## Usage
13
14	Run the bundled script:
15
16	```bash
17	scripts/count-tokens.sh <file-or-directory>
18	```
19
20	### Examples
21
22	Count tokens for a single file:
23	```bash
24	scripts/count-tokens.sh .cursor/rules/main-rules.mdc
25	```
26
27	Count tokens for an entire directory:
28	```bash
29	scripts/count-tokens.sh .cursor/rules/
30	```
31
32	Count tokens for the whole project:
33	```bash
34	scripts/count-tokens.sh .
35	```
36
37	## Steps
38
39	### Step 1: Identify Target
40	- [ ] Determine what the user wants to measure (single file, directory, or set of files)
41	- [ ] Verify the target path exists
42
43	### Step 2: Run Token Count
44	- [ ] Execute `scripts/count-tokens.sh <target>` using the Shell tool
45	- [ ] Capture output
46
47	### Step 3: Interpret Results
48	- [ ] Present the token estimates in a clear table or list
49	- [ ] Flag any files that exceed context layer budgets:
50	- Immediate context: ~2000 tokens
51	- Relevant context: ~3000 tokens
52	- Extended context: ~3000 tokens
53	- [ ] Suggest optimizations if totals are high (split files, trim comments, use references)
54
55	### Step 4: Recommendations
56	- [ ] If a single file exceeds 2000 tokens, suggest splitting or using progressive loading
57	- [ ] If a directory exceeds 10,000 tokens, recommend selective inclusion
58	- [ ] Reference the token efficiency rules for budget guidance
59
60	## How It Works
61
62	The script uses a simple heuristic: `word_count × 1.3 ≈ token_count`. This approximation works well for English text and code. For precise counts, use `tiktoken` or the OpenAI tokenizer.
63
64	Supported file types: `.md`, `.mdc`, `.ts`, `.tsx`, `.py`
65
66	## Rules
67	- The heuristic is an approximation — actual token counts vary by model and tokenizer
68	- For precise counts, recommend `tiktoken` (Python) or equivalent
69	- Use this for quick estimates and budget planning, not exact billing
70
71	## Completion
72	Token estimates displayed with optimization suggestions if any files exceed recommended budgets.
73
74	## If a Step Fails
75	- File not found: Verify the path and suggest alternatives
76	- Empty output: Check if the directory contains supported file types
77	- Inaccurate results: Suggest installing `tiktoken` for precise counts

People who liked this also viewed

6 picks

Type