llama-cpp

Name: llama-cpp
Author: AlexAI-MCP

132stars24forks

How to install llama-cpp?

alexai-mcp/hermes-ccc/llama-cpp

$npx -y skills add AlexAI-MCP/hermes-CCC --skill llama-cpp

Installs into the current project.

Summary

Run quantized LLMs locally with llama.cpp — CPU+GPU inference, GGUF format, OpenAI-compatible server, and Python bindings.

Files · 1

View on GitHub

SKILL.md

1	# llama.cpp
2
3	## Purpose
4
5	- Use this skill to run quantized GGUF models on laptops, workstations, and edge systems.
6	- Prefer it when you need portable local inference without a heavyweight serving stack.
7	- `llama.cpp` is especially useful for CPU-first deployments, low-cost GPU offload, and offline workflows.
8	- Python users typically access it through `llama-cpp-python`.
9
10	## Install
11
12	- Fastest path for Python users:
13
14	```bash
15	pip install llama-cpp-python
16	```
17
18	- Build from source when you need custom acceleration backends or tighter platform control.
19	- Source builds are common for CUDA, Metal, ROCm, Vulkan, and CPU-tuned environments.
20	- Confirm the package imports successfully:
21
22	```bash
23	python -c "from llama_cpp import Llama; print('ok')"
24	```
25
26	## Model Format
27
28	- `llama.cpp` primarily uses the `GGUF` model format.
29	- GGUF packages tokenizer metadata, architecture settings, and quantized weights in one artifact.
30	- Choose a GGUF variant that matches your hardware budget and quality target.
31
32	## Download GGUF Models
33
34	- Hugging Face is the standard source for GGUF checkpoints.
35	- Common repos include:
36	- `bartowski/*`
37	- `TheBloke/*`
38
39	- Typical examples:
40	- `bartowski/Llama-3.1-8B-Instruct-GGUF`
41	- `TheBloke/Mistral-7B-Instruct-v0.2-GGUF`
42	- `bartowski/Qwen2.5-7B-Instruct-GGUF`
43
44	- Store the downloaded file locally, for example:
45	- `models/llama-3.1-8b-instruct-q4_k_m.gguf`
46
47	## Quantization Levels
48
49	- `Q4_K_M`: best balance for many local deployments
50	- `Q5_K_M`: more quality, more RAM or VRAM
51	- `Q8_0`: highest quality among common quantized options
52	- `Q2_K`: very small, but quality drops sharply
53
54	- Start with `Q4_K_M` unless you already know the task is quality-sensitive.
55	- Move to `Q5_K_M` or `Q8_0` for coding, reasoning, or long-form generation where quality matters more.
56	- Use `Q2_K` only for extreme memory constraints or experiments.
57
58	## Basic Python Usage
59
60	```python
61	from llama_cpp import Llama
62
63	llm = Llama(
64	model_path="models/llama-3.1-8b-instruct-q4_k_m.gguf",
65	n_ctx=8192,
66	n_threads=8,
67	n_gpu_layers=0,
68	)
69
70	output = llm(
71	"Explain why GGUF is useful for local inference.",
72	max_tokens=256,
73	temperature=0.7,
74	)
75
76	print(output["choices"][0]["text"])
77	```
78
79	## Important Init Parameters
80
81	- `model_path`: path to the `.gguf` file on disk
82	- `n_gpu_layers`: number of transformer layers to offload to GPU
83	- `n_ctx`: context size, constrained by the model and available memory
84	- `n_threads`: CPU worker threads for prompt processing and generation
85
86	- These four parameters are the first tuning knobs to adjust for almost every deployment.
87
88	## Initialization Guidance
89
90	- Keep `model_path` on a local SSD when possible.
91	- Set `n_threads` close to the number of performant CPU cores, not necessarily total logical threads.
92	- Increase `n_ctx` only after checking memory pressure.
93	- Increase `n_gpu_layers` gradually if the model fails to load or performance is unstable.
94
95	## Generation Call
96
97	- Basic call shape:
98
99	```python
100	result = llm(
101	"Write a short checklist for running a local LLM service.",
102	max_tokens=256,
103	temperature=0.7,
104	)
105	```
106
107	- Access the text with:
108
109	```python
110	print(result["choices"][0]["text"])
111	```
112
113	- Keep `max_tokens` bounded for interactive usage.
114	- Use lower temperatures for summarization, extraction, and tool-style tasks.
115
116	## Chat Format
117
118	- For instruction-tuned models, prefer the chat API:
119
120	```python
121	from llama_cpp import Llama
122
123	llm = Llama(
124	model_path="models/qwen2.5-7b-instruct-q4_k_m.gguf",
125	n_ctx=8192,
126	n_gpu_layers=20,
127	n_threads=8,
128	)
129
130	response = llm.create_chat_completion(
131	messages=[
132	{"role": "system", "content": "You are concise and technical."},
133	{"role": "user", "content": "List three tradeoffs of 4-bit quantization."},
134	],
135	max_tokens=256,
136	temperature=0.4,
137	)
138
139	print(response["choices"][0]["message"]["content"])
140	```
141
142	- Use the chat API when the model card says the checkpoint is chat-tuned or instruct-tuned.
143	- Keep prompt templates aligned with the model family if outputs seem malformed.
144
145	## Streaming Output
146
147	- Stream tokens for responsive CLI or web applications:
148
149	```python
150	stream = llm.create_chat_completion(
151	messages=[
152	{"role": "system", "content": "You are concise."},
153	{"role": "user", "content": "Describe GPU offload in llama.cpp."},
154	],
155	max_tokens=128,
156	temperature=0.3,
157	stream=True,
158	)
159
160	for chunk in stream:
161	delta = chunk["choices"][0].get("delta", {})
162	text = delta.get("content")
163	if text:
164	print(text, end="", flush=True)
165	```
166
167	- Streaming is useful for chat UIs, terminals, and server-sent event bridges.
168
169	## GPU Offload
170
171	- `n_gpu_layers=-1` mean

People who liked this also viewed

6 picks

Type

find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 20262.6M27k
grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026633k182k
vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026573k29k
improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026522k182k
setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026454k182k
triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026424k182k

llama-cpp

132stars24forks

View on GitHub

How to install llama-cpp?

alexai-mcp/hermes-ccc/llama-cpp

$npx -y skills add AlexAI-MCP/hermes-CCC --skill llama-cpp

Installs into the current project.

Summary

Run quantized LLMs locally with llama.cpp — CPU+GPU inference, GGUF format, OpenAI-compatible server, and Python bindings.

Files · 1

View on GitHub

SKILL.md

1	# llama.cpp
2
3	## Purpose
4
5	- Use this skill to run quantized GGUF models on laptops, workstations, and edge systems.
6	- Prefer it when you need portable local inference without a heavyweight serving stack.
7	- `llama.cpp` is especially useful for CPU-first deployments, low-cost GPU offload, and offline workflows.
8	- Python users typically access it through `llama-cpp-python`.
9
10	## Install
11
12	- Fastest path for Python users:
13
14	```bash
15	pip install llama-cpp-python
16	```
17
18	- Build from source when you need custom acceleration backends or tighter platform control.
19	- Source builds are common for CUDA, Metal, ROCm, Vulkan, and CPU-tuned environments.
20	- Confirm the package imports successfully:
21
22	```bash
23	python -c "from llama_cpp import Llama; print('ok')"
24	```
25
26	## Model Format
27
28	- `llama.cpp` primarily uses the `GGUF` model format.
29	- GGUF packages tokenizer metadata, architecture settings, and quantized weights in one artifact.
30	- Choose a GGUF variant that matches your hardware budget and quality target.
31
32	## Download GGUF Models
33
34	- Hugging Face is the standard source for GGUF checkpoints.
35	- Common repos include:
36	- `bartowski/*`
37	- `TheBloke/*`
38
39	- Typical examples:
40	- `bartowski/Llama-3.1-8B-Instruct-GGUF`
41	- `TheBloke/Mistral-7B-Instruct-v0.2-GGUF`
42	- `bartowski/Qwen2.5-7B-Instruct-GGUF`
43
44	- Store the downloaded file locally, for example:
45	- `models/llama-3.1-8b-instruct-q4_k_m.gguf`
46
47	## Quantization Levels
48
49	- `Q4_K_M`: best balance for many local deployments
50	- `Q5_K_M`: more quality, more RAM or VRAM
51	- `Q8_0`: highest quality among common quantized options
52	- `Q2_K`: very small, but quality drops sharply
53
54	- Start with `Q4_K_M` unless you already know the task is quality-sensitive.
55	- Move to `Q5_K_M` or `Q8_0` for coding, reasoning, or long-form generation where quality matters more.
56	- Use `Q2_K` only for extreme memory constraints or experiments.
57
58	## Basic Python Usage
59
60	```python
61	from llama_cpp import Llama
62
63	llm = Llama(
64	model_path="models/llama-3.1-8b-instruct-q4_k_m.gguf",
65	n_ctx=8192,
66	n_threads=8,
67	n_gpu_layers=0,
68	)
69
70	output = llm(
71	"Explain why GGUF is useful for local inference.",
72	max_tokens=256,
73	temperature=0.7,
74	)
75
76	print(output["choices"][0]["text"])
77	```
78
79	## Important Init Parameters
80
81	- `model_path`: path to the `.gguf` file on disk
82	- `n_gpu_layers`: number of transformer layers to offload to GPU
83	- `n_ctx`: context size, constrained by the model and available memory
84	- `n_threads`: CPU worker threads for prompt processing and generation
85
86	- These four parameters are the first tuning knobs to adjust for almost every deployment.
87
88	## Initialization Guidance
89
90	- Keep `model_path` on a local SSD when possible.
91	- Set `n_threads` close to the number of performant CPU cores, not necessarily total logical threads.
92	- Increase `n_ctx` only after checking memory pressure.
93	- Increase `n_gpu_layers` gradually if the model fails to load or performance is unstable.
94
95	## Generation Call
96
97	- Basic call shape:
98
99	```python
100	result = llm(
101	"Write a short checklist for running a local LLM service.",
102	max_tokens=256,
103	temperature=0.7,
104	)
105	```
106
107	- Access the text with:
108
109	```python
110	print(result["choices"][0]["text"])
111	```
112
113	- Keep `max_tokens` bounded for interactive usage.
114	- Use lower temperatures for summarization, extraction, and tool-style tasks.
115
116	## Chat Format
117
118	- For instruction-tuned models, prefer the chat API:
119
120	```python
121	from llama_cpp import Llama
122
123	llm = Llama(
124	model_path="models/qwen2.5-7b-instruct-q4_k_m.gguf",
125	n_ctx=8192,
126	n_gpu_layers=20,
127	n_threads=8,
128	)
129
130	response = llm.create_chat_completion(
131	messages=[
132	{"role": "system", "content": "You are concise and technical."},
133	{"role": "user", "content": "List three tradeoffs of 4-bit quantization."},
134	],
135	max_tokens=256,
136	temperature=0.4,
137	)
138
139	print(response["choices"][0]["message"]["content"])
140	```
141
142	- Use the chat API when the model card says the checkpoint is chat-tuned or instruct-tuned.
143	- Keep prompt templates aligned with the model family if outputs seem malformed.
144
145	## Streaming Output
146
147	- Stream tokens for responsive CLI or web applications:
148
149	```python
150	stream = llm.create_chat_completion(
151	messages=[
152	{"role": "system", "content": "You are concise."},
153	{"role": "user", "content": "Describe GPU offload in llama.cpp."},
154	],
155	max_tokens=128,
156	temperature=0.3,
157	stream=True,
158	)
159
160	for chunk in stream:
161	delta = chunk["choices"][0].get("delta", {})
162	text = delta.get("content")
163	if text:
164	print(text, end="", flush=True)
165	```
166
167	- Streaming is useful for chat UIs, terminals, and server-sent event bridges.
168
169	## GPU Offload
170
171	- `n_gpu_layers=-1` mean

People who liked this also viewed

6 picks

Type