mlx

Name: mlx
Author: itsmostafa

70installs0forks

How to install mlx?

itsmostafa/llm-engineering-skills/mlx

$npx -y skills add itsmostafa/llm-engineering-skills --skill mlx

Installs into the current project.

Summary

Running and fine-tuning LLMs on Apple Silicon with MLX. Use when working with models locally on Mac, converting Hugging Face models to MLX format, fine-tuning with LoRA/QLoRA on Apple Silicon, or serving models via HTTP API.

Files · 1

View on GitHub

SKILL.md

1	# Using MLX for LLMs on Apple Silicon
2
3	MLX-LM is a Python package for running large language models on Apple Silicon, leveraging the MLX framework for optimized performance with unified memory architecture.
4
5	## Table of Contents
6
7	- [Core Concepts](#core-concepts)
8	- [Installation](#installation)
9	- [Text Generation](#text-generation)
10	- [Interactive Chat](#interactive-chat)
11	- [Model Conversion](#model-conversion)
12	- [Quantization](#quantization)
13	- [Fine-tuning with LoRA](#fine-tuning-with-lora)
14	- [Serving Models](#serving-models)
15	- [Best Practices](#best-practices)
16	- [References](#references)
17
18	## Core Concepts
19
20	### Why MLX
21
22	\| Aspect \| PyTorch on Mac \| MLX \|
23	\|--------\|----------------\|-----\|
24	\| Memory \| Separate CPU/GPU copies \| Unified memory, no copies \|
25	\| Optimization \| Generic Metal backend \| Apple Silicon native \|
26	\| Model loading \| Slower, more memory \| Lazy loading, efficient \|
27	\| Quantization \| Limited support \| Built-in 4/8-bit \|
28
29	MLX arrays live in shared memory, accessible by both CPU and GPU without data transfer overhead.
30
31	### Supported Models
32
33	MLX-LM supports most popular architectures: Llama, Mistral, Qwen, Phi, Gemma, Cohere, and many more. Check the [mlx-community](https://huggingface.co/mlx-community) on Hugging Face for pre-converted models.
34
35	## Installation
36
37	```bash
38	pip install mlx-lm
39	```
40
41	Requires macOS 13.5+ and Apple Silicon (M1/M2/M3/M4).
42
43	## Text Generation
44
45	### Python API
46
47	```python
48	from mlx_lm import load, generate
49
50	# Load model (from HF hub or local path)
51	model, tokenizer = load("mlx-community/Llama-3.2-3B-Instruct-4bit")
52
53	# Generate text
54	response = generate(
55	model,
56	tokenizer,
57	prompt="Explain quantum computing in simple terms:",
58	max_tokens=256,
59	temp=0.7,
60	)
61	print(response)
62	```
63
64	### Streaming Generation
65
66	```python
67	from mlx_lm import load, stream_generate
68
69	model, tokenizer = load("mlx-community/Mistral-7B-Instruct-v0.3-4bit")
70
71	prompt = "Write a haiku about programming:"
72	for response in stream_generate(model, tokenizer, prompt, max_tokens=100):
73	print(response.text, end="", flush=True)
74	print()
75	```
76
77	### Batch Generation
78
79	```python
80	from mlx_lm import load, batch_generate
81
82	model, tokenizer = load("mlx-community/Qwen2.5-7B-Instruct-4bit")
83
84	prompts = [
85	"What is machine learning?",
86	"Explain neural networks:",
87	"Define deep learning:",
88	]
89
90	responses = batch_generate(
91	model,
92	tokenizer,
93	prompts,
94	max_tokens=100,
95	)
96
97	for prompt, response in zip(prompts, responses):
98	print(f"Q: {prompt}\nA: {response}\n")
99	```
100
101	### CLI Generation
102
103	```bash
104	# Basic generation
105	mlx_lm.generate --model mlx-community/Llama-3.2-3B-Instruct-4bit \
106	--prompt "Explain recursion:" \
107	--max-tokens 256
108
109	# With sampling parameters
110	mlx_lm.generate --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
111	--prompt "Write a poem about AI:" \
112	--temp 0.8 \
113	--top-p 0.95
114	```
115
116	## Interactive Chat
117
118	### CLI Chat
119
120	```bash
121	# Start chat REPL (context preserved between turns)
122	mlx_lm.chat --model mlx-community/Llama-3.2-3B-Instruct-4bit
123	```
124
125	### Python Chat
126
127	```python
128	from mlx_lm import load, generate
129
130	model, tokenizer = load("mlx-community/Llama-3.2-3B-Instruct-4bit")
131
132	messages = [
133	{"role": "system", "content": "You are a helpful assistant."},
134	{"role": "user", "content": "What's the capital of France?"},
135	]
136
137	prompt = tokenizer.apply_chat_template(
138	messages,
139	tokenize=False,
140	add_generation_prompt=True
141	)
142
143	response = generate(model, tokenizer, prompt=prompt, max_tokens=256)
144	print(response)
145	```
146
147	## Model Conversion
148
149	Convert Hugging Face models to MLX format:
150
151	### CLI Conversion
152
153	```bash
154	# Convert with 4-bit quantization
155	mlx_lm.convert --model meta-llama/Llama-3.2-3B-Instruct \
156	-q # Quantize to 4-bit
157
158	# With specific quantization
159	mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 \
160	-q \
161	--q-bits 8 \
162	--q-group-size 64
163
164	# Upload to Hugging Face Hub
165	mlx_lm.convert --model meta-llama/Llama-3.2-1B-Instruct \
166	-q \
167	--upload-repo your-username/Llama-3.2-1B-Instruct-4bit-mlx
168	```
169
170	### Python Conversion
171
172	```python
173	from mlx_lm import convert
174
175	repo = "meta-llama/Llama-3.2-3B-Instruct"
176	convert(
177	repo,
178	quantize=True,
179	mlx_path="./llama-3.2-3b-mlx",
180	q_bits=4,
181	q_group_size=64,
182	)
183	```
184
185	### Conversion Options
186
187	\| Option \| Default \| Description \|
188	\|--------\|---------\|-------------\|
189	\| `--q-bits` \| 4 \| Quantization bits (4 or 8) \|
190	\| `--q-group-size` \| 64 \| Group size for quantization \|
191	\| `--dtype` \| float16 \| Data type for non-quantized weights \|
192
193	## Quantization
194
195	MLX supports multiple quantization methods for different use cases:
196
197	\| Method \| Best For \| Command \|
198	\|--------\|----------\|---------\|
199	\| Basic \| Quick conversion \| `mlx_lm.convert -q` \|
200	\| DWQ \| Quality-preserving \| `mlx_lm.dwq` \|
201	\| AWQ \|

People who liked this also viewed

6 picks

Type

mlx

70installs0forks

View on GitHub

How to install mlx?

itsmostafa/llm-engineering-skills/mlx

$npx -y skills add itsmostafa/llm-engineering-skills --skill mlx

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Using MLX for LLMs on Apple Silicon
2
3	MLX-LM is a Python package for running large language models on Apple Silicon, leveraging the MLX framework for optimized performance with unified memory architecture.
4
5	## Table of Contents
6
7	- [Core Concepts](#core-concepts)
8	- [Installation](#installation)
9	- [Text Generation](#text-generation)
10	- [Interactive Chat](#interactive-chat)
11	- [Model Conversion](#model-conversion)
12	- [Quantization](#quantization)
13	- [Fine-tuning with LoRA](#fine-tuning-with-lora)
14	- [Serving Models](#serving-models)
15	- [Best Practices](#best-practices)
16	- [References](#references)
17
18	## Core Concepts
19
20	### Why MLX
21
22	\| Aspect \| PyTorch on Mac \| MLX \|
23	\|--------\|----------------\|-----\|
24	\| Memory \| Separate CPU/GPU copies \| Unified memory, no copies \|
25	\| Optimization \| Generic Metal backend \| Apple Silicon native \|
26	\| Model loading \| Slower, more memory \| Lazy loading, efficient \|
27	\| Quantization \| Limited support \| Built-in 4/8-bit \|
28
29	MLX arrays live in shared memory, accessible by both CPU and GPU without data transfer overhead.
30
31	### Supported Models
32
33	MLX-LM supports most popular architectures: Llama, Mistral, Qwen, Phi, Gemma, Cohere, and many more. Check the [mlx-community](https://huggingface.co/mlx-community) on Hugging Face for pre-converted models.
34
35	## Installation
36
37	```bash
38	pip install mlx-lm
39	```
40
41	Requires macOS 13.5+ and Apple Silicon (M1/M2/M3/M4).
42
43	## Text Generation
44
45	### Python API
46
47	```python
48	from mlx_lm import load, generate
49
50	# Load model (from HF hub or local path)
51	model, tokenizer = load("mlx-community/Llama-3.2-3B-Instruct-4bit")
52
53	# Generate text
54	response = generate(
55	model,
56	tokenizer,
57	prompt="Explain quantum computing in simple terms:",
58	max_tokens=256,
59	temp=0.7,
60	)
61	print(response)
62	```
63
64	### Streaming Generation
65
66	```python
67	from mlx_lm import load, stream_generate
68
69	model, tokenizer = load("mlx-community/Mistral-7B-Instruct-v0.3-4bit")
70
71	prompt = "Write a haiku about programming:"
72	for response in stream_generate(model, tokenizer, prompt, max_tokens=100):
73	print(response.text, end="", flush=True)
74	print()
75	```
76
77	### Batch Generation
78
79	```python
80	from mlx_lm import load, batch_generate
81
82	model, tokenizer = load("mlx-community/Qwen2.5-7B-Instruct-4bit")
83
84	prompts = [
85	"What is machine learning?",
86	"Explain neural networks:",
87	"Define deep learning:",
88	]
89
90	responses = batch_generate(
91	model,
92	tokenizer,
93	prompts,
94	max_tokens=100,
95	)
96
97	for prompt, response in zip(prompts, responses):
98	print(f"Q: {prompt}\nA: {response}\n")
99	```
100
101	### CLI Generation
102
103	```bash
104	# Basic generation
105	mlx_lm.generate --model mlx-community/Llama-3.2-3B-Instruct-4bit \
106	--prompt "Explain recursion:" \
107	--max-tokens 256
108
109	# With sampling parameters
110	mlx_lm.generate --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
111	--prompt "Write a poem about AI:" \
112	--temp 0.8 \
113	--top-p 0.95
114	```
115
116	## Interactive Chat
117
118	### CLI Chat
119
120	```bash
121	# Start chat REPL (context preserved between turns)
122	mlx_lm.chat --model mlx-community/Llama-3.2-3B-Instruct-4bit
123	```
124
125	### Python Chat
126
127	```python
128	from mlx_lm import load, generate
129
130	model, tokenizer = load("mlx-community/Llama-3.2-3B-Instruct-4bit")
131
132	messages = [
133	{"role": "system", "content": "You are a helpful assistant."},
134	{"role": "user", "content": "What's the capital of France?"},
135	]
136
137	prompt = tokenizer.apply_chat_template(
138	messages,
139	tokenize=False,
140	add_generation_prompt=True
141	)
142
143	response = generate(model, tokenizer, prompt=prompt, max_tokens=256)
144	print(response)
145	```
146
147	## Model Conversion
148
149	Convert Hugging Face models to MLX format:
150
151	### CLI Conversion
152
153	```bash
154	# Convert with 4-bit quantization
155	mlx_lm.convert --model meta-llama/Llama-3.2-3B-Instruct \
156	-q # Quantize to 4-bit
157
158	# With specific quantization
159	mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 \
160	-q \
161	--q-bits 8 \
162	--q-group-size 64
163
164	# Upload to Hugging Face Hub
165	mlx_lm.convert --model meta-llama/Llama-3.2-1B-Instruct \
166	-q \
167	--upload-repo your-username/Llama-3.2-1B-Instruct-4bit-mlx
168	```
169
170	### Python Conversion
171
172	```python
173	from mlx_lm import convert
174
175	repo = "meta-llama/Llama-3.2-3B-Instruct"
176	convert(
177	repo,
178	quantize=True,
179	mlx_path="./llama-3.2-3b-mlx",
180	q_bits=4,
181	q_group_size=64,
182	)
183	```
184
185	### Conversion Options
186
187	\| Option \| Default \| Description \|
188	\|--------\|---------\|-------------\|
189	\| `--q-bits` \| 4 \| Quantization bits (4 or 8) \|
190	\| `--q-group-size` \| 64 \| Group size for quantization \|
191	\| `--dtype` \| float16 \| Data type for non-quantized weights \|
192
193	## Quantization
194
195	MLX supports multiple quantization methods for different use cases:
196
197	\| Method \| Best For \| Command \|
198	\|--------\|----------\|---------\|
199	\| Basic \| Quick conversion \| `mlx_lm.convert -q` \|
200	\| DWQ \| Quality-preserving \| `mlx_lm.dwq` \|
201	\| AWQ \|

People who liked this also viewed

6 picks

Type