smg

Name: smg
Author: lightseekorg

412stars123forks

How to install smg?

lightseekorg/smg

$git clone https://github.com/lightseekorg/smg

Installs into the current project.

Summary

Engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments. Centralizes worker lifecycle management, balances traffic across HTTP/gRPC/OpenAI-compatible backends, and provides enterprise-ready control over history storage, MCP tooling, and privacy-sen

Files · 1

View on GitHub

README.md

1	<p align="center">
2	<img alt="SMG Logo" src="https://raw.githubusercontent.com/lightseekorg/smg/main/docs/assets/images/logos/logomark-dark.svg" width="80">
3	</p>
4
5	<h1 align="center">Shepherd Model Gateway</h1>
6
7	<p align="center">
8	<a href="https://github.com/lightseekorg/smg/releases/latest"><img src="https://img.shields.io/github/v/release/lightseekorg/smg?logo=github&label=Release" alt="Release"></a>
9	<a href="https://github.com/orgs/lightseekorg/packages/container/package/smg"><img src="https://img.shields.io/badge/ghcr.io-lightseekorg%2Fsmg-blue?logo=docker" alt="Docker"></a>
10	<a href="https://pypi.org/project/smg/"><img src="https://img.shields.io/pypi/v/smg?logo=pypi&logoColor=white&label=PyPI" alt="PyPI"></a>
11	<a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License"></a>
12	<a href="https://lightseekorg.github.io/smg"><img src="https://img.shields.io/badge/docs-latest-brightgreen.svg" alt="Docs"></a>
13	<a href="https://discord.lightseek.org"><img src="https://img.shields.io/badge/Discord-Join%20Us-5865F2?logo=discord&logoColor=white" alt="Discord"></a>
14	<a href="https://slack.lightseek.org"><img src="https://img.shields.io/badge/Slack-Join%20Us-4A154B?logo=slack&logoColor=white" alt="Slack"></a>
15	<a href="https://deepwiki.com/lightseekorg/smg"><img src="https://deepwiki.com/badge.svg" alt="Ask DeepWiki"></a>
16	<a href="https://pytorch.org/blog/lightseek-smg/"><img src="https://img.shields.io/badge/PyTorch-Technical%20Blog-EE4C2C?logo=pytorch&logoColor=white" alt="PyTorch Blog"></a>
17	</p>
18
19	Engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments. Centralizes worker lifecycle management, balances traffic across HTTP/gRPC/OpenAI-compatible backends, and provides enterprise-ready control over history storage, MCP tooling, and privacy-sensitive workflows.
20
21	<p align="center">
22	<img src="https://raw.githubusercontent.com/lightseekorg/smg/main/docs/assets/images/architecture-animated.svg" alt="SMG Architecture" width="100%">
23	</p>
24
25	## Why SMG?
26
27	\| \| \|
28	\|:--------------------------------\|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------\|
29	\| 🚀 Maximize GPU Utilization \| Cache-aware routing understands your inference engine's KV cache state—whether vLLM, TensorRT-LLM, TokenSpeed, or SGLang—to reuse prefixes and reduce redundant computation. \|
30	\| 🔌 One API, Any Backend \| Route to self-hosted models (vLLM, TensorRT-LLM, TokenSpeed, SGLang) or cloud providers (OpenAI, Anthropic, Gemini, Bedrock, and more) through a single unified endpoint. \|
31	\| ⚡ Built for Speed \| Native Rust with gRPC pipelines, sub-millisecond routing decisions, and zero-copy tokenization. Circuit breakers and automatic failover keep things running. \|
32	\| 🔒 Enterprise Control \| Multi-tenant rate limiting with OIDC, WebAssembly plugins for custom logic, and a privacy boundary that keeps conversation history within your infrastructure. \|
33	\| 📊 Full Observability \| 40+ Prometheus metrics, OpenTelemetry tracing, and structured JSON logs with request correlation—know exactly what's happening at every layer. \|
34
35	API Coverage: OpenAI Chat/Completions/Embeddings, Responses API for agents, Anthropic Messages, and MCP tool execution.
36
37	## Quick Start
38
39	Install — pick your preferred method:
40
41	```bash
42	# Docker
43	docker pull lightseekorg/smg:latest
44
45	# Python
46	pip install smg
47
48	# Rust
49	cargo install smg
50	```
51
52	Run — point SMG at your inference workers:
53
54	```bash
55	# Single worker
56	smg launch --worker-urls http://localhost:8000
57
58	# Multiple workers with cache-aware routing
59	smg launch --worker-urls http://gpu1:8000 http://gpu2:8000 --policy

People who liked this also viewed

6 picks

Type

smg

412stars123forks

View on GitHub

How to install smg?

lightseekorg/smg

$git clone https://github.com/lightseekorg/smg

Installs into the current project.

Summary

Files · 1

View on GitHub

README.md

1	<p align="center">
2	<img alt="SMG Logo" src="https://raw.githubusercontent.com/lightseekorg/smg/main/docs/assets/images/logos/logomark-dark.svg" width="80">
3	</p>
4
5	<h1 align="center">Shepherd Model Gateway</h1>
6
7	<p align="center">
8	<a href="https://github.com/lightseekorg/smg/releases/latest"><img src="https://img.shields.io/github/v/release/lightseekorg/smg?logo=github&label=Release" alt="Release"></a>
9	<a href="https://github.com/orgs/lightseekorg/packages/container/package/smg"><img src="https://img.shields.io/badge/ghcr.io-lightseekorg%2Fsmg-blue?logo=docker" alt="Docker"></a>
10	<a href="https://pypi.org/project/smg/"><img src="https://img.shields.io/pypi/v/smg?logo=pypi&logoColor=white&label=PyPI" alt="PyPI"></a>
11	<a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License"></a>
12	<a href="https://lightseekorg.github.io/smg"><img src="https://img.shields.io/badge/docs-latest-brightgreen.svg" alt="Docs"></a>
13	<a href="https://discord.lightseek.org"><img src="https://img.shields.io/badge/Discord-Join%20Us-5865F2?logo=discord&logoColor=white" alt="Discord"></a>
14	<a href="https://slack.lightseek.org"><img src="https://img.shields.io/badge/Slack-Join%20Us-4A154B?logo=slack&logoColor=white" alt="Slack"></a>
15	<a href="https://deepwiki.com/lightseekorg/smg"><img src="https://deepwiki.com/badge.svg" alt="Ask DeepWiki"></a>
16	<a href="https://pytorch.org/blog/lightseek-smg/"><img src="https://img.shields.io/badge/PyTorch-Technical%20Blog-EE4C2C?logo=pytorch&logoColor=white" alt="PyTorch Blog"></a>
17	</p>
18
19	Engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments. Centralizes worker lifecycle management, balances traffic across HTTP/gRPC/OpenAI-compatible backends, and provides enterprise-ready control over history storage, MCP tooling, and privacy-sensitive workflows.
20
21	<p align="center">
22	<img src="https://raw.githubusercontent.com/lightseekorg/smg/main/docs/assets/images/architecture-animated.svg" alt="SMG Architecture" width="100%">
23	</p>
24
25	## Why SMG?
26
27	\| \| \|
28	\|:--------------------------------\|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------\|
29	\| 🚀 Maximize GPU Utilization \| Cache-aware routing understands your inference engine's KV cache state—whether vLLM, TensorRT-LLM, TokenSpeed, or SGLang—to reuse prefixes and reduce redundant computation. \|
30	\| 🔌 One API, Any Backend \| Route to self-hosted models (vLLM, TensorRT-LLM, TokenSpeed, SGLang) or cloud providers (OpenAI, Anthropic, Gemini, Bedrock, and more) through a single unified endpoint. \|
31	\| ⚡ Built for Speed \| Native Rust with gRPC pipelines, sub-millisecond routing decisions, and zero-copy tokenization. Circuit breakers and automatic failover keep things running. \|
32	\| 🔒 Enterprise Control \| Multi-tenant rate limiting with OIDC, WebAssembly plugins for custom logic, and a privacy boundary that keeps conversation history within your infrastructure. \|
33	\| 📊 Full Observability \| 40+ Prometheus metrics, OpenTelemetry tracing, and structured JSON logs with request correlation—know exactly what's happening at every layer. \|
34
35	API Coverage: OpenAI Chat/Completions/Embeddings, Responses API for agents, Anthropic Messages, and MCP tool execution.
36
37	## Quick Start
38
39	Install — pick your preferred method:
40
41	```bash
42	# Docker
43	docker pull lightseekorg/smg:latest
44
45	# Python
46	pip install smg
47
48	# Rust
49	cargo install smg
50	```
51
52	Run — point SMG at your inference workers:
53
54	```bash
55	# Single worker
56	smg launch --worker-urls http://localhost:8000
57
58	# Multiple workers with cache-aware routing
59	smg launch --worker-urls http://gpu1:8000 http://gpu2:8000 --policy

People who liked this also viewed

6 picks

Type