molmo
starsforks
How to install molmo?
graph-robots/open-robot-skills/molmo
$npx -y skills add graph-robots/open-robot-skills --skill molmoVisual pointing and Q&A via the Molmo VLM served from a
SKILL.md
| 1 | # molmo |
| 2 | |
| 3 | Molmo visual pointing + Q&A. The bundle itself is zero-GPU (httpx client |
| 4 | only), but Molmo has **no hosted API** — you must serve it yourself with |
| 5 | vLLM and point `GAP_MOLMO_BASE_URL` at it. If you can't self-host, use |
| 6 | `gemini-er.detect` (hosted Gemini Robotics-ER) instead. |
| 7 | |
| 8 | ## Hosting recipe (vLLM) |
| 9 | |
| 10 | Lifted from the dev tree's run book (`training/README.md`): |
| 11 | |
| 12 | ```bash |
| 13 | # Serve Molmo2-8B on an OpenAI-compatible endpoint |
| 14 | CUDA_VISIBLE_DEVICES=0 PYTHONNOUSERSITE=1 \ |
| 15 | python -m vllm.entrypoints.openai.api_server \ |
| 16 | --model allenai/Molmo2-8B \ |
| 17 | --trust-remote-code \ |
| 18 | --dtype bfloat16 \ |
| 19 | --port 8122 \ |
| 20 | --gpu-memory-utilization 0.5 \ |
| 21 | --max-model-len 4096 \ |
| 22 | --max-num-batched-tokens 4096 |
| 23 | |
| 24 | # Smoke-test |
| 25 | curl -s http://127.0.0.1:8122/v1/models | jq '.data[0].id' |
| 26 | |
| 27 | # Point the bundle at it |
| 28 | export GAP_MOLMO_BASE_URL=http://127.0.0.1:8122/v1 |
| 29 | ``` |
| 30 | |
| 31 | Operational notes from the dev tree: pin Molmo to its own GPU when running |
| 32 | alongside other perception services — under heavy parallel evaluation it |
| 33 | becomes the throughput bottleneck if co-located; on a dedicated GPU you can |
| 34 | push `--gpu-memory-utilization 0.85 --max-num-batched-tokens 8192` for ~2× |
| 35 | perception throughput. The server can also run on a remote machine and be |
| 36 | port-forwarded in (the 4090 real-robot profile did exactly this). |
| 37 | |
| 38 | ## Config |
| 39 | |
| 40 | | Env | Meaning | Default | |
| 41 | |----------------------|--------------------------------------|----------------------| |
| 42 | | `GAP_MOLMO_BASE_URL` | vLLM OpenAI-compatible base URL | — (required) | |
| 43 | | `GAP_MOLMO_MODEL` | Model name served by vLLM | `allenai/Molmo2-8B` | |
| 44 | |
| 45 | ## Notes |
| 46 | |
| 47 | - `molmo.point_prompt` sends the canonical `"Point at <query>"` prompt and |
| 48 | parses all four Molmo point output formats (Molmo2 `<points coords=...>`, |
| 49 | Molmo1 `<point x= y=>`, legacy `<points x1= y1= ...>`, plain `x, y` |
| 50 | fallback), converting normalized coordinates to pixels. `found=False` |
| 51 | means the model emitted no parseable point. |
| 52 | - `molmo.query_yes_no` coerces with the source-verbatim rule: answer is true |
| 53 | iff `"yes"` appears in the lowercased reply. |
| 54 | - Backend unreachable after 3 retries raises `ToolError` (route `on_error`). |
People who liked this also viewed
6 picksType
- find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 2026
- grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026
- vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026
- improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026
- setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026
- triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026
