claude-video-vision
starsforks
How to install claude-video-vision?
jordanrendric/claude-video-vision
$git clone https://github.com/jordanrendric/claude-video-visionGive Claude the ability to watch and understand videos.
README.md
| 1 | <p align="center"> |
| 2 | <img src="./assets/hero.avif" alt="claude-video-vision" width="100%" /> |
| 3 | </p> |
| 4 | |
| 5 | # Claude Code Video Vision |
| 6 | |
| 7 | Give Claude the ability to **watch and understand videos**. |
| 8 | |
| 9 | A Claude Code plugin that extracts frames via ffmpeg and processes audio via multiple backends (Gemini API, local Whisper, or OpenAI API). Claude receives frames as images and audio transcription with timestamps — the plugin is a **perception layer**, not an interpretation layer. |
| 10 | |
| 11 | ## Features |
| 12 | |
| 13 | - **Multimodal perception** — Claude sees video frames directly and reads audio transcriptions with timestamps |
| 14 | - **YouTube URL support** — Pass a YouTube URL directly; the MCP server downloads it with `yt-dlp`, preserving source metadata and captions for context |
| 15 | - **Flexible backends** — Choose between cloud APIs or fully local processing |
| 16 | - **Adaptive extraction** — Claude adjusts fps, time range, and resolution based on your question |
| 17 | - **Auto-installation** — Whisper models download automatically on first use |
| 18 | - **Interactive setup wizard** — `/setup-video-vision` walks you through configuration |
| 19 | |
| 20 | ## Quick Start |
| 21 | |
| 22 | ### 1. Install the plugin |
| 23 | |
| 24 | Inside Claude Code, run these commands **one at a time**: |
| 25 | |
| 26 | ``` |
| 27 | /plugin marketplace add https://github.com/jordanrendric/claude-video-vision |
| 28 | ``` |
| 29 | |
| 30 | Then: |
| 31 | |
| 32 | ``` |
| 33 | /plugin install claude-video-vision |
| 34 | ``` |
| 35 | |
| 36 | The MCP server will auto-install via `npx` from [npm](https://www.npmjs.com/package/claude-video-vision) on first use — no build step required. |
| 37 | |
| 38 | Alternative: local development |
| 39 | |
| 40 | ```bash |
| 41 | git clone https://github.com/jordanrendric/claude-video-vision.git |
| 42 | claude --plugin-dir /path/to/claude-video-vision |
| 43 | ``` |
| 44 | |
| 45 | ### 2. Configure |
| 46 | |
| 47 | Inside Claude Code, run the interactive wizard: |
| 48 | |
| 49 | ``` |
| 50 | /setup-video-vision |
| 51 | ``` |
| 52 | |
| 53 | It will walk you through backend selection, whisper configuration (if local), frame options, and dependency verification. |
| 54 | |
| 55 | ## Usage |
| 56 | |
| 57 | ### Slash command |
| 58 | |
| 59 | ``` |
| 60 | /watch-video path/to/video.mp4 |
| 61 | /watch-video tutorial.mp4 "what language is used in this tutorial?" |
| 62 | /watch-video https://www.youtube.com/watch?v=... "summarize this video" |
| 63 | ``` |
| 64 | |
| 65 | ### Conversational |
| 66 | |
| 67 | Just mention a video file or YouTube URL — Claude will detect it: |
| 68 | |
| 69 | > "analyze this video for me: ~/Downloads/demo.mp4" |
| 70 | > |
| 71 | > "take a look at the first second of ~/videos/bug-report.mov" |
| 72 | > |
| 73 | > "summarize this YouTube Short: https://www.youtube.com/shorts/..." |
| 74 | |
| 75 | Claude adapts parameters automatically: |
| 76 | - "the first second" → extracts at original fps from `00:00:00` to `00:00:01` |
| 77 | - "summarize this 1h lecture" → low fps, full duration |
| 78 | - "what text is on screen at 1:30?" → high resolution, narrow time window |
| 79 | |
| 80 | ## Backends |
| 81 | |
| 82 | | Backend | Audio processing | Cost | Setup | |
| 83 | |---------|------------------|------|-------| |
| 84 | | **Gemini API** | Native (speech + non-speech events) | Free tier: 1500 req/day | `GEMINI_API_KEY` env var | |
| 85 | | **Local (Whisper)** | `whisper.cpp` or Python `openai-whisper` | Free, fully offline | `brew install whisper-cpp` + auto model download | |
| 86 | | **OpenAI API** | OpenAI Whisper API | Paid per usage | `OPENAI_API_KEY` env var | |
| 87 | |
| 88 | **All backends** extract video frames via ffmpeg — Claude always has direct visual access. |
| 89 | |
| 90 | ## Architecture |
| 91 | |
| 92 | ``` |
| 93 | ┌───────────────────────────────────────────────────────┐ |
| 94 | │ Claude Code (your session) │ |
| 95 | │ │ |
| 96 | │ /watch-video ──→ Skill: video-perception │ |
| 97 | │ │ │ |
| 98 | │ ▼ │ |
| 99 | │ MCP tool: video_watch │ |
| 100 | │ │ │ |
| 101 | └────────────────────────┼──────────────────────────────┘ |
| 102 | │ |
| 103 | ▼ |
| 104 | ┌────────────────────────────────────┐ |
| 105 | │ MCP Server (Node.js) │ |
| 106 | │ │ |
| 107 | │ ┌──────────┐ ┌──────────────┐ │ |
| 108 | │ │ ffmpeg │ │ Audio backend│ │ |
| 109 | │ │ frames │ ║ │ (parallel) │ │ |
| 110 | │ └─────── |
People who liked this also viewed
6 picksType
- fastmcp🚀 The fast, Pythonic way to build MCP servers and clients.MCP ServersJul 2026
- playwright-mcpPlaywright MCP serverMCP ServersJul 2026
- ext-appsOfficial repo for spec & SDK of MCP Apps protocol - standard for UIs embedded AI chatbots, served by MCP serversMCP ServersJul 2026
- langchain-mcp-adaptersLangChain 🔌 MCPMCP ServersJul 2026
- mcporterCall MCPs via TypeScript, masquerading as simple TypeScript API. Or package them as cli.MCP ServersJul 2026
- mcp-proxyA bridge between Streamable HTTP and stdio MCP transportsMCP ServersJul 2026
