ROSGPT_Vision
starsforks
How to install ROSGPT_Vision?
bilel-bj/rosgpt_vision
$git clone https://github.com/bilel-bj/ROSGPT_VisionInspired by ROSGPT. Both projects aim to bridge the gap between robotics, natural language understanding, and image analysis.
README.md
| 1 | # ROSGPT_Vision: Commanding Robots Using Only Language Models' Prompts |
| 2 | |
| 3 | [Bilel Benjdira](https://github.com/bilel-bj), [Anis Koubaa](https://github.com/aniskoubaa) and [Anas M. Ali](https://github.com/AnasHXH) |
| 4 | [](https://arxiv.org/abs/2308.11236) |
| 5 | [](https://youtu.be/nYnpzSCaMyw) <a href="https://www.sciencedirect.com/science/article/abs/pii/S0167739X25000184"> |
| 6 | <img src="https://sdfestaticassets-eu-west-1.sciencedirectassets.com/prod/44c3817e58b49348a73e63fb998fb7b2924522e1/image/elsevier-non-solus.png" alt="arXiv" width="100" height="50"> |
| 7 | </a> |
| 8 | |
| 9 | <img src="https://github.com/bilel-bj/ROSGPT_Vision/blob/main/paper.png" width="900" height="600"/> |
| 10 | **Robotics and Internet of Things Lab (RIOTU Lab), Prince Sultan University, Saudi Arabia** |
| 11 | |
| 12 | Inspired by [ROSGPT](https://github.com/aniskoubaa/rosgpt). Both projects aim to bridge the gap between robotics, natural language understanding, and image analysis. |
| 13 | |
| 14 | Collaborators who want to participate in this project, are very welcome. |
| 15 | |
| 16 | ------------------------------------------------------------------------------------------------------------------------------------------ |
| 17 | - **ROSGPT_Vision** is a new robotic framework dsigned to command robots using only two prompts: |
| 18 | - a **Visual Prompt** (for visual semantic features), and |
| 19 | - an **LLM Prompt** (to regulate robotic reactions). |
| 20 | - It is based on a new robotic design pattern: **Prompting Robotic Modalities (PRM)**. |
| 21 | - **ROSGPT_Vision** is used to develop **CarMate**, a robotic application for monitoring driver distractions and providing real-time vocal notifications. It showcases cost-effective development. |
| 22 | - We demonstrated how to optimize the prompting strategies to improve the application. |
| 23 | - LangChain framework is used by to easily customize prompts. |
| 24 | - More details are described in the academic paper "ROSGPT_Vision: Commanding Robots using only Language Models' Prompts". |
| 25 | |
| 26 | |
| 27 | # Video Demo |
| 28 | An illustrative video demonstration of ROSGPT_Vision is provided: |
| 29 | [](https://youtu.be/nYnpzSCaMyw) |
| 30 | |
| 31 | ## Table of Contents |
| 32 | |
| 33 | - [Overview](#overview) |
| 34 | - [ROSGPT_Vision diagram](#rosgpt_vision-diagram) |
| 35 | - [Prompting Robotic Modalities (PRM) Design Pattern](#prompting-robotic-modalities-prm-design-pattern) |
| 36 | - [CarMate Application](#carmate-application) |
| 37 | - [Installation](#installation) |
| 38 | - [Usage](#usage) |
| 39 | - [Citation](#citation) |
| 40 | - [License](#license) |
| 41 | - [Acknowledgement](#acknowledgement) |
| 42 | - [Contribute](#contribute) |
| 43 | |
| 44 | ## Overview |
| 45 | |
| 46 | **ROSGPT_Vision** offers a unified platform that allows robots to perceive, interpret, and interact with visual data through natural language. The framework leverages state-of-the-art language models, including [LLAVA](https://github.com/haotian-liu/LLaVA), [MiniGPT-4](https://github.com/Vision-CAIR/MiniGPT-4), and [Caption-Anything](https://github.com/facebookresearch/segment-anything), to facilitate advanced reasoning about image data. [LangChain](https://github.com/langchain-ai/langchain) is used for easy customization of the prompts. The provided implementation includes the **CarMate** application, a driver monitoring and assistance system designed to ensure safe and efficient driving experiences. |
| 47 | ## ROSGPT_Vision diagram |
| 48 | <img src="https://github.com/bilel-bj/ROSGPT_Vision/blob/main/ROSGPT_Vision.png" width="900" height="600"/> |
| 49 | |
| 50 | ## Prompting Robotic Modalities (PRM) Design Pattern |
| 51 | - A new design approach emphasizing modular and individualized sensory queries. |
| 52 | - Uses specific **Modality Language Models (MLM)** for textual interpretations of inputs, like the **Vision Language Model (VLM)** for visual data. |
| 53 | - Ensures precise data collection by treating each sensory input separately. |
| 54 | - **Task Modality**'s Role: Serves as the central coordinator, synthesizing data from various modalities. |
| 55 | |
| 56 | ** for more information go to [和多智能体(Multi-Agent)，探究AI写小说能力的边界PromptsSep 2024
- ANWS🚀 A spec-driven workflow framework for AI-assisted development. Empowers prompt engineers to build production-ready software through structured PRD → Architecture → Task decomposition. Designed for Antigravity to enforce design-first principles and prevent architectural drift in vibe coding.🗺️Antigravity-Workflow-SystemPromptsMay 2026
