Introduction to TogetheROS.Bot
TogetheROS.Bot is a robot operating system launched by D-Robotics for robot manufacturers and ecosystem developers. It aims to unlock the intelligent potential of robot scenarios, helping ecosystem developers and business customers develop robots efficiently and conveniently, and build competitive intelligent robot products.
TogetheROS.Bot supports running on RDK platforms, and also provides a simulator version for X86 platforms. RDK platforms cover all the features shown in the diagram below. X86 platforms support experiencing some features through image playback, improving algorithm development and verification efficiency, and enabling quick migration to RDK platforms.
TogetheROS.Bot source code is hosted on GitHub under the D-Robotics organization.
Communication Components
Communication is a functional optimization and extension based on the core communication components of ROS2 Foxy/Humble/Jazzy.
Main features are as follows:
The blue parts in the diagram are optimized or newly added modules. The main features of TogetheROS.Bot are as follows:
- Provides "hobot_sensor" to adapt to commonly used robot sensors, saving development time and focusing on core competitiveness
- Provides "hobot_dnn" to simplify on-board algorithm model inference and deployment, unleashing BPU computing power and lowering the barrier to using intelligent algorithms
- Provides "hobot_codec" for hardware-software accelerated video encoding and decoding, saving CPU resources and improving parallel processing capability
- Provides "hobot_cv" for hardware-software accelerated common CV operators, saving CPU resources and improving runtime efficiency
- Provides "hobot Render" for Web and HDMI dynamic visualization, rendering algorithm results in real time (Web only), facilitating demonstration and debugging
- Adds "zero-copy" inter-process zero-copy communication mechanism, reducing data transmission latency and system resource consumption
- Rich middleware software debugging and performance tuning tools, improving problem localization efficiency and facilitating system performance optimization
- Fully compatible with ROS2 Foxy/Humble/Jazzy interfaces, facilitating reuse of rich ROS tool packages and accelerating prototype verification
- Supports minimal and modular trimming, facilitating deployment on resource-constrained embedded products as needed
Boxs Algorithm Repository
Boxs is an intelligent algorithm package based on TogetheROS.Bot launched by D-Robotics for robot manufacturers and ecosystem developers, aiming to improve the efficiency of integrating and deploying intelligent robot algorithms based on the D-Robotics RDK robot operating system.
- Image detection algorithms such as FCOS, YOLO, FasterRCNN, Efficientdet, Mobilenet_ssd;
- Image classification models such as Mobilenet
- Semantic segmentation models such as Unet
- Application algorithm models such as human body detection and tracking, gesture recognition, hand keypoint detection, monocular elevation network, monocular 3D detection, speech processing, etc.
Apps Application Examples
Apps are algorithm application examples developed based on D-Robotics RDK robot operating system Communication and Boxs, aiming to connect the complete pipeline of image input, perception, and strategy, demonstrate application effects, and accelerate customer demo development efficiency.
Glossary
| Term | Description |
|---|---|
| zero-copy | Inter-process zero-copy communication method |
| hobot dnn | Model inference function encapsulation based on BPU |
| SLAM | Simultaneous Localization and Mapping |
| DOA | Direction of Arrival |
| ASR | Automatic Speech Recognition |
| TogetheROS.Bot | TogetheROS.Bot Robot Operating System |
| tros.b | Abbreviation for TogetheROS.Bot |
Feature Support List
| Feature | X3 | X5 | S100 | S600 |
|---|---|---|---|---|
| Data Collection hobot_sensor | ✔ | ✔ | ✔ | ✔ |
| Data Display hobot_render | ✔ | ✔ | ✔ | ✔ |
| Image Encoding/Decoding hobot_codec | ✔ | ✔ | ✔ | ✔ |
| Image Processing Acceleration hobot_cv | ✔ | ✔ | ✔ | ✔ |
| Data Communication zero-copy | ✔ | ✔ | ✔ | ✔ |
| Model Inference hobot_dnn | ✔ | ✔ | ✔ | ✖ |
| Image Publishing Tool hobot_image_publisher | ✔ | ✔ | ✔ | ✔ |
| Text-to-Speech hobot_tts | ✔ | ✔ | ✔ | ✔ |
| Object Detection | YOLO: v2 v3 v5 v8 v10 FCOS MobileNet_SSD EfficientNet_Det | YOLO: v2 v3 v5 v8 v10 FCOS MobileNet_SSD EfficientNet_Det | YOLO: v2 v3 v5 v8 v10 | YOLO: v2 v3 v5 |
| Open-Vocabulary Object Detection YOLO-World | ✖ | ✔ | ✖ | ✖ |
| Open-Vocabulary Object Detection DOSOD | ✖ | ✔ | ✔ | ✔ |
| Image Classification mobilenetv2 | ✔ | ✔ | ✔ | ✔ |
| Image Segmentation mobilenet_unet YOLOv8-Seg | ✔ | ✔ | ✔ | mobilenet_unet |
| Segment Anything mono_edgesam | ✖ | ✔ | ✔ | ✔ |
| Segment Anything mono_mobilesam | ✖ | ✔ | ✖ | ✖ |
| Human Body Detection | mono2d_body_detection | mono2d_body_detection | mono2d_yolo_pose | mono2d_yolo_pose |
| Hand Keypoints | hand_lmk_detection | hand_lmk_detection | hand_lmk_gesture_mediapipe | ✖ |
| Gesture Recognition | hand_gesture_detection | hand_gesture_detection | hand_lmk_gesture_mediapipe | ✖ |
| Face Age Detection and Corresponding APP Examples | ✔ | ✔ | ✖ | ✖ |
| Face 106 Keypoint Detection and Corresponding APP Examples | ✔ | ✔ | ✖ | ✖ |
| Human Body Following | ✖ | ✔ | ✔ | ✔ |
| BEV | ✖ | ✖ | ✔ | ✖ |
| LiDAR Object Detection Algorithm CenterPoint | ✖ | ✖ | ✔ | ✖ |
| Stereo Depth Algorithm | ✔ | ✔ | ✔ | ✖ |
| Stereo OCC Algorithm | ✖ | ✔ | ✔ | ✖ |
| Visual Inertial Odometry hobot_vio | ✖ | ✔ | ✖ | ✖ |
| Intelligent Speech hobot_audio and Speech-Related Examples | ✔ | ✔ | ✖ | ✖ |
| Intelligent Speech Sensevoice | ✖ | ✔ | ✔ | ✔ |
| Vision-Language Model hobot_llamacpp | ✖ | ✔ | ✔ | ✖ |
| DeepSeek Large Language Model hobot_xlm | ✖ | ✖ | ✔ | ✖ |
| Text-Image Feature Retrieval hobot_clip | ✖ | ✔ | ✔ | ✖ |
| Optical Flow Estimation mono_pwcnet | ✖ | ✔ | ✖ | ✖ |
| 4.1 SLAM Mapping | ✔ | ✔ | ✔ | ✔ |
| 4.2 Navigation2 | ✔ | ✔ | ✔ | ✖ |
| 4.9 Intelligent Box | ✔ | ✔ | ✔ | ✔ |
| 4.10 Vision-Speech Box | ✖ | ✔ | ✔ | ✖ |