ROS2 Vision-Guided Waste Sorting System with IRB360 (CoppeliaSim)

Overview

This project implements a complete perception-to-manipulation pipeline for automated waste sorting using ROS2, a Convolutional Neural Network (CNN), and a simulated IRB360 delta robot in CoppeliaSim.

The system captures images from a simulated camera, classifies objects into waste categories, and performs real-time pick-and-place operations to sort them into the correct bins.

---

Features

• ROS2-based modular architecture (camera, classifier, controller)
• CNN-based image classification (TensorFlow/Keras)
• Real-time integration with CoppeliaSim via Remote API
• Event-driven perception using trigger signals
• Robust synchronization using robot_busy state
• Failure handling (missed grasp → reject to default bin)
• End-to-end autonomous pipeline

---

System Architecture

CoppeliaSim Camera ↓ camera_node ↓ /camera/image_raw ↓ classifier_node (CNN) ↓ /waste/class ↓ controller_node ↓ CoppeliaSim IRB360 Robot

---

Technologies Used

• ROS2 (Humble)
• Python (rclpy)
• TensorFlow / Keras
• OpenCV + cv_bridge
• CoppeliaSim (Edu)
• Remote API (legacy)
• NumPy

---

Waste Categories

The classifier predicts 4 classes:

• Carton
• Glass
• Metal
• Plastic

---

How It Works

1. Object reaches the inspection zone.
2. Conveyor stops and triggers image capture.
3. camera_node publishes image + trigger.
4. classifier_node runs CNN and outputs class.
5. controller_node sends command to robot.
6. IRB360 performs pick-and-place.
7. Conveyor resumes.

Failure Handling

If the robot fails to grasp an object:

• The conveyor resumes automatically.
• The object is sent to a default (gray) bin.
• The system continues without blocking.

---

Performance Evaluation

Metric	Value
Throughput	14 objects/min
Avg Cycle Time	~4.29 s/object
Classification Accuracy	100%
End-to-End Success Rate	96%

---

Installation

1. Clone Repository

git clone https://github.com/YOUR_USERNAME/waste-sorting-ros2.git
cd waste-sorting-ros2

2. Build Workspace

cd ros2_ws colcon build source install/setup.bash

3. Dependencies

Make sure you have installed:

• ROS2 Humble

• Python 3

• TensorFlow

• OpenCV

• cv_bridge

---

Running the System

Start CoppeliaSim

Load the simulation scene

Press Play

Run ROS2 Nodes (in order)

• Terminal 1 ros2 run waste_control controller_node

• Terminal 2 ros2 run waste_perception classifier_node

• Terminal 3 ros2 run waste_perception camera_node

---

Project Structure

ros2_ws/
│
├── src/
│   ├── waste_perception/
│   │   ├── camera_node.py
│   │   ├── classifier_node.py
│   │   ├── sim.py
│   │   └── remoteApi.so
│   │
│   ├── waste_control/
│   │   └── controller_node.py
│
├── models/
│   └── classifier_second.h5

---

Key Challenges Solved

• Synchronization between perception and actuation

• Remote API communication conflicts

• Event-driven classification (trigger-based)

• Avoiding duplicate classification

• Handling failed grasp scenarios

• Multi-node coordination in ROS2

---

Future Work

• Improve CNN accuracy with larger dataset

• Implement real robot deployment

• Use ROS2 actions instead of topics

• Replace legacy Remote API with ZeroMQ API

• Optimize robot motion planning

• Add reinforcement learning for grasping

---

Author

Luisa Chávez Vásquez

M.S. Electrical & Computer Engineering

North Carolina State University