Graph-Fused Vision-Language-Action Models for Semantically Safe Dual-Robot Control via Control Barrier Functions

---

Official code and support material for our work on semantically safe dual-robot manipulation using Graph-Fused Vision-Language-Action (GF-VLA) models with Control Barrier Functions (CBFs). This framework enables language-conditioned dual-arm control while guaranteeing collision-free motion through real-time CBF-based safety filtering.

---

✨ Highlights

• Dual-arm VLA control — UR5e + UR10e coordination for manipulation tasks
• CBF safety filter — Minimal modification of nominal actions for guaranteed obstacle avoidance
• Vision-language grounding — Point cloud + language instruction for semantic understanding
• Real-time QP solver — Efficient CBF-QP formulation for online safety filtering

---

🚀 Quick Start

Installation

# Clone repository
git clone https://github.com/YOUR_ORG/GFVLA_CBF.git
cd GFVLA_CBF

# Install dependencies
pip install -r requirement.txt

Run Dual-Arm Demo

# Build "VLA" letters with Jenga blocks (CBF enabled by default)
python hardware/demo_build_vla_2d.py --ur5e-ip 192.168.1.101 --ur10e-ip 192.168.1.102

# Simulation mode (no hardware)
python hardware/demo_build_vla_2d.py --simulation

---

🛡️ Control Barrier Function (CBF) Module

The CBF module provides safety filtering that minimally modifies VLA-predicted actions to ensure collision-free motion during dual-robot manipulation.

Features

Feature	Description
Obstacle avoidance	Keeps end-effectors at safe distance from obstacles (point cloud / detected blocks)
Inter-arm collision	Maintains minimum distance between left and right arms
Workspace bounds	Enforces Cartesian workspace limits
CBF-QP formulation	Solves QP to find closest safe action to nominal control

Usage

CBF is enabled by default. To disable:

python hardware/demo_build_vla_2d.py --no-cbf

Configuration

In HardwareConfig.safety:

Parameter	Default	Description
use_cbf_filter	True	Enable/disable CBF filter
cbf_obstacle_margin	0.08 m	Minimum distance from obstacles
cbf_inter_arm_margin	0.12 m	Minimum distance between arms

Programmatic Usage

from hardware import DualArmHardwareInterface, Obstacle, obstacles_from_point_cloud

# Create hardware interface
hardware = DualArmHardwareInterface(config)

# Obstacles from point cloud (e.g., depth camera)
obstacles = obstacles_from_point_cloud(point_cloud, safety_margin=0.05)

# Option 1: Update obstacles globally
hardware.set_cbf_obstacles(obstacles)

# Option 2: Pass per-action
hardware.execute_action(action, blocking=True, obstacles=obstacles)

---

📁 Project Structure

GFVLA_CBF/
├── hardware/           # Dual-arm hardware & CBF
│   ├── cbf.py          # Control Barrier Function module
│   ├── dual_arm_hardware.py
│   ├── vla_integration.py
│   └── demo_build_vla_2d.py
├── models/             # VLA model definitions
├── vla/                # VLA training & datasets
├── scripts/            # Training scripts
└── assets/             # Figures & resources

---

📄 License

This project is released for research purposes. See individual files for license details.