🤖 Dual-Arm Robotic System Synchronisation

Visual-Guided Decentralised Coordination in PyBullet

📌 Overview

This project implements a dual-arm robotic system in simulation, where:

  • A UR3 manipulator (Worker) performs object manipulation and trajectory execution
  • A Franka Emika Panda (Observer) provides visual feedback via an eye-in-hand camera

The system demonstrates decentralised coordination, where:

No direct joint/state sharing exists between robots — coordination emerges purely through vision-based feedback and control loops.

The primary task involves: - Picking a cube using visual estimation

  • Executing a smooth infinity (∞) trajectory
  • Maintaining continuous visual tracking under noise and occlusion

🧠 Core Contributions

  • 🔄 Decentralised dual-arm coordination
  • 👁ī¸ Visual servoing using image-space error
  • 📐 Mathematical trajectory modelling (Lissajous curve)
  • ⚙ī¸ Physics-based simulation (PyBullet)
  • 📊 Logging + performance evaluation pipeline
  • 🧩 Robust state-machine-driven control (both robots)

🏗ī¸ System Architecture

        Camera (Franka - Eye-in-Hand)
                    │
        RGB Frame → HSV Detection → Centroid (cx, cy)
                    │
            Pixel Error (ex, ey)
                    │
         v_cam → v_world (via R_cam)
                    │
     Jacobian-based Control (DLS Inverse)
                    │
         Franka Joint Velocity Update
                    │
        ─────────────────────────────
                    │
        Stable 3D Estimate (Y, Z)
                    │
              UR3 State Machine
                    │
    Pick → Lift → Infinity Trajectory

⚙ī¸ Mathematical Foundations

📌 Infinity Trajectory (Lissajous Curve)

y(t) = C_Y + A_Y sin(Ī‰t)
z(t) = C_Z + A_Z sin(2Ī‰t)

📌 Visual Servoing Control

Pixel error:

e_x = c_x - W/2
e_y = c_y - H/2

Camera velocity:

v_cam = [ 0,
          K_pix * e_x,
         -K_pix * e_y ]

Jacobian-based control:

q_dot = J^T (J J^T + ÎģI)^(-1) v_world

🔄 Control Logic

đŸĻž UR3 (Worker Arm)

Finite state machine:

  1. WAIT_FOR_CAM – waits for stable visual estimate
  2. APPROACH – moves to pick position
  3. LIFT – lifts object to safe pose
  4. MOVE_∞ – executes infinity trajectory

Smoothing:

p_smooth = Îą p_ref + (1 - Îą) p_smooth

👁ī¸ Franka (Observer Arm)

Detection Pipeline

  • RGB → HSV conversion
  • Red colour thresholding
  • Centroid detection
Tracking States
  • SEARCH
  • TRACKING
  • TEMP_LOST
  • LOST
Control
  • Pixel error → velocity
  • Jacobian inverse
  • Velocity clipping

đŸ“Ļ Movement_with_logs.py (Enhanced Implementation)

📊 Logging System

  • Pixel error: e_x(t), e_y(t)
  • End-effector error: e_EE(t) = || p_EE(t) - p_ref(t) ||

🔁 Stability Improvements

  • First-order smoothing
  • Buffered cube position estimation
  • Cluster-based validation

🧩 Robustness Features

  • Handles noisy detections
  • Handles occlusions
  • Multi-frame consistency checks
  • Kalman Filter for pose estimation

🚀 Tech Stack

  • Python
  • PyBullet
  • NumPy
  • OpenCV

â–ļī¸ How to Run

git clone https://github.com/DyutideeptaB/Dual-Arm_Robotic_System_Synchronisation.git
cd Dual-Arm_Robotic_System_Synchronisation
pip install -r requirements.txt
python Final_with_logs.py

📊 Outputs

  • Pixel error vs time
  • End-effector tracking error
  • Infinity trajectory (reference vs actual)

đŸ”Ŧ Applications

  • Collaborative robotics
  • Industrial automation
  • Visual servoing research
  • Multi-agent systems

📈 Future Work

  • Reinforcement learning integration
  • Real robot deployment
  • Multi-object tracking
  • Dynamic environments

👤 Author

Dyutideepta Banerjee
Physics + AI | Simulation Driven Systems | Computer Vision

⭐ Final Note

This project demonstrates how coordinated robotic behaviour can emerge purely from perception-driven feedback, without centralised control.

GitHub View GitHub Repository