Hand AR Tracker

Real-time hand AR system with ASCII art effects, holographic ribbon, and gesture controls built with Python + MediaPipe.

---

Table of Contents

1. Overview
2. Features
3. Hardware Requirements
4. Installation
5. Quick Start
6. Gesture Controls
7. Keyboard Controls
8. Effect Modes
9. Configuration
10. Project Architecture
11. License

---

Overview

Hand AR Tracker captures your webcam feed and overlays real-time augmented reality effects driven entirely by hand gestures, spacebar presses, and eye winks no buttons, no keyboard, just your body (plus the spacebar for toggling invert mode).

The core effect is a holographic ASCII ribbon that stretches between your two hands, rendering your silhouette as scrolling ASCII scanlines against a black background. Pressing spacebar inverts the entire effect, and a double wink locks the ribbon's position in 3D space.

---

Features

• Dual-hand tracking - up to two hands simultaneously via MediaPipe Hands
• Holographic ASCII ribbon - scanline particle effect renders your silhouette as ASCII art between both hands
• Space-to-reverse - press the spacebar to flip the effect: real video inside ribbon, ASCII art outside
• Double-wink lock - wink twice in one second to freeze the ribbon position in space
• Person + object segmentation - MediaPipe Selfie Segmentation + background subtraction
• Live FPS counter with rolling average
• Click-to-flip camera rotation button in the HUD
• Auto GPU/CPU selection (CUDA or CPU)
• Fully configurable via config.yaml

---

Hardware Requirements

Minimum

Component	Requirement
Camera	Any USB/built-in webcam
Python	3.12.7
RAM	4 GB
OS	Windows 10+, macOS 12+, Linux

Recommended

Component	Recommendation
GPU	NVIDIA GPU with CUDA
Camera	1080p webcam at 60 fps
RAM	8 GB+
CPU	AMD Ryzen 5 / Intel i5 or better

---

Installation

Windows (one-click)

1. Clone the repo

   git clone https://github.com/qenev/hand-ar-tracker.git
   cd hand-ar-tracker

2. Run the installer - creates the virtual environment and installs all dependencies:

   install.bat
   

3. Launch the tracker:

   start.bat
   

Manual installation

git clone https://github.com/qenev/hand-ar-tracker.git
cd hand-ar-tracker

python -m venv venv

# Windows
venv\Scripts\activate
# macOS / Linux
source venv/bin/activate

pip install -r requirements.txt

python main.py

---

Quick Start

start.bat        # Windows one-click launch
# or
python main.py   # manual

The app opens your webcam. Show both hands to the camera to start.

---

Tutorial: Step‑by‑step Walkthrough

1. Launch – run start.bat (Windows) or python main.py.
2. Calibration – the system automatically detects your hands and face. Wait a couple of seconds for the FPS counter to stabilize.
3. Enable the ribbon – perform a double‑pinch (thumb and index together on each hand, then touch the pinched tips). The holographic ASCII ribbon appears between your hands.
4. Reverse mode – press spacebar. The ribbon now shows the real video inside while the rest of the scene turns ASCII.
5. Lock the ribbon – wink one eye twice within one second. The ribbon’s position freezes in space, allowing you to move your hands without moving the effect.
6. Unlock – repeat the double‑wink to release the lock.
7. Toggle off – repeat the double‑pinch gesture to turn the ribbon off.
8. Adjust settings – edit config.yaml to change camera resolution, confidence thresholds, colors, or to enable extra gesture labels.
9. Exit – press Q or ESC (hand‑skeleton toggle) to close the application.

---

Controls Summary

Control	Action
Double‑pinch touch	Toggle ASCII ribbon ON/OFF
Spacebar	Switch to reverse mode (real video inside ribbon, ASCII outside)
Double wink (one eye)	Lock / unlock ribbon position
Q	Quit application
ESC	Toggle hand skeleton visibility
Delete	Show/hide HUD (FPS, flip button, device label)

---

---

Gesture Controls

Most effects are controlled entirely by body gestures - no keyboard required for tracking or locking the AR effects.

Double Pinch Touch - Toggle Ribbon ON/OFF

Bring both hands in front of the camera, pinch your thumb and index finger together on each hand simultaneously, then touch the two pinched fingertips together.

• First double-pinch: Ribbon turns ON (ASCII scanline effect between hands)
• Second double-pinch: Ribbon turns OFF
• If either hand leaves frame while the ribbon is off, the effect auto-resets

Both hands visible → pinch both → touch pinched tips together → toggle

Double Wink - Lock / Unlock Ribbon Position

Wink one eye (not both - a full blink won't count) twice within 1 second to lock the ribbon at its current position in space. The ribbon will stay frozen even if you move your hands away.

• Double-wink while ribbon is ON: Position locks. You can lower your hands - the ribbon stays.
• Double-wink while ribbon is locked: Unlocks the ribbon so it follows your hands again.

---

Keyboard Controls

Key	Action
Spacebar	Switch to reverse mode (real video inside ribbon, ASCII outside)
Q	Quit the application
ESC	Toggle hand skeleton visibility (dots + lines) on/off
Delete	Toggle the entire HUD (FPS counter, flip button, device label) on/off

HUD - On-screen Controls

Element	Location	Action
FPS counter	Top-left	Live rolling-average frame rate
Flip 180 button	Top-center (clickable)	Click to rotate camera 180° for upside-down mounts
Device label	Top-right	Shows CUDA or CPU

---

Effect Modes

Default Mode (no reverse toggle)

Ribbon OFF:  Normal camera feed
Ribbon ON:   Your silhouette rendered as scrolling ASCII scanlines
             between your two hands, against a black background.
             Rest of frame is plain black.

Reversed Mode (after spacebar press)

Ribbon OFF:  Full frame converted to ASCII art on black background
Ribbon ON:   Inside ribbon = clear real video
             Outside ribbon = ASCII art
             (No border - clean hard edge)

Locked Ribbon (after double wink)

The ribbon polygon stays fixed in screen-space at the position captured when the wink fired. Works in both default and reversed modes.

---

Configuration

All settings live in config.yaml.

Camera

camera:
  index: 0          # webcam index (0 = default)
  width: 1280
  height: 720
  fps: 60
  rotate_180: false  # true if camera is mounted upside-down

Tracking

tracking:
  max_hands: 2
  min_detection_confidence: 0.8
  min_tracking_confidence: 0.7
  model_complexity: 1   # 0 = lite/fast, 1 = full/accurate

Renderer

renderer:
  skeleton_color: [0, 255, 0]     # BGR - green
  keypoint_color: [0, 0, 255]     # BGR - red
  keypoint_radius: 6
  skeleton_thickness: 2
  show_fps: true
  show_coordinates: false
  show_gesture_label: false
  show_hand_label: false
  show_device_label: true

Gestures

gestures:
  enabled: false         # enable named gesture labels (Pinch, Fist, etc.)
  pinch_threshold: 0.05  # thumb-to-index distance to count as a pinch
  fist_threshold: 0.85

Device

device: auto   # "auto" | "cuda:0" | "cpu"

---

Project Architecture

File	Purpose
main.py	Entry point, main loop, wink detection, spacebar toggle, mode routing
tracker.py	MediaPipe Hands wrapper + FaceMesh + SelfieSegmentation
renderer.py	All OpenCV drawing: skeleton, ribbon (ASCII + real), HUD
ascii_processor.py	Vectorised ASCII art renderer (CLAHE + char tiles)
gesture.py	Per-hand gesture classification
config.yaml	All runtime configuration
utils/fps_counter.py	Rolling-window FPS counter
utils/device_utils.py	GPU/CPU auto-detection
utils/math_utils.py	Landmark smoothing and coordinate helpers
install.bat	One-click Windows dependency installer
start.bat	One-click Windows launcher
assets/	Reference images and ASCII character data

---

License

MIT License - see LICENSE for details.