An end-to-end web-based system for real-time British Sign Language (BSL) recognition using MediaPipe hand tracking and a deep learning classifier deployed in the browser with ONNX Runtime.
This project supports:
- Real-time hand landmark detection (1-hand and 2-hand signs)
- Neural network–based sign classification
- Confidence scoring
- Dynamic gesture recognition using DTW
- Visual feedback with skeleton overlay
- Browser-side inference (no server latency)
- Uses MediaPipe Hands for 3D landmark extraction.
- Detects left and right hands independently.
- Supports both one-hand and two-hand signs.
- Trained in PyTorch
- Exported to ONNX
- Runs fully in browser via onnxruntime-web
- Input:
- 63 features (single hand)
- 126 features (two hands concatenated)
- Output: Softmax probabilities over BSL alphabet & numbers.
- Z-score normalisation using training statistics (
bsl_norm.json) - Ensures inference distribution matches training distribution.
- Sequence matching using Dynamic Time Warping (DTW)
- Used for gestures like "Yes", "No", etc.
- Live skeleton overlay
- Confidence percentage
- Correct / Incorrect indicator
- View gating for camera positioning