ASL Real-time Detection from Video/Image

Web and mobile application for real-time recognition of ASL signs from a camera or image.

ASL Real-time Detection recognizes American Sign Language letters from a webcam, mobile camera, or imported image. The system detects up to two hands, extracts their landmarks, applies a classification model, and returns the prediction with its confidence level.

01

Problem addressed

Sign language recognition from a camera requires a system capable of accurately detecting hands, interpreting their positions, and providing stable predictions in real-time on the web and on mobile.

02

Objectives

Recognize ASL letters from A to Z.
Recognize del, nothing, and space classes.
Accept images sent via HTTP request.
Process real-time video stream with WebSocket.
Detect up to two hands in an image.
Display the confidence level of each prediction.
Display detected hand landmarks.
Smooth out successive predictions in the video stream.
Allow accumulation of letters in a text area.
Allow translation of accumulated text to Arabic on mobile.

Provide an application accessible on the web, Android, and iOS.

03

Technical solution

The FastAPI backend loads a MediaPipe Hand Landmarker detector and a TorchScript classifier at startup. For each received image, it resizes the image, detects up to two hands, and extracts 21 three-dimensional landmarks per hand. The landmarks are normalized with respect to the wrist and then transformed into a 137-feature vector. The classifier then applies a prediction, a softmax calculation, an argmax, and a confidence threshold set to 0.55. The React/Vite frontend allows the use of a webcam or the import of an image. The Expo mobile application allows the use of the camera, gallery, WebSocket stream, text composition, and translation to Arabic.

04

Architecture

Web User
Mobile User
Web Client React
Vite
Image Import
Webcam Capture
Reconnectable WebSocket
Mobile Client Expo
React Native
Mobile Camera
Image Gallery
Local Settings Storage
Letter Accumulation
Arabic Translation
HTTP Multipart Request
Base64 WebSocket Stream
FastAPI
Health Route
Image Prediction Route
WebSocket Prediction Route
Static Weights Service
Maximum Resizing to 480 pixels
MediaPipe Hand Landmarker
Maximum 2 Hands Detection
Extraction of 21 3D Landmarks per Hand
Normalization relative to Wrist
Construction of 137 Features
TorchScript Classifier
Softmax Calculation
Argmax Selection
Confidence Threshold at 0.55
Majority Vote Smoothing
Prediction Response
Detected Prediction
Confidence Level
Hand Detection State
Optional Landmarks
Number of Hands
Model Artifacts
asl_classifier.pt file
labels.json file
calibration.json file
preprocess.json file
hand_landmarker.task file
Docker
Docker Compose
Backend on port 8091
Frontend on port 5173

05

Documented results

The necessary artifacts for inference are present in the project.
The copies of the classifier have the same SHA-256 hash.
The copies of the MediaPipe model have the same SHA-256 hash.
The model exposes 29 classes.
The classes include the letters A to Z.
The additional classes are del, nothing, and space.
The production build of the React frontend is successful.
The build transforms 38 modules.
Two backend tests are present.
One test checks the health status of the backend.
One test checks the sending of an image.
The backend tests could not be executed in the analyzed environment.
The virtual environment references an inaccessible Python 3.13 interpreter.
The upload test expects an old response structure.
The current response also contains the hands property.
No CI/CD workflow is present.
Docker Compose launches the backend on port 8091.
Docker Compose launches the frontend on port 5173.
No database is present.
No migration is present.
No ORM is used.
The mobile documentation contains outdated translation information.