ISL Recognition

A full-stack system that reads 296 Indian Sign Language words from a webcam as they're signed, builds them into sentences and learns new signs from four examples.

Very few hearing people can read Indian Sign Language, and interpreters are scarce outside big institutions. Everyday conversations, at a counter or in a classroom, still depend on someone in the room knowing the language.

The goal was a recogniser that needs nothing more than a webcam and a browser, runs live, and can be taught a new sign on the spot.

1. 1

   MediaPipe Holistic turns each frame into 75 landmarks (both hands and the pose), a 258-value feature vector. Missing hands are zero-padded so timing stays aligned.

2. 2

   A sliding window of 45 frames feeds a two-layer bidirectional LSTM (256 units each) with an attention layer over time and a dense classification head.

3. 3

   Temperature scaling (T = 3.0) calibrates the confidence, so the interface only commits to a word when the model is genuinely sure.

4. 4

   FastAPI serves REST and WebSocket endpoints; long video uploads go through a Celery and Redis job queue, and the whole stack starts with one docker compose command.

• 92.2% validation and 89.9% test accuracy on the INCLUDE dataset (IIT Madras), across 296 word classes.
• Trained in about 25 minutes on a laptop RTX 4060, with label smoothing and a cosine learning-rate schedule.