🤟 Gesture Connect

A complete web application for sign language recognition and communication, built with React, Flask, Mediapipe, and TensorFlow.

Features

🧍‍♂ App Flow

1. Landing Page:
   • Create Profile or Continue with existing profile
   • Choose from three roles:
     • 👶 Child - Sign language user
     • 🧠 Learner - Learning sign language
     • ❤ Caretaker - Supporting communication

👶 Child Interface

• Live webcam feed with hand tracking
• Mediapipe hand-tracking with red dots on hand landmarks and green connecting lines
• Real-time gesture recognition using TensorFlow
• Text and voice output using Web Speech API
• Detected gestures are automatically spoken aloud

Tech Stack

• Frontend: React + TailwindCSS
• Backend: Flask (Python) with Flask-RESTful
• Machine Learning: Mediapipe + TensorFlow
• Database: SQLite (for profiles and custom gestures)
• Communication: Flask-RESTful API + axios on frontend
• Voice: Web Speech API (text-to-speech)

Installation

Prerequisites

• Python 3.8+
• Node.js 16+
• npm or yarn

Backend Setup

1. Navigate to the project root directory:

   cd gesture_connect

2. Create and activate a virtual environment (recommended):

   python -m venv venv
   # On Windows:
   venv\Scripts\activate
   # On macOS/Linux:
   source venv/bin/activate

3. Install Python dependencies:

   pip install -r requirements.txt

   This will install:

   • flask
   • flask-cors
   • flask-restful
   • mediapipe
   • tensorflow
   • opencv-python
   • numpy

4. Start the Flask server:

   cd server
   python app.py

   The backend will run on http://localhost:5000

Frontend Setup

1. Navigate to the client directory:

   cd client

2. Install dependencies:

   npm install

   This will install:

   • react
   • react-dom
   • react-router-dom
   • axios
   • tailwindcss
   • postcss
   • autoprefixer

3. Start the React development server:

   npm start

   The frontend will run on http://localhost:3000

Usage

1. Start the backend server (from server/ directory):

   python app.py

2. Start the frontend (from client/ directory):

   npm start

3. Open your browser and navigate to http://localhost:3000

4. Create a profile:

   • Click "Create Profile"
   • Enter your name
   • Select your role (Child, Learner, or Caretaker)
   • Click "Start"

5. For Child Dashboard:

   • Click "Start Camera" to begin webcam feed
   • Position your hand in front of the camera
   • Make sign language gestures
   • Red dots will show hand landmarks
   • Green lines will connect the landmarks
   • Detected gestures will be displayed and spoken aloud

Project Structure

gesture_connect/
├── client/                 # React frontend
│   ├── public/
│   │   └── index.html     # HTML template with Mediapipe scripts
│   ├── src/
│   │   ├── pages/
│   │   │   ├── LandingPage.js      # Landing page with role selection
│   │   │   └── ChildDashboard.js   # Child dashboard with webcam
│   │   ├── App.js         # Main app with routing
│   │   └── index.js       # React entry point
│   ├── package.json       # Frontend dependencies
│   └── tailwind.config.js # TailwindCSS configuration
├── server/                # Flask backend
│   ├── app.py            # Flask app with API endpoints
│   └── gesture_service.py # Mediapipe and gesture recognition
├── requirements.txt       # Python dependencies
└── README.md             # This file

API Endpoints

Profiles

• GET /api/profiles - Get all profiles
• GET /api/profiles/<id> - Get profile by ID
• POST /api/profiles - Create a new profile

  {
    "name": "John Doe",
    "role": "child"
  }

Gesture Processing

• POST /api/process-gesture - Process hand landmarks and classify gesture

  {
    "landmarks": [
      [
        {"x": 0.5, "y": 0.5, "z": 0.0},
        ...
      ]
    ]
  }

Gesture Recognition

The application currently recognizes the following gestures:

• OK - Thumb and index finger forming a circle
• Peace - Index and middle finger extended
• Thumbs Up - Thumb extended upward
• Hello - All fingers extended (open hand)
• Fist - All fingers closed

Note: The gesture recognition uses rule-based classification. For production, train a TensorFlow model with your own gesture dataset.

Web Speech API

The application uses the browser's Web Speech API for text-to-speech. Detected gestures are automatically spoken aloud when recognized with sufficient confidence (>50%).

Database

The application uses SQLite to store:

• Profiles - User profiles with name, role, and creation date
• Custom Gestures - Custom gesture definitions (for future implementation)

Development

Adding New Gestures

To add new gestures, modify the classify_gesture method in server/gesture_service.py:

def classify_gesture(self, landmarks: List[Dict]) -> Dict:
    # Add your gesture recognition logic here
    # Return: {'gesture': 'GestureName', 'text': 'Display Text', 'confidence': 0.0-1.0}

Training a TensorFlow Model

1. Collect gesture data (hand landmarks)
2. Train a TensorFlow model
3. Save the model to server/models/gesture_model.h5
4. Update gesture_service.py to load and use the trained model

Troubleshooting

Camera Not Working

• Ensure camera permissions are granted in your browser
• Check that no other application is using the camera
• Try using a different browser (Chrome recommended)

Mediapipe Not Loading

• Check your internet connection (Mediapipe loads from CDN)
• Clear browser cache and refresh
• Check browser console for errors

Backend Connection Issues

• Ensure Flask server is running on port 5000
• Check CORS settings if accessing from different origin
• Verify flask-cors is installed

Future Enhancements

• Learner dashboard for practicing sign language
• Caretaker dashboard for monitoring and support
• Custom gesture training and recognition
• TensorFlow model training interface
• Multi-language support
• Gesture history and analytics
• Real-time communication between users

License

This project is open source and available under the MIT License.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Support

For issues and questions, please open an issue on the project repository.