Skip to content

About

An interactive touchless AI whiteboard featuring real-time MediaPipe hand tracking, gesture controls, and automatic stroke-to-shape recognition.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

Β 

History

62 Commits

Folders and files

Repository files navigation

VisionSketch-AI

VisionSketch-AI is an interactive, touchless virtual whiteboard built with Python, OpenCV, MediaPipe, and NumPy. Draw freehand strokes in the air using your index finger via a standard webcam, and let the real-time computer vision engine automatically detect and snap your strokes into clean geometric shapes.


🌟 Key Features

  • πŸ–οΈ Real-Time 21-Landmark Hand Tracking β€” Low-latency hand landmark tracking powered by Google MediaPipe Hands.
  • ✌️ Dual Gesture-Driven Interaction Modes
    • Selection Mode (Index + Middle fingers raised): Select colors, adjust brush slider, set eraser sizes, or export drawings.
    • Drawing Mode (Index finger only): Freehand air-drawing with live visual feedback.
  • 🎨 6 Vibrant Color Brushes & Interactive Toolbar β€” Red, Purple, Green, Blue, Orange, and Pink selectable from an top toolbar.
  • 🎚️ Virtual Brush Thickness Slider β€” Dynamic brush size adjustment (5 px to 50 px) via an interactive on-screen slider.
  • 🧽 3 Eraser Size Presets β€” Quick toggle panel for Small (25 px), Medium (50 px), and Large (80 px) eraser tools.
  • πŸ“ Isolated Layer Shape Recognition & Snap-to-Perfect β€” Automatic shape classification executed strictly on the active stroke layer (stroke_canvas), preserving existing artwork without canvas wipes:
    • πŸ“ Line: Rotated bounding box elongation check ($\text{long} / \text{short} > 5.0$).
    • πŸ”Ί Triangle: 3-corner polygon approximation (approxPolyDP).
    • ⬛ Square: 4-corner polygon with near-unity aspect ratio ($0.9 \le w/h \le 1.1$).
    • β–ˆ Rectangle: 4-corner polygon with non-square aspect ratio.
    • β­• Circle: Compactness / circularity score ($C = \frac{4\pi \cdot \text{Area}}{\text{Perimeter}^2} > 0.60$).
  • ✊ Fist-Hold Canvas Clear β€” Hold a fist gesture for 2.5 seconds to clear the whiteboard, accompanied by an anchor-locked visual countdown timer near your wrist landmark.
  • 🏷️ Visual Shape HUD Overlay β€” Temporary green outline highlight and on-screen label overlay for 3 seconds upon shape detection.
  • πŸ’Ύ Native Export Dialog β€” Dedicated on-screen Save button spawns a native OS file dialog to export drawings to .png or .jpg.
  • ⚑ Real-Time FPS Counter β€” Live performance overlay.

πŸ“ Repository Structure & Module Breakdown

VisionSketch-AI/
β”œβ”€β”€ VisionBoard.py          # Main application entry point & camera loop
β”œβ”€β”€ HandTrackingModule.py   # MediaPipe wrapper for 21-landmark hand tracking
β”œβ”€β”€ shape_recognition.py    # Contour & geometric shape detection engine
β”œβ”€β”€ shape_generator.py      # Primitive rendering utility (draws clean shapes)
β”œβ”€β”€ toolbar.py              # Toolbar UI loader and color definitions
β”œβ”€β”€ utils.py                # UI sliders, eraser picker, save dialog, clear functions
β”œβ”€β”€ assets/                 # Image assets for color toolbar headers
β”‚   β”œβ”€β”€ Colors.jpg
β”‚   β”œβ”€β”€ Red_Color.jpg
β”‚   β”œβ”€β”€ Purple_Color.jpg
β”‚   β”œβ”€β”€ Green_Color.jpg
β”‚   β”œβ”€β”€ Blue_Color.jpg
β”‚   β”œβ”€β”€ Orange_Color.jpg
β”‚   β”œβ”€β”€ Pink_Color.jpg
β”‚   └── Eraser.jpg
└── requirements.txt        # Project Python dependencies

Module Responsibilities

File Key Class / Functions Description
VisionBoard.py Main Loop Captures camera input, manages dual-layer compositing (canvas & stroke_canvas), dispatches gestures, handles UI overlay and frame rendering.
HandTrackingModule.py handDetector Wraps MediaPipe Hands. findhands() draws landmarks; findpoints() returns [id, x, y] landmark coordinates.
shape_recognition.py shape_recognition() Analyzes active stroke layer contours to classify Lines, Triangles, Squares, Rectangles, and Circles.
shape_generator.py draw_perfect_shape() Renders high-precision OpenCV geometric primitives (lines, rectangles, polylines, circles) onto the canvas.
toolbar.py select_toolbar_tool() Manages toolbar selection logic ($y < 150\text{ px}$) and swaps top toolbar header images from assets/.
utils.py Sliders, Buttons, Clear & Save Renders brush size slider, eraser preset menu, virtual save button, clear function, and native Tkinter file save dialog.

πŸ—οΈ Architecture & Layer Compositing Pipeline

The application separates transient drawing strokes from the main canvas using a dual-layer architecture.

                           β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                           β”‚   Webcam Frame (1280x720) β”‚
                           β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                        β”‚
                                        β–Ό
                           β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                           β”‚ MediaPipe Landmark Trackβ”‚
                           β”‚  (HandTrackingModule)   β”‚
                           β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                        β”‚
             β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
             β–Ό                          β–Ό                          β–Ό
   [Selection Mode]               [Drawing Mode]             [Fist Gesture]
Index & Middle Up           Index Only Up              All 4 Fingers Down
 β”œβ”€ Toolbar Selection        β”œβ”€ Color Brush             └─ 2.5s Countdown
 β”œβ”€ Brush Slider Adjust      β”‚   └─► stroke_canvas          └─► clear_canvas()
 β”œβ”€ Eraser Size Adjust       └─ Eraser
 └─ Save Canvas Dialog           └─► main canvas
                                        β”‚
                                        β–Ό
                          β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                          β”‚   Finger Lifted / Stroke  β”‚
                          β”‚        Transition         β”‚
                          β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                        β”‚
                                        β–Ό
                          β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                          β”‚   shape_recognition()     β”‚
                          β”‚   (On stroke_canvas layer)β”‚
                          β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                        β”‚
                      β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                      β–Ό                                   β–Ό
              [Shape Recognized]                  [Raw Freehand Stroke]
         draw_perfect_shape(canvas)             commit stroke_canvas to canvas
                      β”‚                                   β”‚
                      β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                        β”‚
                                        β–Ό
                          β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                          β”‚  Composite & Mask Frame   β”‚
                          β”‚ img + (canvas|stroke_canvas)
                          β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                        β”‚
                                        β–Ό
                          β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                          β”‚  cv2.imshow("Vision...") β”‚
                          β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

πŸ“ Shape Recognition Logic & Mathematics

Shape classification is performed on the isolated stroke_canvas after grayscale conversion and binary thresholding:

1. Line Detection (Rotated Bounding Box Elongation)

Because approxPolyDP rarely outputs exactly 2 vertices for hand-drawn air strokes, lines are detected using the minimum area rotated bounding box (cv2.minAreaRect): $$\text{Elongation Ratio} = \frac{\max(w_{\text{rect}}, h_{\text{rect}})}{\min(w_{\text{rect}}, h_{\text{rect}})}$$

  • If $\text{Elongation Ratio} > 5.0$, the stroke is classified as a Line.
  • Endpoints are computed by taking the midpoints of the two short edges of the rotated bounding box.

2. Geometric Shape Detection (Triangle, Square, Rectangle, Circle)

For non-line contours, shape detection is performed using Douglas-Peucker vertex approximation (cv2.approxPolyDP) with epsilon parameter $\epsilon = 0.03 \times P$ (where $P$ is contour perimeter):

  • Triangle: 3 approximated vertices ($V = 3$).
  • Square vs. Rectangle: 4 approximated vertices ($V = 4$).
    • Aspect Ratio $R = \frac{w}{h}$ derived from cv2.boundingRect(approx).
    • If $0.9 \le R \le 1.1 \longrightarrow$ Square.
    • Else $\longrightarrow$ Rectangle.
  • Circle: For contours with more than 4 vertices ($V > 4$), circularity score $C$ is computed: $$C = \frac{4 \pi \cdot \text{Area}}{P^2}$$
    • If $C > 0.60 \longrightarrow$ Circle.

πŸ–οΈ Gesture & Control Reference

MediaPipe Landmark Mapping

Gesture Finger State Landmark Condition Mode / Action
✌️ Two Fingers Up Index + Middle raised y_8 < y_6 and y_12 < y_10 Selection Mode (UI interaction & tool configuration)
☝️ One Finger Up Index raised, Middle down y_8 < y_6 and y_12 >= y_10 Drawing Mode (Active drawing stroke)
✊ Fist Hold (2.5s) All 4 fingers down y_8 > y_6, y_12 > y_10, y_16 > y_14, y_20 > y_18 Clear Canvas (Holds 2.5s countdown timer near wrist id 0)

On-Screen UI Layout Coordinates (1280 Γ— 720 Frame)

0 px               240 px    360 px    480 px    600 px    720 px    840 px    960 px   1000 px     1280 px
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” 0 px
β”‚    (Reserved)     β”‚   RED   β”‚ PURPLE  β”‚  GREEN  β”‚  BLUE   β”‚ ORANGE  β”‚  PINK   β”‚(Reserv.)β”‚  ERASER   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ 150 px
  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                 β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”               β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
  β”‚   SAVE    β”‚                 β”‚       BRUSH SIZE SLIDER (5-50px)    β”‚               β”‚  ERASER   β”‚
  β”‚  BUTTON   β”‚                 β”‚ Track: x=620..750, y=180            β”‚               β”‚  PANEL    β”‚
  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                 β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜               β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
  x: 40..130                    x: 450..810, y: 160..210                              x: 1060..1200
  y: 160..210                                                                         y: 180..320
  • Top Color Toolbar ($y &lt; 150\text{ px}$):
    • Red: x: 240 – 360
    • Purple: x: 360 – 480
    • Green: x: 480 – 600
    • Blue: x: 600 – 720
    • Orange: x: 720 – 840
    • Pink: x: 840 – 960
    • Eraser Tool: x: 1000 – 1280
  • Virtual Save Button: x: 40 – 130, y: 160 – 210
  • Brush Size Slider: x: 620 – 750 track line, y: 160 – 210 activation zone.
  • Eraser Size Panel: Appears when Eraser is selected (x: 1060 – 1200, y: 180 – 320):
    • Small (25 px): y: 240 – 265
    • Medium (50 px): y: 265 – 290
    • Large (80 px): y: 290 – 310

πŸ› οΈ Installation & Setup

Prerequisites

  • Python: Version 3.8 to 3.12 recommended.
  • Webcam: Standard USB or integrated laptop camera.

Step-by-Step Setup

  1. Clone or Navigate to the Workspace:

    cd c:\Projects\VisionSketch-AI
  2. Create & Activate Virtual Environment:

    • Windows (PowerShell):
      python -m venv .venv
      .venv\Scripts\Activate.ps1
    • macOS / Linux:
      python3 -m venv .venv
      source .venv/bin/activate
  3. Install Dependencies:

    pip install -r requirements.txt

πŸš€ Running the Application

Launch the main application script:

python VisionBoard.py

Controls & Keyboard Shortcuts

  • q: Quit application and close window.
  • Save Button (UI): Hover in Selection Mode over the Save button at top-left (x: 40-130, y: 160-210) to export your whiteboard drawing.

Camera Note: VisionBoard.py automatically attempts to open camera index 1 (cv2.VideoCapture(1)). If index 1 is unavailable, it automatically falls back to index 0.


πŸ“¦ Requirements & Dependencies

Key libraries listed in requirements.txt:

Package Purpose
opencv-python Image processing, frame capture, contour analysis, and GUI windowing
mediapipe 21-landmark real-time hand detection and tracking
numpy High-performance array manipulation for canvas blending and masking
tkinter Native cross-platform file save dialog (included with standard Python)

❓ Troubleshooting & FAQs

  • Webcam output is black or fails to open: Ensure no other app (e.g. Zoom, Teams) is using your webcam. If using an integrated webcam, VisionBoard.py will automatically fall back to camera index 0.
  • Shape recognition not triggering: Draw deliberate, continuous strokes. Small or erratic strokes ($\text{area} &lt; 100$ or $\text{perimeter} == 0$) are ignored as noise.
  • Save Dialog appears behind the OpenCV window: Click on the taskbar or bring the file explorer dialog to the foreground if it opens underneath the OpenCV full window.

About

An interactive touchless AI whiteboard featuring real-time MediaPipe hand tracking, gesture controls, and automatic stroke-to-shape recognition.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages