VisionSketch-AI is an interactive, touchless virtual whiteboard built with Python, OpenCV, MediaPipe, and NumPy. Draw freehand strokes in the air using your index finger via a standard webcam, and let the real-time computer vision engine automatically detect and snap your strokes into clean geometric shapes.
- ποΈ Real-Time 21-Landmark Hand Tracking β Low-latency hand landmark tracking powered by Google MediaPipe Hands.
- βοΈ Dual Gesture-Driven Interaction Modes
- Selection Mode (Index + Middle fingers raised): Select colors, adjust brush slider, set eraser sizes, or export drawings.
- Drawing Mode (Index finger only): Freehand air-drawing with live visual feedback.
- π¨ 6 Vibrant Color Brushes & Interactive Toolbar β Red, Purple, Green, Blue, Orange, and Pink selectable from an top toolbar.
- ποΈ Virtual Brush Thickness Slider β Dynamic brush size adjustment (5 px to 50 px) via an interactive on-screen slider.
- π§½ 3 Eraser Size Presets β Quick toggle panel for Small (25 px), Medium (50 px), and Large (80 px) eraser tools.
- π Isolated Layer Shape Recognition & Snap-to-Perfect β Automatic shape classification executed strictly on the active stroke layer (
stroke_canvas), preserving existing artwork without canvas wipes:- π Line: Rotated bounding box elongation check (
$\text{long} / \text{short} > 5.0$ ). - πΊ Triangle: 3-corner polygon approximation (
approxPolyDP). - β¬ Square: 4-corner polygon with near-unity aspect ratio (
$0.9 \le w/h \le 1.1$ ). - β Rectangle: 4-corner polygon with non-square aspect ratio.
- β Circle: Compactness / circularity score (
$C = \frac{4\pi \cdot \text{Area}}{\text{Perimeter}^2} > 0.60$ ).
- π Line: Rotated bounding box elongation check (
- β Fist-Hold Canvas Clear β Hold a fist gesture for 2.5 seconds to clear the whiteboard, accompanied by an anchor-locked visual countdown timer near your wrist landmark.
- π·οΈ Visual Shape HUD Overlay β Temporary green outline highlight and on-screen label overlay for 3 seconds upon shape detection.
- πΎ Native Export Dialog β Dedicated on-screen Save button spawns a native OS file dialog to export drawings to
.pngor.jpg. - β‘ Real-Time FPS Counter β Live performance overlay.
VisionSketch-AI/
βββ VisionBoard.py # Main application entry point & camera loop
βββ HandTrackingModule.py # MediaPipe wrapper for 21-landmark hand tracking
βββ shape_recognition.py # Contour & geometric shape detection engine
βββ shape_generator.py # Primitive rendering utility (draws clean shapes)
βββ toolbar.py # Toolbar UI loader and color definitions
βββ utils.py # UI sliders, eraser picker, save dialog, clear functions
βββ assets/ # Image assets for color toolbar headers
β βββ Colors.jpg
β βββ Red_Color.jpg
β βββ Purple_Color.jpg
β βββ Green_Color.jpg
β βββ Blue_Color.jpg
β βββ Orange_Color.jpg
β βββ Pink_Color.jpg
β βββ Eraser.jpg
βββ requirements.txt # Project Python dependencies
| File | Key Class / Functions | Description |
|---|---|---|
VisionBoard.py |
Main Loop | Captures camera input, manages dual-layer compositing (canvas & stroke_canvas), dispatches gestures, handles UI overlay and frame rendering. |
HandTrackingModule.py |
handDetector |
Wraps MediaPipe Hands. findhands() draws landmarks; findpoints() returns [id, x, y] landmark coordinates. |
shape_recognition.py |
shape_recognition() |
Analyzes active stroke layer contours to classify Lines, Triangles, Squares, Rectangles, and Circles. |
shape_generator.py |
draw_perfect_shape() |
Renders high-precision OpenCV geometric primitives (lines, rectangles, polylines, circles) onto the canvas. |
toolbar.py |
select_toolbar_tool() |
Manages toolbar selection logic (assets/. |
utils.py |
Sliders, Buttons, Clear & Save | Renders brush size slider, eraser preset menu, virtual save button, clear function, and native Tkinter file save dialog. |
The application separates transient drawing strokes from the main canvas using a dual-layer architecture.
βββββββββββββββββββββββββββ
β Webcam Frame (1280x720) β
ββββββββββββββ¬βββββββββββββ
β
βΌ
βββββββββββββββββββββββββββ
β MediaPipe Landmark Trackβ
β (HandTrackingModule) β
ββββββββββββββ¬βββββββββββββ
β
ββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββ
βΌ βΌ βΌ
[Selection Mode] [Drawing Mode] [Fist Gesture]
Index & Middle Up Index Only Up All 4 Fingers Down
ββ Toolbar Selection ββ Color Brush ββ 2.5s Countdown
ββ Brush Slider Adjust β βββΊ stroke_canvas βββΊ clear_canvas()
ββ Eraser Size Adjust ββ Eraser
ββ Save Canvas Dialog βββΊ main canvas
β
βΌ
βββββββββββββββββββββββββββββ
β Finger Lifted / Stroke β
β Transition β
βββββββββββββββ¬ββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββ
β shape_recognition() β
β (On stroke_canvas layer)β
βββββββββββββββ¬ββββββββββββββ
β
βββββββββββββββββββ΄ββββββββββββββββββ
βΌ βΌ
[Shape Recognized] [Raw Freehand Stroke]
draw_perfect_shape(canvas) commit stroke_canvas to canvas
β β
βββββββββββββββββββ¬ββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββ
β Composite & Mask Frame β
β img + (canvas|stroke_canvas)
βββββββββββββββ¬ββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββ
β cv2.imshow("Vision...") β
βββββββββββββββββββββββββββββ
Shape classification is performed on the isolated stroke_canvas after grayscale conversion and binary thresholding:
Because approxPolyDP rarely outputs exactly 2 vertices for hand-drawn air strokes, lines are detected using the minimum area rotated bounding box (cv2.minAreaRect):
- If
$\text{Elongation Ratio} > 5.0$ , the stroke is classified as a Line. - Endpoints are computed by taking the midpoints of the two short edges of the rotated bounding box.
For non-line contours, shape detection is performed using Douglas-Peucker vertex approximation (cv2.approxPolyDP) with epsilon parameter
-
Triangle: 3 approximated vertices (
$V = 3$ ). -
Square vs. Rectangle: 4 approximated vertices (
$V = 4$ ).- Aspect Ratio
$R = \frac{w}{h}$ derived fromcv2.boundingRect(approx). - If
$0.9 \le R \le 1.1 \longrightarrow$ Square. - Else
$\longrightarrow$ Rectangle.
- Aspect Ratio
-
Circle: For contours with more than 4 vertices (
$V > 4$ ), circularity score$C$ is computed:$$C = \frac{4 \pi \cdot \text{Area}}{P^2}$$ - If
$C > 0.60 \longrightarrow$ Circle.
- If
| Gesture | Finger State | Landmark Condition | Mode / Action |
|---|---|---|---|
| βοΈ Two Fingers Up | Index + Middle raised | y_8 < y_6 and y_12 < y_10 |
Selection Mode (UI interaction & tool configuration) |
| βοΈ One Finger Up | Index raised, Middle down | y_8 < y_6 and y_12 >= y_10 |
Drawing Mode (Active drawing stroke) |
| β Fist Hold (2.5s) | All 4 fingers down | y_8 > y_6, y_12 > y_10, y_16 > y_14, y_20 > y_18 |
Clear Canvas (Holds 2.5s countdown timer near wrist id 0) |
0 px 240 px 360 px 480 px 600 px 720 px 840 px 960 px 1000 px 1280 px
βββββββββββββββββββββ¬ββββββββββ¬ββββββββββ¬ββββββββββ¬ββββββββββ¬ββββββββββ¬ββββββββββ¬ββββββββββ¬ββββββββββββ 0 px
β (Reserved) β RED β PURPLE β GREEN β BLUE β ORANGE β PINK β(Reserv.)β ERASER β
βββββββββββββββββββββ΄ββββββββββ΄ββββββββββ΄ββββββββββ΄ββββββββββ΄ββββββββββ΄ββββββββββ΄ββββββββββ΄ββββββββββββ 150 px
βββββββββββββ βββββββββββββββββββββββββββββββββββββββ βββββββββββββ
β SAVE β β BRUSH SIZE SLIDER (5-50px) β β ERASER β
β BUTTON β β Track: x=620..750, y=180 β β PANEL β
βββββββββββββ βββββββββββββββββββββββββββββββββββββββ βββββββββββββ
x: 40..130 x: 450..810, y: 160..210 x: 1060..1200
y: 160..210 y: 180..320
-
Top Color Toolbar (
$y < 150\text{ px}$ ):- Red:
x: 240 β 360 - Purple:
x: 360 β 480 - Green:
x: 480 β 600 - Blue:
x: 600 β 720 - Orange:
x: 720 β 840 - Pink:
x: 840 β 960 - Eraser Tool:
x: 1000 β 1280
- Red:
-
Virtual Save Button:
x: 40 β 130,y: 160 β 210 -
Brush Size Slider:
x: 620 β 750track line,y: 160 β 210activation zone. -
Eraser Size Panel: Appears when Eraser is selected (
x: 1060 β 1200,y: 180 β 320):- Small (25 px):
y: 240 β 265 - Medium (50 px):
y: 265 β 290 - Large (80 px):
y: 290 β 310
- Small (25 px):
- Python: Version
3.8to3.12recommended. - Webcam: Standard USB or integrated laptop camera.
-
Clone or Navigate to the Workspace:
cd c:\Projects\VisionSketch-AI -
Create & Activate Virtual Environment:
- Windows (PowerShell):
python -m venv .venv .venv\Scripts\Activate.ps1 - macOS / Linux:
python3 -m venv .venv source .venv/bin/activate
- Windows (PowerShell):
-
Install Dependencies:
pip install -r requirements.txt
Launch the main application script:
python VisionBoard.pyq: Quit application and close window.- Save Button (UI): Hover in Selection Mode over the
Savebutton at top-left (x: 40-130,y: 160-210) to export your whiteboard drawing.
Camera Note:
VisionBoard.pyautomatically attempts to open camera index1(cv2.VideoCapture(1)). If index1is unavailable, it automatically falls back to index0.
Key libraries listed in requirements.txt:
| Package | Purpose |
|---|---|
opencv-python |
Image processing, frame capture, contour analysis, and GUI windowing |
mediapipe |
21-landmark real-time hand detection and tracking |
numpy |
High-performance array manipulation for canvas blending and masking |
tkinter |
Native cross-platform file save dialog (included with standard Python) |
-
Webcam output is black or fails to open:
Ensure no other app (e.g. Zoom, Teams) is using your webcam. If using an integrated webcam,
VisionBoard.pywill automatically fall back to camera index0. -
Shape recognition not triggering:
Draw deliberate, continuous strokes. Small or erratic strokes (
$\text{area} < 100$ or$\text{perimeter} == 0$ ) are ignored as noise. - Save Dialog appears behind the OpenCV window: Click on the taskbar or bring the file explorer dialog to the foreground if it opens underneath the OpenCV full window.