Everything you need to get running

Install

Gesto Labeller — download the latest release, unzip it anywhere, and run GestoLabeller.exe. No Python required. See the full feature guide.

The gesto package — install from PyPI into a fresh virtual environment:

python -m venv gesto_env
# Windows
gesto_env\Scripts\activate
# macOS / Linux
source gesto_env/bin/activate

pip install gesto

Point it at a project folder, pick a mode and a region, and it handles the rest — loading, training, versioned model storage, and live detection that matches how the data was captured. See Dependencies for why the fresh environment matters.

Quickstart

The whole loop — capture, train, detect — in four moves:

1. In Gesto Labeller, create a project, pick a region and mode, and capture ~20–30 samples per class.
2. Note the project folder (use the Copy path button).
3. Train and detect:

gesto train static hands_one "C:\...\gesto_projects\my-signs"
gesto detect static hands_one

A window opens with your webcam, the detected skeleton, and a live probability bar for the closest-matching classes.

Try it without training

No dataset yet? Download a hosted demo model and run detection right away:

gesto detect sequence hands_one --default

This fetches a pre-trained sequence/hands_one model the first time and caches it, so later runs start instantly.

Two modes

Static needs far less data (every captured frame is a training sample) and predicts instantly with no warm-up. Reach for sequence only when two gestures share the same shape and differ by movement.

Model architectures

Both model types come in a lighter and a full-size variant. Small datasets automatically get the lighter one (an oversized network on little data overfits); force a choice with --small or --large. All use the Adam optimizer, softmax output, and sparse categorical cross-entropy loss, and save as model.keras.

Static — Dense network

Input is a single feature vector (63–258 values by region).

Sequence — stacked LSTM

Input is a window of frames (seq_len, dim). LSTM layers use tanh (enabling the fast cuDNN path).

Five regions

The region must match how the project was captured — gesto checks the feature dimension and tells you if it doesn't.

Command line — two styles

Use whichever you like. General — mode and region as arguments:

gesto train static hands_one ./gesto_projects/signs
gesto detect sequence pose --source clip.mp4
gesto image hands_one photo.jpg

Per-combination — one command per mode+region (there's one for each):

gesto train-static-legs ./gesto_projects/stances --epochs 250
gesto detect-sequence-pose --source clip.mp4
gesto image-static-hands-one photo.jpg

Detecting on camera vs video

--source takes a webcam index or a file path:

gesto detect static hands_one                # default webcam (index 0)
gesto detect static hands_one --source 1     # second camera
gesto detect sequence pose --source walk.mp4 # a video file
gesto detect sequence pose --source C:\clips\run.avi

The overlay shows the top matches ranked by confidence, closest first.

Classifying a single image

Static models can classify a still image:

gesto image hands_one photo.jpg              # opens a window with the result
gesto image hands_one photo.jpg --no-show    # just print the prediction
gesto image-static-pose posture.png --version 2 # a specific model version

Drawing landmarks

Landmarks are drawn on the frame by default. Turn them off with --no-draw:

gesto detect static hands_one              # skeleton drawn (default)
gesto detect static hands_one --no-draw    # clean video, no skeleton
gesto image hands_one photo.jpg --no-draw

Training options

Epochs and other hyperparameters are adjustable on any train command:

gesto train sequence pose ./proj --epochs 400 --batch-size 32 --seq-len 30
gesto train-static-hands-one ./proj --epochs 150 --large  # force full model

Python API

Everything the CLI does is available in code:

import gesto

run = gesto.train("./gesto_projects/signs", region="hands_one", mode="static")
gesto.detect("static", "hands_one")                 # camera
gesto.detect("sequence", "pose", source="clip.mp4") # video

# classify a single image with a static model
from gesto.detect import predict_image
label, confidence, probs = predict_image("static", "hands_one", "photo.jpg",
                                         show=False, draw_landmarks=False)

Or drive a model yourself:

from gesto.detect import Predictor

predictor = Predictor.load("static", "hands_one")
vector = predictor.features(holistic_result)   # extract + normalize
probs = predictor.predict(vector)

Where models go

Everything lands under one artifacts/ folder, split by mode then region. Training never overwrites an earlier run — it versions:

artifacts/
    static/
        hands_one/          # model.keras, labels.json
        hands_one_2/        # the next run
        pose/
    sequence/
        pose/
        pose_2/

gesto detect static pose picks the newest version; --version 1 picks a specific one. List everything you've trained with gesto list.

Matching your capture

Predictions are only correct when detection feeds the model the same kind of vector it trained on. gesto mirrors Gesto Labeller exactly:

• The same engine — MediaPipe Holistic, same confidence settings.
• The same landmark order — including one-hand mode preferring the right hand and falling back to the left.
• The same normalization — translation/scale-invariant, verified identical.
• The same mirroring — webcam frames are flipped, video files are not.

Getting good results

• Balance your classes. Similar sample counts per class; gesto applies class weights but balanced data is better.
• Enough samples. ~20–30 static frames per class, or ~15–30 sequences. Small datasets automatically get a lighter model, since an oversized network on little data overfits and collapses to predicting one class.
• Consistent clip length for sequence mode — set "Max frames" in Gesto Labeller so every capture is the same length.

Run gesto inspect <project> to check all of this before training.

Dependencies

gesto uses MediaPipe's legacy solutions API, removed in MediaPipe 0.10.31. It also needs versions of TensorFlow, NumPy and protobuf that agree with that MediaPipe. The package pins a coherent, tested set:

Install into a fresh virtual environment so these don't clash with other projects. To repair an existing environment, pin the set explicitly:

pip install "mediapipe==0.10.21" "tensorflow==2.17.1" \
  "numpy==1.26.4" "protobuf==4.25.9" "opencv-python==4.11.0.86"

Roadmap

The legacy MediaPipe solutions API won't be maintained forever. A future release will move to MediaPipe's newer Tasks API (HandLandmarker, PoseLandmarker), which lifts the version ceiling. That API produces slightly different hand-landmark geometry, so models would need retraining — hence it's a deliberate, separate step rather than a drop-in change.

License

The two parts of Gesto are released under different open-source licenses:

The package's dependencies — TensorFlow, NumPy, OpenCV, MediaPipe, protobuf — are all permissively licensed (Apache 2.0 / BSD). The model architectures are standard Dense and LSTM networks with no proprietary components.

Tutorials