Hand Gesture Recognition Online

How the Hand Gesture Recogniser works

Hand gesture recognition online: what it does and why

Hand tracking used to need a depth camera or a glove. The model on this page, MediaPipe's hand landmarker with a gesture classifier on top, does it from an ordinary webcam frame in a few milliseconds and runs entirely in your browser. It works in two stages: a palm detector finds the hand and crops it, then a landmark model places 21 points — four on each finger and one at the wrist — inside that crop. The skeleton you see connects those points in the order of the bones they sit on.

How the gesture is decided. The 21 points are turned into a small set of angles and distances that do not change with hand size or position — is the index finger extended, is the thumb above the wrist, are the other fingers curled — and a second tiny classifier maps that description onto seven named gestures. The percentage after the name is the classifier's confidence. "No gesture" means the hand is tracked but its shape does not match any of the seven well enough, which is most of what a hand does in normal life.

Left or right. The readout says which hand it thinks it is seeing. The model decides this from the arrangement of the thumb and fingers, on the assumption that it is looking at a mirror image of you, which is how the preview is shown by default. If you untick Mirror, the tool swaps the label so it still names your actual hand.

Why it sometimes loses your hand. The palm detector wants the whole hand in frame, roughly larger than a twentieth of the picture, and reasonably lit. A hand edge-on to the camera, half out of shot, or overlapping the other hand is the hard case; once the detector loses the hand the skeleton disappears until it is found again. Holding your hand a little away from your body, palm towards the camera, is the most reliable pose.

What this is for. It is a demonstration of a capability that is now cheap to build on: the same tracker drives the air-draw tool and the fingerspelling practice tool on this site, and the underlying library is open source, so it is a fair way to judge whether gesture control would work for a kiosk, a presentation clicker, or an accessibility project before writing any code. It is also simply fun to see.

Privacy. The model runs in WebAssembly inside the page. The camera stream, the landmark points and the gesture labels never leave your browser; the only network traffic is the one-time download of the model and runtime, around 12 MB, which is then cached.

Related tools