Face Mesh Online

How the Face Mesh Filter works

Face mesh online: how it works

Every face filter on Snapchat, Instagram and TikTok starts from the same idea: find a few hundred fixed points on the face, such as the corner of each eye, the tip of the nose and the outline of the lips, and track them frame by frame so a drawing can be pinned to them. This page runs the model that does that, MediaPipe's face landmarker, entirely in your browser. It returns 468 face points plus 10 for the irises at camera frame rate, and the overlays are plain canvas drawing attached to a handful of those points.

Why it matters. Face landmarks sit under far more than novelty filters. Virtual try-on for glasses and make-up, VTuber avatars, animation previs, driver-drowsiness monitors, video-call background effects and hands-free accessibility controls all start from the same point set. Seeing the raw mesh on your own face is the quickest way to understand what those products can and cannot know about you: they see geometry and muscle movement, not identity.

How the face mesh works

The 468 points are not found one by one. The model predicts the whole set at once as a deformable mask fitted to your face, so even a point hidden behind a hand gets a best guess. Each point has a fixed meaning. Index 1 is always the nose tip, 33 and 263 are the outer eye corners, 133 and 362 the inner corners, and 61 and 291 the mouth corners. That fixed numbering is what makes attaching props possible, and it is why a filter built for one face works on every face.

The overlay choices show different views of the same data:

• Wireframe mesh draws the triangulation Google publishes for texturing a 3D face. It is what a game engine would wrap a skin around.
• Contours keeps only the lines a portrait artist would draw: eyes, brows, lips, irises and the face oval.
• Dots plots every landmark, which makes the density plain. The eyes and lips carry far more points than the cheeks because that is where expression lives.
• Glasses, Moustache and Face mask are simple props pinned to landmarks.

How the props are attached. Glasses are two circles centred between each pair of eye corners (33 to 133 and 362 to 263). Their radius comes from the eye width, so they grow as you lean in, and the arms run out to the temple points 234 and 454 on the face oval. The moustache sits between the base of the nose and the top lip, and its width is a fraction of the distance between the mouth corners. The mask fills the face oval and cuts the eyes back out. That is the whole trick: measure a distance between two landmarks, scale the drawing by it, and redraw every frame. Commercial try-on apps add a 3D model and lighting, but the anchoring is the same.

Reading the Expression readout

Alongside the points, the model outputs 52 blendshapes, the same set Apple defined for ARKit face tracking. Each is a score from 0 to 1 for one facial action: jawOpen, browInnerUp, mouthSmileLeft, eyeBlinkRight, cheekPuff and so on. The Expression stat shows the name of the strongest one when it scores above 0.5.

• A blendshape name means that action is clearly present. Raise your eyebrows, open your mouth, wink or puff your cheeks and watch it change.
• Neutral means a face is found but nothing scores above a half. A relaxed face usually sits here.
• No face means the model has lost lock. Move into better light or face the camera.

Left and right follow the image the model sees, not the mirrored preview, so a wink with your right eye may read as eyeBlinkLeft. The eye blink timer uses the two eyeBlink scores to count blinks, which shows how one blendshape becomes a measurement.

Where blendshapes are used. Animators drive character rigs directly from these 52 values, which is how iPhone face capture feeds tools such as Unreal's Live Link Face. VTubers map them onto a 2D or 3D avatar so it smiles and blinks with them. Research teams use jawOpen and eye scores for fatigue and attention studies, and assistive software turns a deliberate brow raise or mouth open into a click for people who cannot use a mouse.

Tips for a clean track.

• Light the face from the front. Side light or a bright window behind you makes the far cheek noisy and the mesh jitters.
• Keep your face at least a tenth of the frame tall. At a normal laptop distance that is easy; across a room it is not.
• Turn off "Show camera behind overlay" to see the mesh alone. It shows how well the model is tracking without your face distracting from it.
• Pick a colour that contrasts with the background and raise the line width for screen recordings or projectors, where thin lines vanish after compression.
• Use Snap to save a frame with the overlay baked in.

Pitfalls and limits. The model expects a face roughly facing the camera. It will follow you turned to about sixty degrees, beyond which the far side of the mesh is guessed and the props drift. Only one face is tracked at a time. Glasses and hats are fine, but a hand over the mouth or a very dark room makes it lose lock, and the overlay disappears until it finds you again. The expression label is a raw top score, not an emotion. A smile scores mouthSmile whether or not you are happy, so do not read feelings into it.

Privacy. The model runs in WebAssembly inside the page, and the camera stream never leaves the tab. No frame, point set or expression score is sent anywhere. The browser's network panel shows only the one-time model download of about 12 MB.

Related tools