GEM-X: Full-Body and Hand Pose Estimation from Image and Video

Unified body pose, hand pose, and temporal motion estimation with SOMA

News · Quick Start · Pretrained Models · Evaluation · Training & Evaluation · Documentation

---

Full-body pose and motion from images and video.

Detailed hand pose from images and video.

📰 News

• [2026-10-21] 🚀 GEM-X supports Hand Pose alongside Body Pose and temporal Body Motion, with image/video inference, shared training workflows, and fine-tuning from pretrained Pose checkpoints. Get started with the Quick Start and pretrained models.
• [2025-06] Added ONNX/TensorRT acceleration to the original GEM demo.
• [2025-06] Added motion retargeting to Unitree G1 with SOMA Retargeter.
• [2025-05] Released the original GEM codebase.

🔎 Overview

GEM-X provides Body Pose, Hand Pose, and Body Motion in a shared framework for inference, training, and evaluation. Body and Hand Pose reconstruct 3D geometry from individual images. Body Motion uses temporal context to estimate camera-space and world-space human motion from monocular video. All three models use SOMA geometry.

✨ Key Features

• Full-body and hand reconstruction — dedicated image models for body pose and detailed left and right hand geometry.
• Temporal body motion — person tracking, 2D keypoint observations, and Body Pose features feed a temporal model that predicts camera-space and world-space motion.
• Flexible hand inputs — supplied boxes or RT-DETR detection, ego/exo processing profiles, optional tracking and smoothing, and calibrated camera adapters.
• Robotics showcase — recovered SOMA motion retargeted to Unitree G1 with SOMA Retargeter; see the application demo.
• Training and fine-tuning — train Body Pose, Hand Pose, and Body Motion; fine-tune pretrained weights or resume a complete training checkpoint.
• Visualization and evaluation — render saved predictions, apply optional contact/IK correction, and evaluate with native Hand metrics or custom evaluators.

🧩 Models and Inference Modes

There are three models and four inference modes. The command identifies the model family and input type; --task selects body or hand.

Command	Task	Prediction
pose-image	body, hand	Body or Hand Pose from one image
pose-video	body, hand	Body or Hand Pose independently on each video frame
motion-video	body	Temporal Body Motion with automatic video preprocessing
motion-cache	body	Temporal Body Motion from cached features and observations

pose-video runs the image model frame by frame. Hand tracking and smoothing are optional processing stages; learned temporal prediction is provided by Body Motion. See Model Overview for architecture and output conventions.

🚀 Quick Start

1. Install

Use Python 3.10 or newer and install a PyTorch 2.10 build for your device. From this checkout:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
# Install the PyTorch build for your CPU or CUDA environment before this step.
python -m pip install -e '.[video,hand]'

Install FFmpeg for video processing. For the rendering commands below, install python -m pip install -e '.[render]' and the rendering dependencies in a Linux x86-64 environment. See Installation for CPU/CUDA setup, Docker, ARM64 inference, and optional dependencies.

2. Prepare weights and assets

GEM installs the SOMA 0.3.3 Python runtime. Download the model and geometry files separately, following the instructions for your task:

1. Download the model checkpoints into weights/.
2. For automatic Body Pose or Body Motion, download and extract the YOLOX person detector to weights/yolox_humanart.onnx.
3. Download the SOMA v0.3.3 assets into assets/soma/. The examples below use that directory.

3. Run inference

Body Pose — image

Detect the main person automatically using weights/yolox_humanart.onnx:

gem-infer pose-image --task body --input image.jpg \
  --release body --checkpoint weights/body_s240000.ckpt \
  --soma-assets assets/soma --device cuda --output body.pt
gem-render --task body --input body.pt --output-dir outputs/body

Use --body-detector PATH for another detector location. For calibrated cameras or multiple people, see camera settings and supplied boxes. The same automatic setup supports framewise Body Pose video.

Hand Pose — video

Use the EgoHand checkpoint and a local RT-DETR Hand Detector:

gem-infer pose-video --task hand --input video.mp4 \
  --release hand --checkpoint weights/egohand_s060000.ckpt \
  --hand-detector weights/hand_detector.onnx \
  --soma-assets assets/soma --device cuda --output hands.pt
gem-render --task hand --input hands.pt --output-dir outputs/hand

The default --hand-view exo processes detections independently per frame. For the camera wearer's hands, add --hand-view ego to enable video tracking, crop stabilization, and parameter smoothing. Supplied boxes and calibrated pinhole/fisheye inputs are covered in the Hand inference guide.

Body Motion — video

Configure the local model paths and 2D keypoint detector identifier in body_video_native77.yaml, then run:

gem-infer motion-video --task body --input video.mp4 \
  --config docs/configs/body_video_native77.yaml --device cuda \
  --postprocess --soma-assets assets/soma --output motion.pt
gem-render --task body --input motion.pt --soma-assets assets/soma \
  --device cuda --output-dir outputs/motion

Body Motion produces camera and world views. This example uses --postprocess for the Contact/IK correction shown in the films; omit it for raw predictions. The Inference guide explains input formats, output fields, cached prediction, camera handling, and postprocessing.

📦 Pretrained Models

Download the weights for your task from the following Hugging Face repositories:

Model	Checkpoint	Configuration	Weight source
Body Pose	body_s240000.ckpt	body	nvidia/GEM-X
Hand Pose	egohand_s060000.ckpt	hand	nvidia/EgoHand-v1
Body Motion	Native77 final775	Native77 configuration	nvidia/GEM-X

Hand weights are distributed by the official EgoHand repository; access is managed there. See the download commands.

Body Motion uses Body Pose image features and 77-point 2D keypoint observations. Native29 is an alternate layout for separately trained weights.

See Pretrained Models for checkpoint availability, asset requirements, matching detector identifiers, and compatibility details.

📊 Evaluation

GEM-pose · Body Pose

Single-frame results for the released body_s240000.ckpt. Errors are in millimeters; lower is better.

Dataset	MPJPE ↓	PA-MPJPE ↓	PVE ↓
3DPW	62.20	39.56	72.89
EMDB-1	67.83	43.14	80.27

GEM-motion · Body Motion

EMDB-2 results use original ViTPose, the released native77 final775 checkpoint and Contact/IK postprocessing enabled. The benchmark uses provided camera intrinsics and camera rotations. Lower is better.

W-MPJPE₁₀₀	WA-MPJPE₁₀₀	RTE (%)	Jitter	Foot sliding
267.20	112.76	2.45	34.21	4.22

W-MPJPE₁₀₀, WA-MPJPE₁₀₀ and foot sliding are in millimeters. Jitter uses the reference reporting scale: mean jerk at 30 Hz divided by 10. See evaluation settings and metric definitions for coverage, alignment and aggregation details.

🏋️ Training & Evaluation

Install the training and evaluation dependencies:

python -m pip install -e '.[train,eval]'

Model	Training command	Guide
Body Pose	gem-train pose --task body --config training.yaml	Body Pose training
Hand Pose	gem-train pose --task hand --config training.yaml	Hand Pose training
Body Motion	gem-train motion --config training.yaml	Motion training

Start with the Training & Evaluation guide for data preparation contracts, example configurations, initialization, and full checkpoint recovery. To fine-tune Hand Pose from egohand_s060000.ckpt, use the Hand fine-tuning recipe.

gem-eval pose evaluates Body or Hand Pose; Body Motion uses gem-train motion --validate. Native SOMA25 Hand metrics are included, and Body Pose and Motion support caller-provided evaluators. See Evaluation for configuration and metric definitions.

📚 Documentation

Guide	Contents
Installation	Environment setup, SOMA assets, rendering, and Docker
Pretrained Models	Checkpoints, detector pairing, and local weight setup
Inference	Image, video, and cached inference; camera adapters and visualization
Training & Evaluation	Training entry points, prepared examples, validation, and resume
Body Pose Training	Body sample format, objectives, and initialization
Hand Pose Training	Hand fine-tuning, canonical inputs, and objectives
Evaluation	Release benchmark results, metric definitions, and custom evaluators
Model Overview	Model architectures, conditioning, and outputs
Architecture	Package structure and component responsibilities

For development setup and contribution guidelines, see Contributing.

🏭 Production Deployment with NVIDIA AI for Media

For developers looking to integrate 3D body pose estimation into real-world applications, NVIDIA provides production-ready deployment options through the NVIDIA AI for Media SDK and NIM.

• NVIDIA 3D Body Pose Estimation – AR SDK: GPU-accelerated SDK supporting live video processing, enabling continuous 3D human body pose and motion estimation.
• NVIDIA 3D Body Pose NIM: Containerized inference microservice for integrating 3D body pose estimation into scalable video processing workflows.

Explore the full NVIDIA Augmented Reality SDK for additional AI-powered video capabilities.

📄 License

The default source license is Apache 2.0. Adapted Hand processing files and the source-camera restoration function retain their LicenseRef-OpenMDW-1.1 notices. See ATTRIBUTIONS.md for the complete mapping of file and function exceptions, and OpenMDW-1.1 for their license.

Model weights, geometry assets, and dependencies retain their respective terms; see their distributions and the component notices in ATTRIBUTIONS.md. Media sources and credits are listed in assets/CREDITS.md.