# UniMate for Agents: text-to-animation API on ZeroGPU (agent guide)

> Give a rigged skeleton and a text prompt; get a 60-frame (2 s, 30 fps) motion back as an animated GLB,
> raw motion features, per-frame joint positions and a skeleton preview. One call, synchronous, about 10 s.
> Then refine or extend it with the upstream editing modes: in-betweening (keep some frames, regenerate the
> rest), joint editing (keep some joints, regenerate the others under a new prompt) and motion expansion
> (chain several prompts into one long, continuous motion).
> Model: UniMate (SIGGRAPH Asia 2026), checkpoint unimate_uniml3d_f60_v3 (CC BY-NC 4.0, research use only).

Base URL: https://townbox-unimate-for-agents.hf.space
This is a public Hugging Face Space. Send `Authorization: Bearer $HF_TOKEN` (any Hugging Face account token) with
every request when you can: ZeroGPU time (a few seconds per call) is then billed to that account's daily quota
instead of the small anonymous per-IP allowance, and never to the Space. GET https://townbox-unimate-for-agents.hf.space/api/identity shows who pays
for the headers you send.

Interfaces (same functionality):
- REST: OpenAPI at https://townbox-unimate-for-agents.hf.space/openapi.json, Swagger UI at https://townbox-unimate-for-agents.hf.space/docs, service description at https://townbox-unimate-for-agents.hf.space/api.
- MCP (streamable HTTP): https://townbox-unimate-for-agents.hf.space/gradio_api/mcp/ (schema: https://townbox-unimate-for-agents.hf.space/gradio_api/mcp/schema).
  Tools: animate, inbetween, edit_joints, expand, list_assets, get_result, view_animation (returns a PNG),
  service_info, guide.
- Gradio clients (Python gradio_client, JS @gradio/client): the same functions as endpoints (/animate, ...);
  schema at https://townbox-unimate-for-agents.hf.space/gradio_api/info.

## Workflow (REST)
1. Pick a skeleton: GET https://townbox-unimate-for-agents.hf.space/api/assets
   -> {"assets": [{"id": "unitree_go2", "n_joints": 18, "joint_names": [...], "joint_labels": [...],
                   "parents": [...], "has_mesh": true, "rest_pose_glb_url": "..."}, ...]}
   Bundled: unitree_go2 (quadruped robot, 18 joints), eagle (winged biped, 51 joints), shark (fish, 16 joints).
2. Generate:
     curl -s -X POST https://townbox-unimate-for-agents.hf.space/api/animate -H "Authorization: Bearer $HF_TOKEN" -H 'content-type: application/json' \
       -d '{"asset": "unitree_go2", "prompt": "An object trots forward.", "num_samples": 2, "seed": 7}'
   -> 200 {"job_id": "a_...", "state": "done", "samples": [{"index": 0, "urls": {"animation_glb": "...",
           "motion_npy": "...", "joints_json": "...", "skeleton_mp4": "...", "contact_sheet_png": "..."},
           "root_travel_xz": 4.67, ...}], "timing": {"condition_s", "gpu_s", "outputs_s"}, "provenance": {...},
           "next": [...]}
3. Download the files you need (same Authorization header). Results stay 24 h: GET https://townbox-unimate-for-agents.hf.space/api/jobs/<job_id>.
4. Optional: refine the result with POST /api/inbetween or /api/edit (reference_job_id = that job_id), or build
   a longer motion with POST /api/expand. Every result's "next" list holds ready-made bodies for these.

## Parameters (POST /api/animate, JSON body)
- asset (required): an id from GET /api/assets.
- prompt (required, <= 300 chars): a short caption in the training style. Start with "An object" and describe
  ONE motion, not the character: "An object walks forward.", "An object flaps its wings.", "An object swims and
  turns around.", "An object rears up on its hind legs.", "An object jumps in place.", "An object bites forward."
  The closer to that style, the better it is followed. Longer scene descriptions are followed worse.
- num_samples (1-4, default 1): independent samples of the same prompt (different noise). Samples vary; ask for
  several and keep the best (root_travel_xz and root_height_range in the result help to compare).
- cfg_scale (1.0-15.0, default 3.0): classifier-free guidance. Raise to 5-7 when the prompt is ignored; lower
  for smoother, calmer motion; 1.0 is unconditional.
- seed (integer, default random): same asset + prompt + seed + cfg_scale + num_samples reproduce the samples.
- preview (default true): also render the skeleton MP4 and contact sheet (about 1-2 s of CPU per sample).

## Modes beyond one prompt (the upstream editing options, all verified on this Space)
All three are UniMate's own replacement-style flow sampling (unimate.inference.motion_inbetweening,
motion_editing, motion_expansion): the kept part of a reference motion is pinned at every ODE step, the rest is
denoised under the prompt. They share num_samples, cfg_scale (must be above 1.0), seed and preview with
/api/animate, return the same result shape (plus the fields noted below) and cost about 7 s of GPU time per
sample and segment. In-betweening and editing take a reference from an earlier result of this service (any
mode, same asset, kept 24 h): reference_job_id, reference_sample (default 0) and reference_start (default 0;
the first frame of the 60-frame window, useful on the longer expand results; a window that does not start at
frame 0 is re-aligned to face +Z at its first frame, as upstream does).

1. In-betweening: POST https://townbox-unimate-for-agents.hf.space/api/inbetween (MCP: inbetween)
     {"reference_job_id": "a_...", "reference_sample": 0, "keep_frames": "0,-1",
      "prompt": "An object jumps in place.", "seed": 1}
   keep_frames (default "0,-1": first and last frame): frame indices held exactly at the reference, as a
   comma-separated string or a JSON list; negatives count from the end ("0,1,2,-3,-2,-1" keeps three frames at
   each end). The other frames are regenerated under the prompt (upstream --inbetween --keep_frames). The
   result adds "reference" and the resolved "keep_frames" ([0, 59]) and "generated_frames" (58). Use it to
   change what happens between two poses, to make a clip loop (keep frame 0 and ask for a motion that returns
   to it), or to redo the middle of a segment of an expand result (reference_start = the segment's first frame).
2. Joint editing: POST https://townbox-unimate-for-agents.hf.space/api/edit (MCP: edit_joints)
     {"reference_job_id": "a_...", "keep_joints": "Left Thigh, Right Thigh, Left Shin, Right Shin, Left Foot, Right Foot",
      "prompt": "An object flaps its wings.", "seed": 4}
   keep_joints (required): joint names or anatomical labels of the asset (GET /api/assets/<id>: joint_names and
   joint_labels), case-insensitive, comma-separated or a JSON list; a label such as "Left Wing" selects every
   joint carrying it; every requested name must match. Those joints keep the reference motion over all 60
   frames (positions, the rotation of the bone leading to them, velocities), the other joints are regenerated
   under the new prompt (upstream --motion_edit --keep_joints). Typical: keep the legs of a walking clip and ask
   for an upper-body action. The result adds "reference", "keep_joints" and "edited_joints" (index, name, label).
3. Motion expansion: POST https://townbox-unimate-for-agents.hf.space/api/expand (MCP: expand)
     {"asset": "unitree_go2", "prompts": ["An object trots forward.", "An object turns around.", "An object sits down."],
      "overlap": 10, "seed": 7}
   prompts: 2 to 6 captions, one 60-frame segment each, in order. overlap (1-59, default 10): the first
   `overlap` frames of each further segment are pinned to the last `overlap` frames of the previous one, so the
   motion is continuous and the seam frames appear once: 60 + (60 - overlap) x (prompts - 1) frames in one GLB,
   .npy and JSON (3 prompts: 160 frames, 5.3 s). Each segment is generated in its own +Z-facing frame and rotated
   back into the chain, so a turn in one segment carries into the next (upstream --motion_expand
   --expand_overlap). num_samples x prompts is limited to 8 per call. The result adds "prompts", "overlap" and
   "segments" (index, prompt, [first frame, end frame) in the chain); "prompt" is the captions joined by " | ".
   Measured on this Space (trot -> turn around -> sit down, seed 7): the root path is continuous, but the first
   regenerated frame after a seam can pop (facing jumped 33 deg at the trot-to-turn seam with overlap 10, 48 deg
   with overlap 20, 55 deg with overlap 30; 10 deg at the turn-to-sit seam). Keep the default overlap, prefer
   prompts that follow naturally, and re-run with another seed when a seam shows.
Verified on this Space (2026-11-05, 50 ODE steps): kept frames and kept joints equal the reference to 1.2e-7; a
windowed reference (reference_start 100 of a 160-frame expansion) is re-aligned exactly as upstream does; the
160-frame expansion GLB matches Blender 5.1's importer to 6.6e-6 on every frame. The three verification runs are
examples 7-9 of GET /api/examples. Checks you can run yourself: kept frames / joints of motion_npy equal the
reference's; an expand result's joints_json has 60 + (60 - overlap) x (prompts - 1) frames.

## Outputs (per sample)
- animation_glb: the asset's canonical rest-pose mesh with the skeleton and the generated animation (glTF 2.0,
  one animation named after the prompt, 60 keys at 30 fps, LINEAR; expand results: 60 + 50 keys per extra
  prompt). Plays in any glTF viewer, Blender, three.js, Unity, Godot. The canonical frame: Y up, skeleton
  diameter 2, the character faces +Z at frame 0.
- motion_npy: NumPy array (frames, J, 12), float32: per joint root-relative position (3), rotation in continuous 6D
  relative to the T-pose (6), local velocity (3); the root slot holds height, the facing quaternion and root
  velocity. This is exactly what `python -m unimate.inference.sample` writes; the UniMate repository's
  scripts/run_animate_motion.sh and unimate.utils.motion_utils read it.
- joints_json: {"fps", "frames", "joint_names", "joint_labels", "parents", "positions": [frame][joint][x,y,z]}
  world-space joint positions in the canonical frame, from forward kinematics of the generated rotations.
- skeleton_mp4 / contact_sheet_png: a rendered skeleton (red: root) for a quick look; the PNG shows 6 frames
  from 0 s to 2 s left to right. MCP: view_animation(job_id, sample) returns the PNG.

## Examples
GET https://townbox-unimate-for-agents.hf.space/api/examples lists nine real outputs generated by this Space's own API (six by /api/animate and one
each by /api/inbetween, /api/edit and /api/expand: mode, asset, prompt, seed, cfg_scale, job id, GPU seconds, root
travel, and for the modes the reference, kept frames / joints or segments), each with URLs for the animated GLB,
motion .npy, joints JSON, skeleton MP4, contact sheet PNG and a rendered poster, plus the exact POST body that
reproduces it. Files are served at https://townbox-unimate-for-agents.hf.space/examples/<name>. They are a showcase only: every API call generates
fresh results.

## Your own rigged asset
The Space does not run the Blender-based preprocessing. Preprocess locally with the UniMate repository
(https://github.com/Friedrich-M/UniMate, MIT; needs its `unimate` conda env with the pip `bpy` module):
    python -m data_process.rig_preprocess run --input robot.glb --output_dir out/robot --no_review
    zip -r robot.zip out/robot          # cond.npy, robot.glb (canonical rest pose), summary.json, preview.png
    curl -s -X POST "https://townbox-unimate-for-agents.hf.space/api/assets?filename=robot.zip" -H "Authorization: Bearer $HF_TOKEN" --data-binary @robot.zip
-> 201 {"id": "u_<12 hex>", "n_joints": ..., "has_mesh": true, "next": [...]}; then animate with "asset": "u_...".
Limits: at most 71 joints (simplify game rigs with fingers / face bones), 60 MB per upload. The joint labels and
the facing pair baked into cond.npy drive the model: review them (the tool's annotation.json / REVIEW.md) before
uploading. A cond.npy without a mesh is accepted too (then no animation_glb, only motion_npy and joints_json).

## Reading a result
- state is always "done" (calls are synchronous); errors come back as HTTP 4xx/5xx {"error", "hint"}.
- mode: generate, inbetween, edit or expand; frames: 60, or 60 + (60 - overlap) x (prompts - 1) for expand.
- timing.gpu_s is the ZeroGPU call (model on a half RTX Pro 6000 Blackwell): measured 7 s for one sample on a
  warm worker (10 s wall time with the outputs), 15 s for two samples on a cold worker; inbetween and edit take
  about the same as a sample, expand about 7 s per prompt. GET /api/gpu_check runs a 10 s probe that reports
  the GPU.
- provenance: model repo and revision, checkpoint step, EMA weights, text encoder, code commit, licenses.
- root_travel_xz: distance the root moved on the ground plane over the clip (canonical units; the skeleton is
  2 units across), useful to tell "walks forward" from "in place". root_height_range: min/max root height.

## Errors and hints
- 400 unknown asset / empty prompt / out-of-range parameter: the message says what to change.
- 400 keep_joints names that match no joint (the hint lists the asset's names and labels), keep_frames outside
  [-60, 59], prompts with fewer than 2 or more than 6 captions, num_samples x prompts above 8, cfg_scale 1.0 in
  inbetween / edit / expand.
- 404 unknown job: results are kept 24 h (or until the Space restarts); a reference_job_id must be a result of
  this Space, and its asset must still exist (uploaded assets vanish with a restart).
- 500 with "quota" or "GPU limit": your ZeroGPU quota is exhausted or no token reached the service; send
  `Authorization: Bearer $HF_TOKEN`, check GET /api/identity, or wait for the daily quota to reset.
- 500 "no GPU was available": ZeroGPU had no free slice; retry in a moment.

## Limitations (from the model card)
- No contact model: feet can slide, hover or penetrate the ground; rare topologies and unusual motions can come
  out static, jittery or off-prompt. Many motions and skeletons still fail; try several samples and phrasings.
- 60 frames per prompt. Longer motions come from /api/expand (up to 6 prompts, 310 frames); in-betweening and
  editing take their reference from a result of this service (any 60-frame window of it), not from uploaded
  motion files. The editing modes use a fixed-step Euler solver (50 steps, as upstream) instead of the adaptive
  dopri5 of plain generation.
- Rotations only: the model generates joint rotations and the root path; bone lengths stay those of the rest pose.

## Provenance and licenses
- Code: https://github.com/Friedrich-M/UniMate (MIT), vendored at commit b78c7807; the inference path is the
  repository's own (unimate.inference.generate, motion_inbetweening, motion_editing, motion_expansion,
  unimate.dataset, unimate.models). Mesh driving is a glTF re-implementation of scripts/run_animate_motion.sh
  (no Blender), checked against Blender's importer to 1e-5.
- Weights: https://huggingface.co/Linzhan/UniMate, folder unimate_uniml3d_f60_v3, checkpoint_step_150000.pt
  (EMA weights), revision 174fc755. License CC BY-NC 4.0: non-commercial research use; the training data keep
  their own terms (Truebones ZOO, Mixamo, Objaverse-XL), see the model card.
- Text encoder: google/flan-t5-base (Apache 2.0), frozen.
- Example rigs: assets/examples of the UniMate repository.
- Paper: UniMate: One Unified Model to Animate Diverse Skeletons, Mou et al., SIGGRAPH Asia 2026,
  https://arxiv.org/abs/2609.05415. Project page: https://linzhanmou.com/unimate/ (its 3D showcase is
  access-controlled and is not used here).