Generating Dyadic Context-Aware Gestures for Virtual Avatars via Causal Transformers
MSc thesis at the Max Planck Institute for Psycholinguistics: causal, streaming dyadic co-speech gesture generation with SMPL-H, RVQ-VAE, and multi-GPU cluster training.
This is my MSc AI thesis research at the Max Planck Institute for Psycholinguistics (Nijmegen), a well-known institute for language and multimodal communication. Daily supervision was by Dr. Esam Ghaleb. University supervision at UvA was by Prof. Raquel Fernández. The problem: most co-speech gesture models are monadic and non-causal. They only model one speaker and need future audio. Real conversational agents need dyadic, streaming motion instead. I built a two-stage pipeline on Meta's Seamless Interaction data. Stage 1 is a hierarchical RVQ-VAE on SMPL-H upper-body rotations. Stage 2 is a decoder-only causal transformer with speaker and partner audio, FiLM for personality and volume, progressive ablations, and a custom rendering stack. Scale that mattered: a curated 13.1h high-motion Stage-1 subset, Stage 1 trained ~14h from scratch with DDP on 4x A100, and Stage 2 ~8h on a 150h dyadic subset (200 epochs), all on the MPI computing cluster. Outcome: full thesis, open code and visuals, and a poster at NCCV 2026.
Research problem framing: turned a messy SOTA landscape into clear gaps (monadic vs dyadic, beat vs semantic, non-causal vs causal) and wrote concrete research questions (RQ1-RQ4) with measurable objectives.
Literature to method: mapped related work (SemTalk, DyaDiT, EMAGE, diffusion and flow models, and others) into design choices instead of copying or building upon one paper.
Dataset science / experimental hygiene: designed algorithmic curation for noisy multi-vendor dyadic data (hand boundary filtering, motion-intensity thresholding, VAD padding, role-adaptive speaking/listening segmentation) so models train on signal, not tracking junk.
Hypothesis-driven modeling: two-stage discrete motion tokens plus causal generation. Tested whether continuous personality and partner audio actually help via progressive ablations (timing/VAD, then semantics, then persona, then the full dyadic loop).
Quantitative evaluation design: Fréchet distance, motion diversity, beat consistency. Monadic vs dyadic setups. Short (10s) vs long (30s) clips for stability.
Qualitative scientific judgment: built fast visual loops because MSE alone lies for motion. Audited reconstructions, listening poses, and tracking-artifact denoising by eye.
Scientific communication: wrote the full MSc thesis with clear contribution claims and honest limitations. Presented a poster at NCCV 2026 to the research community.
Independent research ownership: Ownership of pipeline, models, eval, and open release, intended as a baseline for a new dataset at the Max Planck Institute for Psycholinguistics.
ML / systems skills
Deep learning at scale: multi-stage neural pipelines trained from scratch, not notebook demos.
SMPL-H / 3D human motion: parametric body models, 6D rotations, forward kinematics, vertex losses through a frozen SMPL-H layer.
Discrete latent motion (RVQ-VAE): hierarchical codebooks, commitment loss, dead-code refresh, avoiding codebook collapse. Stage 1 as a spatial denoiser prior.
Causal / streaming generation: decoder-only transformer, zero look-ahead, decoupled Layer-1 AR plus residual fine codes, scheduled sampling / exposure-bias handling, classifier-free guidance.
Multimodal conditioning: HuBERT/BERT-style streams, partner audio, FiLM for personality and volume.
Cluster computing / long jobs: MPI computing cluster (SGE), 4x A100 DDP (Stage 1 ~14h / 800 epochs), multi-hour Stage-2 runs, job scripts, uv envs, and long feedback loops.
Training ops / monitoring: Weights & Biases across long runs to iterate toward stable RVQ-VAE and generative behavior.
Data engineering at research scale: Seamless download, multi-worker segmentation, LMDB caches (large max_gb), manifests, and pair-preserving dyadic mode.
Eval and visualization tooling: metrics scripts plus Gradio / headless Pyrender (EGL) rendering for SMPL-H/SMPL-X as a reusable research contribution.
What I built
Data curation pipeline for Seamless Interaction (filter, segment, LMDB).
Stage 1 hierarchical RVQ-VAE on SMPL-H upper body.
Stage 2 causal transformer with dyadic/social conditioning and an ablation suite.
Research is mostly diagnosing failure modes: codebook collapse, exposure bias, silent listening that does not generalize.
Cluster work needs discipline. Long jobs force careful logging, smaller local checks, and patience with feedback loops.
Metrics alone are not enough for motion. You need a fast visual loop or you will trust the wrong runs.
Writing and presenting are part of the skill set, not afterthoughts. The thesis and NCCV poster forced clearer claims and honest limitations.
Scope
Research thesis at the Max Planck Institute for Psycholinguistics. Full technical depth lives on the paper site and PDF. This portfolio page is for the skills and the setting.
