EmoteGPT: 3D Human Facial Expression from Natural Language Descriptions

Precise control of 3D facial expressions from text is crucialfor virtual avatars, animation, and human–computer interaction, yet ex-isting text-to-3D methods jointly generate identity, expression, and tex-ture, making fine-grained expression control difficult. We instead for-mulate text-driven expression synthesis as a regression problem in thedisentangled parameter space of a 3D Morphable Model (3DMM). Thissetting, however, requires paired data linking detailed language to pre-cise expression parameters, which is missing from existing resources. Tofill this gap, we introduce Txt2Emote, a benchmark of diverse 3D facialexpressions with fine-grained textual annotations obtained from GPT-4o and a high-fidelity face tracker, providing both explicit descriptionsdetailing facial features and implicit descriptions referencing the situa-tional context behind the expression. Leveraging this dataset, we presentEmoteGPT, a text-to-3D expression framework based on a Multi-ModalLarge Language Model (MLLM) with a dedicated token to se-mantically ground expression representations, which are then decodedinto 3DMM parameters. We further improves EmoteGPT by augmentingtraining with large-scale image-to-3DMM data, surpasses state-of-the-arttext-to-3D face synthesis methods on emotion recognition metrics andin perceived expressiveness. Integrated into avatar pipelines, our methodenables photorealistic and stylized 3D avatars, as well as expressive 3D-consistent 2D face synthesis from textual input.