AI Choreographer: Music-Conditioned 3D Dance Generation with AIST++

At the heart of the AI Choreographer system is the FACT (Flow-Aware Crossmodal Transformer) model. This innovative architecture leverages the power of transformers, a neural network architecture that has revolutionized natural language processing and is now proving highly effective in other domains, including computer vision and sequential data generation. FACT is specifically designed to handle crossmodal learning, meaning it can process and integrate information from different modalities – in this case, music and 3D human motion. The model takes a short seed motion sequence and a piece of music as input and is capable of generating extended, non-looping dance motions that are synchronized with the rhythm and mood of the music. This ability to maintain coherence and expressiveness over long sequences is a significant advancement in AI-driven dance generation.

The AIST++ dataset stands out due to its impressive scale and comprehensive nature. It is designed to serve as a benchmark for a variety of tasks, including motion generation and prediction, and has the potential to benefit other areas like 2D/3D human pose estimation. As of its release, AIST++ is recognized as the largest 3D human dance dataset available. It encompasses 1408 distinct dance sequences, featuring 30 different subjects performing a wide array of movements. The dataset covers 10 distinct dance genres, incorporating both basic and advanced choreographies. In total, AIST++ provides over 18,000 seconds of motion data, corresponding to more than 10 million images. This vast amount of data allows for robust training of sophisticated AI models.

The research behind AI Choreographer represents a significant contribution to the fields of AI, computer vision, and animation. The development of the FACT model, a crossmodal transformer, addresses the complex challenge of synchronizing visual motion with auditory input. The creation and release of the AIST++ dataset provide the research community with a vital resource for advancing the state-of-the-art in 3D human motion synthesis. The paper detailing this work, authored by Ruilong Li, Shan Yang, David A. Ross, and Angjoo Kanazawa, was presented at ICCV, a premier computer vision conference, highlighting the academic significance of this project. The project also acknowledges the contributions of various individuals and teams for their discussions, code sharing, and support in user study experiments, underscoring the collaborative nature of scientific advancement.