ComfyUI ACE-Step: AI Music Generation Workflow

ComfyUI ACE-Step integrates the ACE-Step music generation model into the ComfyUI environment. ACE-Step uses a hybrid architecture combining diffusion-based generation with Sana's Deep Compression AutoEncoder (DCAE) and a lightweight linear transformer. This enables rapid generation of high-quality music with exceptional control capabilities. It allows users to create original music across various genres and styles using simple natural language prompts and lyrics.

ComfyUI ACE-Step offers several advantages: * **Unprecedented Speed:** Synthesizes up to 4 minutes of music in just 20 seconds. * **Musical Coherence:** Maintains excellent quality across melody, harmony, and rhythm. * **Multilingual Support:** Generates music in 19 languages. * **Advanced Control:** Enables voice cloning, lyrics editing, remixing, and track generation with fine-grained parameters. * **Creative Flexibility:** Supports diverse musical styles, genres, and instruments with various description formats. * **Seamless Integration:** Connects directly to ComfyUI workflows for AI-assisted audio creation.

Using the ComfyUI ACE-Step workflow involves setting up the nodes and parameters within the ComfyUI interface. This includes configuring the text prompts, sampler settings, and audio output options to generate the desired music.

There are several methods for generating music with ACE-Step, including: * **Main ACE-Step Generation Workflow:** Best for creating original music from text descriptions and lyrics. * **Lyric2Vocal:** Tailored for generating high-quality vocals from lyrics. * **Text2Samples:** Specialized for creating instrumental loops and samples. * **RapMachine:** Optimized for generating rap music in various styles.

Key parameters within the ComfyUI ACE-Step workflow include: * **TextEncodeAceStepAudio Node:** * `clip`: Text field for style, genre, and mood descriptions. * `lyrics`: Text field for lyrics with optional structure tags. * `lyrics_strength`: Controls the influence of lyrics on generation. * **KSampler Node:** * `seed`: Sets the initial value for randomization. * `steps`: Number of diffusion steps. * `cfg`: Classifier-free guidance scale. * `sampler_name`: Sampling algorithm. * `scheduler`: Noise schedule type. * `denoise`: Controls the level of noise removal. * **EmptyAceStepLatentAudio Node:** * `seconds`: Duration of the generated audio. * `batch_size`: Number of samples to generate simultaneously. * **VAEDecodeAudio Node:** * `samples`: Input from KSampler. * `vae`: VAE model used for decoding. * **SaveAudio Node:** * `filename_prefix`: Prefix for saved audio files. * `audio`: Player for previewing generated audio.

Advanced techniques include: * **Generating Variations:** Adjust the variance parameter to control similarity to original generations. * **Inpainting:** Selectively regenerate specific audio sections. * **Lyrics Editing:** Modify lyrics while preserving melody, voice timbre, and accompaniment. * **Voice Cloning:** Preserve vocal characteristics when generating new content. * **Style Transfer:** Apply new musical styles to existing compositions.

For general music, be specific about genre, mood, and instrumentation. For instrumental music, specify instruments and musical characteristics. ACE-Step performs best with English, Chinese, Russian, Spanish, Japanese, German, French, Portuguese, Italian, and Korean.

The ACE-Step model was developed by ACE Studio and StepFun. The ComfyUI integration enables seamless music generation within the ComfyUI environment. Full credit is given to the original authors for their innovative work on ACE-Step.