AudioCraft

A

AudioLDM 2

CUHK & Surrey|N/A

AudioLDM 2 is a unified audio generation framework developed by researchers at the Chinese University of Hong Kong and the University of Surrey, capable of producing music, sound effects, and speech from text descriptions within a single model. Building on the original AudioLDM, version 2 introduces a universal audio representation called Language of Audio that bridges the gap between different audio types by encoding them into a shared semantic space. The model combines a GPT-2 language model for understanding text inputs with an AudioMAE encoder for audio conditioning, feeding into a latent diffusion model that generates audio spectrograms which are converted to waveforms. This architecture enables AudioLDM 2 to handle diverse audio generation tasks without requiring separate specialized models for each audio type. The model achieves competitive performance across multiple benchmarks including text-to-music, text-to-sound-effects, and text-to-speech evaluations. AudioLDM 2 generates audio at up to 48 kHz with good perceptual quality for both musical and non-musical content. Released in August 2023 under a research license, the model is open source with code and pre-trained weights available on GitHub and Hugging Face. AudioLDM 2 supports audio inpainting, style transfer, and super-resolution in addition to text-conditioned generation. The model is particularly relevant for researchers studying unified audio generation, content creators needing diverse audio types from a single tool, and developers building comprehensive audio generation systems. Its unified approach to handling speech, music, and environmental sounds makes it a versatile foundation for multi-purpose audio applications.

B

Bark

Suno AI|N/A

Bark is a transformer-based text-to-audio generation model developed by Suno AI that converts text into natural-sounding speech, music, and sound effects. Released as open source under the MIT license in April 2023, Bark goes far beyond traditional text-to-speech systems by generating not only spoken words but also laughter, sighs, music, and ambient sounds from text descriptions. The model uses a GPT-style autoregressive transformer architecture with EnCodec audio tokenizer to generate audio tokens that are then decoded into waveforms. Bark supports multiple languages including English, Chinese, French, German, Hindi, Italian, Japanese, Korean, Polish, Portuguese, Russian, Spanish, and Turkish, making it one of the most multilingual open-source audio generation models available. The model can clone voice characteristics from short audio samples, allowing users to generate speech in specific voices or speaking styles. Bark operates in a zero-shot manner, meaning it can produce diverse outputs without task-specific fine-tuning. Generation includes natural prosody, emotion, and intonation that closely mimics human speech patterns. The model generates audio at 24 kHz sample rate with reasonable quality for most applications. As a fully open-source project with pre-trained weights available on Hugging Face and GitHub, Bark is widely used by developers building voice applications, content creators producing multilingual audio, and researchers exploring generative audio models. The model is particularly valued for its versatility in handling diverse audio types within a single unified architecture and its accessibility for rapid prototyping of audio generation applications.

M

MusicGen

Meta|3.3B

MusicGen is a single-stage transformer-based music generation model developed by Meta AI Research as part of the AudioCraft framework. Released in June 2023 under the MIT license, MusicGen uses a single autoregressive language model operating over compressed discrete audio representations from EnCodec, unlike cascading approaches that require multiple models. The model comes in multiple sizes ranging from 300M to 3.3B parameters, allowing users to balance quality against computational requirements. MusicGen generates high-quality mono and stereo music at 32 kHz from text descriptions, supporting a wide range of genres, instruments, moods, and musical styles. Users can describe desired music using natural language prompts specifying genre, tempo, instrumentation, and atmosphere, and the model produces coherent musical compositions that follow the specified characteristics. Beyond text-to-music generation, MusicGen supports melody conditioning where an existing audio clip guides the melodic structure of the generated output, enabling more controlled music creation. The model achieves strong results across both objective metrics and subjective listening evaluations, producing music that sounds natural and musically coherent for durations up to 30 seconds. As a fully open-source model with code and weights available on GitHub and Hugging Face, MusicGen has become one of the most widely adopted AI music generation tools in both research and creative communities. It integrates easily into existing audio production workflows through the Audiocraft Python library and various community-built interfaces. MusicGen is particularly popular among content creators, game developers, and musicians who need royalty-free background music generated on demand.

M

MusicLM

Google|N/A

MusicLM is a text-to-music generation model developed by Google Research that generates high-fidelity music from text descriptions at 24 kHz. Published in January 2023 alongside a research paper, MusicLM was one of the first models to demonstrate that AI could generate coherent, high-quality music spanning multiple minutes from natural language descriptions alone. The model employs a hierarchical sequence-to-sequence architecture combining SoundStream for audio tokenization and w2v-BERT for audio representation learning, generating music tokens at multiple temporal resolutions that are then decoded into waveforms. MusicLM can produce music in diverse genres and styles based on text prompts describing instruments, tempo, mood, and musical characteristics, maintaining musical coherence and structural consistency across extended durations. The model also supports melody conditioning where users can hum or whistle a melody that guides the generated output, enabling more intuitive music creation workflows. MusicLM generates audio with rich timbral quality and natural-sounding dynamics that represent a significant improvement over earlier text-to-music approaches. As a proprietary Google model, MusicLM is not open source and was initially accessible only through the AI Test Kitchen experimental platform before being integrated into broader Google services. While newer models like MusicGen and Suno have since achieved wider adoption, MusicLM remains historically significant as a pioneering demonstration of high-quality text-to-music generation. The model influenced subsequent research and commercial developments in the AI music generation space and helped establish text-to-music as a viable and rapidly advancing field of AI research.