Generating audio is harder than generating images, and the approaches taken to make it tractable explain everything about how AI-generated music sounds.
An image is a fixed grid of pixels. Three minutes of stereo audio at 44.1kHz is over 15 million samples, each depending on those around it, with structure at every timescale from microseconds (timbre) to minutes (song form).
Generating that directly, sample by sample, is computationally brutal. Every practical approach is a strategy for avoiding it.
Predict each sample from the previous ones. WaveNet was the landmark example.
Quality: excellent — it models audio directly with no compression loss.
Problem: generating 44,100 sequential predictions per second is extremely slow. Impractical for long-form music without heavy optimisation.
The dominant approach. A neural codec compresses audio into a few hundred discrete tokens per second. A transformer generates token sequences. The codec's decoder turns them back into audio.
Quality: good, bounded by the codec. The compression is why generated music often has a slightly glassy quality in cymbals and sibilance.
Advantage: hundreds of tokens per second instead of 44,100 — tractable for full songs.
Generate a spectrogram by iteratively denoising, then convert to audio with a vocoder.
Quality: excellent texture.
Problem: the spectrogram-to-audio step must reconstruct phase information, which is lossy. And global refinement handles long-range structure poorly.
Rather than generating audio, the network predicts parameters for traditional synthesis — harmonic oscillator amplitudes, filtered noise levels, envelope shapes. Real DSP does the sound generation.
Quality: very clean, because the output is synthesised by well-understood algorithms rather than reconstructed.
Advantage: tiny models, small training sets, real-time capable, and inherently controllable.
Limitation: works best on harmonic, monophonic sounds. Less suited to dense polyphonic mixes.
DDSP is what powers most good timbre transfer.
| Sound | Cause |
|---|
| Glassy cymbals, metallic sibilance | Neural codec compression |
| Blurry stereo image | Codec or vocoder phase reconstruction |
| Beautiful texture, wandering structure | Diffusion without a structure model |
| Clean, expressive single instrument | DDSP |
All of these produce rendered audio. However sophisticated the synthesis, the output is a fixed file.
That's why the more useful question for a musician isn't which synthesis approach a tool uses — it's whether the tool gives you something editable afterwards: MIDI you can change, or stems you can rework.
Related reading: how AI generates music, timbre transfer explained, and diffusion vs transformer music models.
Start making music in Veena
Free, browser-based, no downloads required.
Try Veena Free