Stem separation looks like magic and is actually estimation. The model never recovers the original stems — it predicts what they probably were, then divides the mixed audio according to that prediction. Understanding that explains every artifact you've ever heard.
Audio is usually converted into a spectrogram — a picture of which frequencies are present at each moment. In that picture, sources look different: a voice has a characteristic harmonic structure and vibrato, a snare is a broadband transient, a bassline is low-frequency and sustained.
A neural network is trained on millions of examples where both the mix and the true separated stems are known. It learns to look at a spectrogram and predict, for each point in time and frequency, how much of that energy belongs to each source.
That prediction is called a mask. Apply the vocal mask to the mixed audio and you get the vocals.
The problem is overlap. When the vocal and the guitar both have energy at 800Hz at the same instant, the model must decide how to split it. It cannot know for certain — the information genuinely isn't there.
If it splits slightly wrong, you get:
- Watery or phasey vocals — small errors across many frequency bins, heard as smearing.
- Bleed — bass leaking into the drum stem when the kick and bass occupy the same range.
- Smeared reverb tails — reverb belongs to a source but sounds diffuse, so the model spreads it.
- Missing high end — cymbals and vocal sibilance overlap heavily and often get split badly.
Two approaches compete:
Spectrogram-based models work on the frequency picture. They're efficient and good at harmonic content, but the conversion back to audio can lose phase information, which is part of the watery quality.
Waveform-based models, like the Demucs family, work directly on the raw audio samples. They preserve phase better and generally sound more natural, at higher computational cost. Most current high-quality separation uses waveform or hybrid approaches.
The pattern is consistent:
| Separates well | Separates badly |
|---|
| Sparse arrangements | Dense walls of sound |
| Wide stereo placement | Mono-heavy mixes |
| Distinct registers | Instruments in the same range |
| Moderate compression | Heavily limited loud masters |
| Dry recordings | Heavy reverb and delay |
A loud, brickwalled modern master is the hardest case: limiting compresses the dynamic differences the model uses to tell sources apart.
Live separation in DJ software must process in milliseconds, so it uses smaller, faster models and produces more artifacts. Offline separation can use a large model and process the whole file with full context, which is why the same song sounds noticeably cleaner separated at home than on a deck.
- Expect estimation, not extraction. Separated stems are a reconstruction.
- Feed it the best source you have. A lossless file separates better than a 128kbps MP3, because compression already discarded information the model needs.
- Use separation as a starting point. Replacing a badly-separated drum stem with a programmed one often sounds better than fighting the artifacts.
Tools like Veena run separation and then put the stems straight into an editable project, which matters because the usual next step after separating is repairing — and that is editing work.
Related reading: best stem separation tools, AI music tools and stem bleed, and music source separation models.
Start making music in Veena
Free, browser-based, no downloads required.
Try Veena Free