Timbre transfer is the technology behind "hum it and hear a guitar." It's a genuinely different approach from pitch detection, and the difference matters.
Any musical sound can be thought of as two things:
- Performance — pitch over time, timing, dynamics, vibrato, slides, articulation.
- Timbre — the character that makes a trumpet sound like a trumpet: harmonic structure, attack shape, noise components, how tone changes with volume.
Timbre transfer keeps the first and replaces the second.
Most approaches use an encoder-decoder structure. The encoder analyses input audio and extracts a performance representation — fundamental frequency over time, loudness envelope, and features capturing expression — while deliberately discarding timbre.
A decoder trained on the target instrument then generates audio with that instrument's timbre, following the extracted performance.
Many systems use differentiable digital signal processing (DDSP), which combines neural networks with classic synthesis: the network predicts parameters for harmonic oscillators and filtered noise, and the synthesis itself uses well-understood DSP. This produces cleaner results than pure neural generation and needs far less training data.
Pitch-to-MIDI extracts discrete note events — note on, pitch, velocity, note off — and triggers a sampler. Everything between the notes is thrown away: the slide into a note, the vibrato that develops halfway through, the breath at the start of a phrase.
Timbre transfer keeps continuous pitch and loudness, so those survive. It's why a hummed melody transferred to saxophone sounds performed rather than sequenced.
Physically implausible performances. Hum a rapid run and map it to a tuba, and you get something no tuba could play. The result sounds synthetic because it is.
Out-of-range notes. Ask for a note below a violin's lowest string and the model extrapolates badly.
Polyphony. Most timbre transfer is monophonic. Chords generally need separate passes.
Noisy input. Background noise gets interpreted as performance detail and transferred.
- Perform it like the target instrument. Hum a sax line the way a sax player would phrase it — breaths included.
- Record cleanly. Close mic, quiet room. Noise becomes artifacts.
- Stay monophonic. One note at a time.
- Respect the range.
Used well — as in Veena's voice-to-instrument feature — this solves a real problem: you can hear a horn line in your head but can't play horns. Your voice becomes the interface to every instrument.
Related reading: hum to full song, neural audio synthesis explained, and how to turn a guitar riff into a full track.
Start making music in Veena
Free, browser-based, no downloads required.
Try Veena Free