Play a song in any app. It will show up here.
Chords from audio it has already heard
Every 40 ms, 25 times a second, Rubato turns the newest slice of sound into a chord. It never waits for the rest of the song, so it can follow live playing. It runs entirely on this device: the audio is never uploaded.
A constant-Q transform measures how loud each musical note is: 144 bins over 6 octaves, two per semitone. Every window ends at "now".
A small convolutional network spots local patterns: which notes stack into a chord, which are just overtones, where notes start.
Adds context from the last ~32 s, so it knows the key and feel of the song when a moment is ambiguous. A KV cache keeps each step cheap.
A hidden Markov model filters the guesses over time: chords tend to last, and some changes are far likelier than others. It runs forward only, so it adds no delay.
A rolling estimate of the key spells chords the way a musician would: G♯m in E major, D♭ in A♭ major.
The live chord, a steadier big chord that hides split-second blips, and a sheet that writes a chord down once it has held for half a second.
How it was trained
- First on synthetic audio: chord progressions rendered with random instruments, so the labels are exact.
- Then on real recordings labelled by people: US Billboard hits (the McGill Billboard dataset) and 152 Bollywood songs whose chords come from guitar chord sheets, lined up to the audio.
- Every song is shifted into all 12 keys and gets random EQ and bass cuts, so the model listens to harmony, not just the bass line.
What's different
- vs. whole-song tools (upload a track, get chords): those can look ahead and back over the entire recording. Rubato only uses what it has heard, plus a deliberate 120 ms, so it works live on anything you can point a mic at.
- vs. classic live detectors (match each instant against chord templates): Rubato uses a deep model with half a minute of memory and knows 109 chord types, not just major and minor.