Ask HN: Is there an ML model that can go from an audio song to sheet music?
Trying to learn some foreign music where the sheet music is not exactly available (old folk music type stuff) so I was wondering if there is a model or an app that can go from an audio file to sheet music or notation.
I was going to post basic pitch from Spotify but it looks like billconan beat me to it. That said I can give you a bit more advice. The Spotify basic pitch model isn't too good at multi-track input. It's capable of it, but you may actually get better results if you separate out the tracks first and then run them individually through the basic pitch model.
Either way the key words you want to be looking for are "midi transcription" and "stem separation", should help you find more models to try for both steps. Good luck! :)
Is that worth it tough? It seems to me the effort of cleaning up sheet music generated from MIDI would be higher than transcribing the whole piece by hand.
I imagine that if you separated a track into its components you could extract the sheet music for each part individually. I don’t know how sheet music for clapping would work though.
spleeter, demucs (my favorite), etc, can split a song into 4 tracks (drum, bass, vocal, and other). Your other comments mentioned vocals, percussion (hand drums and claps), and organ; on paper it sounds like it should work.
This is called (multitrack) music transcription. There are some commercial solutions (AudioScore, AnthemScore, ...).
For OSS, look at Omnizart [1] and magenta/mt3 [2].
I suppose these models are trained on western / pop music, so they may not work nicely on ethnic music.
I've been working on training models for this problem. Haven't shipped as a product yet because the inference cost is too high (takes 10-15 minutes to transcribe each song). Would love to try it out on the music you're trying to learn and get your feedback. Email address is in my profile.
In order to do this you can use a source/stem separation model like spleeter (https://github.com/deezer/spleeter) and then run the basic pitch model (or any other midi transcription model). There's other you can try which may yield better results, for example: (https://github.com/Music-and-Culture-Technology-Lab/omnizart)
Either way the key words you want to be looking for are "midi transcription" and "stem separation", should help you find more models to try for both steps. Good luck! :)
EDIT: Oh it looks like there's even a stem separation leaderboard on papers with code, neat: https://paperswithcode.com/task/music-source-separation
Then there are open source programs that can convert midi to sheet music.
I have tried the process, but you tend to get artifacts in the midi from Ableton.
You have to know a little bit about music theory to clean it up.
Some people can do this cleanup really quickly.
I suppose these models are trained on western / pop music, so they may not work nicely on ethnic music.
[1] https://github.com/Music-and-Culture-Technology-Lab/omnizart [2] https://github.com/magenta/mt3
This might help. Not exactly what you're looking for.
Imagine how cool, useful, and joyful this tech would be to most people.