I like the concept, but I think it could have been presented differently. I thought it was a generic memory game until I saw that pressing H let you listen to the sounds. The problem is that when the notes are shown initially, it also highlights the block which makes the sound obsolete. Instead if the highlighting is removed and you just let the player figure out based on sound, it would have been way more interesting IMO. I tried this by looking away when the sounds played and I think it works. Perhaps a smaller grid would be needed in that case.
Multiple modes were nice for a jam game, so kudos for that! I think it would also be better if the art were a bit more cohesive though; the somewhat shabby style still kind of works, but the colors clash way too much.