Totally an awesome first game jam entry, there's originality and the idea could become something really great. It'll be interesting to see what you can do with some UI too.
Good job getting into a game jam!