Certainly a creative idea. I think the controls of dragging and menu use are just not convenient at all though. I think what also you could do to take your game to the next level is to add feedback elements (definitely sounds, also some shake and some particle effects).