I once played this similar game concept where you control camera and player separately in another jam and that was badly designed.
Your game however, is a puzzle game and it was executed nicely. It does not get annoying but rather makes us plan the order of actions to do. Well done.