I had already figured out the logic: if you left the banana there and traveled back in time, the button wouldn't be there anymore, so you just had to leave it on the button while in the past, and then you could just walk straight past it. But you had to go back and get the banana. It was a bit complicated, but it was my first game jam and I didn't have enough time to simplify it.