Sorry, i forgot to mention in the instructions that you can only grab a baton at the start of your turn. As for the timers, that is part of an anti-frustration mechanic that increases the opponents minimum and maximum response time after you've already won a 1v1 against them. This is to prevent an RNG fest to get the achievements (times get back to normal ranges after you get them all), but i might have overtuned it too much. I will fix this in my next update.