it only happened once so I'm not sure what the trigger is.
After losing, I respawned back at the beginning of the level and the time was already in the negative. No control at all.