Ok so factory reset did not help, but setting headphones to SBC seemed to solve this. Using LDAC/AAC seems to cause the freezing so if you're able to set your headphones to a lower standard/connection priority, you might be ok.
