Hi! Thanks for letting us know about this problem. We have tested with various controllers here, including Xbox One and Nintendo Switch, and there is no problem with those.
It must be something specific to do with Xbox Series X controllers but we don't have one here to test. We'll investigate and see what we can do about it!
Thank you for playing :)