I have seen this happen with some phones mostly. if it's an issue that persists, you can disable drag and drop from the options. Then you will only have to tap on two gems to swap them.
