You could just kick/ban whoever you thought was the imposter. Or you could ban anyone who disagreed with you. Trust me, that kind of power advantage over other players can get super toxic.