I feel like it should be your choice and not some petty / immature people! Its understood if you feel the need to remove him. But at the end of the day it is your game and people can't control watcha do! sending love !! :D
