In the end it is your decision and who are we to judge.
However being called a perv when you have nothing to prove this is you developing this is just fear talking. Take it from an openly bi man who attend furry conventions and has 5 body shops.
What you do professionally and do at home are your business. Times are changing that is true but who is it to judge someone for likes and fetishes?
I do not know your profession or choices but like I said in the end if you are throwing in the towel because of others you have lost already. BUT if you are doing this for you then do not feel bad.