"Not all men" If I gave you a box of cookies and said some of those cookies are made of shit instead of cookie dough. You'd be weary of all the cookies in the box.
"Not all men" if you were swimming in the ocean and saw a shark. You would get out of the water. Doesn't matter if it's going to attack you or not or if there's a low chance to be bitten. You're getting the hell out.