Yea its a bit odd since they don't chase you. They only hurt you if they decide to move towards you and you dont get out of the way. Do you think it would be helpful if they always attack you if you're next to them?
That would make the game much harder, and the current behavior makes sense. I used the same rule for some enemies in one of my games. It's a nice counterpoint to the aggressive ones who have it in for you.