That makes sense why it was tricky to aim, maybe look into spherecasting instead of raycasting, just one idea.
