I think you have to enable continuous collision detection since the balls sometimes clearly pass a portal but they don't hit it.
