Try to set collision mode for character collider to continuous in Unity instead of discrete, that should help.
