Unlocking the mansion upgrade ought to be simple enough, but the effect it provides covers only dungeons and one-time tasks. For those who gather resources from the mansion and villages instead, the upgrade does not truly support their use case - especially in the early game, as workers are constantly shuffled around.
Admittedly, I also dislike the upgrade at a conceptual level: it eliminates the need to build and manage different kinds of tools, deciding where to invest your limited resources to gain the most benefit. In terms of role play, I also prefer the idea of "stocking the workshop" and letting my workers choose their tools themselves, as opposed to creating a universal multi tool for everyone to use.