The problem really comes down to the fact that modifiers are only applied to focus NPCs. I'm experimenting now with a build that will allow any NPC to be assigned modifiers. Possible downsides in increasing the complexity of the LLM's instructions and therefore making errors more likely but I'll add it if it doesn't degrade performance.