I really like the idea (very creative), but I wish controls felt more snappy.
A part of me thinks that maybe you can switch and swap using just your mouse and you don’t have to be near the npc (something similar to the hacking system in gunpoint). This will introduce complete freedom to the player but still, I think it will be a lot more fun.
Anyway, great idea, keep at it and I am sure you can make a lot more cool stuff!