I found the problem. It does seem half-bugged though. Even after restarts this happens.
If you talk to them without doing the appropriate attack action and canceling the action to get them following you, you just get a cut-off message that says "re following you" and you get locked into an invisible menu, and just become frozen in place until you hit "escape" either 1 or a few times to free yourself.
I kept getting them to follow me, but only by whip, so I ended up just bringing them there, and getting frozen before restarting until I figured out the escape button trick that doesn't always work the first time.
Viewing post in SlaveBar M ~ Episode 1 [v1.4] comments
ah, in those cases you have to press F5 to unstuck things (pretty useful button) or just pressing the RMB outside the menu.
Also, the only way to have a fapet following you is either canceling sex or using the Call Command (you get it after processing in the story) So take that in mind when you need to do a Bring X fapet Quest.
PD: the only indication about a Fapet following you, is a yellow circle with numbers under your HUD