I found the problem. It does seem half-bugged though. Even after restarts this happens.
If you talk to them without doing the appropriate attack action and canceling the action to get them following you, you just get a cut-off message that says "re following you" and you get locked into an invisible menu, and just become frozen in place until you hit "escape" either 1 or a few times to free yourself.
I kept getting them to follow me, but only by whip, so I ended up just bringing them there, and getting frozen before restarting until I figured out the escape button trick that doesn't always work the first time.
ah, in those cases you have to press F5 to unstuck things (pretty useful button) or just pressing the RMB outside the menu.
Also, the only way to have a fapet following you is either canceling sex or using the Call Command (you get it after processing in the story) So take that in mind when you need to do a Bring X fapet Quest.
PD: the only indication about a Fapet following you, is a yellow circle with numbers under your HUD