Yeah i noticed that, if it didn't take your energy too it would be very spam-able.
Whats more, after selecting a new partner for your slave, if you then click a different one in the residents list, the game remembers the partner you previously selected. This means you can make a slave with "swing" unlocked sleep with anyone, even themselves.