I think if you didn't gave them your address you wouldn't be in their route and wouldn't be able to talk to them but I'm not 100% sure, now, in order to go to the pool with them you need to share Samantha with Ken and after a while you'll be able to ask if Ken would like to go to the beach with you, he'll say he doesn't know how to swim and you'll have to rent the whole public pool.
I'm not sure how to edit files so I'd recommend start a new game if you're not in their route.