Uhhhhhh...NO. You're ALLOWED to say no to people. If you didn't want to be with him, that's your choice and he has to respect that. He might be sad, but he'll get over it.
Well that's just immature of him. He could've still been your friend and you could've talked it out, but he chose to block you. And it sucks, but there are other human people out there, and you're going to be ok