At the same time, Peter didn't know where they were the whole time- he didn't know that they went to UA and they had to go back some day.
After about a week of this...he was crushed. He was finally free, but he was lonely. They were his friends, and he missed them so much....but it was more than that. Ever since his curse had been lifted, he felt like a piece of him was....missing. Like he wasn't a real person anymore, just a bunch of pieces.