If it says give him time then just sleep (might ne a few days not just one) and come back ?
I've slept more than 5 nights already, since I was doing Erza's quest line and Levy's after doing Natsu...that's why I'm asking.