There's an rng component to it as well, I had to try it like 4 times myself to get it.
Could be me, but it seems like he mostly walks in on you when you shower in the afternoon.
