That´s also stated inside the game. Multiple days happen between each visit you see, but during those days Sito is still playing with you off camera. Sito specifies also at the start of the dialogue of the second day, but it maybe still so subtle.
Really disagree with the last part btw, I think it would make the game worse. I insist this is about frustration and confusion.