The blue building is actually the post office where you can get packages u cant carry later on. I totally get the confusion though, the house colors are in the mailboxes and that's the only houses you can deliver to.
Thanks so much for playing the game and for the feedback!