Have you visited Lakeshard Village with him for the first time after rescuing him from the Rat King? I was just able to invite him a while ago to visit Bareshade and then ask him again to join the Fair after our visit to the village (Was updating my Top Logan save).
For context, I don't think I was able to invite him pre-discovering Iron Blood Village (And before the point of no return for a specific character during Northcrest Conspiracy). I progressed through the Iron Blood's trials for a bit until I was able to meet Khalgor for the first time. After I have reached the point where I was required to progress more on Northcrest Conspiracy, I was able to ask Tao to visit Lakeshard Village with him.