If you find the girl's pendant in the forest, which i think you can do on the first day or the day after that, you can talk to the people in the first town about it, one of them should mention about seeing him in the Sawmill and should have that appear as an option for you to visit.
You need to do the quest where you cure the Shopkeeper of his Sickness when you see him suffer it in his shop, you can either try to figure out what the items is needed for the Heat Occisia item or you can ask the lizardess in the Ruins about it and she teach it to you so that you don't have to craft it blind, after you give him the item needed from the option that appears in your home, visit him next day and he will be back to normal, after doing that, the Bandit attack can happen on Day 20 and beyond.
If you don't have her Ruins unlocked yet, keep doing the story quests/advancing the days until you unlock the Swamp and find the Lizardman's Head Statue in there to start a puzzle based part in there that will leads to unlocking the Ruins, you don't have to win this one, you can just continue being a sub to the Statue and Rothar if i spell the name of that lizarman right, and he will eventually pull you through the hole and save you from the Swamp Beast if you choose to sub to him during that event.