You get 10 charm and don't activate where you first encounter him then go to the bath house and spend 30 coins then you'll get charm 1 that's 11 then eat the apples or just use that bath house again.
