You have to go back to the queen after asking enough people. Version 1.1 adds a dialogue telling you that explicitely :)
