What happened was I talked to the vagabonds because I happened to go into their camp, and then when Serena came back I talked to them one last time before I could talk to Serena about their case, so Serena ended up helping me clear them before I got to finish the main quest which is to talk to her about how she was doing, so that probably broke something in the system, other than that i have no idea.