The same meaning could also be expressed with: "I didnt think you were the fitness type" or "I would not guess you were the fitness type". So, this is not dirty expression at all. "to peg someone" in this context means something along the lines "trying to identify someone´s character based on observation rather than by getting to know them". Hope the explanation makes sense.