You need to select your "Guile" Skill Book and use it, in the menu that comes up will be an option to increase the Lockpicking specialisation. However, your specialisation level has to match your Skill level, so if you're Guile level is 1 you will need to raise it to 2 before you can raise your Lockpicking to 2.
You can ALSO climb in through the window of the Brothel, there are some Vines outside, so if you have the Climbing Specialisation, you can just climb those and interact with the window.