Have you tried heading to the 'north' part of the bandit hideout? There's a second area to sneak your way through and one more key to acquire if I'm remembering rightly...!
If you have done that and got the key at the top-right of the north room... then your destination is back in the 'south' area (the first area) over to the left / west! You're looking for the bandit leader!
You've got this, I'm sure. Nothing crazy you need to do, you just need to find the place you haven't been!