First of all, here is my information trove. The Lively-class frigate is about halfway down.
You will probably be most interested by Gustav Adolph Wasa.jpg
https://drive.google.com/drive/u/0/folders/0B-pwrC9JR7ahfkpKOGVYYUlCMU13OTFjNlRs...
To take a crack at your questions:
When sources write 'close hauled' I interpret it at as 'full and by' with the sails drawing well, rather than pinching as close as possible, with sails shivering. In other words, closer to 80 degrees than 70. This is because Royal Navy sailing qualities reports contain some surprising speeds for frigates sailing close hauled. For instance, 11 knots forthe Lively-class.
I have never in all these sources heard of a ship being fastest with the wind abeam, however the sailing reports I have seen also lack that granular detail. They only ask about performance close-hauled and with the wind aft. Typically the captains want to brag about their best performance, and supply the best speed, invariably with the ship sailing large (although I suspect that this is usually closer to 100 degrees than 145).
Southampton stability: The ship won't fall over until the line falls beneath the Y axis, which is pretty close to 80 degrees for the fully loaded condition (IIRC). Substantially less for the lighter displacement. However, if the wind can heel the ship to the top of the curve, the ship will then capsize unless there is some intervention or the wind slackens.
Leeway: Your simulation already feels spot on. Negligible leeway in moderate conditions is what Harland describes, increasing drastically in heavy weather or when not enough lift is being produced. Also look at the Gustav Adolph sailing report for this. And bear in mind that the Lively, as the apex of British Napoleonic frigate design, was very weatherly.
Other random thoughts: It is possible that your simulation is a little too harsh with the sail blanketing, but I only base that on the visual appearance of the sails (mizzen making the maintopsail go slack, etc). I also really doubt that we could get the frigate to 11 knots close hauled. Heck, even 13 kts requires a gale. But that is a big mystery for me in general, because clearly these speeds were exceptional. In a simulation they would be routine.
I am reminded of the L'Hermione version of the virtual regatta online game, where the ship maxed out at 7 kts when in reality she has made 13 no problem.