It sounds like the changes you have/will have in the updated version are good ones. At one point I did try to turn the craft perpendicular expecting the extra drag would slow me down, but for reasons that are now obvious didn't have success with it.
I can appreciate wanting to have some complexity, and not just the simplest possible form of the concept. There is a question of, do the quirks make it challenging in a way that's fun to contend with. For me personally, dealing with a lot of inertia in three dimensions like this already feels challenging, but things that add confusion or inputs not doing what I expect them to do, those sorts of "challenges" are harder for me to come to terms with.
I'm still not convinced that having two separate modes the way you have them is either necessary or a good idea. But I'm also curious to know what led you to try that in the first place. I can understand as far as you wanting airplane like flight and helicopter like flight, and so you have two different modes for it, that kind of follows at least on paper. But practically speaking, it doesn't seem like something that arose out of necessity, or maybe I'm just not seeing it.
As I mentioned, I'm not fond of how many things change when you switch modes. I could understand if it was something a little more specific, like enabling a particular kind of stabilizing effect. Then at least the button would have a clearer intention and utility, rather than being a button that just changes "a bunch of things." It's also unclear precisely how/when to make the transition from one mode to the other. You talked about using maneuvering skill to make the transition between modes of flight, but you also have a button that's supposed to cross that transition; how do the two meet?
To give a more concrete suggestion, why not leave the pitch/roll control unchanged between modes and just have the hover mode try to stabilize you vertically or something? If the player wants to flip upside down and crash then let them. That would keep the controls more consistent and make it clearer what the button is for. You could also have the hover mode be an on/off feature rather than creating a logical division between two separate flight modes (which would also help solve your naming issue).