Room 3 was always the one I struggled with. I usually try to dash down, as you mentioned, but even with that you end up surrounded by 6 - 8 giants almost immediately. It's not impossible by any stretch, but it does feel like a significant jump in difficulty compared to the rest of the Yendor rooms.
I gotta admit, I've always considered Dex to be weakest of the attributes. The difference in power that strength and intelligence-based builds give you has always outweighed the need for a few speed points, and the highest I upgrade Dex for non-ranger or duelist builds is usually 12 to get access to Strafe Attack. And there's enough SP-giving items laying around that I'll usually end up with 3 or 4 SP regardless. I dunno, maybe it's just a difference in playstyle though, I tend to prefer heavy-hitting builds to movement-based ones.