Shooting rays only towards corners is an interesting idea! Unfortunately a big part of the lighting right now is from bounce lighting, which doesn't necessarily come from primary light sources like torches. You can actually see the impact of bounce-light if you open the dev UI with Tab and turn off Bounce light, it makes a big difference!
That being said, totally agree that the current cost is unacceptable! My approach in this case was to get something looking the way I wanted first, and then move on to optimization after. That way i have a good idea which features are important to keep, and a nice 'ground truth' renderer to compare against as I try to optimize.
Thank you for both links, I'll definitely dive in and investigate!