Yeah, this is a bit of a pet peeve for me, too. Although, for the characters that use staves, I've managed to make an accurate guess based on what little of a peek I get at the gems (granted, I was on fullscreen and the staves never had a skull on them (at least for me), and I'm pretty sure hammers only have one gem color anyway.
I WAS SUPPOSED TO GRAB THE EYE FIRST!!!!!!!!!!!!!!?!!!!!!!!!!!!! Ugh, every other interaction with the grabber involved clicking the grabber first, so I kind of assumed I'd have to do so there, too. I feel like there should've been some kind of hint that explained that since I typically learn by having instructions.