Because it’s difficult to tell apart at first which stone is what, you punish the player for trying to get what’s in his pocket. I had this feeling entirely because I felt it was my only way to know if stone was actually stone. If you have a way early on to tell them apart, it becomes fair game!
Another option would be to not punish early in the game, and make the amounts bigger as time goes on / as you’re getting closer to retirement? Or, very different option, creating more of a risk/reward situation on a different vendor? For instance: with a more elusive description + a fee to drop stuff in that bin, but if you get it right it’s a huge payoff).
Please let us know if you make an update, I’m quite curious to see how you make the game evolve!!! <3