Yeah, they're just really in love with those gems and will walk face first into them. You can direct them to walk past the gem, or put an arrow so they move somewhere else after mining.
