Small follow-up, I encountered an interesting bug. I obtained two Stalwart Small Charm of the Blessed, one giving +6 HP and the other giving +3 HP. However, if I equip both of them then the game gives me +12 HP.
Edit:The tooltip when hovering over the Phantom Stone gem reads "even when not the weapon isn't selected"