Thats weird, that never happend before in a playthrough. Some harvestable objects takes longer to harvest, so maybe you need to press and hold the key for a bit longer.
