You can right-click a modifier to remove it quickly. I thought about doing the same to add it quickly, but I thought that it would be inconsistent, cause if you have multiple weapons, which gun should it go to? The first one? What if the first one is already full? Should it be added to next one, or just not added at all?