And inverted these same numbers would apply to a "roll Xd4 and take the lowest"!
That's the mod I would make - good things effecting a roll add dice (up to a max of 4) while bad things "remove" dice (down to a max of 4). Below 1d4 is -2d4 and so on; the minus means take the lowest. I would also make 4 the max instead of 5 since they're so similar, and because 4 is a 4 and we're all about 4s. so:
4d4
3d4
2d4
1d4
-2d4
-3d4
-4d4
This allows for games with truly miserable results happening under certain situations which can be ~FUN~ :)
P.S. I love the d4 and have also done some experimenting with it