There might be a bug with the buff.
Edit: Looks like it only works with Swords for now, I've found a fix for it, but it requires a whole new build to be made, so the fix for it is coming in the 0.24 version the latest, if there won't be any major bugs to fix! It's been added to the bug list here:
https://itch.io/t/5495969/bugs-of-build-023