Ability usage, including spellcasting, uses weapon durability as well. Most abilities benefit from stronger weapons for damage calculation.
You can either get more damage from stats (dex for daggers physical damage) or craft more weapons for replacement, although if it takes up to 3 daggers for a single kill it seems your damage might be too low.