Stats going down is really strange, never happened during my tests, the only thing in the game right now that reduce your stats is when the magic buff ends but it is just to return your stats to what they were before the buff.
The buff increase str and agi for some time, the amount and duration depending on how much you invested in magic. Then it goes on cd for a few turns and you can use again.
But it really was supposed to become easier with levels even if the feats are not implemented yet, they will probably come with the next girl in 0.3
Later today I will release another version including an easymode option for those who want it but I will still see if I can balance the game a little more and give the player some buffs while there are no feats.