Make your shield large enough and just forgot about swarms...
Pretty much, this is due to weapons currently not having a level cap.. Scythes too become as big as the screen with enough levels! Gotta put a cap on the scaling, like 20/30 levels per weapon
