I keep changing my mind about this problem. Some people say that the game becomes too easy after certain waves, others (like you) say it gets too hard. The problem may be with the way wave enemies are scaled, or maybe I need to have later waves have more time in-between.
Regardless, which map are you playing on, and on what difficulty?