I think it's because if you use the 4x button, it speeds up the robots, but not the firing rate of the towers.
As mentioned in the previous comments, higher game speed is designed to be more difficult, but I am still looking into this and validating idea of making it "pure time compression" and thus much more casual friendly. It's balance between progression speed (armory upgrades) and ability to actually save time as you get stronger. Any further feedback, especially on the latest version v1.0.22 is and will be much appreciated! https://michalkomar.itch.io/starvoxel/devlog/1127100/release-notes-v1022