Yeah I agree. Sometimes I find myself wanting to refactor something for no real reason other than a vague "to make it better" when there isn't really a need. That just tends to overcomplicate things with no real benefit.
In my latest game though, which started off as a jam game, I really broke some stuff at the end of the jam and then I was absolutely forced to re-think the architecture and refactor the code to fix the problems. So yeah, I do need to avoid the temptation to just over-engineer and refactor with no good reason to do so.