Ok so apparently, when you first play two of the four main worlds, a glitch makes it so that the switches to open the secret world only works on your first visit. That's why only two of them are functional on a return trip. If you missed the other two, you sadly need to restart the whole thing for them to work.
No problem! I'm coming hot off of having properly 100%ed the game, gaining access to the final secret world, it was a good time. I can also confirm, only the Green and Pink Worlds' switches have this glitch. The Yellow and Orange Worlds' switches work as intended and can be activated after missing them on your first trip.