Its a pretty complicated topic and not something that can be learning by itself, it's more a combination of many things. It's generally just using similar artistic techniques and styles for everything, as well as everything kind of fitting a theme. And yes, a shared palette among things you wish to associate is one of those techniques for more cohesion.

There should be similar geometry among parts that want to be cohesive, similar palette, similar sizes, similar level of detail, etc.

In games, generally the first enemies will be introduced in an easy environment for the player to learn about them. Then every enemy henceforth is introduced in reference to that first. If a new enemy looks similar to something you saw before then it's assumed it will behave similar. If a new enemy is stronger maybe in terms of maybe health and attack, then it will usually be bigger, if an enemy is showing new geometry or color, then it is probably going to have new mechanics. If you see a new enemy like the ones previous, except this one has a really big spear, then the player will inherently think that this enemy is the same as the previous one in every way, except that it will use this spear.

You can break conventions, but you have to be really good at introducing these new conventions to players. Because there is a thing called game literacy, and subverting that will more often than not just unecessarily annoy players. Like if you make the left analog stick control the camera, or make up arrow on a controller equal to jump. These things break convention for no good reason and make a player relearn mechanics that they could've already know. And the same goes for art and sign posting.

For my game, I originally had more detailed sprites for the enemies, but the rest of the game was minimalist, so while I personally liked the sprites I made, they just didn't fit the game, and I ended up just making the enemies squares 

This is the kind of thing I've had to fit to, this minimalist style and also the pallette I chose. The enemies are grey because they couldn't be the same color as the player, they couldn't be allowed to blend into the background, and adding too much more color to the scene would've made it muddier, so they became grey.

But then also, the little indicators on my enemies don't fit the pallette that much, because I purposely want them to stand out as they are an important game mechanic, not just art.