Does the Gameboy have the ability to flip/rotate tiles in this case? i.e would "▙  ▛  ▜  ▟" have to be four tiles or could one "▟" represent all of them?

The original Game Boy does not have the ability to recognise rotated tiles sadly. So that would indeed count as four tiles. 

Excellent question. Models prior to GBC indeed didn’t have this ability.

Since this year’s jam is 1-bit, my guess would be GBC isn’t supported, but let’s wait for the organisers to confirm this.

All variations of a background tile are deemed to be unique - rotating or flipping them will not be counted as the same tile. We're going to render this on original DMG hardware, so need to stick to these limits.

The responses are correct. There is no tile flipping/rotating available to reduce the tile count. Use https://gb-studio-tile-count.netlify.app/ to confirm your tile counts before submitting.