for a game that relies on switching characters, i don’t think discouraging the player from switching with a heavy health penalty every switch fits with the design.
in fact the timer/health meter as a whole seems unnecessary.
each character having their own music is a nice touch!