You are fully right about lack of contrast! I encountered this problem trying to make limited color palette, but didn't find solution till the end of jam. It's challenging!
Yeah trying to work with a limited colour palette is always going to be difficult at times but despite this, I do think you did a pretty good job with it :) It's no easy feet but you can still tell the player apart decently well!