I personally don't have any accessibility issues but I still think it's dumb to make the games harder to play for certain people for the sake of principle. Game jam constraints are meant to foster creativity, like the 4 colour rule, but this seems completely unnecessary. It is your jam though, so run it however you like.