Bit late to be handing out advice now that the jam's wrapped, but I saw a few people wrestling with co-op so figured I'd share some stuff.
I actually skipped co-op myself this time. For a physics game, hashing out netcode bugs in a jam timeframe is a nightmare. But if you're set on it, you don't need to stand up your own servers. Both Epic and Steam offer free peer-to-peer connections, so that side of things is pretty much handled for games like these.
For testing, I built an Unreal widget earlier this year that makes multiplayer QA testing way less painful. It launches up to 4 separate PIE processes signed into 4 separate Epic accounts at once, with the Epic overlay enabled on each respective window. And it's not faking anything with null networking, all the traffic actually routes through EOS from inside Unreal, so what you're testing is the real thing.
I keep meaning to release it and I'll get it out one of these days. If that sounds useful, follow my account and I'll post when it's up. Might do a video walking through it too.