Thanks for the feedback! Yes, I made it intentional to "flatten out" a bit as a square/cube + physics sucks. I am stupid and I didn't think about using sphere collider. That would be much simpler :D
Edit: another problem the flattening out 'solves' is that sometimes the cube can just go up/down. This can happen with the round collider as well... Though looking back I could solve it just to give a random horizontal nudge if the ball collides perpendicular to players.