Are you using a deltaTime approach in the Update method or fixedDeltaTime in the FixedUpdate method?
Ah, gotcha. To be fair, physics stuff is usually better off using fixed time for this very reason. This article is really helpful in understanding; maybe it'll be useful to you. The article discusses this from the angle of a different issue, but applying physics using regular deltaTime is impacted by framerate.
Great work, by the way, super creative!