another hotfix came out just know. I found the issue and I hope that it works now.
it is getting much better in stage 1, but in stage 2.... nono, it didnt work....T_T
https://itch.io/jam/game-off-2023/rate/2384304 there is another entry about picking up and placing object, and it works really well. maybe you can find some solution from it?