I think I might've figured it out, the spider drones jumping might have been breaking the raycasts. I have no idea how that works but since the bug - for me at least - only happened periodically and in the Toxic Desert biome exclusively - which is the only one with the spider drones - I have to assume its that.
I disabled the jumping and it seems to be fixed for me, let me know if it were to be still broken for You
This is the most baffling bug I've ever encountered, so thanks for helping me out