I couldn't find anyone else that did this online so I decided to post this so other people can find it in the future. Took me a few attempts over the years to get this going but third time's the charm! Not perfect but not too bad either. Inspired by a similar idea I saw a while ago (link in description).
There's a download link to the unity assetbundle in the description which might or might not work if you drop it into an empty project? First time exporting one so hope it works correctly. The code is commented where it matters but spaghetti nonetheless, so feel free to ask questions (might take a while to respond considering I don't check here often).
Hopefully I'll take this project further, but not 100% sure if that will be the case.