Idk if you watched the video but it probably averaged from 60 down to 20/15 it felt like. System specs:
RX 6600 non XT, Ryzen 7 5800x 8-Core Processor, 32 GB of 3200Mhz DDR4 RAM, Game ran on a 1TB WD Black SSD.
I watched your video and checked your computer's characteristics. As far as I know about game development, I can say that the problem is not in your computer configuration. It's about game optimization, of course! And there is no optimization as such (suddenly! lol). If you're interested, I think the clouds are to blame. Clouds contain a lot of polygons, and since their number changes from level to level, that's probably why fps changes.
I mean I did build this machine to be badass at playing AAA games so yeah I figured my PC was not to blame for a indie horror project running rough 😂. I play games on the daily that (Not to be mean, game looks great as a VHS horror) that look way better graphically and are probably more advanced. Cyberpunk comes to mind besides raytracing cause non raytracing card.