Sorry I probably should've mentioned, I did try that. As far as I can tell there aren't any weather effects in the town are there? At the very least using the item doesn't seem to change much. The lag does stop when I go in buildings though. Maybe it's to do with all the animations of the water and lights or something?
Something else weird I've discovered messing around though is that the lag stops as long as I'm holding the confirm button. Holding enter or z I can run around outside just fine, but the moment I release the button it's back to the slideshow. This makes it possible to get around at least, but reading text or viewing scenes properly might be kinda difficult.