I may have accidentally soft-locked myself. I tried progressing while avoiding the purple orb in one of the save room that triggers the cutscene with the witch.
Once I'm downstairs in the same level as the "1 damsel to unlock" door, I cannot go upstairs anymore.