All I had to do was get the Stage II Time Vortex to slow time down enough to get it consistently. Normally even the first stage is only obtainable after the skip, but by defeating the goblin tribe and taking the goblin spear, you can actually unlock the door to the room where you get the vortex by attacking with the spear through the wall, because the goblin spear is just barely long enough to hit the lever inside that opens the door.