Thanks for trying the game, and for this feedback! :)
I think that's likely an optimal strategy, and it still works fairly well even if you're like me and sometimes need to go back and forth a couple times to be sure which one changes. Ideally there'd be a little while where tracking everything feels overwhelming and then it'd feel like more of an achievement to come up with a process for managing it, but it sounds like it's often too easy to quickly arrive at a good strategy, and then the player misses out on that.
It's interesting to think about what could be added to give the game more depth - one thought is if multiple people were guilty and you had to match them up with what artifact they stole, which would lengthen it but maybe not add anything completely new. A separate second phase with a different puzzle might be more satisfying.
I think it'd also help if I'd added audio, which shouldn't be too bad even in Java. I think it would have been fun to find a lofi or non-intrusive instrumental song that it could play normally or in fast forward or reverse depending on the clock settings being used. Just didn't make time to find a song or add in the code for it yet.
Thanks again!