That's probably something else - weblate "batches" translations and commits several at a time. If your changes don't show up here: https://github.com/itchio/itch/blob/master/src/static/locales/ja.json then weblate hasn't pushed them yet.
You can force weblate to push in the "Tools" menu (there's a "Commit" entry), directly from weblate.itch.ovh
I looked into it some more, and it looks like weblate is messing with file permissions every time it updates one of the language files..
After looking into it a bit, I just set up a cron job that fixes file permissions every 5 minutes. So, there might be a 5-minute period between "new translations being committed" and them being available, which is not ideal, but it's much better than them remaining accessible for days!
