Hi,
I figured, but it's okay, it's not a huge problem.
Yes, it happens even when I don't touch the keyboard or the mouse. It's not every time, but it happens pretty often.
Also, it has trouble translating some words correctly.
It's nothing too serious, though. I'm still happy with the app overall. Keep it up.
Heya,
Thank you very much for the examples, it helps a lot! :)
I have just released 0.5.04. It introduces a translation & text rendering cache. Please try it out and let me know if that solves that specific problem.
Regarding the other issue, I think that is the OCR simply not picking up the words, or they do and the confidence threshold I have set is too high. I am going to add user configurable options for these. I hope I can release a 0.5.05 version tomorrow with that in place.
Once again, thank you, you have been very very helpful.
Hi,
Version 0.5.04 has fixed the text jumping issue I showed in the first picture. I re-tested it with the same manga chapter, and the text didn't jump this time. I also tried it with another manga and it worked fine there too. So far, it looks like that's solved.
Yep, unfortunately, the word recognition problem is still there in this version.
I did notice something interesting with the new version when it comes to games, though: automatic translation doesn't work if Comic mode is on. It won't translate the text. it just shows the original. If I try again, the selection box even disappears. In previous versions, automatic mode worked even with Comic mode turned on. This isn't a huge deal since I can just turn it off, and it works fine, but I figured you should know.
The good news is that with this new version, the text doesn't jump in games anymore either! I re-tested with those same two games, and for now, it seems like the jumping text issue is gone there too. Hopefully it stays that way, but we’ll see if it works with other games too :)
I also have one more observation. It's not a problem, but I thought I'd mention that when the app is running during a game and you use NVIDIA Freestyle/Nvidia filters, the app seems to slightly weaken the filters' effect. For example, if I need the Sharpen+ filter's Intensity at 15 for the right sharpness without the app running, I have to turn it up a lot more to get the same effect when the app is in use. At least that's what I've found with Wuthering Waves.
That's everything. Thanks for the new version! :)
Heya norby,
That's great to hear! I hope it works for most cases. I'll add a user configurable value later in case this doesn't work at all in some scenarios.
You may be able to fix the word recognition now in the 0.5.05 version that I just released. It has a 'Confidence threshold' value you can change for any of the OCR engines. Setting it lower will allow more text to be captured (but can increase the 'error' rate).
If it still doesn't work even if you lower the value, then you'll have to wait for the 0.5.06 version which will have more user-configurable values relevant to OCR.
Thank you very much for the notice on the Comic mode bug. I had not encountered it before, and it is completely breaking that mode whenever anything is moving in the background.. As far as I know, I have now fixed it completely in the 0.5.05 version. Please let me know if it keeps happening!
Thank you for the heads up on the filters - I think I know the cause of this issue, but I am not certain that I know of any way to solve it.. If it is what I think it is, this only occurs in Desktop/Attached mode, and not the Internal mode, correct?
Cheers again for all this feedback! :)
Hey,
Unfortunately, it's still not working for me. I tried setting the Confidence threshold to a low value, but nothing changed. I guess I'll just have to wait for the new version.
Yep, now it's totally working. Automatic translation is functioning perfectly even with Comic mode turned on :)
Yeah, I used it in Attached mode. It doesn't work in Internal mode. It just gets stuck at "Verifying launch".
Alright, no worries! I just uploaded 0.5.06 for you. It adds 'Box threshold' and 'Text threshold' values for you to configure. Try lowering the Box Threshold number and see if that makes the text get picked up! The settings will be found at the bottom of the OCR tab.
Yay! Nice one. Progress.
Ah, okay, just a few specific games I assume? Afaik most games you have mentioned do not use any of the normal graphics APIs, thus Attached is the only one that'd work (or Desktop..)
Unfortunately, it's still no good. I couldn't get it to work at all. I played around with a bunch of different settings, like lowering the Confidence and Box thresholds, trying both automatic and manual Model and Font Selection, and even switching to a different Screenshot Capture API and Text Removal Algorithm but nothing helped.
On that second picture, I forgot to mention another issue besides the word recognition problem: some bubbles were translated correctly, but you could still see the original text or word underneath. That's another bug.
Right now, I only use Nvidia filters with Wuthering Waves, but I'm guessing the same thing would happen in other games if I was using Attached mode.
Got it, I didn't know that. Thanks for the heads-up. :)
Dang, that's interesting.. Could you please send me a few screenshots of problematic Comic book scenes like the example you showed me? Much easier to figure out what's going on if I can do hands-on testing.
About the second bug, yes, unfortunately I am already aware of that one. It's important to fix and I will likely try to find a way to get it done within the next few days.
Thank you again 🙏🏼
I went ahead and made a video of the whole process.
You can find the video here: "Stream Friendly". I couldn't upload it directly since it only allows a maximum of 3MB. I've also included some screenshots below to help explain what's happening.
Stream Friendly: Here are the bugs I spotted in the video:
1. Some bubbles are translated well. But sometimes the original text peeks out a little at the edges, or there's a small gray spot.
2. Similar to the first point, the text translates nicely and the sentence makes sense, but unfortunately, one or more original words pop up underneath the translated text, making it really hard to read. The original text peeks out at the edges here too.
3. There's a word recognition bug where it can't figure out some words, so it gives a nonsense translation and the original word or text is still visible. I wonder if the manga's font is the issue?
4. It's like the first, but the sentence doesn't make sense because it used the English word itself instead of its Hungarian equivalent.
It contains the letter "i," but essentially, it is the English word "mean."
5. The app can't properly translate text that's on background images inside the manga panel, outside of the bubbles.
6. In some places where there are two words one under another, short one-word sentences, it translates nonsensical sentences. To be more accurate, this was a really literal translation. Maybe because they're too short?
I first tried the Gemini API, but manga translation just didn't work. It gave me a '429 too many requests' error. So I used Azure AI for the translation instead because I didn't want to burn through my DeepL character count, even though it resets in a week. Despite that, I decided to redo the video with Deepl this time lol because I realized that even if DXGI doesn't work, I can record with WGC. So I made a video using that method too.
WGC: "Video link"
The translation in the video is flashing and flickering because of that yellow border, but you can still read it if you pause it. I'm not sure why it's doing that in the video, because everything looked completely normal on my screen while I was recording.
Bugs in the video: From what I could tell, the bugs are about the same as the ones with Stream Friendly, except for the issue with the original text peeking out at the edges that wasn't happening here. I also noticed a new bug here: Sometimes it doesn't translate the whole sentence, just shows a single word.
1. Some bubbles are translated well. But there's a small gray spot.
2. Similar to the first point, the text translates nicely and the sentence makes sense, but unfortunately, one or more original words pop up underneath the translated text, making it really hard to read.
3. There's a word recognition bug where it can't figure out some words, so it gives a nonsense translation and the original word or text is still visible. I wonder if the manga's font is the issue?
4. It's like the first, but the sentence doesn't make sense because it used the English word itself instead of its Hungarian equivalent.
5. The app can't properly translate text that's on background images inside the manga panel, outside of the bubbles.
6. In some places where there are two words one under another, short one-word sentences, it translates nonsensical sentences. To be more accurate, this was a really literal translation. This was a little better, but it's still not quite right. Maybe because they're too short?
7. Sometimes it doesn't translate the whole sentence, just shows a single word.
Oh, I almost forgot, but I also gave MangaOCR a try. The Japanese-to-Hungarian translation with DeepL wasn't the best, but it was acceptable. At least with MangaOCR, everything looked pretty okay. The translation was really slow, though. I had to wait at least a minute for it to finish. I edited the video to cut out all those waiting periods. Since I was already making videos, I made a short one for this too, but only with WGC because it doesn't have the text poking out at the edges issue, though it did have the flashing. You can find that one here: "Link".
I did find one bug with MangaOCR: it doesn't work with a Custom API. When I switch to a Custom API and then try to select MangaOCR, the app just freezes. It works fine with all the other translation engines.
So that's all the feedback I have for now.
Dude, thank you! I appreciate all this effort so much. It really is helping me a lot to narrow down bugs & you're showing me some I have not even encountered myself.
I'll respond in the same format as you've put it down;
1. I have not seen this before. It would seem like the Stream Friendly mode has alignment issues for some reason. Do you use really high windows scaling by any chance? I upped mine to my max (175%) but I still couldn't reproduce the bug, so I'm not sure what's happening here.. It could be that I've set up the screenshot window with the wrong sizes somehow.
2. This is a text removal bug. I'm going to try to fix this very soon.
3. Yes, the font might look easy to OCR, but the RapidOCR model is unfortunately really struggling with the separation of spaces here. I have just updated the model to the newest English model, it will be used by default in the next version. I also noticed that 'AH. GOTCHA.' failed to translate - if you use the classic DeepL model here, you will get the exact same result. So in that case, it is a DeepL model issue. The 'next-gen' model works just fine. For the next version, I have added an argument to the DeepL translation request that will prefer the next-gen models by default. This should improve translations by quite a margin for DeepL!
4. This would be some sort of OCR issue. Not entirely sure what's going wrong here. The second pic below here is misplaced I assume..?
5. In these specific images, the issue is angled text. It is not supported at all at the moment, not for OCR and not for rendering. It is an addition that requires quite a bit more thought than I initially had in mind.. so it may take a while before I introduce support for that.
6. This is again a DeepL model issue. It translates to 'MI VAN!?' with the next-gen model. Hopefully that's correct, and as stated above, the next-gen model will be used in the next version! :)
Yep, the 'Too many requests' error is a bit of a headache and something I need to figure out for other API's that are used through Custom API.
The reason it is flashing like that on your recording is because it is impossible to use the Automatic mode with WGC if I do not constantly hide/unhide it to a recording device (in this case, new screenshots!). That solution does not work at all for DXGI, that's why it is always hidden from recordings. Hence why I added the 'Stream Friendly' mode.
1. Yep, this is a text removal issue. I will be improving/adding new algos, and likely add some user-configurable values to fix edge cases.
2. ^ ditto
3. These are all OCR model issues. As stated above, the new English model will be included in the new version, however, it is not perfect and will also cause translation errors.
4. The first pick is the same as above. The second pic is also the same, but the font is specifically an issue here, the SO is picked up as 5O, 50, S0 instead of just alphabetical characters 'SO'.
5. Again, angled text.
6. Same as previous 6.
7. OCR issue. This specific example is massively improved with the new model. There is also a box detection issue here that I will expand upon below.
In these two examples, we have a box detection issue that happens sometimes. All independent text lines should be in their own box for best possible recognition. In these cases, we have some boxes that expand over several lines, which is very problematic.
I will try the specific English box detection model later to see if this isn't a problem with that model. If so, I will add it. The reason it is not already added is that the current model being used thus far has worked very well for all languages, and it would just take up unnecessary space.
MangaOCR is much slower than RapidOCR mainly due to the sheer size of the Manga model. It is not optimized for speed (but I did all I could to trim it down, without losing quality..). In the future, if I ever get enough money, I will create my own models that have a good balance of speed and quality for this app's purpose.
I have not been able to reproduce the last bug you mentioned.. Could you please make a short video of the process?
Fyi, if you weren't aware. When you do debugging like this - try to use the LanguageSwap hotkey so you can see if the OCR is recognising the text correctly or not. It helps a lot to figure out if it is an OCR or a translation issue.
Again, thank you very very much for all of this. You're a hero! :)
*Edit* Forgot to say - Use the CPU for MangaOCR. I think I should disable the GPU selection completely for this, because it is excruciatingly slow on my old 980 Ti.
*Edit 2* Again, forgot to say.. To temporarily fix the text removal issues, try zooming in! Should fix the problems (apart from the Stream Friendly one). You can hold CTRL + scroll up and down in the browser to zoom in/out.
*Edit 3* I tried the older model that is specifically made for English box detection. In this case it works really well, but I am not prepared to set it as default just yet. It may very well be so that it does a lot worse in other scenarios. You can download it from here. Just go to RapidOCR and then import it into the 'Detection' section and select it for use.
