I figured you intended to improve them further, but it's good to get that confirmation. :) I had faith in your project when I looked at the program's extensive update log. I can tell when somebody is passionate over what they are working on, haha.
I did attempt using some of those other model options but indeed, the site to download them at is quite a cryptic place. I used some of the suggestions you linked but they were exceptionally-slow at generating and didn't really give me the best results. They also kinda crashed on me half the time. A proper guide might be handy, I'll give them another shot if you do write one later. :)
Language-specific models I imagine are both faster and more quality, since they only have to page through one language and not everything. I'm personally hoping to mainly use your program to translate Japanese games to English, and occasionally Chinese games too.
Great to hear that! I've tried my best to update weekly, but this year has been quite difficult personally. I really do hope things settle soon; it has been wearing on my passion and motivation quite a bit.
Yes, how do navigate and use the models hosted on Huggingface isn't exactly super straightforward. LLM's do indeed need quite powerful PCs, which is very unfortunate, but within a few years, most PCs should be able to handle them quite efficiently.
Yep, for sure! There is actually some fairly new models specifically for JPN-ENG and vice versa. They seem to be of rather high quality, but I believe they can only be used with shorter text. I am going to write up a guide to set it up soonish since (I swear to god) 80% of all GameTranslate users only download the Japanese-English offline model, haha.
The last couple years have been particularly hard on us too, so I very much sympathize. Passion's such a hard thing to keep up sometimes. We do Twitch streams/YouTube as a profession, and we used to be more passionate (and still can be), but now sometimes I just have to take the day off due to depression. Sucks, but self-care is also important, and fortunately, most reasonable people are sympathetic to you needing breaks. Don't be afraid to take them if you need them. :)
I wish us both an improvement in life. We have some positive prospects of our own, but it's mostly a matter of waiting till we can finally move. (The apartment market is cut-throat in America, so it's been a real challenge.)
On the subject of my profession, I'm eager for this program for that reason too. :D I like to check out weird and obscure stuff on my channels, and I've already played several non-English games which I managed to translate one way or another. It's fun to engage that other side of the world's games on my English channels, where a lot of viewers will not even know these games existed before I cover them.
Yes, indeed, LLMs are a bit too much for my PC I think, hence the slowness. Kind of an older thing I got, I do want to upgrade but the money's not there for it currently. But perhaps language-specific LLMs will manage to be fast enough for this computer to make the process more viable...that's my hope anyways!
As to your last point, very interesting! Shorter text...that might be alright with how I would use these! Take the time that you need on the guide, but I'll definitely get right to reading it when it's done. (...80% of users use the Japanese-English model? Wow, I figured that percentile would be high, but not that high!)
I am sorry to hear that mate. Yes, self-care is super important, and I think one of the most important things that makes all the difference is to not beat yourself up about not being able to cope for the day. Definitely always struggled with that.
I do wish you the best of luck in the apartment hunting, and hope your depression is not long-lasting!
Glad to hear about your reason for using the app! I've had a few content creators asking about the app, so I'm stoked that I was able to add a stream-friendly Capture API. There are for sure a whole other world out there of games that is cut off by language obstacles, which is the main reason I created the application to begin with. Many games, especially indie games, do not have translations for even all the major languages. There are hundreds of millions of people out there who play games by trial and error.. which is how I learned English.
What is your CPU & GPU?
I did upload a guide for Japanese - English yesterday, let me know if you try it and how it works for you. Many thanks! :)
https://itch.io/t/5522705/guide-japanese-english-llama
Thank you! ^_^
Ah yes, as for its stream-friendliness... I use Xsplit (which I think officially makes me a weirdo) for stream capture, and it behaves a little oddly with GameTranslate. I can still capture the untranslated game just fine with no conflicts, and I can kind-of capture the overlay translation separately, but it's weird and the text is surrounded by blackness, so I can't really overlay the translation atop the game. (Well, it might be possible with some 'greenscreening' of the black part of that capture layer; I might have to play around with that more once the program is capable enough for me to stream with it.) I'm not too bothered if I can't capture the English translation in overlay, it's certainly enough just to translate it for myself since I'm fine with reading it out for my audience, but it would be an extra boon if I can get it to capture the translation too.
You're quite right about there being many games out there that are cool enough to struggle through without knowing its language, and I've done that too. That's really cool that's what got you to learn English! What's your native language, if you don't me being curious? We have people from all around the world join our streams, and we are close to quite a few people who are ESL (English as a Second Language) and it's always fun to see and learn about the quirks that a primary language adds to a secondary one.
My specs... They're currently: AMD R5-2600 (Stock AMD Stealth Cooler), Nvidia GTX 1060 6GB, 2x8GB DDR4 @ 3200 MHz. It's not the most robust computer but this old thing's been kicking around for a while and it's been one of my more reliable computers I've owned. :) I hope to update from it at some point...but that move has to come first.
Also, heck yeah, thanks! I've got a little weekend here so I'll follow the guide and run some tests and see how it works out for me.
Xsplit! Yeah, most people only use OBS these days don't they.
That's interesting, are you definitely using the 'Stream friendly' capture API found in the General configuration tab? It may be that Xsplit is trying to filter it out in some way.
My native language is Swedish! :)
I did not give any attention to school whatsoever and dropped out very early, so my English is more or less all the way self-taught through gaming experiences & later moving to England for a while. Yes, languages are very interesting in many ways. Always a fun discussion about each language's quirks as you say.
Oh yeah, that's pretty old PC indeed! Good enough to stream what you want still, so that's nice. Yeah, a comfortable living space is definitely more important.
Cool! I'll try to respond to your tests asap!
I just tested out your new guide, left a reply over there on that post about my experience with it, since it might be useful to others too. :)
On another note, along the way, I also ran some more capture tests with my stream broadcaster (Xsplit). It is currently not possible to capture the game with the translation overlayed on top of it without capturing the entire screen space, but that function's a privacy risk we almost always avoid using since if anything sensitive pops up over the screen in that area it will also get captured and displayed. We typically capture by application process, not by screen-space, as such.
It IS possible to capture the game application, then also capture the translation application on top of that, but the translation application capture weirdly covers the whole screen as black when this is done. (There is technically a game behind all of that blackness lol.)
It is possible for me to then Chroma-key out 'Black' on the translation application capture to let the game be visible, but the text is kinda painful to read in this situation.
If you want to make this program more-accessible to streamers, here's what I suggest... It would be nice if we could customize how the translation overlay displays itself.
1. Customizable background color. If I could control the background behind the text to be a different color (like green). then I could easily chroma it out cleanly. (Or heck, just give the option for a transparent background.)
2. Customizable text color. If I could change the color of the translated text itself, I could pick a color that stands out more-easily on top of the game, especially after chroma-keying the background out.
3. Optional text border. If I could add a border around the text, that would help it stand out atop the game as well.
4. Optional text shadow. Again, would help readability.
Here's a quick mockup of what I'd find ideal personally, if those four customization settings existed for the overlay function:
Even beyond streamers, these suggestions could just be a nice way to customize things for any users. Your approach of putting white translation text in a black box works for now of course, but these are things I would hope for in the future. :)
EDIT: Ah, with a bit more tweaking of how strongly the chroma-key was told to react, I was able to capture just the black box and not black-out the whole game screen. 
That being said, it will sometimes blink the whole screen black for the capture display due to how the black chroma-keying is working to cull out the rest. That would be quite unpleasant for viewers if it keeps up. I might be able to tweak further to stop that from happening, but the customization ideas I shared would solve it for good.
Aha! Xsplit must be breaking how the Desktop/Attached mode works. How I achieve transparency is basically through 'Chroma keying'.
I used to have a different implementation entirely, but it used quite a bit more power to run. I wonder if that solution would work differently in this case.
1. If you mean the background colour of the translation window, this is planned! :)
2. Also planned.
3. Definitely something I'd like to have.
4. Ditto!
Yeah, your example clearly demonstrates how important customization is. Cheers for that!
Thank you for all that testing, this is very interesting. I'm not sure how the chroma-key is set in Xsplit, but the 'transparent' parts are all set to exactly pitch black 0.0, 0.0, 0.0.
By the way - is there a reason you are using the 'Manual' translation window instead of the 'Automatic' one? If so, for your use case, why? It's good for me to understand how users want to use the application and the 'Manual' translation mode has been quite neglected to favour the development of more immersive translations.
(Consolidating my replies here~)
OBS is by far more popular, technically better but... Our previous computer didn't work right with it, which is why we got into Xsplit. Xsplit kinda sucks in some ways but it's serviceable. Fortunately, the two program are similar enough and share enough of the same toolset that if something works for one, it'll probably work for the other.
Ah Swedish huh! Neat~ We have a couple of Swedes in our audience too.
Oh that's interesting, so you do use chroma keying already! I definitely think it's best to use the lower-power method since a utility program like yours should definitely not make too much of a footprint.
1. Yes, basically!
2 - 4. Excellent~
It's certainly functional as it is, I got the blackness problem to mostly settle down. Ran another test and I think it's that...if I click the GameTranslate overlay that pops up over the game, then Xsplit thinks what I clicked is the 'active window' it should capture over the game window. (As I use an 'auto-detect' setting for capturing games.) Basically, Xsplit thinks the GameTranslate overlay is a 'game' if I click on it. If I avoid accidentally clicking any overlay elements this shouldn't be a problem though.
As for your 0,0,0 blackness I figured! I basically...well, here's a pic of that settings menu I configured for chrome keying the overlay right. If I chroma too little blackness, it just covers the whole game window. If I increase it too far though, that's what causes it to eat into the lettering and the text window and look bad. These are the settings I found to work best.
As for the Manual vs Automatic... I didn't really 'understand' Automatic. I initially used it but it didn't...well, literally seem to be doing anything with the games I hooked it to? TBH I sort of gave up tinkering with it very early on, set it to Manual, set my tilde key to be the Fast Key, and just press tilde to start a selection, drag it across the text I want to translate, and let go. Besides...if Automatic is (supposed to, at least) capture the whole screen...that may be messy. If I try to capture a lot of text with the Fast Key drag, like the whole screen in a complicated inventory, the text sort of jumbles up. So I think it's better to capture sections at a time anyways.
I would like to know...how to make Automatic...work, though. I didn't actually get it to do a thing for me, so I can't comment on if it would be useful to me or not.
EDIT: Oh, I see. I tried again and got Automatic mode to work. I didn't realize you also had to still MANUALLY select a capture area while Automatic mode was on and then the model got to work automatically. ...That said, I don't like it very much for a common habit it seems to do that this gif shows...
I think a controlled manual capture is better in most cases. Less-intensive too, I'm sure, than the translation tool running across the whole screen all the time.
And yeah, if there's text all over the screen GameTranslate can...be very messy about displaying it, which basically makes a full screen auto-translate sort of a jumble. In both Manual and Automatic modes. Here...
This is a shop screen: Untranslated, Automatic, and then Manual, respectively. Either type of translation of this screen is quite messy when I try to capture the whole screen's text. Manual's layout is worse than Automatic's, but of course Automatic here was also still flickering the translation to different variations too. All sort of a mess, regardless of what I pick for a screen-wide catch-all capture area.
So yeah, Manual with capturing sections at a time feels better...for right now, anyways. Surprised to hear it's been neglected; it actually feels the best right now, so I would ask that you don't leave it behind too much in the future. :)
We're a tiny country, but we get around!
Yep, indeed it does! Exactly yeah, that was one of two main reasons I changed it, the other being that the 'Automatic' mode didn't work in Desktop/Attached mode without making these changes.
I see. Well, there could potentially be a solution for this, but it sounds like a very niched issue and not something I have time to spend on right now. The idea anyways is to make the Desktop/Attached window not allowed to take any mouse input whatsoever (potentially controlled by a hotkey), as that may solve your problem.
That's pretty cool, with the chroma stuff, never realised streaming applications had that feature (quite obvious now that I think about it..). Thank you for the example!
I can tell someone skipped the capture tutorial..! 😉
I understand that the naming can be confusing. I haven't really spent any time trying to come up with a less confusing way to name the two modes.
Eitherway, the 'Automatic' mode is as you found out executed the same way, just constantly capturing and immersively rendered into the background. It is not a 'Fullscreen' capture mode, this option is planned but not added yet. Thus, what you should use it for is just to capture specific areas that only render text, to avoid it from interfering with the background and gameplay intensive areas.
Manual mode is meant more as a 'translate-on-the-go' alternative.
There has been a previous request to basically mix the two, to be able to select an area to continuously capture & translate, but then render the translated text in a 'Manual' style window. The more examples I see of how people want to use the app, the more I understand the request. I think it is something I will implement soon enough.
Btw, there's some configs to modify to make the 'Automatic' mode less destructive that may work in your favour!
Thank you for your examples and input. Sorry about the late reply! :)
That's understandable, the niche issue thing. (We tend to be niche-focused and doing/checking out weird things.) No worries, I can get by! It's mostly-solved as far as I'm concerned anyways since I found my workarounds. Plus, yes, your idea of making the window not take mouse input would 100% solve this specific issue I'm having, if I can't click it to active, Xsplit can't glitch out on it~
Actually I didn't skip the capture tutorial, I read it a couple times. I just suck at information-retention sometimes. ^_^; Sometimes I need stuff worded to me in a different way for me to fully understand it. It's a learning disability (it made me fail math as a kid, I never got my high-school diploma mostly because I could not understand many abstract concepts in it), and it made me misunderstand something in that tutorial for sure. The 'naming' is probably part of it, but I agree, I can't think of a better name either.
I see, it's not meant for fullscreen. Yeah - that's fine with some games with their text boxes in one place, but some games have their text all over the place, like the one I was testing ironically. (That game is one of those where the text box will be placed high or low based on the player and npc positions on the screen, among other visual tricks.) I guess it's possible to update the screen capture area manually either way, but this is also why I found the Manual mode just made more sense. Again, for games like that, at least. (Manual mode obviously has the huge downside of 'I must stop focusing on game controls to do this', which could be bad with more actiony stuff. I understand the value of Automatic mode for sure.)
Mixing Manual and Automatic could be functional, depending. I think my only real request would be to always let us have a standalone Manual option, just in case the basic direct approach is less buggy than Automatic functions for a person... Even if you don't wanna keep updating Manual mode (if it is a niche thing not used much, understandable), just letting it continue to exist is all I'd really ask. :)
As for making Automatic mode less destructive, could you maybe point out which options I should be exploring more? I did see a number of things that sounded potentially useful, but even with the mouseover tips I wasn't sure what I should be messing with. I did a few changes to some things I thought would help its formatting, but then was further confused because it seemed like I'd made no visual changes whatsoever even though I had changed some options.
Also no worries, feel free to take your time~
