Post by Charem in Update 0.5.6 - New tutorial system! comments

Update 0.5.6 - New tutorial system! » Comments

Viewing post in Update 0.5.6 - New tutorial system! comments

I just tested out your new guide, left a reply over there on that post about my experience with it, since it might be useful to others too. :)

On another note, along the way, I also ran some more capture tests with my stream broadcaster (Xsplit). It is currently not possible to capture the game with the translation overlayed on top of it without capturing the entire screen space, but that function's a privacy risk we almost always avoid using since if anything sensitive pops up over the screen in that area it will also get captured and displayed. We typically capture by application process, not by screen-space, as such.

It IS possible to capture the game application, then also capture the translation application on top of that, but the translation application capture weirdly covers the whole screen as black when this is done. (There is technically a game behind all of that blackness lol.)

It is possible for me to then Chroma-key out 'Black' on the translation application capture to let the game be visible, but the text is kinda painful to read in this situation.

If you want to make this program more-accessible to streamers, here's what I suggest... It would be nice if we could customize how the translation overlay displays itself.

1. Customizable background color. If I could control the background behind the text to be a different color (like green). then I could easily chroma it out cleanly. (Or heck, just give the option for a transparent background.)

2. Customizable text color. If I could change the color of the translated text itself, I could pick a color that stands out more-easily on top of the game, especially after chroma-keying the background out.

3. Optional text border. If I could add a border around the text, that would help it stand out atop the game as well.

4. Optional text shadow. Again, would help readability.

Here's a quick mockup of what I'd find ideal personally, if those four customization settings existed for the overlay function:

Even beyond streamers, these suggestions could just be a nice way to customize things for any users. Your approach of putting white translation text in a black box works for now of course, but these are things I would hope for in the future. :)

EDIT: Ah, with a bit more tweaking of how strongly the chroma-key was told to react, I was able to capture just the black box and not black-out the whole game screen.

That being said, it will sometimes blink the whole screen black for the capture display due to how the black chroma-keying is working to cull out the rest. That would be quite unpleasant for viewers if it keeps up. I might be able to tweak further to stop that from happening, but the customization ideas I shared would solve it for good.

Godnoken61 days ago(+1)

Aha! Xsplit must be breaking how the Desktop/Attached mode works. How I achieve transparency is basically through 'Chroma keying'.
I used to have a different implementation entirely, but it used quite a bit more power to run. I wonder if that solution would work differently in this case.

1. If you mean the background colour of the translation window, this is planned! :)

2. Also planned.

3. Definitely something I'd like to have.

4. Ditto!

Yeah, your example clearly demonstrates how important customization is. Cheers for that!

Thank you for all that testing, this is very interesting. I'm not sure how the chroma-key is set in Xsplit, but the 'transparent' parts are all set to exactly pitch black 0.0, 0.0, 0.0.

By the way - is there a reason you are using the 'Manual' translation window instead of the 'Automatic' one? If so, for your use case, why? It's good for me to understand how users want to use the application and the 'Manual' translation mode has been quite neglected to favour the development of more immersive translations.

Charem60 days ago (4 edits)

(Consolidating my replies here~)

OBS is by far more popular, technically better but... Our previous computer didn't work right with it, which is why we got into Xsplit. Xsplit kinda sucks in some ways but it's serviceable. Fortunately, the two program are similar enough and share enough of the same toolset that if something works for one, it'll probably work for the other.

Ah Swedish huh! Neat~ We have a couple of Swedes in our audience too.

Oh that's interesting, so you do use chroma keying already! I definitely think it's best to use the lower-power method since a utility program like yours should definitely not make too much of a footprint.

1. Yes, basically!

2 - 4. Excellent~

It's certainly functional as it is, I got the blackness problem to mostly settle down. Ran another test and I think it's that...if I click the GameTranslate overlay that pops up over the game, then Xsplit thinks what I clicked is the 'active window' it should capture over the game window. (As I use an 'auto-detect' setting for capturing games.) Basically, Xsplit thinks the GameTranslate overlay is a 'game' if I click on it. If I avoid accidentally clicking any overlay elements this shouldn't be a problem though.

As for your 0,0,0 blackness I figured! I basically...well, here's a pic of that settings menu I configured for chrome keying the overlay right. If I chroma too little blackness, it just covers the whole game window. If I increase it too far though, that's what causes it to eat into the lettering and the text window and look bad. These are the settings I found to work best.

As for the Manual vs Automatic... I didn't really 'understand' Automatic. I initially used it but it didn't...well, literally seem to be doing anything with the games I hooked it to? TBH I sort of gave up tinkering with it very early on, set it to Manual, set my tilde key to be the Fast Key, and just press tilde to start a selection, drag it across the text I want to translate, and let go. Besides...if Automatic is (supposed to, at least) capture the whole screen...that may be messy. If I try to capture a lot of text with the Fast Key drag, like the whole screen in a complicated inventory, the text sort of jumbles up. So I think it's better to capture sections at a time anyways.

I would like to know...how to make Automatic...work, though. I didn't actually get it to do a thing for me, so I can't comment on if it would be useful to me or not.

EDIT: Oh, I see. I tried again and got Automatic mode to work. I didn't realize you also had to still MANUALLY select a capture area while Automatic mode was on and then the model got to work automatically. ...That said, I don't like it very much for a common habit it seems to do that this gif shows...

I think a controlled manual capture is better in most cases. Less-intensive too, I'm sure, than the translation tool running across the whole screen all the time.

And yeah, if there's text all over the screen GameTranslate can...be very messy about displaying it, which basically makes a full screen auto-translate sort of a jumble. In both Manual and Automatic modes. Here...

This is a shop screen: Untranslated, Automatic, and then Manual, respectively. Either type of translation of this screen is quite messy when I try to capture the whole screen's text. Manual's layout is worse than Automatic's, but of course Automatic here was also still flickering the translation to different variations too. All sort of a mess, regardless of what I pick for a screen-wide catch-all capture area.

So yeah, Manual with capturing sections at a time feels better...for right now, anyways. Surprised to hear it's been neglected; it actually feels the best right now, so I would ask that you don't leave it behind too much in the future. :)

Godnoken57 days ago

We're a tiny country, but we get around!

Yep, indeed it does! Exactly yeah, that was one of two main reasons I changed it, the other being that the 'Automatic' mode didn't work in Desktop/Attached mode without making these changes.

I see. Well, there could potentially be a solution for this, but it sounds like a very niched issue and not something I have time to spend on right now. The idea anyways is to make the Desktop/Attached window not allowed to take any mouse input whatsoever (potentially controlled by a hotkey), as that may solve your problem.

That's pretty cool, with the chroma stuff, never realised streaming applications had that feature (quite obvious now that I think about it..). Thank you for the example!

I can tell someone skipped the capture tutorial..! 😉
I understand that the naming can be confusing. I haven't really spent any time trying to come up with a less confusing way to name the two modes.
Eitherway, the 'Automatic' mode is as you found out executed the same way, just constantly capturing and immersively rendered into the background. It is not a 'Fullscreen' capture mode, this option is planned but not added yet. Thus, what you should use it for is just to capture specific areas that only render text, to avoid it from interfering with the background and gameplay intensive areas.
Manual mode is meant more as a 'translate-on-the-go' alternative.

There has been a previous request to basically mix the two, to be able to select an area to continuously capture & translate, but then render the translated text in a 'Manual' style window. The more examples I see of how people want to use the app, the more I understand the request. I think it is something I will implement soon enough.

Btw, there's some configs to modify to make the 'Automatic' mode less destructive that may work in your favour!

Thank you for your examples and input. Sorry about the late reply! :)

Charem57 days ago (1 edit)

That's understandable, the niche issue thing. (We tend to be niche-focused and doing/checking out weird things.) No worries, I can get by! It's mostly-solved as far as I'm concerned anyways since I found my workarounds. Plus, yes, your idea of making the window not take mouse input would 100% solve this specific issue I'm having, if I can't click it to active, Xsplit can't glitch out on it~

Actually I didn't skip the capture tutorial, I read it a couple times. I just suck at information-retention sometimes. ^_^; Sometimes I need stuff worded to me in a different way for me to fully understand it. It's a learning disability (it made me fail math as a kid, I never got my high-school diploma mostly because I could not understand many abstract concepts in it), and it made me misunderstand something in that tutorial for sure. The 'naming' is probably part of it, but I agree, I can't think of a better name either.

I see, it's not meant for fullscreen. Yeah - that's fine with some games with their text boxes in one place, but some games have their text all over the place, like the one I was testing ironically. (That game is one of those where the text box will be placed high or low based on the player and npc positions on the screen, among other visual tricks.) I guess it's possible to update the screen capture area manually either way, but this is also why I found the Manual mode just made more sense. Again, for games like that, at least. (Manual mode obviously has the huge downside of 'I must stop focusing on game controls to do this', which could be bad with more actiony stuff. I understand the value of Automatic mode for sure.)

Mixing Manual and Automatic could be functional, depending. I think my only real request would be to always let us have a standalone Manual option, just in case the basic direct approach is less buggy than Automatic functions for a person... Even if you don't wanna keep updating Manual mode (if it is a niche thing not used much, understandable), just letting it continue to exist is all I'd really ask. :)

As for making Automatic mode less destructive, could you maybe point out which options I should be exploring more? I did see a number of things that sounded potentially useful, but even with the mouseover tips I wasn't sure what I should be messing with. I did a few changes to some things I thought would help its formatting, but then was further confused because it seemed like I'd made no visual changes whatsoever even though I had changed some options.

Also no worries, feel free to take your time~

Godnoken53 days ago

Thank you for your patience! :)

Good to hear, I really hope to be able to make everything work as you wish to use the app soon enough!

Aha! No worries, I fully understand and sympathize with that. It's definitely not a great tutorial anyways, so I'm not surprised it can be confusing 😅

Yep! I really would love to add a proper fullscreen mode, it is just that it adds yet another mode to maintain when I release new features. And the biggest two problems are performance and actually making the OCR good enough so that it isn't constantly finding text that doesn't exist and thus ruins the experience.

Oh don't worry, the Manual mode is going nowhere. It is here to stay! :)

To make Automatic mode less destructive, go to the General tab and lower the "Text Removal Expansion" to the lowest possible value. Also, keep in mind that when you change settings, you need to restart the tool (not the main app) to put these changes into effect :)

Charem52 days ago

Thanks Godnoken. ^^

I can understand the hesitation with fullscreen capture support though, it would increase load after all. Yeah, the performance hit of scanning an entire game window rather than just parts. The OCR should probably be gradually-improved regardless of this fullscreen mode idea but it definitely would necessitate it be refined faster too. (And I imagine the image-recognition aspect of this program is somewhat-tricky to design, it doesn't sound like a simple thing for the program to do.)

Awesome to hear about Manual mode staying. :D And thanks for the tip! I was wondering what those values did. Ah - you know I don't think I was always restarting the tool after adjusting settings so maybe that's why I was 'not noticing anything change' sometimes. Good point! That'll definitely help me tweak this.

Godnoken43 days ago(+1)

Yep, the image recognition part is definitely the trickiest to perfect. I'm actually working on a complete test-bed behind the scenes, which should over time improve the OCR across the board, and potentially push forward a fullscreen solution! :)

Yep, that's on me, haha. I had an unintuitive and bad setup. I hope the recent update made it easier for you to test things out. The next update will make it possible to change these values from the dashboard in the tool too, and more improvements like that are on its way.

itch.io

Viewing post in Update 0.5.6 - New tutorial system! comments