No need to worry about how long replies take. Really. (I didn't get around to testing last week due to other things coming up, too, though I *will* do it this week)
As for the ID thing, yeah, it's more like your first example, just getting an extra value that can be passed along with the request for anyone who has a weird setup like mine. I don't think the LLM should ever see it, and some APIs might break because it's an unknown field. I also don't care if it's stable between sessions: it can be an integer or a UUID or something user-definable; my use-case and design just wants a way to differentiate sources.
What I'm doing is intercepting every JSON post, adding them to a list, making a Chat Completion request based on that, and returning the result.
I have GameTranslate post something simple like this:
```
{
"text": "<captured data>"
}
```
And then a small Python script is building the real request and sending back the response from a llama.cpp server:
```
{
"model": "aya-expanse",
"temperature": 0.65, //other parameters, too, but they're not important to illustrating the idea
"messages: [
{
"role": "system",
"content": [{
"type": "text",
"text": "<my-system-prompt, basically telling it that it's a casual translator that is willing to use slang and assume a conversational tone>"
}]
},
{
"role": "user",
"content": [{
"type": "text",
"text": "<old captured data, held in memory, each as their own message, to build chat-history; a separate request to this proxy resizes the list's maximum capacity, letting me reset it on scene-changes or when more or less history is appropriate; another lets me drop recent messages, in case I don't want them influencing the next production>"
}]
},
{
"role": "user",
"content": [{
"type": "text",
"text": "Translate the following into English without adding any commentary or explanations:\n\n<new captured data>"
}]
},
]
}
```
The response is intercepted and the newly captured data is appended to the "choice" block so I can see the translation and the original text when the overlay is opaque (I also sometimes add a kana line here). And then that slightly modified result is returned to GameTranslate to be presented.
Having the window-ID as an optional parameter that I can set in the initial request would let me easily direct the new text to an appropriate message-history list so that the main dialogue box doesn't get mixed up with a speaker-string. People who just pass things directly simply wouldn't set it, so there would be no impact on working setups.
I do think you could integrate something like what I'm doing into GameTranslate itself (with a bit less flexibility: no control over runtime buffer-tweaking because that would be very hard to present intuitively), but as illustrated by your use of Text Completion (which, yes, is more powerful), by now, people have surely created their own workflows around this feature. Just having REST-JSON opened up a ton of possibilities.