Alright, here is a little demo of how it could look like to give an AI character knowledge of what can be seen on screen and control over what is allowed and what is not: Hotscreen AI Demo
You can see at the bottom of the chat window the classes that are currently on screen and if they are allowed or forbidden on the right are actions that can be chosen by the AI. So with each character reply it can decide weather to allow a class that is currently forbidden or vice versa, that happens all autonomously via a hotscreen bridge within the platform and a custom mod in hotscreen talking to each other. The AI platform can use different services for TTS, SST and text-gen to bring everything to live.