"You seem to fuss about the word token or the definition."
I do, forgive me--but there is a reason for that. It's because I had to comprehend them in order to build my own AI (not a chatbot, something completely different). A token is actually nothing on it's own. A single variable holding an array of strings--that's a token dictionary. The final layer gets softMaxed and produces an index list of probabilities that matches the same amount of indices in your array. Then there's that game of "Battleship" I spoke of. If index 304 sits at 0.989 then the "word" in index 304 of the word dictionary array is chosen by code logic--not the neural network itself.
There is no token use in the beginning. That's where the disconnect is. Tokens are for after the system is done with it's calculations, not before. There is an easy way to both prove and demonstrate this:
Recently, a man was able to get an AI to send him money from that AI's company. (he got caught) He did this by communicating in Morse Code to the chat agent. There are no Morse Code tokens. However, the AI was still able to understand the input. Why it worked as an adversarial prompt, that is also obvious when one stops considering the framework around the system and looks directly at how feed-forward neural networks do what they do.
So that's why I get hung up on it. The way the tech industry uses the word "tokens" to track monetization, it muddies the waters on what a token actually is and what function it performs in the overall architecture.
Check it out--you don't even need tokens at all. This is a short demo video for a completely different kind of AI that I am developing for video game NPCs (a multi-modal dual-input feed-forward/simplicial neural network architecture):