Hey.

I'm thinking about getting into automating the NPC creation process. But I'm wondering what's already out there? Like if you're making a GTA type of open world game, how do you create the behaviours of NPCs?

The same way you do anything, I suppose. It’s just code.

If you want to make a gta style npc you could code behaviours into the npc. If you use Unity you could code State machine behaviours to control the npc 

https://docs.unity3d.com/Manual/StateMachineBehaviours.html

You should definitely take a look on behavior trees. In most cases they are much cleaner solution than state machines, and there are tons of documentation, resources and assets available. They are also built-in solution in many game engines, such as UE4.

https://docs.unrealengine.com/en-US/Engine/ArtificialIntelligence/BehaviorTrees/...