I might have found a solution, it might have been obvious for long unreal engine developers, but im just a newbie with unreal engine. So theres some folders in ue projects that hold temporary files and built files, cache, etc... and all those files are regenerated once the project is opened, so the solution is to add them into .gitignore, and those are: Intermediate, Saved, Build, DerivedDataCache and .vs, the last one is because i use visual studio as my main IDE for UE.
If someone found misinformation pls correct me!