I used OGRE as the renderer in my game and Bullet Physics for collisions and other physics tasks. I added Lua as my scripting engine and built the rest of the engine to glue them all together. Given how sophisticated popular game engines like Unity or Unreal have become, the idea of building a custom engine seems almost like an unproductive thought, but an issue in my case was that I needed to expose a scripting debugger to my players and none of the known engines can do that. Turns out it was a lot of fun just to work on it. So I would definitely say.. go custom if you value the journey more than the destination.