Well.. Fear and Hunger has sort of custom fighting mechanics, though it's a horror game and that approach will not fit your game. I remember good old Polymorphous Perversity (can be found on gamejolt com). The game is full of very diverse sex scenes (the whole thing is like discovering your own sexuality), made on RPG Maker, but a very good example of custom graphics and fight mechanics.
Viewing post in Yes, Brother - [VISUAL UPDATE! v1.3.9b] comments
you can check a little bit of my research on https://maledollstudio.itch.io/yes-brother/devlog/149241/girls-are-lgbt-boys-are-just-gay-the-appropriation-of-the-tag-lgbt-by-lesbian-protagonism :)