I guess the brothel is optional and if we don't set foot in it it'll be fine.
I started Nagatoro's quest "Kill the goblins in the forest", watched the beginning of the scene in the farm village, decided it wasn't worth doing the quest and backed out of accepting it.