There's no right way to write a story IMO! You probably know what's best for the story you want to tell.
If you really want our (or my) opinion, I think you're probably right on cutting filler fluff, BUT I don't think any of what I've read so far counts as filler fluff. It was all plot related, setting up environment, setting up character developmet. This is why I've been loving this story so far. You've managed to add all the juicy romance and character development with the plot building up around it, waiting for no one.
Ultimately it's your story and you get to tell it how you feel it's best! My opinion is simply that you've been doing a great job so far, and I have zero complaints.
Viewing post in Violet Memoir Dev Log! VERY IMPORTANT! comments
I do agree about a lot of the previous stuff not being filler. Some of it has been setting up characterization and dynamics. You can afford more filler early on in your story so the reader can get used to the characters.
But even then, there are still some that could have been snipped down. We didn't entirely need to have.
For example, I don't really consider much of the latest Leigh build as filler. Maybe some of the dialogue with Charlie but most of that build's scenes served a purpose.
However, some scenes like Lucas explaining the book he's reading in an earlier scene of his route really didn't need to be there to that degree. We could have simply learned the basics of the book to infer that Lucas prefers reading stories for escapism.
My mentality about filler is scenes that don't serve a purpose. Even if a fun scene might be nice to read, if it doesn't serve a purpose it's cuttable. However, that scene might still be included because it serves a purpose tonally or for pacing reasons.