You know best what is useful and what isn't! If you're feeling like it's hard to tell sometimes, having an editor to help you with that might be a good idea.
There's no right way to write a story IMO! You probably know what's best for the story you want to tell.
If you really want our (or my) opinion, I think you're probably right on cutting filler fluff, BUT I don't think any of what I've read so far counts as filler fluff. It was all plot related, setting up environment, setting up character developmet. This is why I've been loving this story so far. You've managed to add all the juicy romance and character development with the plot building up around it, waiting for no one.
Ultimately it's your story and you get to tell it how you feel it's best! My opinion is simply that you've been doing a great job so far, and I have zero complaints.
