One more insight, I am playing with outline just to see how it interacts with the issue.
I put a red square behind my text and outline seems to have the same issue of the background shining through:
