If I'm understanding you correctly, you're saying that the phosphors appear too big relative to your game screen when running in a small window. This is actually by design; the shader reads your window size and outputs the phosphors in sync with your actual screen's resolution. For example, if you are using a phosphor mask with 3 columns (red, green, blue), it will always tile every 3 pixels on your real-life screen, and it can't go any smaller without completely breaking the effect. If you need a finer pattern, you can swap the mask out for the included 2-column wide (green, magenta) texture that takes advantage of your screen's RGB subpixel layout to squeeze everything as small as physically possible. If this still isn't acceptable for your needs, then you may want to disable the phosphors when your window size is very small. This effect generally requires an output resolution of 720p at the very least to begin looking correct.
As far as your code, it looks like it should generally work fine. However, this shader comes with built-in aspect ratio correction, meaning that you should be able to resize your output window to any shape at all, and the shader will automatically ensure the simulated CRT screen has the correct 4:3 ratio (or whatever your game is) with reflective borders filling any leftover space. I'd generally recommend just setting the output width and height to the actual window width and height (window_get_width(), etc), and let the shader handle everything else with constant game height, width, and aspect ratio settings.