I'm having a similar issue, where the shader looks completely different in full screen v. windowed. Is there a way to lock the output to a specific resolution so it looks identical no matter what the window size is?
Viewing post in Advanced CRT Shader for GameMaker comments
The shader is designed to tile the phosphor mask against your screen at a 1:1 resolution, as is common in CRT shaders for emulation and retro gaming. There is, however, a parameter in the presets called mask_scale, that will simply resize the mask relative to your screen. If you want a fully consistent look regardless of screen resolution, I would probably recommend disabling the phosphors altogether, or using a neutral mask like the shadow mask. If you're hard-set on it, then what you would need to do is pick your "default" resolution, like 1080p, and then set crt.mask_scale = (1080 / window_get_height()) when necessary, such as after room transitions or when the window resizes. This would ensure there are a consistent number of phosphor triads between resolutions, but it can begin to look very broken at lower resolutions, so use with caution.