Looks like the same issue is in the example project too:
I tried also setting up another project with only a code-created camera but no dice. Maybe its something with my hardware? I recently got of the new 9070xts but I don't really know if that's relevant or not.
My game is pretty early in development so if you wouldn't mind taking a look at it I could send it to you and see if the issue is there on your half.
That doesn't look like a problem with the example project. The circle sprite is stretched and has no anti-aliasing pixels so it will look jagged. Look around the scene and see if there are any of the HD sprites looking incorrect.
The problem you originally posted might also just be because you are rotating a low resolution sprite with no anti-aliasing.
In the example project, if you place something like the military crate objects that cast shadow and rotate it to some random angle, does it also have the same issue?
The crates do look jagged but they also look like that in the room editor so it is probably an anti aliasing issue, which makes sense since the blue squares are only 56x56 in pixels, mainly to get the scale on a 1080p canvas right. I plan to use a scaled down larger sprite once I move past placeholder sprites since I'm going for HD art.
Since the non-shadow version of my sprite does look a lot smoother than the shadow one, does that mean that GM is applying some sort of AA automatically? And how could I apply AA to the shadow caster in order to get the same or similar effect?
Edit: The crates do appear to have the same outline effect when rotated too. They also seem to have some highlights on the edges where it doesn't really make sense to?
That does look normal. Remember we are essentially zooming way in on texture pages here with sprites like this. It is like in a 3D game when you get the camera as close as possible to some texture and see the actual pixels.
One thing you can do to improve it is on the light_engine object in the variables tab, change the shadow_atlas_size and shadow_map_size.
The atlas size is the total size that will be used and you should simply use the biggest your platform supports. The shadow map size is the resolution for shadow maps and bigger will look better. The more you can fit into a single atlas, the faster performance is.
And again, the better aliasing the sprite has will help.
Another improvement is adding your own anti-aliasing as a post processing effect. As you know this is common and hides jagged edges in most games. You can use a post draw event somewhere after the game world is rendered and apply it (turn off automatic draw of application surface, etc.).
Increasing the shadow atlas and map help a lot and combined with display_reset's AA it looks pretty good now, but I'll probably have to keep performance in mind going forward. It's hard for me to judge it now since I don't have much on the screen visually yet but I'll see.
Thanks for the help! Additionally, would you happen to know any preferred AA systems (shaders or whatnot?) I should be able to find some on my own but if you have any recommendable ones that could help too.
