Hello, this is all covered in the rules section:
Your game must be for the Atari Lynx console and submitted as a LNX ROM file with a correctly populated header
You can use any tools or assets you wish and can code in C or Assembly
Having said that, I doubt very much that Game Maker Studio will be able to do what you want. Remember this is a 30+ year old console, so you need to have a very different approach to coding on it compared to modern day hardware.
I suggest you take a look at the following page that we also linked from the jam page...