Are there any specific requirements for the game type beyond "tabletop RPG"? Any system / custom systems? What about more conventional board games, or some kind of RPG-board-game-hybrid?

I ask because I'm interested in games that are similar in spirit to RPGs but that de-emphasize the actual roleplaying aspect, or RPGs where there's less/no distinction between players and game masters, and other sorts of variations on the typical format.

So, I guess the question is how far can I push the definition of "RPG", or what are the essential elements of an RPG for the jam's purposes?

I don't think there's a solid wall between what is and what isn't a table top RPG. Besides, I've seen many games coming out on itch that pushes the boundaries of what TTRPG should be.

I love GM-less RPG. Lovecraft-esque is a really good example.

Some more boundary pushing RPGs I can think of are Microscope and Quiet Year.

One of the most important aspect I see in RPG is open-ended storytelling. Every time you play the game, unique stories can be told.

Coming back to your question, it's an RPG as long as you think it is one. There's no judging in this jam, so no one can really tell you off for not making a "proper" RPG.