As per documentation, tj_encode accepts arrays, TJ objects, TJ booleans, and basic values. It does not accept maps, as there isn't even a way to tell that your value is a map, let alone tell which children of it might be maps.

So you'd want to do

var obj = tj_object("a",10, "b",50);
var json_string = tj_encode(map);
show_debug_message( "result: "+ json_string );

or

var obj = tj_object();
tj_set(obj, "a", 10);
tj_set(obj, "b", 50);
var json_string = tj_encode(map);
show_debug_message( "result: "+ json_string );

if you can describe your use case in more detail, I may have further recommendations.