Here's part of what's happening in the Hardliner Demo to limit mouselook up/down:

In this case "playerHead" is a sprite with camera behavior and an instance variable "zLook". Now I'm lerping the value of this variable to "rawY" (vertical movement) of the mouselock. It's divided by "unitRes" to translate from pixels to units since Z position (or often referred to as Z elevation in context of the plugin) is measured as such.
This is being clamped between the negative and positive constant global variable which defines how far up and down you can look.
Then "zLook" is being applied when using the "Set camera target" action or could be used in a rotation action like you're apparently doing.
Let me know if you have further questions.
