Unfortunately I don't think this documented anywhere. I believe the data type is specific to the backend database we use for ASM. Though I tried running rule_get on an existing monitor to presumably get a valid value and then tried to create a new monitor using the value returned. This also failed with the error:
error>illegal parameter value</error>
<info>could not determine script type</info>
I'll discuss this with our internal ASM team and see if we can get a better answer as to what encoding is used and why the error occurs.