REST job response parsing and undefined variables

[Michael A. Lowry]

We use a REST job to check the status of jobs in another AE system, using the GET executions endpoint. In this use case, we are requesting tasks that have completed. We recently discovered that the executions endpoint sometimes includes in the list tasks that are still running (specifically, in status 1574 / Post Processing ).

The REST job is configured to parse the JSON response and create arrays with the details of the tasks found.

Tasks in that have not completed lack an end_time element in the JSON response. When one of the JSONPath expression fails to match, the corresponding array will not be created. When the problem happens, only the &RESPONSE_END_TIME# array is not created; all of the other arrays are created. If the task with no end time is first in the JSON response, and followed by completed tasks, the &RESPONSE_END_TIME# array will be created only during parsing of the second result in the list. As a result, the index of this array will be off by one, compared to the other response arrays.

While we wait for a fix for the REST API bug, is there a good work-around for this? Is there any way to specify that the arrays should be created even if the JSONPath expression does not match, perhaps will a null or empty value inserted as a placeholder for  the missing result?

[Michael Lumpkin]

Did you receive a fix or acknowledgment this is issue?

While not identical in nature (the array size problem) I've discovered the issue many years back where the array (true) object variable isn't created if the response parsing/expression returns nothing.  One caveat, if you specify the first position of the JSON array instead of *, it will still create the object variable as array (i.e. $.data[1].end_time) even when it doesn't parse or return any JSON results; I know this doesn't help but it's interesting an explicit position that doesn't exist or parse in the response will still result in an object array variable being created (with array size = 1), however traversing the entire array using * doesn't work when parsing fails.

I have 2 work arounds you can probably implement:

Option 1) I don't know why it works but it does.   Declare the same array variable in the post processing but wrap it in a ":IF/:ENDIF" statement that is always false.  It seems to trick the interpreter somewhere/somehow and doesn't complain about the missing variable when referenced later in post processing, but also appears compatible in post processing when array object variable is created as a result valid response parsing.  I don't know how long this will work and I always test it after patching but currently it still works in V24.4.1 both with RA REST and IG REST and worked in V21.0.9.  Personally, I think it is a bug, and Automic should create a minimum Array object, empty, just as they do non-array objects when no result is parsed.

Example Post Processing:

!Trick the interpreter, if no response value is parsed then array object variable will not be created, but a script array variable will be available instead.
:IF 0 = 1
:DEFINE &ARRAYVAR#, string, 1
:ENDIF

:P "Your array always exists and first position value is: &ARRAYVAR#[1]"

If your array parsing doesn't return anything, &ARRAYVAR#[] will still exist as a script array variable and your post processing won't fail, so you can evaluate the result to do something like test the first array value and length to know if you have results.

Option #2:  Change the Parsing Type to Json to XML and XQuery, or Groovy, and write an appropriate Expression to verify the array and set the response variable to the array result or a fixed value you can then test in your post processing.

Here is an example for Json to XML and XQuery (no named parent in this case due to XML), if the array is empty it will return "-1" in the response object array variable else it will return the response array in the same variable.

for $x in $input/uc4TopParent
return 
  if (empty($x/uc4ArrayParent/id))
  then <result>-1</result>
  else $x/uc4ArrayParent/id/text()

Both methods above result in either the actual JSON response array (when parsed correctly) in an object array variable, or either an empty script array variable (option #1) or an object array variable with a dummy value (option #2).

-------------------------------------------

[Michael A. Lowry]

Thanks for the good ideas. I believe we implemented a similar trick to fool the parser and ensure that the arrays were defined.

-------------------------------------------

[Eric Lontz]

This can be accomplished with updating the JSON Path Expression.
by adding a filter for the end_time field to the as a prerequisite to parsing the response data.

by replacing [*] with [?(@.end_time)] you will only parse results where the end_time element is present. You don't need to update the end_time parsing.

$.data[?(@.end_time)].parent
$.data[?(@.end_time)].alias
$.data[?(@.end_time)].activation_time
$.data[?(@.end_time)].start_time
$.data[*].end_time

-------------------------------------------