Load default prompt set values from a VARA object using {VARA,KEY,COLUMN} references

[Michael A. Lowry]

Has anyone tried to create a prompt set whose default values come from a VARA object? I thought this might be a nice way to store and recall each user’s most recently used values. Here is what I have in mind:

The &$USER# variable is resolved to the ID of the executing user; then the enclosing VARA object reference (in the format {VARA,KEY,COLUMN}) is resolved.

The only problem with this approach is that if the specified VARA object, key, or column does not exist, the VARA object reference will not be resolved at all, and this may not be desirable.

[joel_wiesmann_automic]

Hi Michael

We do use this feature very heavily with various variables - some are internal, some are calculated during runtime.

If the {} notation does not resolve the vara or key wasn't found - this seems to be very relieable, in about all cases it was either a typo or the key was forgotten to add. It can also be added when the prompt is open and when it's being submitted it will resolve. We did only use this feature with STATIC VARAs, I don't know how it behaves with SQL VARA / other types.

Hope this helps, regards
Joel

//Edit: AE 10.0.7

[Michael A. Lowry]

The only problem with this approach is that if the specified VARA object, key, or column does not exist, the VARA object reference will not be resolved at all, and this may not be desirable.

It occurred to me that I could use an EXEC VARA as a wrapper between the prompt set and the actual the static VARA where previously used settings are stored. The EXEC VARA would check to see if values already exist, and automatically fill in initial values if there are none present.

Because an EXEC VARA can have only one result column though, I would have to create a separate wrapper EXEC VARA for each prompt set value whose default I want to load this way.

[joel_wiesmann_automic]

I agree with that. I guess this has been done to provide backwards compatibility.

By the way.. we use this feature also for environment-specific values.

VAR:
CONFIGKEY          |{VAR,CONFIGKEY/&$CLIENT#}
CONFIGKEY/0010 |PROD
CONFIGKEY/0020 |TEST

If you put {VAR,CONFIGKEY} into your promptset it will resolve to PROD / TEST depending on whether the solution run's on client 0010 or 0020. Very useful for transportable solutions.

[Michael A. Lowry]

The Automation Engine apparently does not like it when the default value of a prompt set element contains a curly-brace reference to an EXEC VARA. Running such workflow hangs the Java GUI completely. If I kill the associated java process and log in again, I see that the workflow is in the Waiting for user input status. If I select the workflow and choose Modify task parameters, the GUI hangs again.

I also tried submitting the workflow in the Automic Web Interface. The web UI does not hang, but the prompt set “Request” never appears, and the workflow remains in the Waiting for user input status.

The EXEC VARAs work fine on their own. This seems like a bug.

Oh, and the AWI lacks the Modify task parameters function. I added this to the list of missing AWI features.
Update: This function was added in v12 and is called Open User Input Dialog.

[Michael A. Lowry]

Because an EXEC VARA can have only one result column though, I would have to create a separate wrapper EXEC VARA for each prompt set value whose default I want to load this way.

I thought I might have found a clever way around this, by creating a data sequence containing multiple columns in the SCRI called by the EXEC VARA.

! Read values from a static reference VARA, addding keys that do not exist.
! It is a bit like NOTFOUND=IGNORE, but for curly-brace VARA object references.

! Read input parameters from calling EXEC VARA 1: user, 2: VARA value column
:READ &Ref_VARA_Name#,,
:READ &Ref_VARA_Key#,,
:PRINT "Reference VARA name   : &Ref_VARA_Name#"
:PRINT "Reference VARA key    : &Ref_VARA_Key#"
:SET &Value_1# = GET_VAR(&Ref_VARA_Name#,&Ref_VARA_Key#,1,)
! If GET_VAR returns a single space, it usually means the key was not found.
:IF &Value_1# = " "
:  PRINT "Creating initial user defaults."
:  PUT_VAR &Ref_VARA_Name#, &Ref_VARA_Key#, "", "", "", "", "", ""
:ENDIF
:SET &Value_2# = GET_VAR(&Ref_VARA_Name#,&Ref_VARA_Key#,2,)
:SET &Value_3# = GET_VAR(&Ref_VARA_Name#,&Ref_VARA_Key#,3,)
:SET &Value_4# = GET_VAR(&Ref_VARA_Name#,&Ref_VARA_Key#,4,)
:SET &Value_5# = GET_VAR(&Ref_VARA_Name#,&Ref_VARA_Key#,5,)

:SET &ProcHnd# = CREATE_PROCESS(NEW)
! Write column headings
:PRINT "Writing column headings to data sequence."
:SET &RC# = PUT_PROCESS_LINE(&ProcHnd#,"Value 1;Value 2;Value 3;Value 4;Value 5",";")
! Write row of 5 values from reference VARA
:PRINT "Writing values to data sequence:"
:PRINT "&Value_1#;&Value_2#;&Value_3#;&Value_4#;&Value_5#"
:SET &RC# = PUT_PROCESS_LINE(&ProcHnd#,"&Value_1#;&Value_2#;&Value_3#;&Value_4#;&Value_5#",";")
! Send result back to calling EXEC VARA
:PSET &Result# = SAVE_PROCESS(&ProcHnd#)

The variables &Ref_VARA_Name# and &Ref_VARA_Key# are passed to the SCRI by the calling EXEC VARA. The idea is that the &Ref_VARA_Name# is the name of a static VARA that might be missing some keys. Curly-brace-style VARA object references would resolve to null values in cases where the key is missing. I’m not sure what key to reference in the curly-brace-style VARA reference though, and the bug is preventing much progress anyway.

[Pete Wirfs]

We do this in promptsets quite a bit for environment variables and state variables.  We use both static and SQL variable objects.   But we have never tried it with an EXEC variable object, so your findings are good to know.  

If I remember right, I believe this can also get into "waiting for user input" status if a variable returns a result that does not pass promptset edits, such as if it goes into a drop down list box which restricts acceptable values.

[joel_wiesmann_automic]

Very cool idea with the "add default if not found" Michael. Unfortunately I'm on V10 and have no EXEC VARAs yet, but I will keep that in mind (hopefully the misbehaviour will be resolved until then).

[Michael A. Lowry]

I opened INC00128431 for the bug “Workflow hangs in 'Waiting for user input' if default value of a prompt set element contains reference to EXEC VARA.”
I will post an update when I learn more.

[Michael A. Lowry]

I updated my script a bit, to make it possible specify the values to write if the key is not found.

Input parameter	Description
&Ref_VARA_Name#	Static VARA to read.
&Ref_VARA_Key#	Key to read from the VARA.
&Ref_VARA_Defaults#	Defaults to write to the VARA, if the key is not found. Separate values using the delimiter.
&Ref_VARA_Delimiter#	Delimiter to use when splitting defaults field.

! Read values from a static reference VARA, addding keys that do not exist.
! It is a bit like NOTFOUND=IGNORE, but for curly-brace VARA object references.
! Defaults can be specified in the third parameter.

! Read input parameters from calling EXEC VARA
:READ &Ref_VARA_Name#,,
:READ &Ref_VARA_Key#,,
:READ &Ref_VARA_Defaults#,,
:READ &Ref_VARA_Delimiter#,,
:PRINT "Reference VARA name      : &Ref_VARA_Name#"
:PRINT "Reference VARA key       : &Ref_VARA_Key#"
:PRINT "Reference VARA defaults  : &Ref_VARA_Defaults#"
:PRINT "Reference VARA delimiter : &Ref_VARA_Delimiter#"
! Check whether key already exists in reference VARA
:SET &VARA_Key# = GET_VAR(&Ref_VARA_Name#,&Ref_VARA_Key#,KEY,)
! If GET_VAR returns a single space, it means the key was not found.
:IF &VARA_Key# = " "
:  PRINT "Key not found. Writing initial defaults."
:  DEFINE &VARA_Defaults#,STRING,5
:  FILL &VARA_Defaults#[] = STR_SPLIT(&Ref_VARA_Defaults#,&Ref_VARA_Delimiter#)
:  PRINT "1: &VARA_Defaults#[1]"
:  PRINT "2: &VARA_Defaults#[2]"
:  PRINT "3: &VARA_Defaults#[3]"
:  PRINT "4: &VARA_Defaults#[4]"
:  PRINT "5: &VARA_Defaults#[5]"
:  PUT_VAR &Ref_VARA_Name#, &Ref_VARA_Key#, &VARA_Defaults#[1], &VARA_Defaults#[2], &VARA_Defaults#[3], &VARA_Defaults#[4], &VARA_Defaults#[5]
:ENDIF
:SET &Value_1# = GET_VAR(&Ref_VARA_Name#,&Ref_VARA_Key#,1,)
:SET &Value_2# = GET_VAR(&Ref_VARA_Name#,&Ref_VARA_Key#,2,)
:SET &Value_3# = GET_VAR(&Ref_VARA_Name#,&Ref_VARA_Key#,3,)
:SET &Value_4# = GET_VAR(&Ref_VARA_Name#,&Ref_VARA_Key#,4,)
:SET &Value_5# = GET_VAR(&Ref_VARA_Name#,&Ref_VARA_Key#,5,)
:SET &ProcHnd# = CREATE_PROCESS(NEW)
! Write column headings
:PRINT "Writing column headings to data sequence."
:SET &RC# = PUT_PROCESS_LINE(&ProcHnd#,"Value 1;Value 2;Value 3;Value 4;Value 5",";")
! Write row of 5 values from reference VARA
:PRINT "Writing values to data sequence:"
:PRINT "&Value_1#;&Value_2#;&Value_3#;&Value_4#;&Value_5#"
:SET &RC# = PUT_PROCESS_LINE(&ProcHnd#,"&Value_1#;&Value_2#;&Value_3#;&Value_4#;&Value_5#",";")
! Send result back to calling EXEC VARA
:PSET &Result# = SAVE_PROCESS(&ProcHnd#)

I have attached an XML export containing a set of AE objects that demonstrates this approach.

Attachment(s)

GET_VAR_OR_SET_DEFAULTS demo uc4_export.xml   6 KB 1 version

[Michael A. Lowry]

OK, now I’m really in uncharted waters. I thought I might be able to load the initial default values using SQL VARAs.
https://us.v-cdn.net/5019921/uploads/editor/wm/o1xsujrios4d.png" width="1080">
This does not work as hoped. This sets &Ref_VARA_Defaults# to the first 1000 characters of the SQL query in the referenced SQL VARA. :smile: This works fine if the referenced VARA is a STATIC VARA, so the problem may be limited to SQL VARAs. Moreover, the documentation indicates that variables are allowed in EXEC VARA parameter values. I suppose this is another bug.

I opened INC00128529 for the problem “SEC SQL VARA object references in EXEC VARA parameters not correctly resolved.”
I will post an update here when I have news.

Update: I was able to get the SEC SQLI VARA object reference to resolve to a value by omitting the key and column. E.g.,

{UC4.SCM.V134.TRANSFER.GET_FIRST_PREFIX.VARA_SQL}

This retrieves the first value from the first key, so it obviously will not work in situations where one wants to retrieve a different  key/value.

[Michael A. Lowry]

Comment deleted.

[Michael A. Lowry]

Now that I have had time to do more testing, I can provide an update.

Problem summary: VARA object references in the parameters of an EXEC VARA are not correctly resolved when both the following criteria are met:

• The VARA object reference points to an SEC_SQLI VARA
• The VARA object reference takes the form {VARA,RESULT,1}

Instead of being resolved to the result of the SQL query, the VARA object reference in the parameter is resolved to the first 1023 characters of the SQL query.

Possible work-arounds: The problem does not happen if any of the following criteria is met:

• The VARA object reference points to a  to STATIC VARA
• The VARA object reference takes the form {VARA} (no key or column is specified)
• The VARA object reference takes the form {VARA,,COLUMN} (no key is specified)
• The VARA object reference takes the form {VARA,KEY,COLUMN} (a known key is specified)
  

I have not tested this with other types of SQL VARA object, so I do not know if the problem also affects SQL, SQLI, or SEC_SQL VARA objects.

[Michael A. Lowry]

I realized that the keyword RESULT is meant to refer to a column, and not a key. I then discovered that it is possible to read values from a dynamic VARA object such as an SEC_SQLI VARA specifying the column, but not specifying the key.

• InVARA object references, e.g.,  {VARA,,COLUMN}
• In GET_VAR script commands, e.g.,  GET_VAR(VARA,,COLUMN)

This format — specifying the column, but not the key — is not documented anywhere that I could find.

Using a VARA object reference of the form {VARA,,1} in the EXEC VARA parameter, I was able to work around the above problem.

However, I identified two new problems:

1. A single VARA object reference of form {VARA,,1} in an EXEC VARA parameter works fine. If a second VARA object reference is added to the same parameter though, it is not resolved correctly. Instead, it is resolved to the same value as the first VARA object reference in the parameter.
2. VARA object references in an EXEC VARA parameter that take the form {VARA,,RESULT} do not work at all. The first character of the VARA name is replaced by another character. (In my case, the letter U was replaced by w).
   

I am not sure if the {VARA,,COLUMN} format is fully supported, but it does work, at least in some contexts. I will update the incident and let Automic Support know about my new findings.

[Michael A. Lowry]

Michael Lowry wrote:

I opened INC00128431 for the bug “Workflow hangs in 'Waiting for user input' if default value of a prompt set element contains reference to EXEC VARA.”

This has been confirmed as a bug. PRB00129192 was opened for this problem. The same problem also affects v12. Unlike the Java UI, the AWI does not hang when encountering this situation. However, it also does not correctly display the prompt set, and the workflow remains in the state Waiting for user input.

[Michael A. Lowry]

A new problem ticket has been opened for the three additional problems I identified with variable resolution.
 
PRB00129234 SEC SQL VARA object references in EXEC VARA paramters not correctly resolved

1. {VARA,KEY,COLUMN} references of the type {VARA,RESULT,1} in the parameters of EXEC VARAs are not correctly resolved if they point to an SEC_SQLI VARA. The parameter is replaced with the text of the SQL query instead of the result.
2. A single VARA object reference of form {VARA,,1} in an EXEC VARA parameter works fine. If a second VARA object reference is added to the same parameter though, it is not resolved correctly. Instead, it is resolved to the same value as the first VARA object reference in the parameter.
3. VARA object references in an EXEC VARA parameter that take the form {VARA,,RESULT} do not work at all. The first character of the VARA name is replaced by another character. (In my case, the letter U was replaced by w).

[Michael A. Lowry]

I have an update: the problem described in PRB00129192 does not appear to affect AE v12.0.1. I also hear from Automic Support that they are unable to reproduce the problems with v11.2.4.3.

[Michael A. Lowry]

Now that I have found a solution to the aforementioned problems, I have proceeded to build a quite workable solution. The prompt set defaults are set using an EXEC VARA.
https://us.v-cdn.net/5019921/uploads/editor/fe/7zagx29cn1d7.png" width="515">
This EXEC VARA looks like this:
https://us.v-cdn.net/5019921/uploads/editor/t6/3jg0rdjinxp3.png" width="974">
The EXEC VARA runs a script, and passes four parameters. The script is the same one I described above. It simply loads from a static VARA the values that the user submitted the last time he submitted the prompt set. The settings are stored in values 1-5 of the static VARA, and the key is simply the user ID. If this is the first time the user has used the prompt set, this key will not exist. In this case, the default values in the third EXEC VARA parameter will be stored in the VARA, and used for the prompt set defaults. Two of these initial default values are dynamically generated using SQL VARAs.

When the user has submitted the prompt set, a few lines of scripting save the prompt set values in the static VARA:

:SET &VARA_Name# = UC4.SCM.V134.TRANSFER.LAST_USED_SETTINGS.VARA_STATIC
:PRINT "Saving prompt set values to user defaults VARA &VARA_Name#"
:PUT_VAR &VARA_Name#, &$USER#, &PROJECT#, &BATCHNAME#, &SOURCE_UC4_SYSTEM#, &SOURCE_PROFILE_VARA#, &OP_INST_LOCATION

The only problem now is that the prompt set has grown beyond five values, so a single static VARA is no longer sufficient to store all of the user’s last-used settings. For this reason, I am looking into alternatives, including storing last-used settings in XML.

[Michael A. Lowry]

Michael A. Lowry wrote:

The only problem now is that the prompt set has grown beyond five values, so a single static VARA is no longer sufficient to store all of the user’s last-used settings. For this reason, I am looking into alternatives, including storing last-used settings in XML.

I decided to store the additional settings in a second row in the static VARA. I made a duplicate EXEC VARA that simply passes &USER#_2 as the second parameter instead of &USER#.
https://us.v-cdn.net/5019921/uploads/editor/u2/bx3ofd3lyz6i.png" width="528">
This works like a charm.

[Michael A. Lowry]

Unfortunately, I have run into yet another problem with this approach. Intermittently the prompt set does not appear, due to an error during loading the defaults. There are a couple of errors that I have run into:

1. U00010000 '0001', Object '' was not found. Please enter the correct name.
2. 00020440 Runtime error in object 'UC4.SCM.V134.TRANSFER.LIST_DEPL_VARAS_UNDER_SUBFOLDER.JOBS_SQL(1)', line '00019': Non numeric input '' not allowed.
   

After comparing the logs of many different failures, I have come to the conclusion that the problem is due to the fact that defaults are not loaded sequentially, or based on dependencies between fields in the prompt set. Instead, they are loaded more or less simultaneously. If one prompt set’s default depends on a value from another one that has not yet been loaded, an error occurs.

I will see if I can fix this by hard-coding fallback defaults into the SCRI/JOBS objects called by the EXEC VARA.

[Michael A. Lowry]

I suspect that prompt set defaults are not evaluated in any particular order, and that the defaults may be evaluated in a different order each time a prompt set is activated.

Would someone from Automic please confirm or refute this hypothesis?

[Michael A. Lowry]

I tried to set up a quick test to validate my hypothesis. The defaults are not being evaluated.

What is required to ensure that {VARA,KEY,COLUMN} references in prompt set defaults are evaluated during prompt set activation?

[Michael A. Lowry]

If I change one of these references so that it points to a STATIC VARA object, then all of them are evaluated. In this example, I changed the first default:

Upon activating the prompt set, all 10 defaults were evaluated:

So it seems that evaluation of {VARA,KEY,COLUMN} references in prompt set defaults is not triggered if all of these references point to EXEC VARA objects. This seems like a bug.

[Michael A. Lowry]

There’s another problem. When multiple prompt set defaults in the same prompt set contain {VARA,KEY,COLUMN} references, the executable object executed during evaluation of these defaults is not kept distinct for each one.

Here is how I discovered this problem. To keep track of the order in which the defaults are being evaluated, I created a separate EXEC VARA for each field in the prompt set. Each EXEC VARA pointed in turn to its own unique SCRI object.
https://us.v-cdn.net/5019921/uploads/editor/5n/7dd5d74huho6.png" width="813">
The first default is still pointing to a STATIC VARA object, because of the aforementioned problem.

When I activated this prompt set by running the containing workflow, the last-referenced SCRI object UC0.GET_VAR_OR_SET_DEFAULTS_10.SCRI was executed nine times. None of the other SCRIs was run at all. Interestingly, although the wrong executable object was executed for all but one evaluated default, the EXEC VARA parameters were set correctly, and distinctly for each one.

This seems like a bug too. It is very clear now that no one at Automic anticipated that anyone would try to do something like this.

[Michael A. Lowry]

Oh, one more thing: I inserted a :WAIT statement into the SCRI to get a try to shed light on the order in which the prompt set defaults are evaluated. It looks like they are evaluated bottom-up. In my test, the first one evaluated was the one in the field Variable 10, followed by the one in Variable 9, Variable 8, and so on. The last default evaluated was the one in the Variable 2 field.

[Pete Wirfs]

I concur they are done bottom-up.  In addition, if it finds a "redundant" request, it will only execute it once.  The way I learned this, is we pull Oracle sequence numbers into our promptsets. (SQL: select <seqname>.nextval from dual)   One of our applications pulls 5 unique numbers at a time, so logically you would think you should be able to use the same UC4 variable 5 times.  But when I did so, it returned the same value 5 times.  My understanding is that this is a performance feature.

So now we use 5 unique (redundant) UC4 variables with the same SQL statement.  But the sequence numbers returned are definitely executed bottom-up, because the 5 sequence numbers are provided in descending order, instead of ascending order.

[Michael A. Lowry]

On 8 June 2017, Pete Wirfs wrote:

I concur they are done bottom-up.
…
But the sequence numbers returned are definitely executed bottom-up, because the 5 sequence numbers are provided in descending order, instead of ascending order.

I wonder what on earth the reason could be for evaluating them bottom-up. I suppose this decision was made long before prompt set defaults could contain anything particularly interesting, such as references to objects that actually do something in the system.  B)

I am confident now that the evaluation of PRPT defaults is initiated sequentially, bottom-up. However, experience tells me that without a :WAIT statement in the called executable object, the actual executions may not actually run sequentially. I suspect that some of the errors I have seen during prompt set activation have been caused when these tasks ran simultaneously, or ran in a non-sequential order. These problems seems to happen more frequently in busy systems.

I opened support tickets for the two bugs I found, and also opened a ticket to get clarification on exactly how PRPT defaults are evaluated.

               

Ticket	Description
INC00134379	EXEC VARA references not evaluated in PRPT defaults.
PRB00134504	Executable object run from EXEC VARA in {VARA,KEY,COLUMN} references in PRPT defaults is not kept distinct.
REQ00134427	How are prompt set defaults evaluated?

[Michael A. Lowry]

We probably will not rely on the above approach until some of the bugs have been worked out in the underlying features. However, I thought I would share a couple of the other ideas I have come up with, based on the saving of user-specified prompt set values:

1. A prompt set control that resets all fields
2. A prompt set control that immediately saves currently-specified values

1. Reset all fields

The variable set by this checkbox is not actually used in the parent workflow. However, the checkbox is configured so that any change to it resets most of the other fields in the prompt set.

The PRPT control elements that were reset are indicated by a thin black surrounding box.

2. Save defaults

The combo box is a dynamic control that accesses an EXEC VARA. This VARA runs an SCRI that immediately saves all of the currently-specified values to the static VARA used to save user defaults.

:READ &SOURCE_UC4_SYSTEM#,,
:READ &PROJECT#,,
:READ &BATCHNAME#,,
:READ &OP_INST_LOCATION#,,
:READ &SEND_EMAIL#,,
:READ &CONFIG_UC4_STAGE#,,
:READ &ADMIN_SUBFOLDER#,,
:READ &RULES_VARA#,,
:READ &SOURCE_PROFILE_VARA#,,
:INCLUDE UC4.SCM.V134.TRANSFER.SAVE_LAST_USED_SETTINGS.JOBI
:SET &VARA_Name# = UC4.SCM.V134.TRANSFER.LAST_USED_SETTINGS.VARA_STATIC
:PRINT "Verifying that settings were saved to user defaults VARA &VARA_Name#"
:SET &VARA_SOURCE_UC4_SYSTEM#   = GET_VAR(&VARA_Name#, "&$USER#_1", 1)
:SET &VARA_PROJECT#             = GET_VAR(&VARA_Name#, "&$USER#_1", 2)
:SET &VARA_BATCHNAME#           = GET_VAR(&VARA_Name#, "&$USER#_1", 3)
:SET &VARA_OP_INST_LOCATION#    = GET_VAR(&VARA_Name#, "&$USER#_1", 4)
:SET &VARA_SEND_EMAIL#          = GET_VAR(&VARA_Name#, "&$USER#_1", 5)
:SET &VARA_CONFIG_UC4_STAGE#    = GET_VAR(&VARA_Name#, "&$USER#_2", 1)
:SET &VARA_ADMIN_SUBFOLDER#     = GET_VAR(&VARA_Name#, "&$USER#_2", 2)
:SET &VARA_RULES_VARA#          = GET_VAR(&VARA_Name#, "&$USER#_2", 3)
:SET &VARA_SOURCE_PROFILE_VARA# = GET_VAR(&VARA_Name#, "&$USER#_2", 4)
:SET &PUT_VAR_RC# = 0
:IF &SOURCE_UC4_SYSTEM# <> &VARA_SOURCE_UC4_SYSTEM#
:  SET &PUT_VAR_RC# = 1
:ENDIF
:IF &PROJECT# <> &VARA_PROJECT#
:  SET &PUT_VAR_RC# = 1
:ENDIF
:IF &SOURCE_UC4_SYSTEM# <> &VARA_SOURCE_UC4_SYSTEM#
:  SET &PUT_VAR_RC# = 1
:ENDIF
:IF &BATCHNAME# <> &VARA_BATCHNAME#
:  SET &PUT_VAR_RC# = 1
:ENDIF
:IF &OP_INST_LOCATION# <> &VARA_OP_INST_LOCATION#
:  SET &PUT_VAR_RC# = 1
:ENDIF
:IF &SEND_EMAIL# <> &VARA_SEND_EMAIL#
:  SET &PUT_VAR_RC# = 1
:ENDIF
:IF &ADMIN_SUBFOLDER# <> &VARA_ADMIN_SUBFOLDER#
:  SET &PUT_VAR_RC# = 1
:ENDIF
:IF &RULES_VARA# <> &VARA_RULES_VARA#
:  SET &PUT_VAR_RC# = 1
:ENDIF
:IF &SOURCE_PROFILE_VARA# <> &VARA_SOURCE_PROFILE_VARA#
:  SET &PUT_VAR_RC# = 1
:ENDIF
:IF &PUT_VAR_RC# <> 0
:  PRINT "Not all settings were saved successfully to user defaults VARA &VARA_Name#"
:ENDIF
:SET &ProcHnd# = CREATE_PROCESS(NEW)
! Write column headings
:PRINT "Writing column headings to data sequence."
:SET &RC# = PUT_PROCESS_LINE(&ProcHnd#,"Result",";")
:IF &PUT_VAR_RC# = 0
:  SET &RC# = PUT_PROCESS_LINE(&ProcHnd#,"Defaults saved successfully.",";")
:ELSE
:  SET &RC# = PUT_PROCESS_LINE(&ProcHnd#,"Defaults NOT saved successfully.",";")
:ENDIF
! Send result back to calling EXEC VARA
:PSET &Result# = SAVE_PROCESS(&ProcHnd#)

The result of saving the values to the static VARA is returned in the combo box pop-up menu.

This allows the user to save the currently-specified values for next time, even if he plans to cancel the prompt set.

These are admittedly kludges. If prompt sets offered a simple Button control, these capabilities could be implemented in a more consistent and intuitive way.

[Michael A. Lowry]

I submitted a couple of enhancement requests related to my recent experiments:

1. Buttons in Prompt Sets.

With the advent of EXEC VARA objects, it is now possible to execute an object in the Automation Engine system just by accessing a VARA object.

This  means that it is possible to define a dynamic prompt set element (such  as a combo box) whose data source is an EXEC VARA object, such that  clicking on the combo box executes a script or other executable object.  (See this discussion thread for an example.)

It  should be possible to add ordinary buttons to prompt sets, to make the  best use of this capability and to make possible more interactive and  dynamic prompt sets.
https://us.v-cdn.net/5019921/uploads/editor/26/p7aohhypm9i6.png" width="1073">
It should be possible to use the result from the VARA in another element, e.g. a Header/Text element.

2.Allow configuration of how defaults in Prompt Sets are evaluated

Currently (in v12.0.2), default values in prompt sets are evaluated in the following way:

• Evaluation of defaults isinitiated bottom-up, from the bottom-most element to the top-most element;
• Evaluation of defaults may not always beexecutedsequentially. This means that sometimes:
  • Defaults may be evaluated in a non-sequential order;
  • Two or more defaults may be evaluated simultaneously.

It should be possible to configure both of these things for each prompt set:

1. Whether to evaluate prompt set defaultstop-down orbottom-up.
2. Whether to evaluate prompt set defaultssequentially orsimultaneously.
   • Sequentially: Evaluation of one default must be completed before evaluation of the next default will be initiated.
   • Simultaneously: Evaluation of all defaults is initiated at the same time.

When these ideas have beenaccepted for voting, please consider voting for them.

[Michael A. Lowry]

Pete Wirfs wrote:

I concur they are done bottom-up.  In addition, if it finds a "redundant" request, it will only execute it once.

While I’m waiting for a response to REQ00134427, I ran some additional tests to see how the evaluation of prompt set defaults works. It is not strictly bottom-up.

As petwir discovered, there is some intentional optimization to reduce redundancy. I made a prompt set whose defaults are based on the values of other prompt set variables.

Note that variables 2 through 5 have defaults based on the previous (lower) variable, and variables 6 through 9 have defaults based on the next (higher) variable. Regardless of whether the dependent variables are above or below in the prompt set, they were resolved correctly.
https://us.v-cdn.net/5019921/uploads/editor/ed/uzp9y1d7907p.png" width="484">
My more complicated use case involves one more level of abstraction: prompt set variables are used in parameters of EXEC VARAs that are in turn used in defaults of other prompt set fields.

But at least in this very simple case, wherein prompt set variables are used directly in the defaults of other fields, the Automation Engine appears to be clever enough to determine the order in which the defaults should be resolved. It is definitely not just a simple bottom-up process.

[Michael A. Lowry]

When testing various combinations of PRPT defaults, I accidentally removed the static VARA object that is the ultimate source of the values retrieved by the EXEC VARA–SCRI combination used in these PRPT defaults. When I did this, the next time I tried to start the workflow, it just hung indefinitely in the state Resolving variables. No errors appeared in the Messages window.
https://us.v-cdn.net/5019921/uploads/editor/f7/ll1kcxu9wvhs.png" width="490">
Initially I thought there might be some sort of race condition when resolving the defaults, so I inserted a :WAIT statement into the EXEC VARA used to load these defaults, and re-ran the workflow; this did not help. Eventually, I found the root cause by looking at the WP trace.

WPsrv_EXP2_trc_007_10.txt:20170620/115341.220 - U00020331 Runtime error in object 'UC0.LOAD_SETTINGS.JOBI', line '00017': U00003705 Variable 'UC0.SETTINGS.VARA_STATIC' not found.


My intuition is that this error condition should be handled differently:

• The workflow should end with statusFAULT_OTHERinstead of hanging in statusResolving variables;and
  
• The error message should be displayed in theMessageswindow so that the submitting user can easily see why the workflow did not start.

I opened INC00134839 for this.

[Michael A. Lowry]

I started a separate thread to keep track of the various support tickets related to prompt sets and EXEC VARA objects.

[Michael A. Lowry]

I made a couple of objects to help me quickly test VARA object references.
https://us.v-cdn.net/5019921/uploads/editor/lt/hblycvtlsu3p.png" width="525">

If the specified key is not found, or if the specified value column of a key is empty, <key not found> or <empty value> will be displayed, respectively. I have attached the EXEC VARA, the SCRI, and a static VARA to use for testing. Enjoy.

Attachment(s)

GET_VAR uc4_export.zip   2 KB 1 version

[Michael A. Lowry]

Here's another quirk of the Java UI. It allows inserting variables (object/script variables or references to VARA objects) in the Executable object field of EXEC VARA objects, even though variable resolution is not performed on this field.

6/21/2017 7:47:50 AM -  U00010000 '0001', Object '&EXEC_NAME#' was not found. Please enter the correct name.

6/21/2017 7:50:25 AM -  U00010000 '0001', Object '{UC0.TEST2.VARA_STATIC,ABC,1}' was not found. Please enter the correct name.

Note that the Variables button is enabled when the Executable object field has focus, and that the variable names are color-coded. These two things usually indicate that the field supports variable resolution.

The Automic Web Interface lacks color coding of variables in most fields, but it has a $x button in many fields that support variable resolution. This button is not present in the Executable object field of EXEC VARA objects in the AWI. In its place is a little gray folder icon.

Clicking this button opens an object picker dialog box.


This object picker displays only executable objects. This makes more sense.

Variable resolution is not performed on the contents of the Executable object field. This means that if you want to be able to choose dynamically which executable object to run, you must create a separate EXEC VARA for each executable object, and then choose dynamically the name of the EXEC VARA to access.

[Michael A. Lowry]

Michael A. Lowry said:

I made a couple of objects to help me quickly test VARA object references.
https://us.v-cdn.net/5019921/uploads/editor/lt/hblycvtlsu3p.png" width="525">

If the specified key is not found, or if the specified value column of a key is empty, <key not found> or <empty value> will be displayed, respectively. I have attached the EXEC VARA, the SCRI, and a static VARA to use for testing. Enjoy.

I’m working on a more flexible version of the above EXEC VARA & SCRI. The new version will be able to read any specific value, all values from a row, all values from a column, or all values matching filters. Coming soon!

[Michael A. Lowry]

Michael A. Lowry wrote:

Pete Wirfs wrote:

I concur they are done bottom-up.  In addition, if it finds a "redundant" request, it will only execute it once.

...at least in this very simple case, wherein prompt set variables are used directly in the defaults of other fields, the Automation Engine appears to be clever enough to determine the order in which the defaults should be resolved. It is definitely not just a simple bottom-up process.

After some initial discussion, REQ00134427 was converted to INC00134427. In the context of this incident, I spoke with an Automic developer about how the Automation Engine evaluates prompt set defaults. The developer explained that the AE evaluates defaults top-down, and performs multiple passes, as many times as necessary, to ensure successful evaluation of all variables in PRPT defaults.

The results of my testing suggest that this is not a complete description of how it actually works.

• When PRPT defaults contain references to EXEC VARAs, it is possible to see exactly how many times these VARAs are evaluated, by looking at the statistics of the called executable object. These tasks also appear as child tasks of the executable object on which the PRPT is defined. When I defined these defaults in such a way that each one pointed to a different EXEC VARA, and these EXEC VARAs all had unique parameters, I never saw a case in which the called SCRI was executed more than once with the same parameters.
  
• Even so, the PRPT defaults wereusually but not alwaysevaluated in such a way that independent variables were evaluated first, and dependent ones later.
  
• I observed several cases where the data that were retrieved during evaluation of one PRPT default were mistakenly inserted into a different PRPT default (or even into multiple ones). I suspect this could be a manifestation of the bug described inPRB00134504. (See alsothis discussion.)
  
• I observed several instances wherein the data source VARAs were accessedbeforeall defaults had been evaluated. I am not sure why the AE would need to access the data sources before evaluating all variables in defaults; perhaps it wants to ensure that each default value is a valid value (one of the values returned by the data source). This problem happens only intermittently.

In the prompt set I am working with, although each default is specified by its on EXEC VARA, they all call the same SCRI, just with different parameters. Because I know the order in which these defaults must be evaluated, and because they all call the same SCRI, I am beginning to think that the best way forward is to hard-code the order of evaluation into the SCRI. In other words, the SCRI will test whether certain variables have values yet, and if they do not, it will look up and set their values in the correct order. This way, regardless of which PRPT default is evaluated first, before the called SCRI actually does any of the work of evaluating that particular element’s default value, it will look up and set the default values of all of the dependent variables. Effectively this is a way of forcing the order of evaluation. I’ll post an update later with the results.

I have also seen another problem wherein VARA object references of the form {VARA,KEY,COLUMN} in EXEC VARA parameters are not resolved correctly. Only two of my nine EXEC VARAs are set up in such a way. I’m not sure how to work around this problem. The referenced VARA in these cases are SQL VARAs. Perhaps I can more easily track whether & how they are being accessed by using EXEC VARAs that run SQL JOBS objects instead.

[Michael A. Lowry]

Michael A. Lowry wrote:

I have also seen another problem wherein VARA object references of the form {VARA,KEY,COLUMN} in EXEC VARA parameters are not resolved correctly. Only two of my nine EXEC VARAs are set up in such a way. I’m not sure how to work around this problem. The referenced VARA in these cases are SQL VARAs. Perhaps I can more easily track whether & how they are being accessed by using EXEC VARAs that run SQL JOBS objects instead.

Unfortunately, this did not help. I discovered that {VARA,KEY,COLUMN} references to EXEC VARA objects are not correctly resolved when they are used in the parameters of other EXEC VARA objects. The {VARA,KEY,COLUMN} reference resolves to the label of the data sequence instead of the contents of the data sequence. E.g.,

RESULT	Value 1
001_PROCESS_001	001_PROCESS_001

The EXEC VARA that fails to resolve correctly in the parameter works fine when used elsewhere. This problem might be related to PRB00135286, but it seems like a distinct problem. I opened INC00136560 for this. (See this discussion.)

Update: I identified this problem in v11.2.1HF1. AE v11.2.5 and v12.0.2 do not appear to be affected.

[Michael A. Lowry]

Michael A. Lowry wrote:

Because I know the order in which these defaults must be evaluated, and because they all call the same SCRI, I am beginning to think that the best way forward is to hard-code the order of evaluation into the SCRI. In other words, the SCRI will test whether certain variables have values yet, and if they do not, it will look up and set their values in the correct order. This way, regardless of which PRPT default is evaluated first, before the called SCRI actually does any of the work of evaluating that particular element’s default value, it will look up and set the default values of all of the dependent variables. Effectively this is a way of forcing the order of evaluation. I’ll post an update later with the results.

This turned out to be a bit trickier than expected. To force the order of evaluation of defaults, the SCRI called by the EXEC VARAs is actually accessing a few of those EXEC VARAs again. This approach risks starting a recursive loop.

:IF &$ACTIVATOR_TYPE# = "JOBP"
:  SET &DEP_VARS_RESOLVED# = "&DEP_VARS_RESOLVED#"
:  IF &DEP_VARS_RESOLVED# <> "YES"
:    PSET &DEP_VARS_RESOLVED# = "YES"
:    PRINT "Resolving variables with dependent variables."
:    INCLUDE UC4.SCM.V140.TRANSFER.RESOLVE_PREFIX.JOBI
:    INCLUDE UC4.SCM.V140.TRANSFER.RESOLVE_SOURCE_SYSTEM.JOBI
:    INCLUDE UC4.SCM.V140.TRANSFER.RESOLVE_CONFIG_SYSTEM.JOBI
:    INCLUDE UC4.SCM.V140.TRANSFER.RESOLVE_ADMIN_FOLDER.JOBI
:  ENDIF
:ENDIF

I added two checks to prevent unlimited recursion, and to eliminate unnecessary/redundant executions of the SCRI:

• During evaluation of PRPT defaults, the activator type of the SCRI task will be JOBP, because the PRPT is defined on a workflow. If the SCRI is running as a child process of an SCRI object (or anything other than a JOBP), the :INCLUDE statements will be skipped. As a result, recursion can go only one level deep.
• The :INCLUDES are executed only if the&DEP_VARS_RESOLVED#variable isnot set toYES. When the :INCLUDES that resolve the variables are executed once, the variableis set toYES.
  

The first check works. The second doesn’t. Even if the &DEP_VARS_RESOLVED# variable is set to YES after the SCRI runs the first time, the next instance of the SCRI does not see the change, and so it runs the :INCLUDES again needlessly. I can think of two possible explanations:

• The SCRI tasks run to evaluate the prompt set defaults are run simultaneously, not sequentially.
• The SCRI tasks run to evaluate the prompt set defaults are run in independent execution environments, such that changes made to object variables in one task are not visible to other tasks.
  

I’m a bit flummoxed by this one.

[Michael A. Lowry]

Michael A. Lowry wrote:

Even if the &DEP_VARS_RESOLVED# variable is set to YES after the SCRI runs the first time, the next instance of the SCRI does not see the change, and so it runs the :INCLUDES again needlessly. I can think of two possible explanations:

• The SCRI tasks run to evaluate the prompt set defaults are run simultaneously, not sequentially.
• The SCRI tasks run to evaluate the prompt set defaults arerun in independent execution environments, such that changes made to object variables in one task are not visible to other tasks.
  

I’m a bit flummoxed by this one.

I would like to get a more complete understanding of the execution environment of these tasks. Perhaps someone from Automic can answer the questions below. When an executable object is called during evaluation of a {VARA,KEY,COLUMN} reference to an EXEC VARA in a prompt set default:

1. What object variables does the task inherit?
2. If object variables are set in this task, at what level are these changes applied?
3. Is there any way to set a variable in this task, so that this variable value is accessible by other similar tasks (other tasks started during evaluation of PRPT defaults)?
4. Is it necessary to use a uniqueresult variable for each EXEC VARA, to prevent them from colliding?

Thanks in advance.

[Michael A. Lowry]

I added the above four questions to INC00134427. I hope to be able to report back soon with the answers.

[Michael A. Lowry]

Michael A. Lowry wrote:

...The SCRI tasks run to evaluate the prompt set defaults are run simultaneously, not sequentially.

I had a thought: I could use a SYNC object to force these tasks to run one at a time. I’ll try this when I find time.

[Michael A. Lowry]

I added a SYNC dependency to the SCRI I’m using to resolve EXEC VARA references in PRPT defaults. I observed the following about the tasks started in the course of this variable resolution:

• These tasks are executed sequentially, although the order is a bit unpredictable.
  
• These tasks do not inherit object variables from the parent workflow.
• These tasks do not inherit values from the prompt set.
  
• Changes made to object/script/PRPT variables in these tasks arenot seen by other such tasks. Each of these tasks appears to start with its own independent and pristine execution environment.
  
• Attaching a SYNC dependency to the called executable object appears to reduce the likelihood that default values from one PRPT element will be mistakenly inserted into other elements. This bug might be the same as the one described inPRB00134504. (Seethis discussionfor a list of related bugs.)
• The first three times I executed the workflow after adding the SYNC object, the workflow executed acompletely unrelated objectthat is not mentioned anywhere in the workflow or its child objects.  :#  This problem has not returned since then. (I opened an incident for this.)
  

[Michael A. Lowry]

I think I figured out a bit more about what’s going on. Executable objects started by EXEC VARAs do not inherit any variables from the calling task.

In this simple example, the most straightforward way to pass information from the SCRI1 to SCRI2 is via parameters in the EXEC VARA. However, there is no way to set any object variables in SCRI1 from SCRI2. See the attached objects for an example.

The best way I can think of to pass information from SCRI2 back to SCRI1 is:

• During execution, SCRI2 writes the information to a separate static VARA.
• When it has finished processing the EXEC VARA, SCRI1 reads the information from the static VARA.

I think what I’ll do is change SCRI2 to write a temporary static VARA whose name is based on the run ID of SCRI1 (passed as an additional parameter). Then, as soon as SCRI1 has the information it needs, it will remove the temporary static VARA.

Attachment(s)

EXEC_VARA_TESTING uc4_export.xml   6 KB 1 version

[Michael A. Lowry]

I updated my workflow based on this idea.
https://us.v-cdn.net/5019921/uploads/editor/l6/1nod3be6hjev.png" width="751">
Each of the default values in the prompt set is set using an EXEC VARA. All of these EXEC VARAs call the same SCRI, only with different parameters.
https://us.v-cdn.net/5019921/uploads/editor/9y/7kzb224q5dei.png" width="751">
The first time the EXEC VARA is evaluated, the child SCRI looks for the temporary static VARA. If it does not find the temporary static VARA, the SCRI creates it, performs some additional lookups using another set of EXEC VARAs, and writes these details to the temporary static VARA.
The next time the main EXEC VARA runs, it will find the temporary static VARA, and will load some values directly from there. Subsequent runs of the SCRI load the remaining values from EXEC VARAs, using as parameters details read from the temporary static VARA.

The whole point of this is to ensure that the values are loaded in a particular order.

Unfortunately, it doesn’t work. The workflow hangs in Waiting for user input without starting any child tasks; the workflow task cannot be cancelled or even manually set to complete; and an unhandled error appears in the Quarantine.

                 

Message no.	Status symbol	Reason for failure	XMLStream
3002	Unprocessed	Signal 11 (SEGV) at (nil) with   code 128	RSLVQ,Client=100,UserIdNr=1234567

The user ID is my user, and I presume RSLVQ is related to variable resolution.

[Michael A. Lowry]

Alright, folks: do not try what I described above.  :#  Two WPs died with segmentation faults, and shortly afterward 11 WPs hung completely. Fixing the problem required stopping the AE server, killing (-9) the hung WPs and performing surgery on the database to delete the corrupted tasks. We are back up and running now.

Thanks to Andrew_Garland_7890 & Michiel_Verhoeven_4432 for their quick, professional, & friendly  support. I will report back here if and when the problem is confirmed as a bug.

In future, I’ll do my bleeding edge AE experimentation in one of our experimental systems.  o:)

[Michael A. Lowry]

Today I reproduced the problem in one of our experimental systems, using the same workflow. I’m pretty confident now that this is a bug.

Update 2017.09.18: I opened INC00137672 for this. I also updated the list of problems with prompt sets and EXEC VARA objects.