Cross package input component validation

[Mathew Goldstein]

A weaknesses of the input component validation during package cast is that the scope of the input component validation is limited to the current package. This is a problem because input components intrinsically tend to be common to many programs and therefore the same input components are likely be utilized by program elements across multiple different packages.

How do we deal with this issue?

One possibility is to create two separate common packages for the assembler and compile time input components and the subprogram input components that are scheduled to be deployed on a given date. Then create a third set of packages containing the other program elements scheduled to be deployed on that date.

So, for example, one package (IC1) could contain only the assembler and compile time input components scheduled to be deployed on a given date, another package could contain only the called subprograms scheduled for that same date (SP2), and a third set of multiple packages could contain all other program elements scheduled for that same date. IC1 would be cast with input component validation and then executed first, followed by SP2, followed by all of the other packages.

Does anyone do that? Does it work? It seems to me that in a large organization with many developers working on many different projects it could be difficult to coordinate the common packages for the common input components.

Another approach to deal with single package scope limitation of cast time input component validation could be to repeat the input component validation a second time just prior to package execution. The reason to do this is that other packages may have already moved the common input components to the next stage. So while the package failed input component validation when it was cast, it is nevertheless now safe to proceed with executing that package since another package remedied the within package scoped missing input component failure. However, there is a problem of circular input validation dependencies between multiple packages. Package P1 depends on an input component in Package P2 which in turns depends on an input component in Package P1 so neither package can pass input component validation at execution time. Such dependencies can exist between any two or more packages in many different possible combinations.

So this presents an issue of detecting, and eliminating, circular input component dependencies across multiple packages. ACM currently does not track input component dependencies across the cast, but not yet executed, packages.

Some input component ideas:
https://community.broadcom.com/idea/input-component-validation-mandate-first-cast-only
https://community.broadcom.com/idea/block-package-execution-if-input-component-omitted
https://community.broadcom.com/idea/cast-and-execute-merged-packages
https://community.broadcom.com/idea/package-input-component-relationship-report
https://community.broadcom.com/idea/repeat-input-component-validation

------------------------------
[FirstName]
------------------------------

[Mathew Goldstein]

The package cast input component validation has two limitations: It knows the future status only of those input components that are within the current package and it knows only the current status of the other input components as they are in ENDEVOR at the time the cast is executed. It knows nothing about the contents or status of any of the other packages and how those packages will change the input component validation context when they are executed.

This is where co-/pre-requisite package information comes to the rescue. The information needed to manually complete the input component validation is in those packages. This is why when the input component validation fails SCM staff needs to know the co-/pre-requisite packages to obtain the missing information about those within package, point in time, “missing” input components, so that they can then make a reasonably accurate decision/prediction regarding whether or not everything is good to go or not.

So I think we can manage the input component validation issue by requiring each package to contain a memorandum identifying the co/pre-requisite packages and then implementing the following procedure whenever a production package fails input component validation due to one or more missing input components:

• For each program element with a commented move action added to the production package SCL by ENDEVOR during the failed package cast:
  • For each co-/pre-requisite package identified in the memorandum the SCM staff will:
    • Locate the program element in one of the co-/pre-requisite packages.
    • If the program element is not found in any of the co-/pre-requisite packages (or there are no co-/pre-requisite packages identified in the memorandum) then execution of the package will remain blocked. The developer(s) will be notified that the package was abandoned because there is at least one missing input component that is not being moved to the next stage by any of the co-/pre-requisite packages identified the memorandum.
  • If all the input component program elements were in one of the co-/pre-requisite packages, then the co-/pre-requisite packages should be cast and executed prior to casting the package again with input component validation. If two or more of the related packages form a mutual dependency circle such that casting them all with input component validation is not possible then the packages will be cast with warnings and executed in any sequence. [Note: If the co-/pre-requisite packages are scheduled to be executed after the package with those co-/pre-requisites then the developers for those packages should be contacted and asked to update their memorandum to re-schedule the execution of the packages so that the co-/pre-requisites are executed on the same day, or before, the package with those co-/pre-requisites.]
  • Otherwise, the developer(s) will be responsible for remedying their production package’s missing input component problem and then notifying the SCM staff to attempt to cast the production package again. The remedy will be context sensitivity, following are some possibilities:
    • If one or more co-/pre-requisite packages are missing from the memorandum, then add them to the memorandum and create a package increment to move the memorandum.
    • To bypass unwanted updates to input components in the current migration path switch to another migration path (start again in entry stage ID U of that other migration path). Create a package increment to move the program elements.
    • If multiple updates were made to the same input component in a quality assurance testing stage and your program is using an earlier update, then either regenerate your program element and test it again with the newest level of the input component and create a package increment to move the program element. or arrange for that input component to be moved with history (OPTIONS HISTORY) to preserve both levels of the input component when it is moved to production.
    • If different and incompatible updates are needed for the same input component, create a second copy of that input component with a new name and use that second copy. Create a package increment to move the new and revised program elements

It would still be helpful if ENDEVOR had some additional functionality to better enable us to escape the single package and point in time limitations of the package cast input component validation. How about an ability to execute the input component validation again after the package is in-approval or approved status (after it was successfully cast with warnings)? How about an input component validation report on the entire/merged content of multiple packages?

------------------------------
[FirstName]
------------------------------

[Mathew Goldstein]

A REXX exec can export a package's SCL and then read that SCL to obtain the element information. The REXX exec can then create and submit a job that executes VALIDATE actions on those elements to report on potential input component issues. 

------------------------------
[FirstName]
------------------------------

[Mathew Goldstein]

Maybe the existing package inspect action can include a new option to also optionally execute element validate actions? And maybe package inspect actions can be launched alongside package execute actions from panel CISP5000 or CISP5001 ?

https://community.broadcom.com/idea/validate-element-actions-during-inspect-package-action

------------------------------
[FirstName]
------------------------------

[Dave Harding]

Hi Mathew,  

You asked "Does anyone do that? Does it work? It seems to me that in a large organization with many developers working on many different projects it could be difficult to coordinate the common packages for the common input components."

My last company could not do this as we had Internal Audit imposed lead-time requirements for package cast; all 'release-related' packages were cast at least 4-days in advance of the execution window. Even if that lead-time limitation wasn't imposed, I don't think we would have been successfully getting developers to coordinate with common components in a first wave set of packages– I just wouldn't want to base a procedure off something like that. If you have a Release Mgmt Team that might work but certainly some type of oversight would be necessary to make this successful.

 

I assume you've put this question to the community because (probably more than once) someone turned off package validation and that was the root cause of critical input components getting left behind. I'm not sure this will work at your company but an alternate approach might be to use (1) package inspect – that will catch the sync, security, & signout issues and (2) a post-install reporting process to catch things left behind. No, I don't have a utility to do this but it seems possible to build a list of all input components from recent element changes. Compare that list against what still sitting in your production staging area (i.e. wherever your code was last assembled / compiled). Such a report could be acted against immediately with an emergency change request.

[Mathew Goldstein]

We have a similar procedural complication where many packages are sometimes being cast by a fixed deadline which can be long before some of those packages are executed. Also, promotion packages reset and cast automatically for the move to the next stage.

There was a recent incident which prompted a focus on this input component complication, but that particular incident was not as you described. The root causes were a series of developer mistakes. The developer was making a current production fix to a load module that was already being updated for future production, Instead of selecting one of the alternate parallel development migration paths the developer selected the same migration path where the update to that same load module was already being worked for a future deployment. When the production package with the fix for current production was cast it failed input component validation (missing the input component for the future deployment). The developer apparently decided to override that failure (the package was then cast by SCM staff with warnings), The developer also did not test the load module for any impacts of what was supposed to be the future deployment update. Those three mistakes resulted in a production failure that the developer then blamed on ENDEVOR for not generating/binding again into production (of course we should not be slamming production with untested changes). When I vigorously disputed his request that ENDEVOR be reconfigured to generate the load module (without the unwanted and missing input component) into production the developer complained that ENDEVOR should have blocked the package with a missing input component from being executed. which is what ENDEVOR actually did initially by failing the cast. It thusly occurred to me that our procedure of asking the developers if their package with missing input components should be cast with warnings is deficient. Neither the developers nor the SCM staff who cast the packages are skilled enough to make such evaluations accurately on their own. We need a better, more effective/foolproof, way of evaluating when packages with missing input components should NOT be merely cast again with warnings and then executed.

------------------------------
[FirstName]
------------------------------

[Joseph Walther]

Very good point, Matthew, and I could not agree with you more when you said " We need a better, more effective/foolproof, way of evaluating when packages with missing input components should NOT be merely cast again with warnings and then executed.".

One method for improving the quality of production installations might be the automation and evaluation of application testing. 
See if one of these sites might be applicable for you: 

https://medium.com/modern-mainframe/how-to-leverage-endevor-processors-to-test-batch-applications-6247a9dfdafa
https://medium.com/modern-mainframe/mainframe-test-scripting-made-easy-using-test4z-31cfbe71caca
https://medium.com/modern-mainframe/how-test4z-makes-batch-testing-easier-fc445cd29176

[Joseph Walther]

Yes a REXX can do this for packages that are CAST, by just reading the Package action list. There is no need to parse the SCL. 
Still. It is an interesting idea you have here.

How are your packages named?  Would it be effective at your site if the package validation was performed on packages with a common prefix - rather than just one package?  For example, if the packages were prefixed with CCID123* and the validation was performed across all CAST packages with that prefix, would you see that as a helpful approach?

[Mathew Goldstein]

I wrote a REXX that reads the SCL section of the package detail report, outputting SET or VALIDATE actions for each SET or MOVE action respectively, while removing any OPTIONS. The REXX exec will be combined with another REXX, a panel, and a JCL skeleton. Individual package ID must be specified (the panel can prompt for more than one) and that restriction, while not a fatal flaw, is definitely inconvenient for us. Your question is a bullseye. We currently execute INSPECT PACKAGE using wildcarded package ID. If we could also execute the VALID ELEMENT for those same wildcarded package ID then we would.

------------------------------
[FirstName]
------------------------------

[Joseph Walther]

What if (perhaps as an interim solution):

• Have your REXX output the same element action that it finds in the package (not converting to VALIDATE statements)
• Run the REXX for the list of packages, eliminating duplicate element actions, and accumulating the output results 
• Place the accumulated element action results into a new (temporary) package, and INSPECT (or CAST) it. Then, you will get the collective package validation you need.
• You may then elect to keep or Delete the temporary package

Surely there must be other sites that would benefit from such a procedure.

[Mathew Goldstein]

Starting with a wildcarded package ID, how is a list of packages that were cast with warnings obtained? Will this require a CSV report? Once we have that starting list of packages I am confident that I can code everything else.

------------------------------
[FirstName]
------------------------------

[Joseph Walther]

It seems you would want all packages CAST with the common prefix - regardless of whether Cast warnings were given. (A package that had warnings might be dependent on a package that had no warnings). The CSV syntax to LIST PACKAGE ID could be the one to use.  Something like this perhaps:
    LIST PACKAGE ID 'PKPRFX*'  .... WHERE STATUS = (INEDIT,INA,APPROVED,DENIED)
                                                                        CAST FROM DATE 01/01/99 TIME 00:00. 
I can provide JCL detail if needed.

On the other hand, if you think that only packages with CAST warnings are the ones you want, then the CONRPT72 - the "Package Detail" report from the USER menu, can provide the detail. It shows the CAST Report for each selected package but might need another REXX to extract package names that are showing warnings. CONRPT72 also allows you to report on a wildcarded package ID.

[Eoin OCleirigh]

Hi Mathew/Dan,

Yes I think starting with a CSV (LIST PACKGE ...) and then filtering as appropriate. Just looking for records with a STATUS of 'IN-EDIT' and a 'CAST RCD EXIST' = Y (ALPKG_RS_FLG_CARE) should find you all packages that were CAST but are still in IN-EDIT status (likely because they failed the cast) would give you a good place to start.  You could then pull the cast reports for those packages to find the element/reasons for the failure - or just process the actions to see what was going on.

But Dan's earlier suggestion building a "super package" by copying a selection of "sub-packages" (matching a CCID perhaps, or what ever makes sense from your site standards) and just executing a "dummy cast" to validate all the components, of all the sub-packages, against each other (not even sure you need to de-duplicate the elements, thought that is a nice optimization).  But then by validating at the "super package level" you could validate all the components across the "component packages" - so that even if those, sub-packages weren't valid on their own - they were OK when merged - and so by proxy so long as ALL the sub-packages validate OK under the "super Package" - they must be OK and could be cast with ignore/warning on components.  Then the Super Package could be deleted again - remember it is ONLY used to validate ALL the components, and probably should not be executed (although the old reasons of maximum package size are probably no longer relevant) .

Again it'd be nice if Rexx structures were provided that mapped the CSV layouts - I use something like this to parse the bits I needed for a recent project...

/* we have a record... parse & Check it's the type and version we expect */
parse var PKH.n , /* Extract ALPKG_RS_DATA */
07 ALPKG_VER 09, /* Version number - expect x'0003' */
09 ALPKG_EYE 13, /* Eye Catcher - Expect 'LPKS' */
13 ALPKG_RS_PKGID 29, /* Package ID */
30 ALPKG_RS_COMMENT 80, /* Description */
80 ALPKG_RS_PKG_TYPE 90, /* Package Type */
116 ALPKG_RS_STAT 128, /* Status */
138 ALPKG_RS_EXRC 142, /* Package Exec RC */
207 ALPKG_RS_CAU 215, /* Cast UserId */
227 ALPKG_RS_EXBD 234, /* Exec Begin Date */
234 ALPKG_RS_EXBT 239, /* " " Time */
239 ALPKG_RS_EXED 246, /* Exec End Date */
246 ALPKG_RS_EXET 251 /* " " Time */

ALPKG_RS_CAU = strip(ALPKG_RS_CAU) /* remove trailing spaces */

if ALPKG_EYE \== 'LPKS' then
Say "RXLISTPK - Expecting a Package Header eye-catcher 'LPKS'",
"Found '" || ALPKG_EYE || "'"

if ALPKG_VER \== '0003'x then
Say "RXLISTPK - Expecting a API version '0003'x",
"Found '" || c2x(ALPKG_VER) || "'x"

...

/* We have a Package header record... continue processing... */  

[Mathew Goldstein]

The more I think about this, the more what you are saying above about creating and casting a consolidated package makes sense. If, each time a package cast fails, we created a consolidated package and cast it to see if it has any of the same missing input components reported in the failed package. If not then that is good evidence it is safe to re-cast the failed package, this time with warnings. However,, the consolidated package cast will fail because the program elements are already assigned to other cast packages. Will those cast failures prevent the package cast from proceeding to the input component validation?

VALIDATE ELEMENT is mistake, that will not help. So if the consolidated package will fail without completing the input component validation then we have no solution for this problem. If that is the case then the only remaining remedy that I confident would work would be to allow cast packages that were previously cast with warnings to be cast again.