Laura:
Lucky you! I think that you have thought of the same issues that we have faced and are on the right track with your approach.
While this isn't quite a Magnum Opus it may be a bit more than you want or need, but once I got started, I just couldn't stop! Let me know if you need further detail explanation or additional rationale for our decisions.
First a bit of history - Initial use of AE, né UC4, was not well controlled and most any application type development group that wanted to automate one or more of their processes was able to do so as the product was available to most anyone that asked. It was considered a utility, there for the asking with virtually no standards or policies for usage.
It was soon realized that this approach was not very effective for company wide automation. This became very evident when scheduling of tasks quickly started to cross system boundaries and it was difficult to establish proper predecessor and successor dependencies across these boundaries. The good news was this was discovered well before there were a significant number of objects created and executing in production. The bad news is that there are still some of these objects actively in use.
This resulted in the creation of an enterprise automation group that centralized the development of objects for all application systems and platforms. Users submit requirements and the automation group will create or modify objects as appropriate on their behalf. The requesters are not required to have much if any AE specific knowledge and only need to be able to communicate their application's requirements effectively. The automation group makes it so that application developers do not also need to be AE savvy. Obviously over time these users become more AE aware and conversant which is sometimes not a particularly good thing since they want to tell us what to do and how to do it.
Anyway, enough about the past. Here are, in relatively broad strokes, the standards and methods that we enforce and use to accomplish the company's goals. As with many implementations we think that we got the basics right and continue to evolve as we find smarter ways to do things. And, as usual, there are exceptions but we work very hard to minimize those. Many of our standards exist so as to eliminate the need to have our group involved as the application systems evolve and only participate when AE objects must be modified such as adding new tasks or dependencies.
Our overall goal is provide cross platform automation of batch type processes for the enterprise. We are a centralized IT organization and our systems and users are in a single time zone. The current operating platforms are mainframe, Windows, SAP, Unix and its variants. We currently have about 135,000 objects in our Production Client. The automation group currently consists of 3 employees that spend perhaps 50-60% of their time on AE type work. We are soon to add a fourth to replace some part-time contractors and there is a push to automate many more processes so our workload is increasing.
In addition we have a philosophy that production processes must not be directly dependant on AE facilities to successfully execute. The assumption is that if, for whatever reason, AE is unavailable these processes can still be executed manually though obviously with probably much difficulty. Our basic use of AE is as a "traditional" task scheduler and thus far this approach has served us well.
In no particular order:
1. No application JCL (shell, commands, etc.) in a Job object beyond a single line to execute a "framework" command. All application systems must develop this single command, usually a Unix or Windows shell that accepts as a parameter the object's name. That command is responsible for doing whatever is necessary to execute the job's JCL by establishing paths to directories and such to locate the needed resources. We will provide assistance if needed to develop this framework command, though we are not responsible for it. This framework command must operate the same whether called from the command line in any environment manually and when used by an Agent via AE. This ensures that whatever is executed in a non-AE development environment will operate the same when automated in production thus reducing application development effort and increasing accuracy.
The decoupling of the application JCL from Job objects keeps the automation group focused on AE type development and no involvement with initial and subsequent modifications that may occur to a job's JCL.
2. Additional parameters may be passed to the framework, however they are limited to things that AE inherently knows such as: Client number, process flow name, dates, times, restart status, etc. We also, for example, can pass a file name used in a previous File Transfer task since many of them contain a time-stamp in their To File name.
3. Objects transport unmodified from the AE object Development Client to QA (one AE System) and Production Client (separate AE System). The source for all transports is always the Development Client. If modifications are required to objects in QA to support those activities the source development objects are never modified unless there was an error in an object's definition. There are other Clients that the application developers may use as needed. Objects from the Development Client may be transported to them and the application developers may modify or create objects as needed. Objects from these Clients are never transported to the QA or Production Clients and are considered expendable.
Each Client has its own set of Variable objects that contain Client specific values that are substituted at generation time to tailor the various attributes that are unique to that Client an application system. These are the usual attributes such as: Login, Agent, file names, directories, etc and are usually accomplished using the PUT_ATT script statement contained in the Process or Pre Process tabs by common Include objects that are used to perform this runtime modification.
4. There is extensive use of common scripting logic Include objects. Many utility type functions have been created that are inserted as needed in the various object's Process tabs. They perform things like parsing delimited variable lists, activate objects, determine standard information such as object name/number, parent name/number, Agent to use (we do not use Agent Groups), Notification object usage, etc.
5. Early on each application system had its own common Includes to perform their attribute assignment. We have since developed a single set of common Includes that are capable of handling all of the required attribute setting regardless of the application system. We use the Archive Key1 to specify a "Role" that the object is to assume. The Includes use this to determine attributes to set, again such as Host/Agent, Login, framework path and command, shell type, etc. This technique has made the implementation of new application systems extremely quick and easy. We have not, as yet, gone back to convert the previous unique application system Includes to use the Role technique but this is something that is being considered.
6. Object naming standards were developed early on. They are rather simple in that the first two characters, followed by an underscore, identifies the application system. This identifier also establishes user ownership, associated Login objects and approval processes. We only allow the underscore as a delimiter and not the period; this just exists for consistency and readability purposes. The user has virtual carte blanche on naming their objects except we prohibit including the object type in its name such as xx_POSTING_JOB as it is redundant and many objects have relatively long names. Each application's objects are segregated in a Folder prefixed by the two character code followed by its description, i.e. xx_ACCOUNTS_PAYABLE. There are sub-folders within these folders that further segregate the objects by type, i.e. xx_JOBS, xx_SYNCS, xx_VARIABLES, etc. We also discourage the use of a scheduling indicator in the name such as xx_DAILY_PROCESS or xx_MONTHLY_EXTRACT as the actual scheduling of some number of objects tend to change over time.
7. Email addresses used in Notification objects are always mail lists, never individuals. Users add or delete themselves to the list depending on their needs. We do not use the SEND_MAIL function but instead provide a generalized facility using the Unix sendmail command for sending email messages that includes the ability to substitute AE data such as: Object name/number, Client, Status, etc. into the message's content. The email's text and addressing is totally under the user's control as it is a text file that the generalized facility reads. This is often used in Post Process, Runtime evaluation's Else/Execute object or as a successor task.
8. The Development Client exists only to verify the proper execution of the requested new or modified objects and is never used for application development testing execution. While all users have full Read authority it is only the automation group that has Write and Execute. As stated previously this is the source for transport to other Clients.
9. The automation group has no routine production support duties. Once objects are verified as meeting the application's requirements and transported to the Production Client their day to day monitoring falls to the application's operation group. They are responsible for coordinating with the Operations Center to effect scheduling changes, restart and recovery actions. The automation group rarely is involved with the Production Client's operation except when providing second-level problem determination analysis of failed tasks. We have only Read authority to the Production Client and usually have little or no application specific knowledge.
The application group is totally responsible for all runbook type documentation of their systems and any associated AE objects. The objects do not contain any application documentation or comments beyond the object's Title. All of that type of information is in the application's documentation and as stated is their responsibility though we have on occasion provided input if there is any AE type information regarding the activation or execution of their objects.
10. We tend to avoid using some of the "newer" features. Even though we are still shamefully on V8 we believe that Automic has been rather remiss in fully implementing some of them. A prime example is the availability of Alias and Prompt and their limitations; just search on either of those words in the forum and read the gyrations that some are going through. We perform a fair amount of impact analysis and rely primarily on the Search dialog to accomplish this. We are hesitant to implement any new functionality whose use cannot readily be detected via this standard search or usable with script statements or functions . We also are resistant to using SQL to determine this type of information though that appears to be the default position that is often being taken by Automic. Since we already have a rather rich catalog of common Include functions and techniques to accomplish most all of our requirements there will need to be some compelling business reason to adopt some these newer features.
11. No tasks require user supplied information via AE objects. None of the objects use READ type functions to prompt or acquire information for execution. It is the responsibility of the application developer to provide whatever may be needed for execution outside of AE. This data must be supplied via application files or interactive type transactions or dialogs that prompt for control or execution time information. Generally if some type of data is needed before a task can execute a File Event object is used to wait as appropriate if the application code does not perform the needed detection itself.
12. A well defined change control/management procedure is pretty vital. We have a fully documented process that describes how to make a request, test it, and transport to production with the appropriate approvals. One of the biggest challenges that AE offers is its inability to allow concurrent object development. We sometimes have issues regarding the timing of multiple modification to the same object, mostly Process Flows, that needs to be made and transported to the Production Client in rapid succession. We really have not yet developed a solid technique to handle this and we occasionally struggle with satisfying our users needs.
While AE has object versioning there is not, to our knowledge, a reasonable way to pick and choose which version to activate for any given execution. It would sure be nice it there was some inherent facility to do so. Perhaps a Product Idea is in order.
13. All objects are defined with Generate at runtime checked. Given how our common Includes operate by performing the PUT_ATT and changes are sometimes made to the Client specific Variables as needed we are assured of getting the most current value. We have modified the Automic supplied object Templates in Client 0 of our non-production system in the Development Client to ensure that proper attributes (most notably the Generate) are set when we define new objects. We are aware that this is a potential maintenance issue when changing AE versions but feel the positives far outweigh the negatives.
14. Minimize the number of Calendar objects. Initially every application group wanted their own Calendar and associated keywords. There was a significant duplication of Keywords and maintenance became very labor intensive. We are down to one Calendar that all systems use with a couple of exceptions and those are very system specific though we wish to fold those into the single Calendar eventually.
There are still more things we do but I believe that the above are the most salient ones. Hope this helps and you didn't doze off. If I think of anything else major that I missed I will update this discussion.
Good luck!
Mark