Automic Workload Automation

Expand all | Collapse all

Best Practices for New Automic Implementations

laura_albrecht_automic

laura_albrecht_automicJan 04, 2016 09:05 AM

  • 1.  Best Practices for New Automic Implementations

    Posted Dec 30, 2015 05:21 PM
    I am currently in the process of building out new Automic systems - Development, Test and Production and have lots of ideas on how best to create these systems with all I've learned about Automic over the past x years.  But I'd love to hear the pros / cons of my ideas as well as any other Lessons Learned out there.  Imagine you could start all over..... what would you do differently?  Let me know your pain points - perhaps I can avoid them.  :smile: One main goal I have is to limit the changes (ideally eliminate entirely) needed to move objects from one system (Development) to another (Production).  This  means implementing policies / standards like: - Agent Names either need to be set dynamically via PUT_ATT (and pulled from an Include so no hard coding) or use Agent Groups to keep agent names the same between environments. - Login objects either need to be set dynamically via PUT_ATT (and pulled from an Include so no hard coding) or keep the login object names the same between environments. - Any environment specific information is stored in an include vs. hard coding in individual objects.  Any other ideas out there?  If for some reason, I cannot eliminate all changes between the environments any suggestions on the best way to handle that?  Was thinking Transport Case (vs. Export/Import) since that maintains the folder structure.  And I believe the DB Change Utility allows you to edit the Transport Case file although I have limited knowledge there.  Folder structure would be exactly the same between the environments. Was planning on having a strong object naming convention and got lots of tips reading through the Community posts. I think I plan to use Queues for each application / time zone to address time zone issues, but am waiting for my Development system so I can confirm this will work the way I envision.  Any thoughts are appreciated.  Thanks in advance! Laura Albrecht


  • 2.  Best Practices for New Automic Implementations

    Posted Dec 30, 2015 05:22 PM
    Not sure why it's not saving any of my formatting.  Sorry for the tough read.


  • 3.  Best Practices for New Automic Implementations

    Posted Jan 01, 2016 06:07 AM
    Hi Laura, which version are you going to use for the re-implementation? 


  • 4.  Best Practices for New Automic Implementations

    Posted Jan 04, 2016 09:05 AM
    It's a brand new deployment - we're going straight to 11.2.


  • 5.  Best Practices for New Automic Implementations

    Posted Jan 04, 2016 11:34 AM
    Laura:

    Lucky you!  I think that you have thought of the same issues that we have faced and are on the right track with your approach.

    While this isn't quite a Magnum Opus it may be a bit more than you want or need, but once I got started, I just couldn't stop!   Let me know if you need further detail explanation or additional rationale for our decisions.

    First a bit of history -  Initial use of AE, né UC4, was not well controlled and most any application type development group that wanted to automate one or more of their processes was able to do so as the product was available to most anyone that asked.  It was considered a utility, there for the asking with virtually no standards or policies for usage.

    It was soon realized that this approach was not very effective for company wide automation.  This became very evident when scheduling of tasks quickly started to cross system boundaries and it was difficult to establish proper predecessor and successor dependencies across these boundaries.  The good news was this was discovered well before there were a significant number of objects created and executing in production.  The bad news is that there are still some of these objects actively in use.

    This resulted in the creation of an enterprise automation group that centralized the development of objects for all application systems and platforms.  Users submit requirements and the automation group will create or modify objects as appropriate on their behalf.  The requesters are not required to have much if any AE specific knowledge and only need to be able to communicate their application's requirements effectively.  The automation group makes it so that application developers do not also need to be AE savvy.  Obviously over time these users become more AE aware and conversant which is sometimes not a particularly good thing since they want to tell us what to do and how to do it. 

    Anyway, enough about the past.  Here are, in relatively broad strokes, the standards and methods that we enforce and use to accomplish the company's goals.  As with many implementations we think that we got the basics right and continue to evolve as we find smarter ways to do things.  And, as usual, there are exceptions but we work very hard to minimize those.  Many of our standards exist so as to eliminate the need to have our group involved as the application systems evolve and only participate when AE objects must be modified such as adding new tasks or dependencies.

    Our overall goal is provide cross platform automation of batch type processes for the enterprise.  We are a centralized IT organization and our systems and users are in a single time zone.  The current operating platforms are mainframe,  Windows, SAP, Unix and its variants.  We currently have about 135,000 objects in our Production Client.  The automation group currently consists of 3 employees that spend perhaps 50-60% of their time on AE type work.  We are soon to add a fourth to replace some part-time contractors and there is a push to automate many more processes so our workload is increasing.

    In addition we have a philosophy that production processes must not be directly dependant on AE facilities to successfully execute.  The assumption is that if, for whatever reason, AE is unavailable these processes can still be executed manually though obviously with probably much difficulty.  Our basic use of AE is as a "traditional" task scheduler and thus far this approach has served us well.

    In no particular order:

    1. No application JCL (shell, commands, etc.) in a Job object beyond a single line to execute a "framework" command.  All application systems must develop this single command, usually a Unix or Windows shell that accepts as a parameter the object's name. That command is responsible for doing whatever is necessary to execute the job's JCL by establishing paths to directories and such to locate the needed resources.  We will provide assistance if needed to develop this framework command, though we are not responsible for it.  This framework command must operate the same whether called from the command line in any environment manually and when used by an Agent via AE.  This ensures that whatever is executed in a non-AE development environment will operate the same when automated in production thus reducing application development effort and increasing accuracy.

    The decoupling of the application JCL from Job objects keeps the automation group focused on AE type development and no involvement with initial and subsequent modifications that may occur to a job's JCL. 

    2. Additional parameters may be passed to the framework, however they are limited to things that AE inherently knows such as: Client number, process flow name, dates, times, restart status, etc.  We  also, for example, can pass a file name used in a previous File Transfer task since many of them contain a time-stamp in their To File name. 

    3. Objects transport unmodified from the AE object Development Client to QA (one AE System) and Production Client (separate AE System). The source for all transports is always the Development Client.  If modifications are required to objects in QA to support those activities the source development objects are never modified unless there was an error in an object's definition.  There are other Clients that the application developers may use as needed.  Objects from the Development Client may be transported to them and the application developers may modify or create objects as needed.  Objects from these Clients are never transported to the QA or Production Clients and are considered expendable.

    Each Client has its own set of Variable objects that contain Client specific values that are substituted at generation time to tailor the various attributes that are unique to that Client an application system.  These are the usual attributes such as: Login, Agent, file names, directories, etc and are usually accomplished using the PUT_ATT script statement contained in the Process or Pre Process tabs by common Include objects that are used to perform this runtime modification. 

    4. There is extensive use of common scripting logic Include objects.  Many utility type functions have been created that are inserted as needed in the various object's Process tabs.  They perform things like parsing delimited variable lists, activate objects, determine standard information such as object name/number, parent name/number, Agent to use (we do not use Agent Groups), Notification object usage, etc.

    5. Early on each application system had its own common Includes to perform their attribute assignment.  We have since developed a single set of common Includes that are capable of handling all of the required attribute setting regardless of the application system.  We use the Archive Key1 to specify a "Role" that the object is to assume.  The Includes use this to determine attributes to set, again such as Host/Agent, Login, framework path and command, shell type, etc. This technique has made the implementation of new application systems extremely quick and easy.  We have not, as yet, gone back to convert the previous unique application system Includes to use the Role technique but this is something that is being considered.

    6. Object naming standards were developed early on.  They are rather simple in that the first two characters, followed by an underscore, identifies the application system.  This identifier also establishes user ownership, associated Login objects and approval processes.  We only allow the underscore as a delimiter and not the period; this just exists for consistency and readability purposes. The user has virtual carte blanche on naming their objects except we prohibit including the object type in its name such as xx_POSTING_JOB as it is redundant and many objects have relatively long names.  Each application's objects are segregated in a Folder prefixed by the two character code followed by its description, i.e. xx_ACCOUNTS_PAYABLE.  There are sub-folders within these folders that further segregate the objects by type, i.e. xx_JOBS, xx_SYNCS, xx_VARIABLES, etc. We also discourage the use of a scheduling indicator in the name such as xx_DAILY_PROCESS or xx_MONTHLY_EXTRACT as the actual scheduling of some number of objects tend to change over time. 

    7. Email addresses used in Notification objects are always mail lists, never individuals.  Users add or delete themselves to the list depending on their needs.  We do not use the SEND_MAIL function but instead provide a generalized facility using the Unix sendmail command for sending email messages that includes the ability to substitute AE data such as: Object name/number, Client, Status, etc. into the message's content.  The email's text and addressing is totally under the user's control as it is a text file that the generalized facility reads.  This is often used in Post Process, Runtime evaluation's Else/Execute object or as a successor task.

    8. The Development Client exists only to verify the proper execution of the requested new or modified objects and is never used for application development testing execution.  While all users have full Read authority it is only the automation group that has Write and Execute. As stated previously this is the source for transport to other Clients.

    9. The automation group has no routine production support duties. Once objects are verified as meeting the application's requirements and transported to the Production Client their day to day monitoring falls to the application's operation group.  They are responsible for coordinating with the Operations Center to effect scheduling changes, restart and recovery actions.  The automation group rarely is involved with the Production Client's operation except when providing second-level problem determination analysis of failed tasks.  We have only Read authority to the Production Client and usually have little or no application specific knowledge.

    The application group is totally responsible for all runbook type documentation of their systems and any associated AE objects. The objects do not contain any application documentation or comments beyond the object's Title.  All of that type of information is in the application's documentation and as stated is their responsibility though we have on occasion provided input if there is any AE type information regarding the activation or execution of their objects.

    10. We tend to avoid using some of the "newer" features.  Even though we are still shamefully on V8 we believe that Automic has been rather remiss in fully implementing some of them.  A prime example is the availability of Alias and Prompt and their limitations; just search on either of those words in the forum and read the gyrations that some are going through.  We perform a fair amount of impact analysis and rely primarily on the Search dialog to accomplish this.  We are hesitant to implement any new functionality whose use cannot readily be detected via this standard search or usable with script statements or functions .  We also are resistant to using SQL to determine this type of information though that appears to be the default position that is often being taken by Automic.  Since we already have a rather rich catalog of common Include functions and techniques to accomplish most all of our requirements there will need to be some compelling business reason to adopt some these newer features.

    11. No tasks require user supplied information via AE objects.  None of the objects use READ type functions to prompt or acquire information for execution.  It is the responsibility of the application developer to provide whatever may be needed for execution outside of AE.  This data must be supplied via application files or interactive type transactions or dialogs that prompt for control or execution time information.  Generally if some type of data is needed before a task can execute a File Event object is used to wait as appropriate if the application code does not perform the needed detection itself.

    12.  A well defined change control/management procedure is pretty vital.  We have a fully documented process that describes how to make a request, test it, and transport to production with the appropriate approvals.  One of the biggest challenges that AE offers is its inability to allow concurrent object development.  We sometimes have issues regarding the timing of multiple modification to the same object, mostly Process Flows, that needs to be made and transported to the Production Client in rapid succession.  We really have not yet developed a solid technique to handle this and we occasionally struggle with satisfying our users needs.

    While AE has object versioning there is not, to our knowledge, a reasonable way to pick and choose which version to activate for any given execution.  It would sure be nice it there was some inherent facility to do so.  Perhaps a Product Idea is in order. 

    13. All objects are defined with Generate at runtime checked.  Given how our common Includes operate by performing the PUT_ATT and changes are sometimes made to the Client specific Variables as needed we are assured of getting the most current value.  We have modified the Automic supplied object Templates in Client 0 of our non-production system in the Development Client to ensure that proper attributes (most notably the Generate) are set when we define new objects. We are aware that this is a potential maintenance issue when changing AE versions but feel the positives far outweigh the negatives.

    14. Minimize the number of Calendar objects.  Initially every application group wanted their own Calendar and associated keywords.  There was a significant duplication of Keywords and maintenance became very labor intensive.  We are down to one Calendar that all systems use with a couple of exceptions and those are very system specific though we wish to fold those into the single Calendar eventually.


    There are still more things we do but I believe that the above are the most salient ones.  Hope this helps and you didn't doze off.  If I think of anything else major that I missed I will update this discussion.

    Good luck!

    Mark



  • 6.  Best Practices for New Automic Implementations

    Posted Jan 04, 2016 11:59 AM
    Wow Mark!  This was great.  Thanks for taking the time to pull this all together / type this up.

    All very interesting point and some I will definitely be looking at using.  Can you go into more detail on # 5?  Do you mean that you have an include that has SQL job information and so SQL jobs in the Archive Header 1 field specify "SQL" thus using those specific settings?

    Thanks again and I'll search the site for those other 2 topics.  GREAT info Mark. 


  • 7.  Best Practices for New Automic Implementations

    Posted Jan 04, 2016 12:29 PM
    The Role processing is accomplished just using an Archive Key 1 value and having the various Process tabs use the appropriate Include.

    For example, the Key has "ROLE=INFORMATICA" on a Job object and the common Include in the Pre Process has a  
    :SET &mekey1 = GET_ATT(ARCHIVE_KEY1)
    followed by logic that is used to read various keywords from the Variable object that contain Client specific values:
    Keyword                 Value
    INFORMATICA_HOST_1     XXXXX01
    INFORMATICA_HOST_2     XXXXX07
    INFORMATICA_HOST_3     XXXXX02
    INFORMATICA_HOST_4     XXXXX08
    INFORMATICA_LOGIN       LOGIN.UNIX.SOME.AUTHORITY
    INFORMATICA_SHELL     RUN_%APPL_JOB,-bash

    Note that the %APPL will be substituted with that Client's "environment" value, like P for production, Q for QA, etc.  By convention, we use strings that begin with the % sign that need some type of substitution that the common Includes will perform.



  • 8.  Best Practices for New Automic Implementations

    Posted Jan 04, 2016 04:08 PM
    So basically you have a single include that has information for all applications / objects.  By using the ROLE, then within the include it would have IF / ELSE statements pointing it to the appropriate values to use.  Is that right?

    Interesting idea - certainly would make maintenance a whole lot easier.


  • 9.  Best Practices for New Automic Implementations

    Posted Jan 04, 2016 04:39 PM
    That is correct.  

    There is actually a single Include that is used in all of the Process and Post Process type tabs except for a Job object.  This determines the object's type and performs its needed settings or other decisions..

    There is a very short single Include for a Job's Process tab that just executes the framework shell and if it is Windows inserts the "@set retcode=%errorlevel" JCL statements.   It is obviously dependent on all of the variables set during the Pre Process. 


  • 10.  Best Practices for New Automic Implementations

    Posted Jan 05, 2016 03:06 PM
    Strong standards and administration saves lots of headaches down the road.  We did a switchover from V6 to V9 in 2014 and decided to take advantage of new features when possible.  We are small enough that we decided to re-code everything from scratch.  

    The first thing we did was to identify the most common types of objects and added our own templates to support them.  We then told everyone they had to use these templates.  These templates were used in probably 95% of our objects, and the bulk of their code is in includes.  We have different includes for the different templates.

    For our environmental differences we did something similar to UC_CLIENT_SETTINGS; We put all of the settings in a variable and then reference that variable.  In our case we set up more than one variable to support different categories of settings. (If I had it to do over again, I think I would have put them all in a single variable so there would be fewer places to look for these settings.)

    It took a bit of effort to design and set up how our problem alerts were going to function.  Good to get this done early.

    It also took a bit of effort to set up role based user security.  I had to keep tweaking it as new requirements popped up.  (for instance only a few people are authorized to open reports associated with HR runs.)

    We use agent groups so that our objects are more portable across clients.  However there are still a few places inside of the (V9) product where you can not use an agent group and you are still required to use a specific agent name.  So when we use transportcase we still have the need to alter the exported file before importing it to a different client.   (TransportCase has some drawbacks too.  There is no perfect change control method yet.)

    We were able to do all calendaring with a single calendar object.  There should be no need to use any static keywords at all, except in very rare occasions.  (We have 2 such occasions; this company has a floating holiday in December that does not follow any rules, and our State sometimes has furlough days that cause the State datacenters to be closed.)


  • 11.  RE: Best Practices for New Automic Implementations

    Posted Aug 26, 2019 11:19 AM
    Edited by Scott Hughes Aug 26, 2019 11:19 AM
    One way to improve the use of agent groups in the more modern versions is to use a vara.exe which returns the agent.  The script that the vara.exe executes uses prep_process_hostg to resolve the agent.  This vara.exe is used where the AE does not allow you to put HOSTG entries.

    Another idea for transport standardization is to use the dbchange utility, it basically does a find and replace of certain fields in the transport case.

    ------------------------------
    Scott Hughes
    Senior Automation Engineer
    Verizon
    Albuquerque, NM
    ------------------------------



  • 12.  RE: Best Practices for New Automic Implementations

    Posted Aug 26, 2019 11:38 AM
    If you don't mind me asking:

    What's vara.exe? Where can it be obtained?​


  • 13.  RE: Best Practices for New Automic Implementations

    Posted Aug 26, 2019 11:47 AM
    Its a scripting function introduced in 11.2  The vara.exe is referenced either by get_var or on a pre_process or post_process condition, and other ways as well.  VARA.EXE is defined to run some other object, that object returns a value based on process tab.  The vara.exe has a parameter page to route promptset data to the called object.

    image.png

    image.png
    image.png
    --
    ~~~~~~~~~~~~~~~~~~~~~~~~
    Scott Hughes
    Verizon  (505) 373-7872
    scott.hughes1@verizon.com 
    ~~~~~~~~~~~~~~~~~~~~~~~~





  • 14.  RE: Best Practices for New Automic Implementations

    Posted Aug 27, 2019 01:24 AM
    ​Oh, it's an EXEC Variable. Sorry, the term vara.exe made me consider this to be a DOS or Windows-style command line executable, thought for a while you had a consultancy solution there.

    Cheers for clarifying.


  • 15.  Best Practices for New Automic Implementations

    Posted Jan 06, 2016 08:45 AM
    Thanks petwir !! What places can't you use an agent group?  I can check and see if that still applies in 11.2.

    Also, what are the drawbacks of the transport case?  So far that's the way I'm leaning.  I found a thread with a white paper about a Transport Case Management System which seems like a pretty cool way to automate migrations of objects across systems / clients, so I'm getting more info on that from my account rep.

    Can you expand a little more on your "templates"?  We customized the client 0 object templates a little at my last job, but it was basically something that only came up as people were creating "new" objects.  People could (and would) still modify the templates removing our defaults if they wanted.  We had to put extra checks in for that.   :)

    I am definitely leaning toward managing a variable / associated include for settings.  This seems like the easiest way to manage these types of things.  Once I get a system up I'll be testing this out and Mark_Hadler_430's option as well with the roles.  I assume you require this include to be in every object like he does.

    Thanks again - these are all great ideas!


  • 16.  Best Practices for New Automic Implementations

    Posted Jan 06, 2016 11:07 AM
    LauraAlbrecht608310 

    Just a note about our use of the Role technique.  Each application system has their own 
    Variable object that contain Client specific values.  This was done since the choice of a role name is up to the application and we did not want to have to manage possible duplication as we would have had to do if they were all in a single Variable.  

    We did devise a method to accomplish the ability to use a single Variable but it was rejected as being a bit too cumbersome.  The basic method was to include the two character system code as a prefix on the Variable's Keywords and then ignore it once the value was retrieved.  So, for example there could be Keywords for xx-INFORMATICA and yy-INFORMATICA though their Archive Key would just be ROLE=INFORMATICA.

    Also there is not specifically a requirement that every object contains a ROLE= Archive Key, just that all of the objects use the common Includes that support it.  If an object does not have a ROLE= specified then there are default values associated with the system code in the Variable object and those will be used.




  • 17.  Best Practices for New Automic Implementations

    Posted Jan 06, 2016 11:55 AM
    I have 11.2 on a test system.  I confirmed that I can't use an agent group when doing a "CHECK FILE" precondition on a workflow task.  I vaguely remember there were other places in the system where this is an issue, but I can't seem to remember them at this time.

    TransportCase does not allow you to export from folder-A and import into folder-B.  We've had developers working under personal folder structures, and when it was promoted to PROD, PROD was then polluted with personal developer folders.  (You'll need a standard for your folder structures!)

    I've written a poor-man's TransportCase system to simplify all of the tasks involved of exporting from one system, importing into another, and automatically changing agent names.  We accomplish this by running EXPORT and IMPORT workflows and filling out a single promptset.  I also built a DEPLOY workflow that imports a TransportCase file into multiple non-production clients.  This is good enough for this small shop, but larger shops (Like Mark's) need to satisfy higher levels of checks and balances than we do.

    Templating; As two examples, we needed to have a standard way of coding .net console apps and oracle stored procedures.  I created our console app template by cloning the windows job template and customizing it to use our standard console app include and standard promptsets.  I created our oracle stored procedure template by cloning the oracle DB job template and customizing it to use our standard oracle stored procedure include and standard promptsets.   They both use the same promptset object to collect environment variable data and pass it to the include.  The reason we pass the environment variables through promptsets is so testers can easily override the environment variables. For instance it is common to want to run a test against database X even though one is testing their software from environment Y.  We limit which environment variables they can modify in this way.