Automic Workload Automation

Expand all | Collapse all

Automic version 12.2 version control

  • 1.  Automic version 12.2 version control

    Posted 07-26-2018 06:24 AM

    Hi all,

     

    Has anyone tried to start using the new Version control released with version 12.2?

     

    We have started to test this out and come into a few issues, using VSTS as our repository.

    1. When setting up REPOSITORY_SSH_DIR this only seems to work with out error if the key directory is set in both the system and client settings when not using the SSH directory of the installed operating system.
    2. Is there a way to unlink the branch from the initialised version and link to a new branch or refresh this if a branch has been deleted in from the repository? The issues throw an error if the branch no longer exists and has to be recreated before the link will work again. 
    3.  If there is an outstanding a pull request you can not commit until the pull request has been completed this means overwriting the committed changes. The issues is the pull request error on attempt and then the version control become unusable in the client it connected too. Below was the output.The above was a roll back to the initial repository but changes had been made but not committed which caused this error.
    4. The UC_VERSIONCONTROL_EXCLUDE only gets created after the client has been initialised with the repository. It seems when creating new branches any objects in this branch are also been added to the new branch and not been excluded. It would be nice to version control on folder structures or at least exclude entire folder structures from being part of specific branched version.
    5. The merge of branches always pics up all items due to changes in version Id last run time

     

    I am wondering if anyone has over come any of the above or if these are known issues?

     

    Below has been used for the most part but not sure if there is more to this documentation. 

    documentation/12.2/AWA Versioning

     

    Thank you

    Regards

    Chris.



  • 2.  Re: Automic version 12.2 version control

    Posted 08-07-2018 07:00 AM

    I have been playing around with version control and managed to get a client connect to a GiT repository. also using the .gitignore file to stop changes in some folders be reflected into the repository.

     

    I have also managed to set up a few objects and use wild card * to stop commits and objects and folder containers. By adding these into the new VARA UC_VERSIONCONTROL_EXCLUDE. Using the combination of .gitignore and the vara I have managed to release only what /i want from one client into another in a controlled way.

     

    I am now looking for a way of being able to stop certain files and folders being pulled down from the repository in to the version controlled client. Is there any way of doing this from Automic?

     

    Also some of the errors have required me to edit the folder directory to complete the pull request. That was created to correct the issue or create a new pull request from VSTS to complete correctly without error in the next pull from the repository.

     

    So far the version control work ok for what we need to do. Create a development client and commit changes required in to the repository and then create a pull request into a new branch using VSTS and completing this and then pulling the change down into another Client. This tracks the change and makes controlling changers easier with the options to lock down branches using security setting in the repository. Issue with losing the active branch when a branch has been locked down trying to see if this can be fixed with permissions changes in the repo.

     

    Thank you

     

    Regards

    Chris



  • 3.  Re: Automic version 12.2 version control

    Posted 08-08-2018 09:08 AM

    HI,

    I am trying to use GIT LAB  with version control, but I cannot get it work.

    When i put  ssh url using ssh://git@servername:group/file.git i get this error when i want to select available branch

     

    If i want to Create new branch 

     

    I have tested access to git using user that AE is running on and using GIT Bash and it works just fine if change url to ssh://git@servername/group/file.git but that format set in UC_CLIENT_SETTINGS is not working,

    Since i am getting this 

     

     

    I am not very good with GIT and CI/CD topics yet, repository is called client-15 so i am not sure why AE is looking for repo named after OS Path 

     

    Thanks

    Jerry



  • 4.  Re: Automic version 12.2 version control

    Posted 08-08-2018 09:58 AM

    Hi Jerry,

     

    I have found issues with this setup myself. The SSH URL is set at client level seems to be OK from my side.

     

    The issues I have had is the the SSH key directory needs to be set in both UC_CLIENT_SETTINGS and UC_SYSTEM_SETTINGS for REPOSITORY_SSH_DIR and they need to be the same. 

     

     

    You need to make sure the REPOSITORY_SSH_DIR is linked to your private key which can be seen by the Automation engine. This also has to be paired with your public key created in Git. Then you need to GiT clone the URL and check the finger print matches the public finger for you public key in you repository.  Once all paired up you should be able to get a connection through to you GiT repository.

     

    I have found you need to create a new branch from Automic for the branch to work correctly. We have been using a new Repo for Automic with no other object in there as well.

     

    It has created a copy of the files in DIR?/Automiation.Platform/Automication/Engine/bin/ae-filesystem/ in this directory there is a file for each client and this is where the location of the file structure and XML are stored between Automic and GiT(had an issue with a unknown file stopping the pull and after removing the file from GiT it was still in this location once delete the pull worked). We use a windows file structure connected to SQL server. 

     

    You can try Connect to your Git repos with SSH | Microsoft Docs but not sure exactly what your setup is but could still do similar with GiT.Bash i would think.

    So maybe more info here Connecting to GitHub with SSH - User Documentation 

     

    Regards

    Chris



  • 5.  Re: Automic version 12.2 version control

    Posted 08-08-2018 10:27 AM

    Hi,

    so SSH DIR has to point to actual keyfile or just Directory ? lets say i keep key on D:\keys and key is bla.key 

    so REPOSITORY_SSH_DIR should have a vaule of D:\keys\bla.key ? 

     

    ==We are running AWA on Windows ===

     

    Git clone run from git bash using that key works fine 

     

    But even if i try to create new branch i get this annoying empty box error 



  • 6.  Re: Automic version 12.2 version control

    Posted 08-08-2018 10:51 AM

    Hi,

     

    The key file has to be the directory which has been paired with the public key.

     

    So we have created on the C: dirve a new Automic user which has the .ssh directory this was added to GiT 

    by using Git cmd start-ssh_pageant.cmd from the tools\GiT\cmd

     

    then from tools\Git\usr\bin adding the user to the key by running ssh_add for the directory you rsa private key is stored 

     

    Then cloned from the directory the .ssh folder is in by doing CD to this directory up to .ssh then git clone the SSH URL from here

     

    Once done this can be added as the REPOSITORY_SSH_DIR in both system and client at least when first doing the initialisation or your repository. The client also needs to be stopped i Automic for the Initialisation so make sure it has been disabled before linking it to a new branch.

     

    This is the first time I have also started to use this version of Automic and connecting it with the REPO it has been a few weeks of trial and error. There is not much info in the Automic documents on how this needs to be set.

     

    Regard

    Chris  



  • 7.  Re: Automic version 12.2 version control

    Posted 10-02-2018 09:47 AM

    I came back to the topic and found issue that for some reason AE cannot read known_hosts in the folder where user is :

     

    U00045015 The previous error was caused by 'org.eclipse.jgit.errors.TransportException: "git@*********:user/c15.git: UnknownHostKey: *********. RSA key fingerprint is XXXXXXXXXXX"' at 'org.eclipse.jgit.transport.JschConfigSessionFactory.getSession():163'.

     

    known hosts is there, GIT commands from git bash works just fine



  • 8.  Re: Automic version 12.2 version control

    Posted 10-03-2018 03:55 AM

    Hi,

    Below is held in the AE engine. which is on the a windows server.

    When you create the key pair as attached link shows. It creates the id_rsa and known_host files in the user folder that s creating the key pair. 

    Connect to your Git repos with SSH - Azure Repos | Microsoft Docs  

     

    This is the directory you ad to you UC_CLIENT_SETTINGS and UC_SYSTEM_SETTINGS. I have found it seems to not work to well if its not in both. Even though the documentation says either can be used and the system is default.  

    REPOSITORY_SSH_DIR should look like c:\users\username\.ssh\.

     

     

    Regards

    Chris



  • 9.  Re: Automic version 12.2 version control

    Posted 10-12-2018 05:31 PM
    • Anyone set Versioning with Git successfully on the environment that have 2 Automation Engines in Active-Active configuration on two different servers?
    • Do you put the SSH keys into the same directories on both servers?
    • Is this feature would use one of the servers as the main one to communicate with Git? How do we configure which server should be used?

     

    @Automic/CA Support, Can the documentation on the topic be made more clear since versioning using Git is new important feature of the product, we believe.

     

    Thanks,

    Vlad



  • 10.  Re: Automic version 12.2 version control

    Posted 10-18-2018 04:14 AM

    No. We have not tried this yet. I have only run this from the one Engine.

    I would be interested to know how this works though as well. Thanks



  • 11.  Re: Automic version 12.2 version control

    Posted 10-23-2018 09:42 AM

    Hi,

     

    I just got answers from our development team. Here is some additional input:

     

    • Version Management was developed so that local copies are kept on every AE servers. A local copy of the branch is located in the /AutomationEngine/bin/ae-filesystem/<client number> directory. These copies are synchronous.
    • Files "id_rsa", "id_rsa.pub" and "known_hosts" must reside at the same location on all servers. However each physical machine must have its own distinct ssh key.

     

    Hope this helps.

     

    Best regards,

    Antoine



  • 12.  Re: Automic version 12.2 version control

    Posted 10-23-2018 06:39 AM

    Hi All, 

     

    Many thanks to JerryDanysz604416 who found a workaround regarding the following Error: 

     

    U00045015 The previous error was caused by 'org.eclipse.jgit.errors.TransportException: "git@*********:user/c15.git: UnknownHostKey: *********. RSA key fingerprint is XXXXXXXXXXX"' at 'org.eclipse.jgit.transport.JschConfigSessionFactory.getSession():163'.

     

    This occurs if the Known Hosts File has an ecdsa encryption.

     

     

    This command has to be run and output pasted to known_hosts file

    ssh (githostname) ssh-keyscan -t rsa (githostname)






  • 13.  Re: Automic version 12.2 version control

    Posted 02-27-2019 05:30 AM

    Hi,

     

    maybe someone can help me here.

    I have set up a git repository ("git init") on the AE server (not out of necessity but out of convenience) in the directory .../repo.

    I have also created a branch therein (let's say "ae_branch").

    Then, I have set up git versioning in the AE by configuring UC_CLIENT_SETTINGS accordingly, using

    REPOSITORY_SSH_DIR (/home/user/.ssh)

    and

    REMOTE_REPOSITORY_URL (ssh://user@ae-server01:22/.../repo

     

    I have successfully done the following:

    1) use the "user" to clone/pull/push/fetch/... into the original repository on ae-server01

    2) use "Version Control" > "Select Available Branch" in AWI (the dropdown shows the branches I have created via command line).

    This IMO means that

    * The user can successfully authenticate to interact with the git repository on ae-server01

    * AE has access (I see the branches)

     

    However, when I click "Initialize", it will take some time but then won't work; instead, I get an error message:

    [HTTP 500] (45110) An Automation Engine internal error occured.: No detail information available.

     

    Does anybody have an idea how to fix that?

     

    Regards



  • 14.  Re: Automic version 12.2 version control

    Posted 02-27-2019 05:45 AM

    Hi Friedrich,

     

    Have you checked to see what been pulled into your local Automic directory on the Automic server?

     

    I have seen extra files in Git branches like for example the README.md file stopping the objects being imported correctly any non Automic file need to be removed from your branch.

     

    A second issue I have seen was down to return carriage being added to XML after export and convert in Git that caused an error when code was pulled back into Automic these were removed and set to a space and then the pull worked.

     

    Regards

    Chris



  • 15.  Re: Automic version 12.2 version control

    Posted 02-27-2019 07:15 AM

    Hi Chris,

     

    thank you for your quick reply.

    I guess, by "local Automic directory" you mean .../server/bin/ae-filesystem/<Client>? If so: The client-directory (in my case that'd be "0010") was not even created.

    I have tested the behaviour with both, a completely empty repository and one that had an (empty) test-file in it.

     

    The export should happen all by itself I guess, with the target folder being .../server/bin/ae-filesystem/<Client>?

     

    Regards



  • 16.  Re: Automic version 12.2 version control

    Posted 02-27-2019 07:42 AM

    Hi,

    Yes the /server/bin/ae-filesystem/<Client> is where the local files for Automic from what I know.

     

    Do you have git installed on you Automic server and can you get to your git branch through git locally and see all your files and versions in git.

     

    If you can I have not seen this issue. sorry.

     

    Regards 



  • 17.  Re: Automic version 12.2 version control

    Posted 02-27-2019 07:54 AM

    Hi,

     

    yes, I can.

    The git "origin" repository is on the AE-server, I can create different repos on different servers that can easily access it, see the branches and files therein etc.

    And also, the AE sees the branches in there (proofing that it can connect).

     

    Thanks again!



  • 18.  Re: Automic version 12.2 version control

    Posted 02-27-2019 08:32 AM

    OK not sure, Only one thing are you using high availability and have more than 1 AE server.

    If so do is it work on both.

    If it dose not can you force it to use the one it dose work on to see if the folder structure is then created.

     

    If not for both the above, not sure what the issue is sorry. 



  • 19.  Re: Automic version 12.2 version control

    Posted 03-01-2019 03:58 AM

    Hi,

     

    no HA-setup in our system.

    However, I have done some more tests and analysis:

    1) If I manually create the server/bin/ae-filesystem/<Client> directory, the AE will delete it as soon as I click on "create new branch".

    2) If I use the REST-API to try the same thing, I get the following response after approximately 5-10 minutes:

    curl -D- --user "AUTOMIC/AUTOMIC:AuToMiC" -X POST -H "Content-Type: application/json" "http://server01:8088/ae/api/v1/10/repositories" -d '{ "branch_name": "newbranch", "force_delete_existing_client": true }'

    <5-10 minutes>:

    HTTP/1.1 200 OK Date: Thu, 28 Feb 2019 13:42:43 GMT Content-Type: application/json Content-Length: 86 Server: Jetty(9.4.8.v20171121) { "client" : 10, "status" : "ERROR_PERMISSIONS", "number_affected_objects" : 0 }

     

    I wonder WHICH permission error occurs here...

     

    Thanks & regards



  • 20.  RE: Re: Automic version 12.2 version control

    Posted 07-26-2019 12:07 PM
    I am about to start playing with this feature.  I am not so sure it is going to help or make more work.  There are only a few of us using Automic so change management isn't that hard.  I helped with the US Consulting Solution which leverages the utilities and pushes changes from client to client using transport cases.  The failure of this design is the inability to quickly roll back, in some cases new objects simply have to be deleted from production when changes go awry.

    My major question is how to manage dev uat and prod.  Prod is the main/head and I do a pull to dev to start a branch?  How do I get that branch to UAT before committing into Prod?  Maybe I need a pivot client that contains head and then prod only pulls my releases from that?
    • Dev pull from master to create changes
    • Check in changes to master
    • Uat pull from master to check changes
    • test fix etc
    • commit changes to master
    • Prod pull from master to implement changes
    • repeat
    I wish someone had a demo or white paper to share.  I tried to do that when I was at Automic but I ran into an error exporting XML variables to the repo and lost steam on the project waiting for resolution.


  • 21.  RE: Re: Automic version 12.2 version control

    Posted 07-26-2019 01:12 PM
    I don't use this feature either in our environment but I can share how we accomplish something similar.

    • We have 1 production client that is set up in a read only state. Say client 100. This is where all processing happens.


    • We use client copy to make a copy of the production client (100) on a set interval and copy it to client 10. This copy is also in our production system.


    • Each user that develops in Automic has their own Nonprod client.. This is in a separate nonproduction system.


    • A developer asks to check out objects from production client 10. The Transport script moves the objects into their personal nonprod client and then moves the objects into a 'checked out' folder in production client 10. So, if someone has an object checked out, no on else can modify or check out that object.


    • When changes are done being made to an active production workflow the developer checks back in the objects to client 10 and then promotes them to the read only client 100 for execution.

    As for testing in UAT, SPT Etc, the user can transport into a shared UAT, SPT environment. However, only objects from the users personal dev client can be checked back in. the UAT SPT environments are all dead ends.

    So far this has been an awesome system for us. This totally eliminates any developers stepping on any one elses work. In the event that there is stale objects left out in the check out folder due to development going stale or the business needs change we just schedule the client copy of client 100 back to 10 and reset the client.