DX Infrastructure Management

 View Only

CA on CA Tech Tip - Pushing archive packages via the UIM REST API 

May 30, 2018 03:20 PM

Hi Community,

At times I need to push packages to a lot of robots but selecting each one in AC or IM is super time consuming. In the Python library that we provided it has a method push_pkg and that is the basis for this code. Essentially it works by

  1. Getting the access information for the REST API, the package we want to push and the file with the list of robots to target from a configuration file
  2. We build a SQLite DB containing the relationship between the robot and the hub as this is required by the call to ADE. This is done using get_hubs and get_robots as well as some new code to build a robot db
  3. We then read the list of target robots. For each robot we find the hub that hosts it and use this information to push the named package to it with ADE. NOTE: We verify that ADE handles the push but we do not verify successful package deployment (more code required for that).

The code for executing this is available in GitHub as push-pkg-by-robot.py. Feel free to change or improve to match your use cases. There is also a sample configuration file now to help with the setup. Post comments and questions below.

 

rtirak

JAllen11

 

Disclaimer: I am a CA employee. I work in the CA IT department. The code is not sanctioned by CA support or development. Test before using in production.

Statistics
0 Favorited
1 Views
0 Files
0 Shares
0 Downloads

Tags and Keywords

Comments

Oct 24, 2018 04:35 PM

Hi Robert,

 

Sounds like you are trying to add new AWS accounts using this method. I guess MCS will not work as an easier interface or you want it to be completely programmable.

 

Have you tried the steps manually to confirm that this will work? Just walk through it to make sure it will actually work before you go through all the trouble.

 

Also I would start a new thread this has nothing to do with the original post so makes it harder for others to help.

Oct 24, 2018 06:22 AM

Alquin

That seemed to fix the issue thanks. A little more context on what I am trying to do with this that maybe you can give me some advice. We want to use a python script to that can create a new profile in the AWS probe. So when we go to put a new service from AWS into production we can add the new profile to the existing AWS probe. So when Looking I was going to use the Probe API and with PUT add a new section to the probe configuration and the different keys needed in each section so that the profile gets created from the config change automatically. The values of the instance/host name, Access Key, and Secrete Key will be passed to the python script some how and those will be used as the new section for the new instance is created and then the keys are added by the script and then saved. Than the probe would be deactivated, and then activated for a restart.

 

Hopefully this will give you some more context as to why i trying to use the API to add sections/key values to a probe with the API and python, and if you are aware of a more efficient way to do this, or a a different probe or API to accomplish this please let me know. I figured I would run it by you as it seems trying to add all the keys at one time with PUT and the probe API seems to be a little difficult when adding multiple sections and multiple keys all at one time.

Oct 23, 2018 09:28 AM

Try data=dumps(data) in your post.

 

I will have time to look further into this tomorrow afternoon if that does not work.

Oct 23, 2018 08:50 AM

Alquin

 

I seem to be getting the following error with this: The request sent by the client was syntactically incorrect, the basic code is below any thoughts as to why I would be getting that error? 

 

import requests
from requests.auth import HTTPBasicAuth

# defining the api-endpoint
API_ENDPOINT = "http://TSTNIMBUS01/rest/probe/Domain/Hub2/tstnimbus02/aws/config"

headers = {
'Content-Type': 'application/json',
'Accept': 'application/json'
}


#basic source code just for testing

a = 1
b = 2
print(a + b)


# data to be sent to api

data = {
'configkey': [
{'key': '/setup/resttest', 'type': 'string', 'value': 'ignoreme'}
]
}


# sending post request and saving response as response object
r = requests.post(url = API_ENDPOINT,auth=HTTPBasicAuth('admin','adminpassword'), verify=False, headers=headers,data= data)

# extracting response text
url = r.text
print("The URL is:%s"%url)

 

 

 

Alquin Gayle wrote:

 

Hi Robert,

Unfortunately I did not provide any code that directly does this. I recommend 2 things. 

  1. Start a new thread with the question so others can have a look. 
  2. Download a REST API client and try it there first it really helps to see it there and then build the code from that sometimes.

I am a little tied up right now but if I get the time and I am still around I will see what I can do to help mid to late next week.

Alquin Gayle wrote:

 

Hi Robert,

Unfortunately I did not provide any code that directly does this. I recommend 2 things. 

  1. Start a new thread with the question so others can have a look. 
  2. Download a REST API client and try it there first it really helps to see it there and then build the code from that sometimes.

I am a little tied up right now but if I get the time and I am still around I will see what I can do to help mid to late next weeimport requests

from requests.auth import HTTPBasicAuth

 

# defining the api-endpoint
API_ENDPOINT = "http://TSTNIMBUS01/rest/probe/Domain/Hub2/tstnimbus02/aws/config"

 

headers = {
'Content-Type': 'application/json',
'Accept': 'application/json'
}


# your source code here

a = 1
b = 2
print(a + b)


# data to be sent to api

data = {
'configkey': [
{'key': '/setup/resttest', 'type': 'string', 'value': 'ignoreme'}
]
}


# sending post request and saving response as response object
r = requests.post(url = API_ENDPOINT,auth=HTTPBasicAuth('administrator','nimbusrocks'), verify=False, headers=headers,data= data)

# extracting response text
pastebin_url = r.text
print("The pastebin URL is:%s"%pastebin_url)

Oct 19, 2018 10:27 AM

Hi Robert,

Unfortunately I did not provide any code that directly does this. I recommend 2 things. 

  1. Start a new thread with the question so others can have a look. 
  2. Download a REST API client and try it there first it really helps to see it there and then build the code from that sometimes.

I am a little tied up right now but if I get the time and I am still around I will see what I can do to help mid to late next week.

Oct 19, 2018 09:46 AM

Alquin

 

I have been working more with python and the requests library. I am still having a bit of trouble with this and was wondering if you had done it by chance or worked it out, or had some type of example code that I could look at. What we want to do is use the API to set Probe configuration values. I know in the API documentation there is a "Set Single probe configuration value" however I have not been able to use the python you have provided and create something to get this to work. We want to be able to do this so that we can configure keys and values inside the probe. For example we would configure a probe and add keys to is config like /setup/resttest and than add the value to that key of some value. Do you happen to have or could you provide me some type of sample python that would do something like this, the Set Single Probe Configuration? Below is a snippet from the documentation.

 

Jun 05, 2018 02:05 PM

rtirak

Based on the log there is a problem finding the automated_deployment_engine probe or if the path to it is correct then the call to it is not making the connection. 

  1. Verify the ADE path is correct (case sensitive)
  2. Can you push to that robot directly from AC using the archive with the ADE you are using?

Jun 05, 2018 01:22 PM

Alquin

 

I installed the werbservices_rest prove on the primary hub as well to test. I am now able to actually access and see the hub and it finds the robot however I am now getting the following error about the ADE

 

2018-06-05 13:06:56,739 - urllib3.connectionpool - _new_conn - DEBUG - Starting new HTTP connection (1): prdnims01
2018-06-05 13:06:57,717 - urllib3.connectionpool - _make_request - DEBUG - http://prdnims01:80 "POST /rest/probe/NMS/MAIN/PRDNIMS01/automated_deployment_engine/callback/deploy_probe HTTP/1.1" 500 None
2018-06-05 13:06:57,717 - root - push_pkg - DEBUG - The ADE package deploy response I/O error on nim session (C) com.nimsoft.nimbus.NimNamedClientSession(Socket[addr=/10.1.50.137,port=48020,localport=52907])
2018-06-05 13:06:57,717 - root - push_pkg - ERROR - Failed to make deploy probe call for package Remove_UIM to robot pocwin2016test1
2018-06-05 13:11:57,753 - urllib3.connectionpool - _new_conn - DEBUG - Starting new HTTP connection (1): prdnims01
2018-06-05 13:11:58,262 - urllib3.connectionpool - _make_request - DEBUG - http://prdnims01:80 "POST /rest/probe/NMS/MAIN/PRDNIMS01/automated_deployment_engine/callback/deploy_probe HTTP/1.1" 500 None
2018-06-05 13:11:58,262 - root - push_pkg - DEBUG - The ADE package deploy response I/O error on nim session (C) com.nimsoft.nimbus.NimNamedClientSession(Socket[addr=/10.1.50.137,port=48020,localport=61135])
2018-06-05 13:11:58,262 - root - push_pkg - ERROR - Failed to make deploy probe call for package Remove_UIM to robot pocwin2016test1
2018-06-05 13:11:58,262 - root - main - INFO - Pushed Remove_UIM package to 0 robots
2018-06-05 13:11:58,262 - root - main - INFO - End of push-pkg-by-robot

 

 

The ADE probe sits on our Primary hub as well. Seems like its trying to push the package and getting an error.

Jun 04, 2018 02:54 PM

rtirak I have only ever deployed the webservices_rest package to the UMP. However the documentation says, "Deploy the webservices_rest package from the local hub archive to a robot running the wasp probe". In the old days that used to only be the UMP but these days it could be any on the Primary, UMP or CABI. If you deploy it elsewhere or get a solution from support and it works please let us know.

Jun 04, 2018 01:22 PM

Alquin

 

So it seems the issue is our network segmentation that's blocking this. Can I install the webservices_rest probe onto the Primary Hub and use is from there rather than the UMP or will I lose functionality?

Jun 04, 2018 10:00 AM

There is an issue connecting to the hub port "Unable to open a client session for 10.1.156.254:48002: Connection timed out: connect" so something is getting blocked some where. The best option at this point is to take the API URL being called and stick it in a browser (or API client) and see if it works or you get the same result. If it is still broken then there is an issue with the environment that we are missing if it works then I am not sure why it is giving this error and something that should be address with the support team in a case.

 

If you exclude this hub, do the other hubs work?

rtirak

Jun 04, 2018 08:57 AM

Same result with HF3 seeing the same 500:

 

</head><body><h1>HTTP Status 500 - (80) Session error, Unable to open a client session for 10.1.156.254:48002: Connection timed out: connect</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>(80) Session error, Unable to open a client session for 10.1.156.254:48002: Connection timed out: connect</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>java.lang.RuntimeException: (80) Session error, Unable to open a client session for 10.1.156.254:48002: Connection timed out: connect

 

 

any other ideas? Or any thoughts on how I might be able to trace this down?

Jun 04, 2018 08:34 AM

The version we are using is webservices_rest Version 8.51 and we are UIM 8.5.1 SP1. I can install that HF3 and see what results I get since this works for you.

Jun 04, 2018 08:22 AM

rtirak

If the hub is online and is not timing out then it should work like all the other hubs in your environment. Where you run the script is not relevant as long at that location has access to the REST API. Once it connects to the REST interface all the calls are made based on how the API does them from that point on. It is a 500 error so what version of the webservices_rest package are you using? We are using 8.51hf3 with UIM 8.5.1 SP1.

Jun 04, 2018 07:19 AM

So what does not make any sense is, this hub is not actually down. It should be accessible without issue. When you say its not accessible with in my environment do you mean from where I am running the script? The hub in question is a secondary hub that I am able to see all the Robots on in the IM and have no issue managing them or it.

May 31, 2018 05:38 PM

rtirak

 

I think I got lost at profiles . However, I think the answer is no. To push the packages we are utilizing the deploy_package callback for the automated_deployment_engine probe. That only allows you to specify a package name based on what I can tell from the parameters (call backs are not documented that I know of). That said the sections defined in each package typically have instructions for different operating systems, setting dependencies or pushing configuration. In AC you cannot specify which section (I think this is what you are calling profile I had to look it up to get the right name myself) only the package it is the same in the code provided. 

May 31, 2018 04:01 PM

rtirak That hub is not accessible in your environment. Adding that hub to the exclude list will allow you to work around that issue until the hub is back online. You would not be able to push packages to it anyway.

May 31, 2018 02:37 PM

Alquin also I seem to be getting this error when I am running to push a package it seems my connection is timing out to one of my hubs. Any idea why that would be? Below is the output:

 

2018-05-31 14:25:30,427 - urllib3.connectionpool - _new_conn - DEBUG - Starting new HTTP connection (1): prdnimump01
2018-05-31 14:25:51,893 - urllib3.connectionpool - _make_request - DEBUG - http://prdnimump01:80 "GET /rest/hubs/NMS/hubNP1/robots?inCurrentDomain=true HTTP/1.1" 500 None
2018-05-31 14:25:51,893 - root - get_robots - DEBUG - Get robot response was <html><head><title>Apache Tomcat/7.0.72 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - (80) Session error, Unable to open a client session for 10.1.156.254:48002: Connection timed out: connect</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>(80) Session error, Unable to open a client session for <hubIP>:48002: Connection timed out: connect</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>java.lang.RuntimeException: (80) Session error, Unable to open a client session for <hubIP>:48002: Connection timed out: connect

 

2018-05-31 14:25:51,903 - root - get_robots - ERROR - Failed get robots

May 31, 2018 01:07 PM

Alquin

 

Is it possible to run a profile of the package that is pushed from the script? For example the NEXEC probe is one of the packages we are pushing to a robot. In that package there are 2 profiles configured. Am I able to start each one of them from the script as well?

May 31, 2018 09:06 AM

in the maintenancemode etc it was in the data file like this:

 

[data]
systems_list_file = systems.txt

May 31, 2018 09:03 AM

rtirak

 

Good questions. The hubs are simply separated by spaces I have updated the sample configuration file.

 

exclude = hub1 hub2 hub3

 

The file name containing the list of robot names is data\\targets.txt. I should move this to the configuration file it is currently hard coded which is a bad practice.

May 31, 2018 08:38 AM

Alquin

 

With the config file and the 'exclude' line. do we list the hubs with just a comma in between if we want to exclude multiple hubs. For Example if we have 4 hubs and we want to exclude 3 of the 4 would the line look something like this:

               

             exclude = HubPI,HubDEV,HubTST

 

 

Also in your examples which file is the file that holds the actual robot names that you want the package deployed to?

 

Thank

Related Entries and Links

No Related Resource entered.