Tech Tip: robot/probe version checker and report tool

[Luc Christiaens]

If there is something common between all clients I visited in the past it is the difficulty they had to have a clear view on what probes/packages and their versions that were installed in their environment.
Via the good old Infrastructure Manager we could drag and drop a new version of a probe, but it was not always easy to see/check the total picture/results.
---
The attached tool, available in Perl source and compiled format, will try to get the highest available version from your LOCAL archive and compare this with the information available from the local, on each robot, stored information.
Input:
To help you create specific target reports you have several:
- sql filter parameters: -lo, -lh, -lr and -lp
- on the output of the created sql query you can apply regex parameters where you can filter on: origin, hub, robot, probe, os_major_os_minor, user_tag1 and user_tag2

Output:
- html report with in red all robot/probe that have lower versions installed
- PU txt file with commands ready to be executed to upgrade the probes (they are NEVER executed/deployed/upgraded automatically)


This example was run with parameters -ex"n"
(-ex"n" is to report also the probes that have a correct version installed)
In the PU txt file you can find:

Doc file is also included in package/attachment
Remarks, suggestions, problems and questions are very welcome.

In version 2.1, takes a totally other approach than previous versions.  
Initially we started, in 1.0, from a list of probes installed on robots coming from cm_nimbus_probe, but this had the limitation that we didn't see/report all installed non-probe packages. (java, app_disco_, c++ distrib,..)

Version 2.0:
- Via the 4 available SQL selection parameters we create a list of servers 
- for each selected server we issue a controller callback: inst_list_summary
- on this packages list we will apply the regex filters and create a report and when needed PU commands (in a txt file)
Version 2.3:
- some additional code to recognize test fixes and hot fixes
- -dp: delete package parameter, this make it possible to remove, in a controlled way, packages in your environment
- nimsoft_generic.pm can now be located in the same directory as the tool or in perl/lib

This latest version is posted lower (nimsoft_check_packages_version_v2.3.zip)

#uim #package #check #tool #perl #commandline #report #version #installed  ​​​​​​​​​​

Attachment(s)

nimsoft_check_probe_version_1.0.zip   27.01 MB 1 version

[Garin Walsh]

Very nice.

I invented a similar wheel a couple months ago when I discovered a significant inconsistency that occurs in the installed version information. There are several sources to identify version (_status callback, get_probes callback, getrobots, local log files, vesions.txt, installed.pkg, cm_nimbus_robot, cm_nimbus_probe, etc.). And, while Nimsoft isn't an outlier in this regard, any time you have many places the same information is stored it is bound to be wrong at times. I have observed (not proven but observed) three events that confound this - if you downgrade a probe install the version information will get out of sync, if you install a package with a dependency and the install of the dependency fails, the version of the dependency will sometimes be recorded as a successful install, and if you have a package with multiple tabs and the first tab is successful and a subsequent tab is not then the package will get recorded as a successful install.

Because of this, in my tool, I added the additional check where for each installed probe I issued the _status callback and compared that version (the one returned by the running probe) to what discovery_server populated in cm_nimbus_probe. 

The results were a little disturbing as I found, across about 7k robots, about 250 cases where the installed version inventory value didn't match the running probe.

Might not be a big deal for some but consider the log4j issue recently where you have to identify the systems that have a probe version with an exploit and you have to ask the question about generating the list of systems that are vulnerable - is a one in 300 probes (250 failures in 7,000 robots with an average of 10 probes) or 1 in 30 affected robots error in the list acceptable? Probably not.

Fixing the inconsistency is a **** matter of redeploying the mismatched probe again (except in the downgrade scenario - there it was necessary to install the latest version).

Back to the tool, a question, how do you deal with version strings like 9.34HF1? Most places in Nimsoft, this data is stored as a number and so the fix number is lost.

[Luc Christiaens]

Hi Garin,
Thanks for your valuable input.
- adding a callback, like probe_list to a controller, seems to return the correct version info with HF included. Very good tip, I will add this 
- for the version info from local archive I was/am struggling with this HF info because the build number is not following correctly incremental (sometimes it does and for other probes it doesn't)
Therefor I opened an idea.
But this morning i had a new idea to solve this problem.  When replacing version 9.32hf2 by 9.321 and 9.32_hf3 by 9.323 the version to deploy would be correct. (now i need only to add the callback to obtain the correct installed version)
So I hope to have soon an updated version with the extra callback + document better the (strange) logic in obtaining probe/package versions

[Guillaume Briere]

Hello,
We've created a grafana dashboard with that information

You select the HUB, then robot and it shows you the probe installed and their version.

We also present the robot version with graph

You could also modify the query to select a specific probe and it could you show the same graph but for the cdm probe instead of the robot.

You can filter on every columns and export it as csv if you want.

We use the /uimapi endpoint to build the dashboard.

Thanks

[Guillaume Briere]

Running curl -X GET "https://<operatorconsole>/uimapi/archive/list" -H "accept: application/json" will list every packages inside the archive so you can compare it with the data from the result of running curl -X GET "https://<operatorconsole>/uimapi/hubs/<domain>/<hub>/<robot> -H "accept: application/json"

Or build a grafana dashboard that do that for you.

Thanks

[Luc Christiaens]

Hi Guillaume,
Is this something that you can package so that we can install this?
Seems to be very cool.

[Guillaume Briere]

Hello,
I wrote a post about grafana and UIM a while ago: https://community.broadcom.com/enterprisesoftware/communities/community-home/digestviewer/viewthread?GroupId=1315&MessageKey=f65f96f7-4a22-499d-9fb3-5ff17d38eaa6&CommunityKey=170eb4e5-a593-4af2-ad1d-f7655e31513b&tab=digestviewer

There's all the information needed to start using it.

[Luc Christiaens]

Thanks to remarks from Garin there is already a version 1.1
- new option -cc"y" will add an extra callback list_probe to the robot controller to obtain the installed version
- when running with -cc"y" and the target robot is not responding no job_add will be created for that robot/probe
- modified logic to detect (or at least trying to) Hot Fixes in the probe name (_hf, -hf and hf)

Attachment(s)

nimsoft_check_probe_version_1.1.zip   27.02 MB 1 version