DX Infrastructure Management

Expand all | Collapse all

Custom Probe - Probe Configuration Archive

  • 1.  Custom Probe - Probe Configuration Archive

    Posted 07-07-2014 10:35 PM
      |   view attached

    This is a custom java probe I have written to help alleviate some probe configuration issues.  Attached is the probe package itself (just drag to the archive), an information document and a basic .jrxml file that can be imported into Unified Reports. This will be an ongoing development process so please provide your feedback here in this thread with issues and feature requests and I will do my best to address them in future versions.

     

     

    Click Here to Download Probe (Working Link)

     

     

    Probe Features

     

    This probe will archive every probe configuration in your Nimsoft environment on a customizable interval and automatically do configuration differentials between the current and previous configurations. This will allow a Nimsoft administrator to track configuration changes in their environment automatically.

     

    TECHNICAL INFORMATION

     

    On startup the probe will talk to the data_engine at the path you specified to detect if you have the required database tables and if not it will create them in the Nimsoft database for you. These two tables are PCA_CFG and PCA_CHANGES.

     

    PCA_CFG TABLE

     

    This table contains a record of each probe configuration. It stores the hub, robot, probe, probe version, filesystem path to configuration file, complete configuration file in a column, a record id and archive date.

     

    PCA_CHANGES TABLE

     

    This table contains a record of each probe configuration change. It stores the hub, robot, probe, configuration section, configuration key, previous and current configuration value, previous and current archive date and previous and current record ids.

     

    HOW IT WORKS

     

    The probe uses the native callbacks provided in many probes to get the probe configurations.

    1. It uses the gethubs callback on the hub probe to get a list of available hubs
    2. Loops through each of those hubs to get a list of robots using the hub’s getrobots callback
    3. Grabs a list of probes and information on each probe from the controller via the probe_list callback
    4. Saves the configuration file from the robot by using the text_file_get callback
    5. Inserts information about the configuration into the database
    6. Compares the current and previous configuration to determine if there were any changes
    7. If there are changes, writes those changes to the database

    NOTES

     

    • As this is the first probe I have ever written in Java there may be some verbose logging that will be removed in future versions.

    Attachment(s)



  • 2.  Re: Custom Probe - Probe Configuration Archive

    Posted 07-07-2014 11:34 PM
    This looks very promising and I know many customers who have asked for something like this.


  • 3.  Re: Custom Probe - Probe Configuration Archive

    Posted 07-07-2014 11:36 PM

    BUG LIST

     

    • Need to change the HUB address to just hub name in PCA_CHANGES
    • Turn down logging


  • 4.  Re: Custom Probe - Probe Configuration Archive

    Posted 07-08-2014 10:15 AM

    Hi Bryan,

     

    The package seems to be broken and can't be imported to archive

     

    -jon



  • 5.  Re: Custom Probe - Probe Configuration Archive

    Posted 07-08-2014 10:53 AM

    Seems Google modified the probe package somehow, I have changed the original link and included the new link in this post.

     

    Download Probe

     

     



  • 6.  Re: Custom Probe - Probe Configuration Archive

    Posted 11-19-2014 04:16 PM

    Hi

    I have deploy the probe on my primary test machine , but I recieved next message in the log file :

     

    Nov 19 14:13:50:380 [main, probe_configuration_archive] ****************[ Starting ]****************
    Nov 19 14:13:50:380 [main, probe_configuration_archive] Version 1.0
    Nov 19 14:13:50:380 [main, probe_configuration_archive] CA Technologies
    Nov 19 14:13:50:392 [main, probe_configuration_archive] port=48034
    Nov 19 14:13:50:431 [main, probe_configuration_archive] Login to NimBUS is OK
    Nov 19 14:13:51:432 [main, probe_configuration_archive] initProbe()
    Nov 19 14:13:51:432 [main, probe_configuration_archive] Setting callbacktimer
    Nov 19 14:13:51:432 [main, probe_configuration_archive] Gathering Configurations
    Nov 19 14:13:52:290 [main, probe_configuration_archive] Create PCA_CFG table.
    Nov 19 14:13:52:290 [main, probe_configuration_archive] Error when running on timer method in main thread. Reason is (1) error, Not able to callback for timer cbOnInit.1000. Reason is java.lang.NullPointerException

     

     

    Have you a idee of the problem



  • 7.  Re: Custom Probe - Probe Configuration Archive

    Posted 11-19-2014 04:21 PM

    Can you look into the NIS database and look to see if the PCA_CFG table was created?



  • 8.  Re: Custom Probe - Probe Configuration Archive

    Posted 11-19-2014 04:34 PM

    HI,

    Table is not created .

    What is exactly the archive-directory ?

     

    Joel

     

     



  • 9.  Re: Custom Probe - Probe Configuration Archive

    Posted 11-19-2014 04:37 PM

    What database are you using, MSSQL, MYSQL or Oracle? I've only tested it on MSSQL and may need to make some adjustments for the others.

     

    The archive-directory is the directory on your robot you want the configuration files copied to.



  • 10.  Re: Custom Probe - Probe Configuration Archive

    Posted 11-19-2014 04:44 PM

    I using SQL2008r2.

     



  • 11.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-16-2015 02:36 PM

    Bryan:

    I am getting a connection refused trying to download the probe.



  • 12.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-16-2015 09:17 PM

    I have switched hosting and cannot edit the original post, the new download location is http://192.99.166.70/nimsoft/probe_configuration_archive_1.0.zip



  • 13.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-17-2015 12:15 PM

    Thanks Bryan.  I noticed you had originally tested with MSSQL and I am wondering if this could be used for Oracle.  I tried deploying and got the Java Null Pointer exception mentioned in this thread.  Would it be enough just to create the two tables manually in Oracle or is there other queries that would need to be modified?



  • 14.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-17-2015 12:18 PM

    I would have to add the Oracle thin libraries and make some modifications on the connectivity. I am currently working on a project that connects to Oracle so adding this to the probe shouldn't be that difficult. Give me a few days and I'll try and get a new version up.



  • 15.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-20-2015 12:51 PM

    I'm also getting the NullPointerException right after I start the probe. Anyone getting this to work without the error.  I used the http://192.99.166.70/nimsoft/probe_configuration_archive_1.0.ziphttp://192.99.166.70/nimsoft/probe_configuration_archive_1.0.zip download link above.Any workaround or suggestions on what needs to be done to fix.  My database is Microsoft SQL2008R2 and my security is Windows Authentication.

     

    Mar 20 10:43:40:439 [main, probe_configuration_archive] log level=3

    Mar 20 10:43:40:439 [main, probe_configuration_archive] file retention=30

    Mar 20 10:43:40:439 [main, probe_configuration_archive] table retention=90

    Mar 20 10:43:40:439 [main, probe_configuration_archive] Setting probe archive directory at : C:\probe_archive

    Mar 20 10:43:40:439 [main, probe_configuration_archive] Ignoring probes: [cdm, spooler, hdb, snmptd, qos_engine, qos_processor, mpse, ppm, alarm_enrichment, remote_control]

    Mar 20 10:43:40:440 [main, probe_configuration_archive] ****************[ Starting ]****************

    Mar 20 10:43:40:440 [main, probe_configuration_archive] Version 1.0

    Mar 20 10:43:40:440 [main, probe_configuration_archive] CA Technologies

    Mar 20 10:43:40:448 [main, probe_configuration_archive] port=48044

    Mar 20 10:43:40:483 [main, probe_configuration_archive] Login to NimBUS is OK

    Mar 20 10:43:41:485 [main, probe_configuration_archive] initProbe()

    Mar 20 10:43:41:485 [main, probe_configuration_archive] Setting callbacktimer

    Mar 20 10:43:41:485 [main, probe_configuration_archive] Gathering Configurations

    Mar 20 10:43:42:094 [main, probe_configuration_archive] Create PCA_CFG table.

    Mar 20 10:43:42:098 [main, probe_configuration_archive] Error when running on timer method in main thread. Reason is (1) error, Not able to callback for timer cbOnInit.1000. Reason is java.lang.NullPointerException



  • 16.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-20-2015 03:35 PM

    This probe was one of my earliest Java creations, there may be some significant issues with the database connectivity. I will do my best to produce a new version with some additional debugging early next week.



  • 17.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-20-2015 04:55 PM

    I'm just excited at the thought of capturing all the configs. Thanks for putting it out there for all to use.



  • 18.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-23-2015 10:07 AM

    What are the differences in this and the audit probe?



  • 19.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-23-2015 10:13 AM

    This will archive the entire .CFG file and then run a diff against the previous configuration. A lot of people have requested the ability to easily archive their configurations, this was just my attempt at that while adding the diff feature as well.



  • 20.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-23-2015 10:32 AM

    Here is a new version of the probe, I just tested it with MSSQL 2012 and UIM 8.2 beta.

     

    http://192.99.166.70/nimsoft/probe_configuration_archive_1.02.zip

     

    If this doesn't work for you please let me know.

     

    Bryan



  • 21.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-23-2015 12:06 PM

    Bryan, thanks for updating the probe. Looks like my issue is the same. I see it shows it's the updated version in the log but I'm still getting the NullPointerException when it trys to create the database. Any chance it has something to do with Microsoft Credentials vs SQL credentials?

     

    Mar 20 10:52:08:101 [main, probe_configuration_archive] ****************[ Stopped ]****************

    Mar 23 09:58:24:715 [main, probe_configuration_archive] data_engine path = data_engine

    Mar 23 09:58:24:715 [main, probe_configuration_archive] log level=3

    Mar 23 09:58:24:715 [main, probe_configuration_archive] file retention=30

    Mar 23 09:58:24:715 [main, probe_configuration_archive] table retention=90

    Mar 23 09:58:24:715 [main, probe_configuration_archive] Setting probe archive directory at : C:\probe_archive

    Mar 23 09:58:24:715 [main, probe_configuration_archive] Ignoring probes: [cdm, spooler, hdb, snmptd, qos_engine, qos_processor, mpse, ppm, alarm_enrichment, remote_control]

    Mar 23 09:58:24:716 [main, probe_configuration_archive] ****************[ Starting ]****************

    Mar 23 09:58:24:716 [main, probe_configuration_archive] Version 1.02

    Mar 23 09:58:24:716 [main, probe_configuration_archive] CA Technologies

    Mar 23 09:58:24:728 [main, probe_configuration_archive] port=48044

    Mar 23 09:58:24:761 [main, probe_configuration_archive] Login to NimBUS is OK

    Mar 23 09:58:25:764 [main, probe_configuration_archive] initProbe()

    Mar 23 09:58:25:764 [main, probe_configuration_archive] Setting callbacktimer

    Mar 23 09:58:25:765 [main, probe_configuration_archive] Gathering Configurations, running inital GetConfigs method

    Mar 23 09:58:25:776 [main, probe_configuration_archive] Database Driver: MS SQL Server

    Mar 23 09:58:25:792 [main, probe_configuration_archive] jdbc:sqlserver://wmstsql04.cs.ball.com:1433;databaseName=NimsoftSLM

    Mar 23 09:58:27:668 [main, probe_configuration_archive] Create PCA_CFG table.

    Mar 23 09:58:27:669 [main, probe_configuration_archive] Error when running on timer method in main thread. Reason is (1) error, Not able to callback for timer cbOnInit.1000. Reason is java.lang.NullPointerException



  • 22.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-23-2015 12:09 PM

    It might, I'm just grabbing the data_engine configuration information. Does the machine you're running the probe on have direct access to the database?



  • 23.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-23-2015 12:19 PM

    I'm running the Probe Configuration Archive on the Primary Hub, but my credentials to the database are required to be NT domain credentials rather than SQL credentials and the SQL database is on another server WMSTSQL04 rather than the HUB server. Have you had any more interaction with joel.ingels above? He reported the same issue.



  • 24.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-23-2015 12:23 PM

    I will test with Windows credentials and see if that is indeed the issue.



  • 25.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-23-2015 01:16 PM

    It is definitely the Windows Integrated Authentication. I will have to do some testing to see exactly how to accomplish the connectivity. Hope to have something before the end of the day.



  • 26.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-24-2015 10:57 AM

    Ok, I have made some changes to the probe and updated the sqljdbc libraries to the latest version. Integrated security should now work. You will need to put the full path to your data_engine in the configuration file, even if its on the Primary Hub (/domain/hub/robot/data_engine). The probe assumes you are running a 64 bit operating system, if you happen to be using a 32 bit OS you will need to go into the sdk/auth/x86 folder and rename the .dll file (just remove .rename). Then in the probe EDIT not configuration, change the java.library.path to x86 instead of x64.

     

    Here is the download link again

     

    http://192.99.166.70/nimsoft/probe_configuration_archive_1.02.zip

     

    If you have any questions or problems you can post here or email me at bryan.morrow@ca.com

     

    Bryan



  • 27.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-24-2015 03:57 PM

    Okay, works like a charm. Thanks for all the hard work.  Looks like Windows Authentication now works.

     

    Danny



  • 28.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-24-2015 09:01 PM

    Glad that it works, please make sure it runs again after 24 hours and let me know.



  • 29.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-27-2015 02:51 PM

    Thank you Bryan! This works great.

    Question:

    1. How do you import the Unified Report file to UR? I'm not that familiar with UR. Where do I find this report w/in the tree structure w/in UR?
    2. Also How can I figure out why this might not be backing up all robots under a specific node? Under one hub have 200+ robots. The probe is only creating 67 of the 200. Any ideas?

    Thank you,

    Daniel



  • 30.  Re: Custom Probe - Probe Configuration Archive

    Posted 03-31-2015 12:35 PM

    Just added to the thread. To add the UR report for this probe do the following. I am doing this thru the UMP - Unified Reports Portlet.

    1. Open and log into UMP.

    2. Reports - Unified Reporter

    3. Expand Public - Nimsoft Monitor

    4. You can r-click and Add Folder to create a "Probe Configuration Report"

    5. Then select the new folder and r-click Add Resource > JasperReport

    6. In the "Set Up" Give this report a name:  Robot Probe Config Report.

    7. In the "Locate the JRXML File: Upload a Local File" hit the Browse button and find the probe_configuration_report.jrxml

    8. Select DataSource, and "Select data source from repository" then choose the Nimsoft, UIM, NimsoftSLM DB that should of previously created when setting up Unified Reports.

    9. Hit Submit when done.



  • 31.  Re: Custom Probe - Probe Configuration Archive

    Posted 04-20-2015 11:40 AM

    Hi Bryan,

     

    Any chance to have a version for MySQL as well? It would fit perfectly to one of my customers (MSP) who is trying to reinvent this wheel by himself.

     

    Thanks,

    Fernando



  • 32.  Re: Custom Probe - Probe Configuration Archive

    Posted 04-20-2015 11:43 AM

    This should work with MYSQL at the current version of 1.02 provided above.



  • 33.  Re: Custom Probe - Probe Configuration Archive

    Posted 09-03-2015 12:05 PM

    Bryan, this looks good, does it require a license key?

     

    Thank you



  • 34.  Re: Custom Probe - Probe Configuration Archive

    Posted 09-03-2015 12:16 PM

    This version should not, if you grab it from the Marketplace it does.



  • 35.  Re: Custom Probe - Probe Configuration Archive

    Posted 12-02-2015 11:59 AM

    Having an issue with the report coming back empty, with no data. everything looks good in the log file for the probe configuration archive and it is running. Any ideas?



  • 36.  Re: Custom Probe - Probe Configuration Archive

    Posted 12-04-2015 12:04 PM

    It has been a while since I have used this probe so please forgive me if I am wrong. I believe the report shows the configuration changes that was made. So if you have not changed any configuration since you have added the probe the report will come back blank. It wasn't until I started changing probe configurations that I saw the report populate with data. Have you made any configuration changes since configuring the probe?



  • 37.  Re: Custom Probe - Probe Configuration Archive

    Posted 12-17-2015 03:39 PM

    This is great, thanks!

     

    We have it running in our test environment but a little concerned moving to PROD.

     

    I've noticed:

    1) It's not honoring the time schedule option.  For instance, setting time to 300 does not do the check every 5 minutes.  I assume it keeps the full 24 hour schedule.

    2) Every time the probe restarts, it'll redo any configuration grab and rewrite the full config file regardless if there are any changes. Is this a bug or feature?

     

    Feature request for a large environment:

    1) It would be nice to limit the scope of which hubs it queries, that way we could run these on individual hubs (which could be geographically far away).  Whitelist + Blacklist for hub names would be amazing.

    2) Specify time of day to actually run if possible, or only run on Sunday, etc.

     

    Thanks again.

    Erik



  • 38.  Re: Custom Probe - Probe Configuration Archive

    Posted 01-22-2016 09:13 AM

    I was able to get it to run and then run a report from the UMP. But it isn't including all of the probes that I have deployed. I can see the data that it grabbed and saved to a flat file but since we didn't change any of the configurations, it isn't pulling the (only set up once) probes into the report. I tried running the probe again and since I haven't changed anything, it is still the same. Any ideas?

    thanks!



  • 39.  Re: Custom Probe - Probe Configuration Archive

    Posted 04-19-2016 08:38 AM

    Hi Bryan,

     

    I just want to say that you have done a really great job.

     

    I was wondering, I am running the probe in my Primary Hub which is separated from the DB server, I am using SQL credentials but I am getting the error below:

     

    Apr 19 13:31:47:905 [main, probe_configuration_archive] ****************[ Starting ]****************

    Apr 19 13:31:47:905 [main, probe_configuration_archive] Version 1.02

    Apr 19 13:31:47:905 [main, probe_configuration_archive] CA Technologies

    Apr 19 13:31:47:913 [main, probe_configuration_archive] port=48024

    Apr 19 13:31:47:939 [main, probe_configuration_archive] Login to NimBUS is OK

    Apr 19 13:31:48:941 [main, probe_configuration_archive] initProbe()

    Apr 19 13:31:48:941 [main, probe_configuration_archive] Setting callbacktimer

    Apr 19 13:31:48:941 [main, probe_configuration_archive] Gathering Configurations, running inital GetConfigs method

    Apr 19 13:31:48:942 [main, probe_configuration_archive] Controller address for primary hub: /CIB-UK/CIB-UK-711-0001/LONS00112372/controller

    Apr 19 13:31:49:955 [main, probe_configuration_archive] Database Driver: MS SQL Server  SQL Authentication

    Apr 19 13:31:50:961 [main, probe_configuration_archive] jdbc:sqlserver://LONC00112148H:1433;databaseName=CA_UIM

    Apr 19 13:32:05:795 [main, probe_configuration_archive] Create PCA_CFG table.

    Apr 19 13:32:05:797 [main, probe_configuration_archive] Error when running on timer method in main thread. Reason is (1) error, Not able to callback for timer cbOnInit.1000. Reason is java.lang.NullPointerException

     

    As you can see, I am using SQL credential, however it does not work. Anybody has any idea about it?

     

    I would like to use this probe because seems to be really useful.

     

    Thank you very much in advance



  • 40.  Re: Custom Probe - Probe Configuration Archive

    Posted 04-22-2016 12:04 PM

    I am having a similar problem as to some in this thread, I've created a separate thread: probe_configuration_archive  Using this on 8.4 and not sure what the fix may be at this point as turning up the loglevel doesn't seem to shed any light on the issue.



  • 41.  Re: Custom Probe - Probe Configuration Archive

    Posted 08-29-2016 11:16 AM

    This is an old version of the probe, please use the latest release that includes an internal H2 database and multi-threading.

     

    Configuration Archive - A Custom Multi-threaded UIM Probe