Make the SDGTW probe Only Auto Resolve ServiceNow Incident if the Status is set to a specific status, other then that leave it alone

[Robert Tirak]

What I would like if possible is a way so that the sdgtw probe has an incident it opened from an alarm, and the alarm clears that the sdgtw probe will check the incident status. If the incident status is still 'Active' then go ahead and close the incident.

If the status is anything else, leave the incident alone and do not auto close it even though the alarm cleared.

Has anyone done anything like this with the sdgtw probe and ServiceNow, or does anyone have any ideas on how I could go about it?

------------------------------
IT Administrator
------------------------------

[Kathy Maguire]

Robert -

What you are asking for is not possible with the current design of the sdgtw probe.  When the probe is configured to automatically close ServiceNow incidents when the corresponding UIM alarm is closed, the sdgtw probe does not first check the current status of the incident in ServiceNow, it just issues the SOAP request to set the status to the configured value (either Closed or Resolved).

My recommendation is to open an Enhancement Request by posting an idea to the Ideation site for the DX Infrastructure Management Category for Program Management to consider including in a future release of the sdgtw probe.

------------------------------
Kathy Maguire
Technical Support Engineer 4
Broadcom
------------------------------

[Robert Tirak]

Kathy,

Ok I understand it cant not be done within the actual probe. My question though is not limited to the probe. I would assume that this also could be accomplished via some type of Python, LUA script and auto operator. Possibly something like turning off the auto resolution completely and then re-creating the functionality of the auto resolution piece in a small script.

Possibly an auto operator that triggers the script on all clears.

- Then the script itself goes and checks to see if the alarm that created the red critical does have a ServiceNow Incident assigned to it (which it should)
- If it does then going and checking the current status of the specific incident. If the incident is still in the Active Status then Resolve the incident and close
- If the incident is in any other status, then disregard the incident leave it as it sits and close.


Would this be something that you would agree should be possible @Kathy Maguire , and do you think this would achieve what I am trying to do?

Again the biggest issue is with having the auto resolution on (which we want so that things that do clear and need no action the incident is resolved automatically) when something does need done or documented in the incident even though the alarm has cleared. Or the incident gets moved to a Status like "in progress" and someone is working on the incident, and out of nowhere it gets closed because the alarm has cleared while they are working on the issue.


​

------------------------------
IT Administrator
------------------------------

[Kathy Maguire]

Robert -

To implement your recommended solution, you would have to also close/resolve the incident in ServiceNow after you have identified that the current incident status is Active from the LUA script that you create for the AutoOperator profile you define for processing all alarms with a Clear severity.  You would also have to deactivate the auto-close incident option in the sdgtw probe.

The drawback to this is that if there are any delays in the LUA script (which can happen with all of the work you are asking this script to do), you will delay the processing of all UIM alarms.  The nas probe is single threaded and can only process one alarm at a time.

Another option would be to create a LUA script to parse each clear alarm to see if there is a ServiceNow incident associated with it, then write the pertinent information to identify the ServiceNow incident for the clear alarm to a file.  You could then create a profile in the logmon probe to read this file and execute a script to login to ServiceNow, check the status of the corresponding incident, and if the status is active set the status to closed/resolved.  By doing this, you assure that the nas will continue to process all alarms and any delays caused by the interaction with the ServiceNow Service Desk application will only delay the closing of ServiceNow incidents and will not delay UIM alarm processing.

------------------------------
Kathy Maguire
Technical Support Engineer 4
Broadcom
------------------------------

[Robert Tirak]

@Kathy Maguire ok that's makes sense. That sounds like a really good idea, and a better solution that you recommend.

question do you possibly have anyone else who has done something like this, that might have the LUA script that I could take a look at? I am def not a LUA pro. Or possibly have a LUA script that does something close to why we want to achieve that I could reference?​

Kathy you stated to parse the clear alarm. Will the clear alarm contain the info if an incident already exists for an alarm?

------------------------------
IT Administrator
------------------------------

[Kathy Maguire]

Robert -

You configure the sdgtw probe to use one of the alarm custom_N fields which the probe will populate with a string similar to the following when it creates an incident in ServiceNow:

<Service Desk connection name>$<Service Desk system id>$<alarm severity>$<incident number>

If you can connect to ServiceNow from a utility outside of UIM and execute queries with that utility to check the current status of an open incident then close/resolve the incident if the current status is set to "Active", then you should be able to use either the <SericeDesk system id> or <incident number> components of the custom_N field of the closed uim alarm.

Here is a simple stupid example of a LUA script that you can execute for all of your alarms with a clear severity (this example assumes that you have the sdgtw probe configured using the custom_4 UIM alarm field for the incident information):


local function isempty(s)
return s == nil or s == ''
end

a = alarm.get()

if a ~= nil then
   if not isempty(a.custom_4) then
      out = "C:\\ServiceNow\\"..a.nimid
      buf = a.nimid.." : "..a.custom_4
      file.create(out,buf);
   end
end

This will check the custom_4 field of the current alarm and the field is not empty, will created a file in the C:\ServiceNiw directory using the nimid for the name of the file with contents similar to the following:

PV43692809-89786 : ServiceNow$asdwj2321asdavaw3rav$critical$INC0020456

The profile you generate in the logmon probe will watch for any file in this directory then execute a script to read the contents of the file parsing out the incident information, then use the utility you have which make the connection to ServiceNow, checks the status of the incidentm and close/resolve the incident if the status is still set to Active.  The script can then delete the file from the C:\ServiceNow directory to prevent filling your file system with these files.

------------------------------
Kathy Maguire
Technical Support Engineer 4
Broadcom
------------------------------

[Robert Tirak]

@Kathy Maguire we do currently have the custom_N fields setup to populate with that information from the SDGTW. Thank you very much for the example LUIA, I will attempt to get this together this morning in our development environment and report back here anything that comes up as well as my outcome. Stay tuned, and thanks for the help.​

------------------------------
IT Administrator
------------------------------

[Robert Tirak]

@Kathy Maguire for the LUA script I should be triggering for each and every Clear is that correct? So I would be taking the LUA example you sent and then triggering that on every clear we would get correct? ​

------------------------------
IT Administrator
------------------------------

[Kathy Maguire]

Robert -

You are correct.  You would create a script AO Profile that only matches on alarms with a clear severity.  Then execute your LUA script to parse out the contents of the custom_N field you have configured in the sdgtw probe to a file that can be read by the logmon probe.  Only alarm close messages from UIM alarms that had incidents created in ServiceNow should have anything in that custom_N field, all other clear alarms will be ignored.

------------------------------
Kathy Maguire
Technical Support Engineer 4
Broadcom
------------------------------

[Robert Tirak]

@Kathy Maguire One question, will it be a problem instead of in your LUA using a file name of the alarm ID of just making the file name actually static, like incidents.txt or something like that? Or will that cause a problem?

Also I have been looking and saw that you had said the logmon probe, however I was wondering did you mean the dirscan probe to watch the file location for any file that gets created? OR no did you mean the logmon probe?

------------------------------
IT Administrator
------------------------------

[Kathy Maguire]

Robert -

You could design a LUA script to write each alarm custmon_N entry into the same file, but unless you also add logic to clean up the entries that you have processed from that file, it will continue to grow over time.  In that case, you have to be make sure that you only remove the lines after you have successfully processed the corresponding incident in ServiceNow.

I did mean the logmon probe since this is the only probe that you can configure to monitor a file, then execute a script when your profile finds a match in the file that is being monitored.  You should be able to extract the ServiceNow incident information from the UIM alarm custom_N field found in the monitored log file and provide that information as an argument to a script that you will have to write which will connect you to the appropriate ServiceNow instance, check the status field of the incident and then close/resolve the incident if the status is Active.  To prevent the file system from filling up, this script will then have to do clean up of the monitored log file to remove the entry that was processed.  This last part was the reason that I provided a LUA script that wrote the UIM alarms to individual files - it is easier to delete a file than to parse a file for a specific line then delete that line.

------------------------------
Kathy Maguire
Technical Support Engineer 4
Broadcom
------------------------------

[Robert Tirak]

@Kathy Maguire

Ok I understand what your saying. Yes I see the LUA script you provided is creating a file for each alarm and the name of the file is the alarmID. My questions is but writing each one to there own file, how are you telling the logmon which one is the correct one to be looking at each time? The logmon probe does not have the ability to look at the directory and see when there is a file in it scan the file.

So how are you proposing that I am able to tell the logmon probe which file to scan each time?  The logmon probe is watching a specific file like a log file. Which is the same file with a bunch of different entries in it, however what you seems to be explaining and have given me is one file, for each alarm id with the incident inside. So How is it possible to configure the logmon probe to look for when a directory has a new file in it and then open the file and scan? Understand what I am saying?​

------------------------------
IT Administrator
------------------------------

[Kathy Maguire]

Robert -

Assuming that your LUA script writes all of these files to the same directory (for example C:\ClosedAlarms) and the files all have the same extension (for example .txt), then in the profile that you create in logmon, you would specify the following for the file pathname:

C:\ClosedAlarms\*.txt

Make sure that you use the Updates mode (and not cat) so that the logmon probe will only process the one line the LUA script adds to the file one time and not on every monitoring interval.  Then the logmon probe will scan all files in the C:\ClosedAlarms directory with the .txt file extension and process each one that exists in the directory on the next scan interval.  You can then set up a match string in the profile similar to the following:

/^(.*)\s:\s.*\$(.*)\$.*\$(.*)$/

From this you can setup variables for the 3 match patterns - these are the fields delimited by (.*) in the match pattern.
The first variable will contain the nimID, the second the ServiceNow incident system ID, and the third the ServiceNow incident number.  See the logmon Use Case Examples -> Configure Custom Variable in Watcher Rule docops page for setting up these custom variables:

You can then pass these 3 variables as arguments to the script that you configure to connect to ServiceNow, check the status of the corresponding incident and close/resolve that incident if the status is Active, then delete the file from the c:\CLosedAlarms directory for the nimId.

------------------------------
Kathy Maguire
Technical Support Engineer 4
Broadcom
------------------------------

[Robert Tirak]

@Kathy Maguire I just wanted to update this, sorry it took me a little longer then expected. I have successfully implmented what you have suggested along with a python script to interact with Service now and handle the Auto resolution of incidents rather then using that feature in the sdgtw probe.

I appreciate your help and collaboration in getting this achieved. It is very much possible, and the ability to create these rules now opens up a larger ability with the process we have in place currently. Thanks for you help. ​

Just as a Summary of the above, in what was done to accomplish this:

1. Basic LUA Script that is triggered by Auto Operator on all clears, examining if the incident was created from alarm that the clear is acting on

2. Logmon probe set to watch and pull incident and alarm info from text file when created by above LUA.

3. Python script (utilizing pysnow library) to check the incident status and make proper updates/closure to the incident if it meets requirements.



------------------------------
IT Administrator
------------------------------