DX Unified Infrastructure Management

Hi Everyone,

   We have recently transitioned new client and they have linux infrastructure. We have 1 client hub server and 9 linux robots. In IM everything seems to be working but we cannot create a group in USM as it is unable to find the hub server as well as any client robot servers. Weirdly, if we go to Performance reports we can see that they have stats. Also we configured switches in SNMPCollector probe and those are the only 2 devices which are showing in USM.

Have already raised a case with CA support until they get back thought i would ask in community. What is the issue here and how do we go about troubleshooting.

Any help is appreciated

Herin

I guess the robots are not registered in CM_COMPUTER_SYSTEM table . Do you see any of the robots listed if you run a query like the one below 

select * from CM_COMPUTER_SYSTEM where name like 'xyz'

I have seen this happening in certain versions of robot running on Linux and the updating the robot package fixed it in some instances.

Also try clearing the niscache on these robots and see if this helps 

Hi Phani,

    Thanks for your reply, We ran the query and couldnt find any robots listed. 

Can you please advise what does updating the robot package means because they are already have the latest robot 7.80 is there any other update we need.

and how would we clear the niscache on these robots ? 

Pretty new with nimsoft so trying to get my head around & apologies for too many questions 

Cheers

If you are already on 7.80, then you can ignore the robot update. Robot update usually means deploying the updated robot_update package to upgrade the robot to latest version if you are on old version 

One way to clean the niscache - connect to the server and..

1. Stop the nimbus service by executing: <Nimsoft Install>nimsoft/bin/niminit/stop. (usually /opt/nimsoft/bin/)
2. Locate the folder <Nimsoft Install>\Nimsoft\niscache\ (e.g. /opt/nimsoft/niscache)
3. Delete all the files found in this folder, leaving the empty "niscache" folder behind

4. Start the nimbus service

There is another way to do it remotely using the callbacks , but this usually dosent work that well (at least for me )

Knowledge Base Articles 

To know more about niscache..check out

Knowledge Base Articles 

Thanks Phani,

   Followed the 1st KB article to clean niscache remotely. how much time does it take to show up in USM. As a test have cleaned niscache from 3 servers

Not more than few minutes ...wait for around 15 min see if they show up in CM_COMPUTER_SYSTEM

Hi Phani,

    Waited for 45 min nothing has shown up under CM_COMPUTER_SYSTEM and other thoughts?

Ok, as I mentioned the remote call dosent always work for me that well..so I end up clearing the niscache by directly connecting to the user machine . Would it be possible for you to try that?

Also try to increase the log level of the discovery_agent running on that HUB and see if its actually picking up these robot information in the discovery_agent log files or if there are any errors noted in the log file 

Ctrl+Right click  "discovery_agent" and you should see "Raw Configure". Under setup folder increase the log level to 3 and see if write some additional info to logs 

Hi Phani,

   I have increased the log level to 3 in discovery agent probe. We dont have access to the client linux server so we have advised the client to do test on the hub server as well as on few robots to check if this works.

Will give you an update soon.

So we cleared the niscache folder directly from the server. It has been more than 1 hour but it still doesnt show up 

Ok..few other things to check ...

Hope the niscache was cleared as per the above process? - stop robot ...clear niscache folder...start robot..

Do you have the "probe_discovery" queue's configured on the customer HUB and also the proxy/tunnel HUB? There needs to be a "attach" queue on customer HUB and "get" queue on proxy/tunnel HUB

Configure Discovery Queues - CA Unified Infrastructure Management - 8.2 - CA Technologies Documentation 

Discovery User Guide 7.0 

Do you see any useful information written to discovery_agent log after you have reset the niscache?

Hi Phani,

   probe_discovery queue is configured on the customer hub as attach and proxy/tunnel hub as get queue

Discovery_agent log doesnt show anything after reset of niscache

Ok ..it seems to be getting bit tricky now 

as usual...lets check few other things 

What version of Linux are you using? For both the HUB and Robot?

If you search through the discovery_server log on the Primary HUB, do you see any information related to the new HUB showing up in the log file ? If you hardly see any info...you might need to increase the log level and check 

Do you see any similar messages as of below for your new hub in discovery_server log?

.
24 Nov 2016 16:19:17,088 [probeTask-6] TRACE com.nimsoft.discovery.server.agent.config.DAVersionMismatchNotifier - Discovery Agent /Domain/hub/hub_name/discovery_agent has valid version 8.41.

24 Nov 2016 16:19:18,392 [daConfigWorker-2] DEBUG com.nimsoft.discovery.server.agent.config.DiscoveryAgentConfigUpdater - Sending configuration to agent /Domain/hub/hub_name/discovery_agent (version 8.41)

24 Nov 2016 16:19:18,535 [nisCacheWorker-4] DEBUG com.nimsoft.discovery.server.nimbus.scan.NisCacheUpdater - No new nis cache elems for /Domain/hub/hub_name

Hi Phani,

    Linux version for hub servers and robots : Linux 3.10.0.327.22.2.el7x86_64.

In discovery server cannot find any information for the new hub server or robots. 

Hello.

Just to confirm - How it goes after restarting discovery_server probe ?

If above does not help, just to confirm - Is the robot accessible from the Primary HUB Server ?

Regards,

Yu

There could be something wrong with the discovery_agent on the customer HUB or the discovery_server. On the customer HUB open the HUB probe and click on the Status tab, under Subscribers/Queues tab do you see anything  Queued or Sent for probe_discovery queue?  Also check the Subjects tab

Similarly on the Tunnel HUB if you look at the status of probe_discovery queue do you see anything  Queued or Sent?

In addition to further debug - increase log level to 5 on discovery_agent probe on customer HUB and then pick one of the server and try the below. This time try the "_reset_device_id_and_restart" callback

1. Navigate to the robot via Infrastructure Manager and select the controller probe and press Ctrl-P. The probe utility window will pop up.
2. Click the Options Icon - (second icon from the right) Select the 'Expert Mode' checkbox, Click Ok.
3. Now, in the "Probe Commandset" select _nis_cache_clean and click the Green Arrow to send the command request.
4. Now, in the "Probe Commandset" select _reset_device_id_and_restart and then click the Green Arrow to send the command request.
5. The niscache is now clean and the robot is restarting.

while doing this take a look at the discovery_agent log  if it is picking up any info related to this robot and also monitor the probe_discovery queue if any information is being sent 

Hi Phani,

    We have attach queue configured for probe discovery on remote hub server and get queue configured on proxy/tunnel server and we can see sent messages numbers and I have checked it every 15 min and it changes so it means that it is working 

Discovery agent probe log is set to 5 but after following the niscache cleanup , discovery agent log is not showing any information

it does restart the robot but doesnt show anything in logs.

Better option would be to have support review the log files and see if there is any issue with the discovery_agent or the discovery_server probe . That would be the best way forward 

Hi Phani,

   I think the issue is resolved. CA support did check the logs and got me to disable to discovery agent and discovery server.

Then turn on Discovery agent and discovery server. They advised me to check it after 2-hours but it still didnt show anything at all. They advised discovery server logs only shows 1 linux robot. During the weekend i think after 48 hours or something i was able to see that all the linux servers are showing in USM

Finally i hope the issue is resolved. I am just doing a final check by running SQL query

Thanks for all your help. Appreciate it mate.

Cheers 

Good to know 

Cheers 

Hi, Herin.

Just to confirm - Is your robot (missing one) environment VM cloned off the other robot images ?

Hi Yu,

   No all the servers are physical servers and they are not cloned. 

Hi All,

Yes there is no SQL query for CM_COMPUTER_SYSTEM and not found from USM however those hubs and robots CDM QoS metrics show in PRD.

Those 3 hubs are connected to the primary by tunnel via the proxy.

Only 2 of hubs host names are detected from the discovery_server log not by none of  IP address of hubs.

I hope it helps for further discussion.