Dashboard Charts stop plotting

[Anon Anon]

Hi,

 

We are running the following config: UMP v2.5.1, Nimsoft v5.1.1 running on virtualization/Windows Server 2008 R2 - 1 server for IM, 1 server for database, 1 server for UMP.

 

I have noticed that after some time, the plotting of the graph on the charts stop and it seems that there is no data to be plotted. I notice this happening to QoS type data be it charts or counters. The QoS data is visible in the SLM, however.

 

Anyone come across this situation before ? Please advise.

 

Thanks.

 

Han

[Anon Anon]

Can you give more detail on which charts and counters stopped displaying data? If it's dynamic views, try to "rebuild" the dashboard by clicking on the Rebuild button next to the Mini Map at the top right. Or is it custom dashboards?

[Anon Anon]

These are the chart widgets. It's a dashboard for a cisco router. I don't see the "Rebuld" option beside the Mini Map - am I missing something ?

 

Refer to my attachment. When we logon to the dashboard see the chart with data for some 2 hours + in history, then it goes on for 5 to 10 minutes and stops plotting.

[Anon Anon]

Just some feedback...

 

I see this issue with charts on custom dashboards. Sometimes they don't populate on the first load of the web page. It takes a refresh or two before they show up. I observed this on our old in-house instance and even with our  new NoD instance. I have also observed the 'gaps' that you refer to in the charts once they have been up for a while, a refresh clears this up though.

 

I added more memory in the WASP probe under Java startup parameters (from 256/512 up to  512/1024) and that seemed to help, but it does not resolve it completely for us. I don’t know if adding the additional memory is a fix for the issue or just a coincidence.

[Anon Anon]

We are having the same issue with custom dashboards (on UMP 2.5.1, NMS 5.11).  I fear this may be an issue with the entire 2.5.1 UMP version - We have seen the issue in upgraded deployments (from 2.1.2) as well as completely new, clean lab-environments.  Every instance where we're using the 2.5.1 version of the portal does not update custom dashboards after their initial load.

 

Scenario 1:

• Custom dashboard containing 1 chart (over 1 hour interval), tied to QoS data from CDM (CPU Usage)
• cdm probe's CPU Usage polling interval is set to 10 seconds
• Data is confirmed (via SLM) to be updating every 15 sec, and inserted into the database very shortly after being sent/received (e.g. difference between insert time and sample time is <5 sec)
• Dashboard chart remains static - No scrolling, no updates, no changes whatsoever

Scenario 2:

• Custom dashboard containing 1 chart (over 1 hour interval), tied to QoS data from CDM (CPU Usage)
• Dashboard also contains a text object ($VAR), linked to NIS datasource using query:  SELECT GETDATE()
• cdm probe's CPU Usage polling interval is set to 10 seconds
• Data is confirmed (via SLM) to be updating every 15 sec, and inserted into the database very shortly after being sent/received (e.g. difference between insert time and sample time is <5 sec)
• Text object on dashboard updates at expected interval (60 sec, as set in dashboard_engine) - Query is obviously running, and text updates as expected
• Dashboard chart remains *scrolls* (X-axis time increases over time), but no new data is displayed. The initial data points just move further left over time, with empty space as it scrolls.

At the same time, even though the dashboard charts do not display any new data, SLM can be refreshed and new data points are readily seen.

 

Tried:

• Changing the update intervals in dashboard_engine (both smaller and larger for "Client data update", "NIS data retrieval", and "Dashboard NIS queries"). Text object updates more or less frequently (depending on interval), but no effect on charts.
• Performance tuning:  64-bit JDK installed and configured in controller, wasp & dashboard_engine tuned to 2GB min 6GB max each, addional PermGen space granted to dashboard_engine (2GB max) - This is a 20GB machine so this RAM is available to allocate, although it never appears to approach these limits.

Opening a support ticket - will post back with any progress...

[Anon Anon]

Hi,

 

Any luck with this ?

[Anon Anon]

I am just wondering whether no one else if facing this problem ?

[Louis Savage]

I am seeing the same issue as well. Also put in a case about this. With the lates ump and nms installs.

[Anon Anon]

Hi,

 

I am also facing this issue however it's prevented to some extent by increasing the timeout parameter from 60 to 120 for "Dashboard NIS queries" in dashboard_engine probe. The thread can be accessed at http://forum.nimsoft.com/t5/General/Dashboard-queries-don-t-load-on-initial-load/td-p/11427 .

 

Moderators / Administrators : Is there a way two threads discussing similar issues can be clubbed ?

 

Regards,

Amit Saxena

 

[Louis Savage]

I have updated my case and put in this forum url in hopes that support starts to correlate the issue as this is obviously not a single one time quirk if multiple people are starting to notice this.

[Anon Anon]

It seems that Engineering has acknowledged the issue, and they have been working to fix it.  I have received several hotfixes so far.  While these have not yet been successful in resolving the issue, I have continued to gather logs and submit them for review. So ultimately they appear to be working towards a resolution.  I'll let you know if I have any additional updates.

[Anon Anon]

Hi Matt,

 

Good to know that engineering team is working on it and we will receive a fix soon from them. Please let  us know once they have an update on this.

 

Regards,

Amit Saxena

 

[tdsnoc]

We are experiencing a very similar problem, (and we have a case open) however with a couple slight differences:

 

1) We are using UMP 2.1.2

 

2) Sometimes graph lines just start trailing off and sometimes they disappear completely. I'm quite confident those two are related because graphs that do one often then do the other (it goes both ways, sometimes they're empty then start trailing pieces in, sometimes they start trailing and then go blank).

 

I have created test dashboards that pull the same QOS half a dozen different ways in one page (multiple graphs, gauges, text fields, etc), and some of the graphs will be empty.

 

It's not a browser/refresh/cache issue -- colleagues and I have noticed the same graphs going blank at the same time, and we were on different systems in different locations.

[Anon Anon]

Hi,

 

Did you got any popup dialog box, specifically with an error message ? Also have you checked the logs to see whether there is any information that might help in resolving this issue ?

 

Regards,

Amit Saxena

 

[tdsnoc]

Hi Amit,

 

No, there are never any UMP error messages. I have also submitted megabytes of logs (up to loglvl 5) from the dashboard_engine, wasp, etc, and no cause has been found yet.

 

-Martin

[Anon Anon]

Hi Matrin,

 

As mentioned in another thread discussing this issue (refer my earlier post for details), I got following error message displayed in popup dialog box.

 

Perhaps that might help in this case.

 

Usually caused by dashboard engine communication interruption.
Details: 5:Common.SRV_DATA_UPDATE_FAILED_SERVER_FAULT: !Common.SRV_DATA_UPDATE_FAILED_SERVER_FAULT!

 

 

Regards,

Amit Saxena

 

[Anon Anon]

All -

 

This issue seems to have been acknowledged as a product issue with 2.5.1.

 

Good news though - It appears that the newest Beta (2.5.2) completely resolves the issue. All tests performed in our lab environments have been successful thus far.  I have heard word that it may be going to GA shortly, but am not sure of the date.

Regards, 
Matt

[Anon Anon]

 

We also have a hotfix released this week for these issues. Please contact me if you would like this. I also encourage everyone to test 2.5.2 which contains the same fixes.

 

Robert Latham

[Anon Anon]

When do patches like this get pushed to Nimsoft on Demand customers? 

[Anon Anon]

Some updates on this issue, which I thought was gone due to all the updates etc, but....... 

 

This problem is really getting old and I hope Nimsoft can solve this once and for all. I also wondering, once again, whether we (those on this thread) are the only ones in the world facing this problem.

 

We are now using NMS 5.61, with UMP 2.61, OS version is W2K8 R2.

The probe that we are monitoring for the chart is exchange_monitor and the version is 3.50 (newest version).

 

The chart is for OWA Response time and User Count-Mailbox ( Please note that the QoS data is shown in the SLM BUT for some strange reason unknown at the moment, the charts will stop plotting the line at a given time ALTHOUGH the data keeps coming into the SLM)

 

Attached is the Screenshot of when the chart stopped and the SLM data taken simultaneously. 

 

The mystery deepens as The chart is working fine in other dashboards (Sql_Response) & CDM (dynamic dashboard) charts are working.

 

Temporary solution is restarting the Dashboard Engine ( the charts back to normal) - but this is not a sustainable of viable solution.

 

We had same problem with UMP 2.51 and 2.52.

 

We changed some of the raw_configure values in Dashboard Engine as the support team did this previously for us when we were using the UMP2.52.

Problem still the same. Values we have changed as below:-

<data> 

alarm_processor_batch_size = 0 to 500 
qos_processor_batch_size = 0 to 500 

<updateintervals> 
update_ums_interval = 60 to 59 
dynamic_views =60 to  61 
dashboards = 30 to 31 

 

 

 

 

 

 

 

[Anon Anon]

I still see this issue once in a while as well.   UMP 2.52 NMS 5.60.

 

I'm planning to upgrade to the latest next week.

[Anon Anon]

Still the same on 6.0 

[tdsnoc]

We just upgraded to NMS 5.61 with UMP 2.6.2 and the problem has finally gone away...

[Anon Anon]

The problem is still present with UMP 2.6.2 and NM 5.6.1. But today I got a new dashboard_engine 3.53 off Nimsoft. This new version is not in the Archive

[Anon Anon]

I've been seeing this quite a bit as well, even with later versions of the UMP (we haven't officially upgraded to 6.0, although new versions of all the probes have been deployed).

 

The biggest positive differences I have seen is a result of changing player_connection_pool_size = 100 in dashboard_engine.cfg, and also increasing memory usage in the DAP probe (which needs to happen in raw config).

 

We're upgrading to UMP 6 in a week or so, so hopefully I'll have better results soon.

[Anon Anon]

After upgrading to  nms 6.2 and UMP 6.2.x some month ago this is stil a daily returing issue for us. Waiting to be claimed bij my customers for presenting wrong data . Like wrong temperature reading in their MER.

 

Not only graphs but all qos values are suffering form this. In one custom dashboard with several qos values presented, it is a mix of wrong and right updated values. (looking at the timestamp) In industrial environment that could be killing. Note: in industrial environments (PLC's, OPC servers, scada, etc) they use the time stamp for an extra check and validaiton of the reading. Maybe a suggestion for Nimsoft developer.

 

If there is an update or fix on this issue please let me know.

[Anon Anon]

FYI - I'm also having this problem with NMS and UMP both on latest versions.

 

I'm hoping Flash is scrapped soon in favour of HTML5!!

[Anon Anon]

Same here with 6.2.

Charts stop plotting random, and depending on which UMP I open the dashboard I see graphs or not, they work again randomly and it is being a nightmare !

[Anon Anon]

same with 7.5

[Anon Anon]

Good to hear this is being worked on. I am experiencing this too and have a case opened.

[Louis Savage]

I have verified charts are working in 2.5.2.0