Missing data in performance graphs

[VMRulez]

I've just been reveiwing some performance graphs and have realised a lot of holes - I have several ESX Servers with a VC2.5 on a SQL database. Data is missing for 10minutes intervals 1 or 2 hours. In some cases the gaps are 1 or two hours without any data.

I haven't noticed any problems with the real time graphs. I've attached a graph for one of the VMz for the last day's statistics

Any thoughts will be appreciatd

[emba]

hi,

we experienced the same here, although not as dramatically as in your graphs. the reasons could be network issues because performance data is send over the wire by UDP. you could also try to decrease the level of performance data gathered (to 1) or increase statistical collection thread limit (both in VC). hostd (1h "current" view) is being queried real-time whereas all other intervals are queried from the DB. if there's a zero value logged in the DB you will see the gaps. is your network connection between VC and COS reliable? what are the ping times?

cu

[VMRulez]

Ping reply time is 1ms

VC is connected at 1GB, all hosts are talking at 1GB as well, DBServer is on 100MB

[cshenberger]

I have the same issues with VC. Everything was fine for a while and now I see gaps of 5 minutes every hour in the "Past Day" Charts. If I look at any other chart everything is fine. I have checked the time sync based on comments from other posts that I have seen, my VC server is at most 6 seconds different than my ESX hosts. I haven't found where to change the thread limit in VC 2.5 so that I can change that limit which is another setting that I have seen many other posts mention.

My setup is ESX Hosts 3.0.2 b 61618 VC 2.5 SQL database on the VC server.

Cshenberger

[csimwong]

Using ESX 3.5, VC2.5

I have similar issues. I have random 10 minute gaps missing when I bring up the charts. I do get random alerts from the VC that hosts get disconnected but reconnected soon after, but the alerts don't coincide with the random 10 minute gaps.

[AndyN]

I see the exact same issue in our VC 2.5 Update 1 / ESX 3.5 deployment. Here is what I am seeing.

I am running VC 2.5 on a dedicated physical host, specifically a HP Proliant DL380G2 with 6GB of memory. I am pointing to a seperate SQL DB server, also dedicated to hosting this single database instance. The SQL server is a HP DL380G5 quad core with 16GB of memory. I am running VC 2.5 Update 1, build 84767. This is a very lightly loaded lab environment with a small number of hosts, 4, and only 16 Winows 2003 Server virtual machines spread between 4 hosts. Two of the ESX hosts are 3.5 Update 1 (build 84374), the other two are 3.0.2 (build 63195). I am running SQL 2005 SP2 64-bit.

The gaps in data appear for all 4 counters: CPU, Memory, Disk and Network, but only if you look at "Past Day". Any other time frame, such as Real Time or Past Week, I see no gaps.

I have verified that this is occuring for Statistics Level 1 and 2. (VC 2.5 appears to have no setting for "Statistics Collection Thread Limit" as VC 2.0.x did).

The gaps appear for data on ESX hosts as well as individual VMs. The gaps appear on ESX 3.5 (build 84374) hosts and their VMs, as well as ESX 3.0.2 (build 63195) hosts and their VMs. The gaps appear to be occuring aproximately once per hour for about 5 minutes.

Also, this is a complete new install of VC 2.5 and the SQL database. I have re-installed VC 2.5 Update 1 specifing a re-initialized database (deleting any previous DB). The problem persists.

[VMRulez]

I was in a VMware class not too long ago where the lecturer tactfully avoided the question.

The only think I can point to is that there's something funky about VC and SQL database over the network. I have a lab with 6ESX hosts on a local SQLExpress database and they're are no gaps whatsoever.

[grEEdy_Berlin]

I have this problem with a local SQLExpress-DB too.

And i also get the HA-warnings in the VC-Eventlog for a few hours for 5 minutes after the full hour every time i did something in VC like migrating or creating a VM.

I would really like to find a solution for that issue. I have an open case because of the HA warnings. Today i discovered the gaps in the perfomance-charts and i will update the case with that.

Really strange behaiour.

[seafish]

Hey did you ever get any resolution to your problem with the perf graph gaps ? We are deploying a huge installation of vmware and just came across the problem ourselves.

Thanks,

Seafish

[VMRulez]

Nothing yet. That being said, I have just deployed update2 of virtual center yesterday. Not that I am expecting it to fix the problem.

Are you on a SQL database with your VC as virtual?

[seafish]

We are have the sql on a physical currently and the VC is a virtual. We are running update 2 on our systems this was the first time we observed the behaviour.

[AndyN]

An update to our "gap" problem.

Our current VirtualCenter version level is now 2.5 Update 2 (build 104215). In our lab, we have several ESX hosts at various levels all now being managed by our VC 2.5 Update 2 box. Gaps still appear when looking at Past Day data on ESX hosts version 3.0.2 (build 63195). Newer ESX builds such as 3.5 Update 1 (build 82663) and 3.5 Update 2 (build 110268) do not have any gaps in ESX host charts. So it appears that on ESX hosts themselves, the performance data gap issue may be ESX version related.

If, however, I highlight an individual VM and look at it's Past Day - CPU, memory, etc. performance chart, I still see random 10 minute gaps. This happens on any ESX host version level. We have examined our SQL data base to see if there are any "zero" values and have found none.

[Arnaud_D]

Hi, I have the exacly the same issue about missing data, (ESX 3.5 buil 110268, VC 2.5 build 64192) from time to time on my hosts (DL585) & VM perf. charts...

[AlexPT]

Hey guys,

Just thought I'd chip in and say we are getting the same. ESX 3.5 update 1 hosts with VC 2.5 update 1 pointing to an Oracle 9.2 database. 10 minute gaps randomly. So I would say its not just SQL related.

I think I'll be logging a call to Vmware.

[pldoolittle]

Add me to the list. Mine's not random, but rather 10 minutes every hour. I thought it was performance related and have cleaned the database, added chip, memory, etc so that VirtualCenter is barely moving the performance meters. No change.

My versions are: VC 2.5.0 build 84767/ SQL express on VC box. ESX 3.0.2 Update 1

Configuration: VC running as a VM in cluster. 2 ESX hosts, 25 VM's SAN storage, lightly loaded. Hosts about 60% utilized.

[TMSio]

Same issue here....

[redkat]

Hi AndyN!

I have the same issue, but as i understand there are two

types of problem with gaps.

First:

Every hour 5 minuts

Second:

Random gaps for 10 minuts.

Did you solve this problem? And in Update 4 as I read the problem of gaps will be solved, but what type of gaps :smileyhappy: first or second, do you now?

[AndyN]

I have not seen the "gaps" since 2.5 U3. I installed 2.5 Update 4 (build 147633) a week ago in our lab and I do not see any gaps. From my knothole, it looks like they might have fixed it.

[jimbo667]

I am still seeing the random 10 minute gaps in the performance graphs in VC2.5u4 and correlating spikes to zero in the new performance overview graphs :smileysad:

[redkat]

HI! I wrote earlier that there are two types of problem with "gaps"

First in past day graph.This is the problem of clearn procedure (procedure wrote by vmware)in ms-sql(msde)

I fix it by reinstalling msde.

Another problem with "gaps" - is random 10 min gaps. About this problem i wrote to vmware tech support and

they answer me that this is well known problem. They will fix it in ESX 3.5 U4. Will be waiting U4 together :smileyhappy:

Best Regards,

Vereschagin Sergey.

[jimbo667]

LOL I thought the fix was meant to be in VC 2.5u4.

But you would have thought that they would have at least fixed them to show off there new Performance Overview graphs.

[malaysiavm]

Please noted that you may lost your history performance data if you remove any ESX host from vCenter.

Regards,

Jason

Malaysia VMware Communities - http://www.malaysiavm.com

[Chris_Howard]

I've not seen this specific issue, however its worth checking that the 3 statistics rollup SQL Server scheduled jobs are working without error.

If you found this helpful please consider the use of the Helpful /Correct buttons to award points. Thanks !!

[cshenberger]

It seems that this goes away with an upgrade to VC 2.5 U1 or U2. We have upgraded 2 instances of VC and both of them have not had this problem since.

[AlexPT]

We are at 2.5 UD1 and it didn't fix it. UD2 may fix it though???.....

[mikepodoherty]

VC 2.5 U2 and ESX 3.5. U2 - Oracle 10g - occassional 10 minute gaps.

Servers are blade servers in same and/or different chassis and on 2 different networks.

[AndyN]

The problem apprears to have gone away with VC 2.5 Update 3. But boy do I even have problems running the U3 install and hosing my SQL DB.....but that's another thread.

[jimbo667]

I can confirm that it is still present in VC 2.5 U3 as we have two environments at that level, and another at VC 2.5 U2, that still have the hourly 10min gaps in the stats. Would love to fix the problem but I dont seem to be getting anywhere.

[nnnnnnj]

I've installed VC2.5U3 for a customer before Christmas and noticed this morning the same behaviour - 10 minute gaps, only in the Past Day sampling, and in this case it's only in VM data not host.

Has anyone had a decent answer to this yet?

Tony Lamont

[wallakyl]

I have several clients with the same issue. However not all of my clients are experiencing it. I have seen it in VC 2.5 (no update) and VC 2.5 U3, so it doesn't seem to matter what update they are on. I haven't been able to find out why some experience it and some don't. Support hasn't been very helpful thus far in admitting the bug. If anyone gets anywhere with support, please let me know.

[jimbo667]

Spoke to VMware the other day and got "Known issue scheduled for update 4". Can't wait for another flawless upgrade :smileylaugh: .

[VMRulez]

The interesting this is VMware has known about this for quite a while and two major updates, 2 and 3, have passed without this being addressed.

Oh well.

[macltrc]

I had a service request open with VMware which is now closed. I spoke to a technician yesterday (Jan 21, 2009) and he indicated that this is a known bug. Unfortunately, the details are "confidential" and the bug report is only available internally to VMware. He also indicated that there is no public knowledge base article that customers could reference or track.

Apparently the bug is expected to be fixed in Update 4, or Patch 11. Update 4 currently has an expected GA date of March 6, 2009 and Patch 11 currently has no public ETA according to the technician. As described by some previous posters in this thread, the specific bug in question supposedly only affects the "Past Day" performance graph views (which is what I am experiencing).

I have no idea why this is considered "confidential" unless it's embarrassing to VMware (which arguably, it should be since it has persisted for so long). There aren't really any security implications as far as I'm aware or would expect from such an issue. So VMware could probably be a little bit more transparent about this bug.