VMmark 2.1 results in QOS score of 0 on the mail server

[Rishi2012]

We were looking for some quick assistance as to why our QOS  score for a 2 tile test is resulting a 0 score for the mail server? We are trying to identify if this is a configuration issue on our part that is causing the issue. We have met all the requirements set forth in the VMmark 2.1 benchmark guide from an infrastructure standpoint so we believe the quality is there. I am attaching our latest score results. Any assistance would be greatly appreciated.

[Joshua Schnee]

Hi,

Typically, this is due to a configuration issue.  When I reviewed your results I see almost as many task exceptions as there are tasks dispatched.  I would recommend that you run Loadgen manually, pointing it at the config file under c:\vclient0\vmmark2-mailserver-0.xml.

Remember not to change any of the populated values except for run time.  Doing this should help uncover what may be causes all the exceptions.

[Rishi2012]

We have a 2 tile benchmark where the prime client is acting as the client for the 1st tile. Our client 1 is hosted in a different cluster the our 2 tiles of VMs. When I run the loadgen test manually on both client0 and client 1 it successfully completes. We have not altered our vmmark config file beyond the run time and adding the variables unique to our enviorment.  Do we have a next step to figure out why QOS is 0 for the mail servers in our score file?

[Joshua Schnee]

When Loadgen completed, did it report any exceptions?

[Rishi2012]

Do you mean under the load generator status on the results page? If so, the count for the number of task exceptions for client 0 and client1 reads as follows:

Client 0 (Prime Client)-1648068

Client 1-1648068

looking at the task exception statistics, I see the source of all the exceptions is the following message:

Microsoft.Mapi.MapiExceptionLogonFailed

I verified the login crednetials in our vmmark2 config file are accurate. is there another source for this logon failure?

[Joshua Schnee]

There are many reasons why you might be getting this error so I'm not sure.  How did you verify that the login credentials used were correct?

[Rishi2012]

I verified by using those credentials to login to the mail server . i was able to logon successfully. Are there some other top hitters that causes this message?

[Joshua Schnee]

I'm unaware of any other main reasons for getting this error within VMmark2.  Can you zip up your entire results directory from your last run so I can review?

[Rebecca Yo]

Hi, can you give the error description for Microsoft.Mapi.MapiExceptionLogonFailed? For example, Unable to make connection to the server or Unable to open message store.

[Rishi2012]

Where do I get the details of the error message Microsoft.Mapi.MapiExceptionLogonFailed? i do not see anymore details on the LoadGen status screen other than error I posted.

[Rebecca Yo]

On the client, go to C:\Program Files\Exchange Load Generator and look in the log files LoadGenInit* and LoadGenSim*.

Thanks, Rebecca

[Rishi2012]

Thanks. Looks like it is the message store:

Microsoft.Mapi.MapiExceptionLogonFailed: MapiExceptionLogonFailed: Unable to open message store. (hr=0x80040111, ec=-2147221231

[Rebecca Yo]

This is most likely a permissions error. I would echo jschnee's request for you to zip the results directory from your problem run and attach to this thread, please. Another place to look for more informative errors is the Event Viewer on your Mailserver VM, under Windows Logs > Application.

[Rishi2012]

Thank you both for your assistance thus far. I have attached the files from our last run. Looking at the application event log on the mail server VMs. We did notice the the following MS exchange IS error. Which should be remedied by the following KB:

http://technet.microsoft.com/en-us/library/bb232050(v=exchg.80).aspx

Once that setting was checked, the mail databases would mount. However, after checking the status of the setting after they were mounted, it was unchecked again. It may take new off-line backup of the database to preserve this setting through a reset, or it may take something else?

[Rebecca Yo]

Hi, could you create a zip file with all the files from your run and attach the zip file please?

It should not be necessary to check "This database can be overwritten by a restore". To clarify, when that box is not checked, are your databases listed as unmounted?

If you would like to try checking the box and running again, follow the directions on page 152 of the VMmark Benchmarking Guide to delete and remake the Microsoft Exchange database.

[Rishi2012]

Sorry for my delay in response but I havezipped up my results file. In addtion to the issue, i ahven't found a solid answer as to how the QOS score is calculated? What does a 0 really mean? Our other VMs have a score assigned to the QOS.

[Rebecca Yo]

A 0 means the Mailserver QoS metric is not being measured at all because the workload is failing. All Exchange actions attempted on the Mailserver are failing. LoadGen is responsible for reporting each action's latency. Loadgen is reporting the latency for all Exchange actions as 0, therefore, the harness also reports it as 0. However, the failed tasks are being recorded as 'completed' although they are not completed correctly, which is why it appears that work is being done.

I have a couple of questions for you:

It's not necessary to check "This database can be overwritten by a restore". To clarify, when that box is not checked, are your databases listed as unmounted?

I would recommend following the directions on page 152 of the VMmark Benchmarking Guide to delete and remake the Microsoft Exchange database. Also please check there is enough disk space on your Mailserver VM.

I saw very odd VC and hostnames containing a large number of v, c, l, d, and X's in your results files. These names were inconsistent even within the same file. Are you running on a non-English version of Windows? VMmark has been tested only on English windows and is not supported on other languages. Or, did you intentionally obfuscate the names of your hosts, LUNs and VC in these files before uploading them?

Thanks,

Rebecca

[Rishi2012]

It's not necessary to check "This database can be overwritten by a restore". To clarify, when that box is not checked, are your databases listed as unmounted?

Yes they were unmounted, and that is why the team attempted to check this box.

I would recommend following the directions on page 152 of the VMmark Benchmarking Guide to delete and remake the Microsoft Exchange database. Also please check there is enough disk space on your Mailserver VM.

We have attempted the step to delete and remake the DB from pg 152 but were unable to have success with this QOS score being greater than 0. Also our Mail server VMs meet the storage and memory requirements stated the Benchmark guide.

I saw very odd VC and hostnames containing a large number of v, c, l, d, and X's in your results files. These names were inconsistent even within the same file. Are you running on a non-English version of Windows? VMmark has been tested only on English windows and is not supported on other languages. Or, did you intentionally obfuscate the names of your hosts, LUNs and VC in these files before uploading them?

yes I intentionally obstructed the names of the servers, VMs, and user name and passwords. I probably should have followed a standard to prevent confusion. There that is why you see this inconsistency.

Thanks,

Rishi

[Rebecca Yo]

The databases should be mounted on their creation. One reason for the databases not to mount is that there is too little disk space on the VM's C drive. Even if you have made your VM with correct size disks, the virtual hard drive of the VM may become filled up with logfiles if you are encountering a large number of exceptions while running the workload, as you have been.

In the Mailserver VM, please check that there is a reasonable amount of free disk space (more than 1 GB) on C:\, E:\ and F:\. Could you tell me how much space remains on these drives, please?

If this is not clearly the problem, could you delete the databases as described on page 152 and recreate the databases. As soon as you create the databases, observe whether they are mounted or not. If not, please check the Event Viewer log  (Windows Logs > Application) to see if there's an error recorded there from the database creation. If any errors are reported during the database creation process, please report those here too.

Thanks,
Rebecca

[Rishi2012]

On our 2  mail server VMs, i see this for the available space that is somewhat consistent.

C Drive-2.44 GB

Database E Drive-25.5 GB

Backup F Drive-1.56 GB 

[Rebecca Yo]

Thank you, it looks like there is enough space on the mail server VMs. The databases should be mounting when you create them, and needs to be mounted to work correctly. Could you please delete and remake the databases again? If the databases are not mounted after you create them, please check the Event Viewer log  (Windows Logs > Application) to see if there's an error recorded there from the database creation.

[Rishi2012]

Hi, I recreated the DBs on Mailserver0 and they are showing mounted. I did not have issue in the steps provided by the benchmark guide.  So I have proceeded to the steps of running loadgen from the prim client0 to verify the results. I was able to initialize LoadGen without issue and am now trying to backup the DBs lpriv1 and lpriv2. However when I am running the defrag step on lpriv1, I get the following error:

operation terminated with error 1808 jet_errdiskfull, no space left on disk.

It looks like the erro became present after 75% of the defragmentation was complete. We confirmed from a prior post there was enough space on the drives on the VM. Could this be a cause to the mail server issues?

[Rebecca Yo]

Hi Rishi, that's great that the databases are mounted. When we earlier confirmed there was enough space on the VM, that was when the old database was present on the Mailserver VM's drives, so it's not relevant now.

Before you recreated the database, did you delete the old database so that the E: and F: drives were completely empty? The directions are under "Deleting the Microsoft Exchange Database" in the Benchmarking Guide.

[Rishi2012]

Hi. yes it stated to delete all the files which i did. However do we need to delet all the subfolders?

[Rebecca Yo]

Hi Rishi, yes, please delete all the files and folders on disks E: and F: so that both of these drives are completely empty.

Rebecca

[Rishi2012]

The C drive has very little space on mailserver 0. i seemed to overlook this issue.Only 1.57 MB available oput of 31.9 GB. Could this be the issue as to why I receive the space issue when i attempt to defrag the database as part of the backup process?

[Rebecca Yo]

Yes, 1.57 MB is too little space on the C drive and would create  problems in the mailserver even independently of the database defrag  step.

Two things: you will need to make sure that E: and F: are completely empty and you'll need to free up space on the C drive.

To create more empty space on the C drive, the first thing you should try is to delete Exchange Logs:

Browse to C:\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group
Delete E000000*.log files that are 1024 KB in size.
Empty the recycle bin, and check how much free space exists on C:\.

Secondly, please delete and remake your Exchange database according to the doc, and please delete all files and subfolders on E: and F: before continuing to create the database again.

Thanks,

Rebecca

[Rishi2012]

I deleted all those file it freed up 491 MB on the C drive out of the 31.5 GB. Is there other opportunities to free up space? Also could this be the very reason why the QOS score is recording a 0 when we run the vmmark benchmark test?

[Joshua Schnee]

You might check to see if this is related to the Transport Service.  Look at the instructions in the latest benchmarking guide, under the section "LoadGen Crashes If Exchange Server Transport Service Stops".  SInce you've previously used the VM it might be that the mail.que is filling up all the C drive.

One note:  the instructions say to change directories to where the mail.que resides, but you might need to execute the command from your E or F drive since you're out of space.

ex (in command prompt)

E:

eseutil /d "C:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue\mail.que" 

[Rishi2012]

I attempted the troubleshooting step of Loadgen crashing due to Microsoft Exchange Transport service crashing and I still seem to eb having issues. My C drive continues to hagec very little space. It only has 54.9 MB out of 31.9 GB of space. What other opportunities do I have to delete space outside of delting logs from the First Storage Group Folder?

Thanks,

Rishi

[Rebecca Yo]

Could you please tell us the size of your mail.que file in C:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue? If the mail.que file was filling the C: drive, running eseutil should have decreased its size.

Also, to answer your earlier question, the lack of space is a likely cause as to why your QoS is being reported as 0.

Thanks,

Rebecca

[Rishi2012]

Thanks. The size of the mail.que file is 2,064 KBs. 

[Rebecca Yo]

Okay, mail.que isn't the problem, so we've eliminated our usual causes of low disk space.

The next thing I can think of is for you to determine the size of the C:\ProgramFiles\Microsoft\Exchange Server directory, by right clicking on the directory name in Explorer and viewing Properties.

I'd also recommend double checking that the recycle bin has been emptied, and looking through the contents of your C: directory or desktop to see if there are any unusually big or extraneous files there.

[Rishi2012]

Yes i double checked the recycle bin before the previous post and it was empty. The size of Exchange Server directory is 1.07 GBs. 

[Rebecca Yo]

Alright, that's a normal size for the Exchange Server. Whatever is taking up the space must be outside of that folder. As there is something taking up a massive amount of space that is not normally there (along the lines of 6 GB), I would recommend browsing through the C drive using process of elimination to find very large folders.

[Rishi2012]

Thanks Rebekah. We are considering increasing the space on the C drive to eliminate this as an issue versus trying to determine what to reduce on the C drive.  We currently have 41.4 MB out of 31.8 GB available. How much space would recommend to add to get a quality QOS score on the mail server?

Could anthing outside of the Mail Server space issue be a cause of a 0 QOS score for our mail servers?

Thanks,

Rishi

[Rebecca Yo]

Increasing the size of the mailserver drive sounds like a good step to troubleshoot the problem. I believe increasing the drive by 5GB would be sufficient, or more, if you wanted to be extra safe. Remember to increase the size of the C partition, too. Runnning with enlarged drives on the Mailserver would produce VMmark results that cannot be published, but that may or may not matter to you.

However, I am not sure that enlarging the disk size would fix the problem long term. The lack of disk space may be a symptom of the real problem if it is caused by large log files, possibly logging errors. Finding the source of the occupied space could tell you what is the underlying problem.

I'm surprised that your Mailservers are running at all, with that little space. Although it may not be the root cause, at minimum you should resolve that issue before considering anything else.