Unconsolidated VMs have poor performance

[Linklight]

So recently, I have an issue with a specific virtual machine and some backup software. I am currently using HP Data Protector 7.01 in conjunction with some P4500 G2 storage systems. I have run into an issue for the last few backups where the full backup of the VM removes the snapshot it creates but does not consolidate the disk automatically. When I come into the office on Monday morning, there are a ton of redo logs and the VM needs to be consolidated. This isn't a big deal in and of itself as it seems that I have plenty of storage to deal with the issue, however until I do a consolidation on the virtual machine, performance of that virtual machine is awful. It may have to do with the fact that this virtual machine has 5 RDMs attached to it. Any ideas or insight would be helpful. I am hoping that understanding why the number of redo logs degrades the server's performance.

[vmroyale]

"When a snapshot is created, the VMware product produces an additional delta file. Each successive snapshot produces an additional file. When a disk operation is performed within the guest, the disk I/O is recreated by parsing each snapshot delta file in the chain. This produces additional disk overhead on the host because more than one file must be opened and processed to recreate the I/O data for the guest operating system. " - from http://kb.vmware.com/kb/1008885

[akkayyakapisett]

It is expected and as the read /write operations have to gone through multiple disks.

So consolidate to increase the performance

Venkat

http://www.peeradmin.com

[Linklight]

Here's what I am confused about though. Realistically, I am only taking one snapshot. If I am only taking one snapshot, would this still qualify this specific scenario? Theoretically, my backup software should only be taking a snapshot at the beginning of the job. At the termination of the job, the snapshot should be removed. According to the vmware snapshot manager, there are no snapshots available however the virtual machine still states it needs to be consolidated and when I look at the data store, there are a ton of ctk.vmdk files and a bunch of 00000n.vmdk files. The storage folder looks like the attachment. 

[continuum]

Looks like you have 3 snapshots - if Snapshotmanager does not display them it means that the vmsd file is invalid

[kermic]

.ctk files are OK, they're used for changed block tracking and allow the backup software to back-up only the blocks that have changed since last backup (incremental) instead of copying full vmdk every time.

What you should probably worry about is the number and sizes of 00000*.vmdk files. Those actually are your snapshot data. VMware recommends that for production the average snapshot life cycle should not be longer than few hours, max 1-3 days in some cases. But essentially you want to make it dissappear as soon as you can.

There will be disk IO performance impact on VM that has snapshots, especially during write operations. Basically every virtual disk IO that your guest issues can result in multiple physical disk IOs (AFAIR if VM wants to change a block that already exists somewhere up the chain, the block is first copied to current delta and only then vmkernel does the write, so it's 1 Read + 2 Writes instead of just 1 Write IO) so you're essentially doubling or tripling your disk load, queues fill up, so does the latency  and VMs are getting "slow".

Snapshot delete or consolidate operation is an extremely disk-intensive process which is why your'e seeing performance drops. The bigger the snapshot, the more data needs to be copied from deltas to base vmdks and since your disk spindles can produce only as much IO as they can, the operation takes longer and you're starting to notice the impact.

Basically it should be so that most of the time your VM has no snapshots at all, backup process kicks in and creates one, then does the backup and cleans up the snapshot afterwards. I would probably start with determining why the backup software does not clean up after backup job has finished. Involve your backup vendor's support if you can. This is a quite popular problem with backup products, those guys are solving hundreds of cases like this on daily basis so most probably your vendor will have a quick solution.

Snapshotting is a great feature if used appropriately (life cycle for a running prod VM snapshot ideally limited to few hours max) however it can do a lot of harm if you misuse it or leave your snapshots unattended (including data loss).

Hope this helps.