The entire size of VMDK files is backed up when backing up with Veritas Backup Exec 16

[JanEberhardt]

Hi,

we got a problem with the tape backups of our VMs. I have been in contact with the Veritas support but after a lot of tries we think this problem is on VMWares side. Before I go into the bug description, I will explain our current setup:

• Hardware
  • HP ProLiant with 2 HDs for Windows Server 2012 R2 and Backup Exec 16
  • HP ProLiant VMWare Center without any additional HD
  • Synology DSM 6.1 with Synology Raid (8 HDDs)
  • Synology is providing space for VMs via iSCSI LUN
  • IBM Ultrium-HH7 Tape drive with 8-slot robotic library
• Use case
  • Multiple VMs (Windows/Linux) with thin provisioned virtual volumes
  • Some of them turned off, most are running
  • Windows 10, Ubuntu Server 16.04, Linux Mint 10
  • Full backup of the vms to tape directly every Friday, incremental on other days
  • Backups are done with snapshot technology

The problem we have is now, that VMWare is always reporting the wrong used block info to Backup Exec, which then backing up too much. In example, we have a data store server (Ubuntu 16.04) with a 7 TB thin provisioned volume where approx 4.4 TB is used. Backup Exec gets reported that all blocks are used and therefore is backing up the whole 7TB. This means that we have to use more tapes and the backup process will take much more time for data, which is not even data.

I have tested the backup multiple times with Veritas support on smaller machines. We have tried to break down the problem, but are not able to get the used block info which VMWare reports and therefore are not able to confirm if this is an issue within the VMWare API. Earlier it helped to reset the CBT, but lately even that does not work anymore. In addition, new VMs are backed up properly (I had a test VM with a 16GB virtual hard drive, installed Ubuntu 16.04 from the same ISO file as always and backup was 2.2GB).

Is there any setting to get debug information from the VMWare center (like used blocks info)?

Are there any known issues with working on iSCSI with snapshots/vmdks?

Kind regards

Jan Eberhardt

[daphnissov]

What version of vCenter Server is this and what is the version and build number of the ESXi hosts? On your Synology, what hardware is it? What type of iSCSI LUN did you configure? Is it the thin LUN with VAAI optimization, or whole volume on LUN?

[JanEberhardt]

1. ESXi 6.5.0 Update 1 (Build 5969303)
2. VMTools build 10247 (most of them, one is a bit lower, but if that's the issue, we can update it)
3. Synology RS3411RPxs, DSM 6.1.3-15152 Update 3
4. LUN: thin provisioned, advanced file LUN with VAAI optimization

[daphnissov]

Other than your Ubuntu file server, can you list the configuration of those VMs? Specifically what is the VM hardware version and the VMDK type? Screenshots are fine if you want to do that.

[JanEberhardt]

Here you go. :3

Where can I find the VMs Hardware version?

[daphnissov]

Click on the VM and it'll show in its configuration.

[JanEberhardt]

Well... I did, but didn't find something called 'Hardware version', you mean the value of the 'Compability' field, which notes VM Version 13 on most (if not, please be more precise where to find it).

Two of them have version 7 (CentOS) and version 11 (Ubuntu, 15.93 GB used space) and both are backed up properly:

CentOS: 2.76 GB used space incl. 361MB host memory; 2,311,094,223 bytes backed up

Ubuntu: 15.93GB used space incl. 3.53GB host memory; 12,822,350,032 bytes backed up

I think I once tried to 'update' vmtools at the Ubuntu server and accidently replaced it instead... it now looks like a problem with the new VM version. But the issue would now be to figure out what has changed so that it works properly...

Thanks for going into the issue.

[daphnissov]

The screenshot you provided appears to be from the ESXi embedded host client. Are you not running vCenter?

Well... I did, but didn't find something called 'Hardware version', you mean the value of the 'Compability' field, which notes VM Version 13 on most (if not, please be more precise where to find it).

Yes, this is known as the VM hardware version. What version is this 7TB Ubuntu VM?

[JanEberhardt]

daphnissov: It has VM version 13 and VMTools version 10247.

[imacfj]

Ive seen stuff like this before

is there a large amount of data churn? i.e is a lot of data added and removed regularly?

if so, then the backup software includes this as a changed block even if it ultimately is empty, and still backs this up

it sounds like the backup software is backing up the actual VMDK rather than the volume within the guest. has this worked previously?

are you able to browse any files within the restore options, or just see the VMDK?

[JanEberhardt]

The amount of data written to the store is decent, I'd say... it's our main store and developers using it regulary for test data. But it's not as much as the difference suggest. We don't have any data on the amount of written blocks, but it is some GB per week. We also had this scenario in mind and therefore I have used the CBT reset on the machines multiple times. Sometimes that helped to reduce the amount of written data in the backup, but it never has on our big data store.

Anyway, the restore option is on vmdks, but that was intended since for being able to browse through the files, we need to install the BEX client to our Windows VMs and provide extra credentials. It was the easier solution to just let VMWare handle the backup/restoration process of the VMs.

[Dave_the_Wave]

One thing I seem to notice for Windows vm's, it appears that BE16 is backing up the guest OS files, and not the entire singular vmdk/vmx.

When I try to restore a vm, I was expecting whole vm snapshots to choose from, but instead it's showing me the C:\ drive and all its folders that I can nit pick from.

Granted, I don't have to install the BE Agent on the Windows vm, but I might as well, because then it's back to the old fashioned drive share backup where I can choose what I want.

(For anyone else that wants to give this a shot, BE has always been a 60-day free trial.)