Getting all Snapshots in a large environment with vRO fails regularly

[FlorianZimmer]

Hi all,

as part of automating the deletion of snapshots, I build a workflow that requests all VMs from the VCs and then infers all snapshots. This, however, regularly fails.

We are talking about an environment with multiple VCs in ELM and a few thousand VMs. All the scripts that I will go into work perfectly fine, complete in seconds and have never even once failed in a test environment with one VC with about 100 VMs. The Orchestrator is on version 8.9.0.

The built-in action "getPropertiesForAllSnapshotTask" executes for about 30 minutes and has about a 70% failure chance. The errors that have been observed are:

Unable to convert object, 'Stub: moRef = (ManagedObjectReference: type = VirtualMachineSnapshot, value = snapshot-***, serverGuid = null), binding = https://***:443/sdk' plugin exception reason : convertToResult() --> Finder 'VC:VirtualMachineSnapshot' : unexpected error 'ch.dunes.model.sdk.SDKFinderException: convertToResult() --> Finder 'VC:VirtualMachineSnapshot' : unable to invoke read method : 'description''

and

TypeError: Cannot read property "connectionState " from undefined (Dynamic Script Module name : getPropertiesForAllSnapshotTask#6)

I was able to mitigate the latter by writing my own version of getPropertiesForAllSnapshotTask and reduce the failure rate to about 40%.

It implements the check "if (typeof vm != 'undefined') {"... and also excludes vcls VMs  and templates with "if(vm.runtime.connectionState.value=="connected" && !vm.config.template && vm.name.match(/(vcls)\w*/g) == null){"

The former, however, I wasn't able to resolve, even after writing a completely new function to get all VMs as seen below and checking the properties of all VMs one by one instead of collecting all VMs (the built-in "getPropertiesForAllSnapshotTask" uses "getAllVirtualMachines" and then checks for snapshots) and then checking for snapshots.

for each (vCenter in vCenters){
System.log(vCenter.name);
var clusters = vCenter.getAllClusterComputeResources();
for each (cluster in clusters) {
System.log(cluster.name);
var vms = vCenter.allVirtualMachines
for each (vm in vms) {...

This, however, takes 18 hours (!!!) before always failing with a "TypeError: Cannot read property "template" from undefined (Dynamic Script Module name : MY_getPropertiesForAllSnapshotTask#17)"

 

Does anyone know whats going on here? I have already spent way too much time on this. I am suspecting that the workflow fails because there are so many VMs in such a large environment, but I have no idea as to how to proceed.

[VMware-SimonSparks]

I have created a solution for this in TypeScript using the Aria Build Tools

https://github.com/vmware/build-tools-for-vmware-aria

Challenge Accepted, I know this question was a LONG time ago but I suspect the problem will be even worse now given there will be LOTS more VMs.

This is a helper function.

public serverFindAllForType<T>(@notNull @notEmpty @required strObjectTypeName: string, strXPathQuery?: string): T[] {

	if (strXPathQuery) {
		strXPathQuery = `xpath:${strXPathQuery}`;
	}

	let arrObjectType: T[] = Server.findAllForType(strObjectTypeName, strXPathQuery) as T[];

	return arrObjectType;
}

This is the main function

public VcVirtualMachineGetAllSnapshotDetails(): Properties {

	let objRegExp: RegExp = /(vcls)\w*/g;

	let strXPathQuery: string = `runtime/connectionState='connected' and config/template=false`;

	let arrVcVirtualMachine: VcVirtualMachine[] = this.serverFindAllForType<VcVirtualMachine>("VC:VirtualMachine", strXPathQuery);

	arrVcVirtualMachine = arrVcVirtualMachine.filter((objVcVirtualMachine: VcVirtualMachine): boolean => {

		return objRegExp.test(objVcVirtualMachine.name) === false;

	});

	let objProperties: Properties = new Properties();

	arrVcVirtualMachine.forEach((objVcVirtualMachine: VcVirtualMachine): void => {

		let objVcVirtualMachineFileLayoutEx: VcVirtualMachineFileLayoutEx = objVcVirtualMachine.layoutEx;

		let arrVcVirtualMachineFileLayoutExFileInfo: VcVirtualMachineFileLayoutExFileInfo[] = objVcVirtualMachineFileLayoutEx.file;

		let arrVcVirtualMachineFileLayoutExSnapshotLayout: VcVirtualMachineFileLayoutExSnapshotLayout[] = objVcVirtualMachineFileLayoutEx.snapshot;

		arrVcVirtualMachineFileLayoutExSnapshotLayout.forEach((objVcVirtualMachineFileLayoutExSnapshotLayout: VcVirtualMachineFileLayoutExSnapshotLayout): void => {

			let intDataKey: number = objVcVirtualMachineFileLayoutExSnapshotLayout.dataKey;

			let objVcVirtualMachineFileLayoutExFileInfo: VcVirtualMachineFileLayoutExFileInfo = arrVcVirtualMachineFileLayoutExFileInfo.find((objVcVirtualMachineFileLayoutExFileInfo: VcVirtualMachineFileLayoutExFileInfo): boolean => {

				return objVcVirtualMachineFileLayoutExFileInfo.key === intDataKey;

			});

			objProperties.put(objVcVirtualMachineFileLayoutExFileInfo.name, objVcVirtualMachineFileLayoutExSnapshotLayout.key);

			System.log("Name : " + objVcVirtualMachineFileLayoutExFileInfo.name + " - Size : " + objVcVirtualMachineFileLayoutExFileInfo.size);
		});
	});

	return objProperties;
}

I don't have a LARGE environment that I can test this in to see if it will complete faster which will result in a higher probability of success as I suspect you are hitting timeouts not actual issues.

------------------------------
Simon Sparks
Automation & Orchestration Consultant
VMware Professional Services CoE
North West England, UK, Europe
Broadcom Employee
------------------------------

[Thorlund]

I think no matter how you do it, it will be a heavy process, as you need a handle to every VM to check for snapshots, and then further down to attach to the snapshot(s). I have tried many different ways, I found this code the fastest for my case. I need to get all snapshots for all vms, and stored the details in in a SQL database, means loop vms, check vm, get snapshot, write to SQL. You can add your own action - in this example it just writes the VM and snapshot name. I have multiple VCs and need to preform the scan on all, so I dispatch it as a parallel process (multiple workflows runs). 

On around 5 VC (in parallel) I can scan in total around 25.000 VMs in total in 4-5 minutes. I am still not happy with the speed, but I dont think I can be done much faster; at least I haven't found a way for it. 

So check out this example. The "vc" is a specific vCenter/SDK you specify in you workflow input. 

var vms = vc.getAllVirtualMachines();

       

for each(var vm in vms)

{

    if(vm.snapshot)

    {

        System.log("");

        System.log("vm: " + vm.name);

        var rootSnapshots = vm.snapshot.rootSnapshotList;

        for (var j in rootSnapshots)

        {

            getSnapshotInfo(rootSnapshots[j]);

        }

    }

}

function getSnapshotInfo(snapshotTree)

{

    System.log("snapshot: " + snapshotTree.name);

    var childSnapshots = snapshotTree.childSnapshotList;

    for (var i in childSnapshots)

    {

        getSnapshotInfo(childSnapshots[i]);

    }

}

-------------------------------------------