PowerCLI

 View Only
Expand all | Collapse all

vSphere automation based on VM custom attribute values

  • 1.  vSphere automation based on VM custom attribute values

    Posted Feb 23, 2019 04:57 PM

    Hi LucD​ and Co. I'm about to take on an interesting pet project surrounding automation of vSphere "things" based on custom attribute values found on VMs and I'm posting an open question for ideas as to how to go about doing this. To be very clear, I'm not asking for people to hand over code to me--only a methodology for those who have worked with custom attributes in PowerCLI before. I know it's not difficult, per se, but since there are always multiple ways to skin a cat I'm wondering what the shortest, cleanest, most efficient path may be here.

    Background

    VMware PKS deploys K8s clusters as virtual machines. These clusters can be of various sizes and contain one or more masters and one or more workers. Each node is represented by a single VM. Every VM belonging to the same cluster has a custom attribute written into a field called "deployment". An example value of this "deployment" field is "service-instance_9c854520-8e43-4566-845e-111e6a1e8425". Additionally, a second field is present called "job" which specifies the role type for each VM, values of which are either "master" or "worker". All VMs in the same deployment are also connected to the same NSX-T logical switch which appears in vSphere's inventory as a custom switch type. For example, all VMs in the same deployment that have "service-instance_9c854520-8e43-4566-845e-111e6a1e8425" written into that custom attribute would exist on a logical switch named "pks-9c854520-8e43-4566-845e-111e6a1e8425". There are other logical switches created that also begin with this name but end with different suffixes.

    Problem Statement

    While this system is all well and good, there are some problems created and therefore gaps I wish to close:

    1. When a cluster is created containing multiple masters, if those masters land on the same vSphere cluster there are no anti-affinity DRS rules automatically created. This obviously creates the possibility that a single ESXi host failure takes out more than half of the masters effectively rendering a K8s cluster down.
    2. All VMs from a deployment/K8s cluster have names created in the format of "vm-{GUID}" and are dumped into the same vSphere inventory location. This makes it difficult for administrators to quickly pinpoint VMs associated with the same deployment/K8s cluster.

    Goals

    Based on the above problems/gaps, I wish to

    1. Automate the creation of anti-afinity DRS rules. In order to do this, a few different pieces of logic have to be implemented.
      1. VMs have to be scanned for the "deployment" custom attribute value. Alternatively, the logical switch can be scanned, but this object is not reported by the "Get-VDportgroup" or "Get-VirtualPortgroup" cmdlets.
      2. VMs need to be matched up based this "deployment" value. Alternatively, the logical switch can be scanned.
      3. The "job" value needs to be read on those sharing the same "deployment" value and, if there are more than 1 master found, the names of all masters need to be stored in memory.
      4. If all masters reside in the same vSphere cluster, an anti-affinity rule should be created with all masters as members forcing them to run on separate hosts.
    2. Automate the creation and organization of all VMs in a given deployment into a vSphere folder. Some slightly separate logic is needed here.
      1. VMs have to be scanned for the "deployment" custom attribute value.
      2. VMs need to be matched up based on this "deployment" value.
      3. A new folder should be created in a given inventory location with the name matching "deployment" value.
      4. All VMs sharing the same "deployment" value should be moved into the newly-created vSphere folder.

    Now that you have all the information, how would you tackle this from a high-level perspective? I think the emphasis needs to be on speed and as much reduced impact on the vCenter API as possible because these PKS VMs could exist in a sea of thousands, and recursing through the entire vCenter inventory would probably not go well. Interested in ideas, commentary, and questions.

    Message was edited by: Chip Zoller Added note about these NSX-T logical switches are not returned by PowerCLI available cmdlets in modules VMware.VimAutomation.Core or .Vds.



  • 2.  RE: vSphere automation based on VM custom attribute values

    Posted Feb 23, 2019 05:49 PM

    Without actually testing the code against the requirements, I think Group-Object might be your best friend here.

    It will let PowerShell do the required grouping for you.

    Since skeleton code is easier to read, compared to a lot of text (at least for me), I would start testing and validating with below code.

    I added some comments, documenting the main steps.


    PS: why CustomFields and not Tags?

    $caDeployment = 'deployment'

    $caJob = 'job'

    $rootFolderName = 'Project'


    # Folder where all 'deployment' folders shall go

    $folder = Get-Folder -Name $rootFolderName


    Get-VM -Name vm-* |

       Group-Object -Property {$_.CustomFields[$caDeployment]} |

       where {$_.Name} |

       ForEach-Object -Process {

       $deployment = $_.Name

       # Group the VMs in a deployment on job type

       Group-Object -InputObject $_.Group -Property {$_.CustomFields[$caJob]} |

       ForEach-Object -Process {

       # Handling the masters

       if ($_.Name -eq 'master') {

       $_.Group | Select Name, @{N = 'Cluster'; E = {Get-Cluster -VM $_ }} |

       Group-Object -Property Cluster |

       ForEach-Object -Process {

       # More than 1 master

       if ($_.Group.Count -gt 1) {

       $sRule = @{

       Cluster = $_.Name

       Name = "$($deployment) Anti-affinity rule"

       VM = $_.Group.Name

       KeepTogether = $false

       Confirm = $false

       }

       # Create the anti-affinity rule

       # Not taking into account when there are more masters than ESXi nodes

       New-DrsRule @sRule

       }

       }

       }

       # Create the VM folder for this deployment and move all VMs in there

       $targetFolder = New-Folder -Name "Deployment $($deployment)" -Location $folder -Confirm:$false

       $_.Group | Move-VM -InventoryLocation $targetFolder -Confirm:$false

       }

    }



  • 3.  RE: vSphere automation based on VM custom attribute values

    Posted Feb 23, 2019 06:07 PM

    Thanks very much, Luc! Although I didn't intend for you to spend your Saturday helping me, I thank you for that code :smileyhappy: Let me play with it for a while and see what I can come up with. I think you're spot on that Group-Object is the way to go here and eliminates a lot of manual matching effort.

    PS: why CustomFields and not Tags?

    This is just how PKS works, and there's no way to either add Tags in addition to CAs or use one in place of the other. CAs wouldn't be my choice, but that's what they did...



  • 4.  RE: vSphere automation based on VM custom attribute values

    Posted Feb 23, 2019 06:10 PM

    I just happened to be stuck on some other code, saw your thread passing by, and since I find tackling questions easier with code than long winded sentences... :smileygrin:



  • 5.  RE: vSphere automation based on VM custom attribute values

    Posted Feb 23, 2019 10:32 PM

    After some tweaking it seems to work great, except the anti-affinity rule(s) doesn't get created. Honestly haven't looked into that much though. Now I just need to make it more idempotent so it can be scheduled and only perform those actions if not done (which should be easy with some -ErrorAction SilentlyContinue statements.



  • 6.  RE: vSphere automation based on VM custom attribute values

    Posted Feb 23, 2019 10:45 PM

    You might want to display the content of $_.Group just before the If line, where the number of masters is checked.



  • 7.  RE: vSphere automation based on VM custom attribute values

    Posted Feb 23, 2019 11:04 PM

    Seems like posh isn't liking double grouping or nested group structures of some sort because it's returning no objects there. I'll have to fiddle with this since I've not used Group-Objects before.



  • 8.  RE: vSphere automation based on VM custom attribute values

    Posted Feb 23, 2019 11:07 PM

    Time to use the VSC debugger (it has breakpoints) :smileywink:



  • 9.  RE: vSphere automation based on VM custom attribute values

    Posted Feb 23, 2019 11:30 PM

    I see what it is. The second grouping (Group-Object -InputObject $_.Group -Property {$_.CustomFields[$caJob]}) is returning null in the Name property, so the code basically stops executing there since the if statement is going to be false. What's also funny is the formatting in tabular form. If I check this snippet in ISE, the "Group" name have VMs formatting without comma separation (like not an array?) versus if I just execute the first Group-Object statement.

    Get-VM -Name vm-* | Group-Object -Property {$_.CustomFields['deployment']} | where {$_.Name -like 'service-instance_*'} | ForEach-Object -Process {Group-Object -InputObject $_.Group -Property {$_.CustomFields['job']}}



  • 10.  RE: vSphere automation based on VM custom attribute values

    Posted Feb 24, 2019 07:03 AM

    Can you show some sample out from

    Get-VM -Name vm-* | Group-Object -Property {$_.CustomFields['deployment']} | where {$_.Name -like 'service-instance_*'}

    and

    Get-VM -Name vm-* | Group-Object -Property {$_.CustomFields['deployment']} | where {$_.Name -like 'service-instance_*'} |

    ForEach-Object -Process {

       Group-Object -InputObject $_.Group -Property {$_.CustomFields['job']}

    }



  • 11.  RE: vSphere automation based on VM custom attribute values

    Posted Feb 24, 2019 08:52 AM

    Get-VM -Name vm-* | Group-Object -Property {$_.CustomFields['deployment']} | where {$_.Name -like 'service-instance_*'} | fl

    Name   : service-instance_9c854520-8e43-4566-845e-111e6a1e8425

    Count  : 6

    Group  : {vm-785dd36f-d984-4700-9b8d-54381758d141, vm-46e24f81-d4ad-4722-9cc3-1c70b631663d, vm-14876478-6e6e-4a00-933f-7bafecbef991, vm-bcc0550b-5203-48f0-a674-d59543ddb5e0...}

    Values : {service-instance_9c854520-8e43-4566-845e-111e6a1e8425}

    and

    Get-VM -Name vm-* | Group-Object -Property {$_.CustomFields['deployment']} | where {$_.Name -like 'service-instance_*'} |

    ForEach-Object -Process {

      Group-Object -InputObject $_.Group -Property {$_.CustomFields['job']} | fl

    }

    Name   :

    Count  : 1

    Group  : {vm-785dd36f-d984-4700-9b8d-54381758d141 vm-46e24f81-d4ad-4722-9cc3-1c70b631663d vm-14876478-6e6e-4a00-933f-7bafecbef991 vm-bcc0550b-5203-48f0-a674-d59543ddb5e0 vm-063992a7-c9f3-4cee-9287-27e4a4ac15b1 vm-73d25407-4e22-4e70-a2af-825b33ede569}

    Values : {$null}

    Sorry I didn't provide this earlier. Was being hurried to dinner :smileyhappy: Boarding a flight in a few hours so may get time to play around some more with the second grouping to figure out why it's not picking that Name up.



  • 12.  RE: vSphere automation based on VM custom attribute values
    Best Answer

    Posted Feb 24, 2019 01:18 PM

    Ok, there seems to be an "issue" when using Group-Object with the InputObject parameter.

    And there was a logic error in my original skeleton.

    This should work better.

    $caDeployment = 'deployment'

    $caJob = 'job'


    $rootFolderName = 'Project'


    # Folder where all 'deployment' folders shall go

    $folder = Get-Folder -Name $rootFolderName


    Get-VM -Name vm-* |

       Group-Object -Property {$_.CustomFields[$caDeployment]} |

       where {$_.Name} |

       ForEach-Object -Process {

       $deployment = $_.Name

       # Group the VMs in a deployment on job type

       $_.Group | Group-Object -Property {$_.CustomFields[$caJob]} |

       ForEach-Object -Process {

       # Handling the masters

       if ($_.Name -eq 'master') {

       $_.Group | Select Name, @{N = 'Cluster'; E = {Get-Cluster -VM $_ }} |

       Group-Object -Property Cluster |

       ForEach-Object -Process {

       # More than 1 master

       if ($_.Group.Count -gt 1) {

       $sRule = @{

       Cluster = $_.Name

       Name = "$($deployment) Anti-affinity rule"

       VM = $_.Group.Name

       KeepTogether = $false

       Confirm = $false

       }

       # Create the anti-affinity rule

       # Not taking into account when there are more masters than ESXi nodes

       New-DrsRule @sRule

       }

       }

       }

       }

       # Create the VM folder for this deployment and move all VMs in there

       $targetFolder = New-Folder -Name "Deployment $($deployment)" -Location $folder -Confirm:$false

       $_.Group | Move-VM -InventoryLocation $targetFolder -Confirm:$false | Out-Null

    }



  • 13.  RE: vSphere automation based on VM custom attribute values

    Posted Feb 24, 2019 01:50 PM

    Nice, Luc, that's it exactly. Need to go back and compare your code to mine so I understand this for future reference. Cheers to you for the help, Luc! Now on to the next task!