PowerCLI

 View Only
Expand all | Collapse all

Execute multiple Powershell tasks in parallel

  • 1.  Execute multiple Powershell tasks in parallel

    Posted Jul 11, 2019 08:00 AM

    Currently scenario:

    Read an xml file and Deploy VM's, Customize them and then invoke Installation.

    The above process takes almost 25 minutes for each VM. So in a foreach loop if i have 4 VM's, it takes around 1.5 ~ 2 hours

    Is there any way to perform the task in parallel and reduce the execution time.



  • 2.  RE: Execute multiple Powershell tasks in parallel

    Posted Jul 11, 2019 08:09 AM

    Yes, you could use background jobs (with Start-Job).



  • 3.  RE: Execute multiple Powershell tasks in parallel

    Posted Jul 11, 2019 10:23 AM

    I tried using workflow as below,

    $VM = "R511-SRVA-WyU3", "R511-SRVB-0DV1"

    workflow pbatch{

    param ([string[]]$VM)

    foreach -parallel($a in $VM){

    $installscript = "hostname"

    $invokeinstalltask = Invoke-VMScript -ScriptText $installscript -VM $a -GuestUser "abc" -GuestPassword "Pass" -RunAsync

    }

    }

    pbatch -VM $VM

    but it gives me error as " Invoke-VMScript You are not currently connected to any servers. Please connect first using a Connect cmdlet. ".

    The error is strange because i dont get this error if i run it in a simple foreach loop.

    I Tried using Jobs as well with below example but the JOBs didn't stop even after 2 hrs. Ideally, the Get VM should not take 2 hrs when the VM name is provided.

    $abc = "R511-SRVA-WyU3", "R511-SRVB-0DV1"

    $block = {

    param ([string[]]$a)

    Get-VM -Name $a

    }

    $MaxThreads = 2

    #Remove all jobs

    Get-Job | Remove-Job

    foreach($a in $abc){

        While ($(Get-Job -state running).count -ge $MaxThreads){

            Start-Sleep -Milliseconds 3

        }

        Start-Job -Scriptblock $Block -ArgumentList $a

    }

    #Wait for all jobs to finish.

    While ($(Get-Job -State Running).count -gt 0){

        start-sleep 1

    }

    #Get information from each job.

    foreach($job in Get-Job){

        $info= Receive-Job -Id ($job.Id)

    }

    #Remove all jobs created.

    Get-Job | Remove-Job



  • 4.  RE: Execute multiple Powershell tasks in parallel

    Posted Jul 11, 2019 10:54 AM

    To use background jobs, you have to reconnect inside the job.

    See my Running a background job.



  • 5.  RE: Execute multiple Powershell tasks in parallel

    Posted Jul 18, 2019 05:35 AM

    Hi LucD, i tried the way you did parallel jobs it works fine with powercli functions, but i am not able to use my custom function.

    it fails with error as

    The term 'Get-vspherevmname' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify

    that the path is correct and try again.

        + CategoryInfo          : ObjectNotFound: (Get-vspherevmname:String) [], CommandNotFoundException

        + FullyQualifiedErrorId : CommandNotFoundException

        + PSComputerName        : localhost

    Can you please help?



  • 6.  RE: Execute multiple Powershell tasks in parallel

    Posted Jul 18, 2019 06:51 AM

    Where and how do you define your function?
    The job environment is a new PS environment, you might have to dot-source the file that contains your function inside the job.



  • 7.  RE: Execute multiple Powershell tasks in parallel

    Posted Jul 18, 2019 08:28 AM

    currently i have the Function created as a separate ps1.

    I run that ps1 file before executing my main script.

    Sorry i am not an expert on Powershell, but your guidance will help..

    Do you have any example where you have executed custom Function in parallel..



  • 8.  RE: Execute multiple Powershell tasks in parallel

    Posted Jul 18, 2019 08:33 AM

    You have to make your function known to the PS engine.
    The PS engine in a background job can be considered as new.

    So all functions that the calling script knows, are not known in the background job.

    That is why you need to dot-source the .ps1 file inside the background job.
    In the below example I assume the file is called myfunction.ps1 which contains the function Do-MyFunction.
    Note the space between the 2 dots when dot-sourcing a .ps1 file.

    $block = {

       param ([string[]]$a)


       . .\myfunction.ps1

       Do-MyFunction

    }



  • 9.  RE: Execute multiple Powershell tasks in parallel

    Posted Jul 18, 2019 09:14 AM

    Hi LucD, I am still getting the error as, "Get-CIVM You are not currently connected to any servers. Please connect first using a Connect cmdlet."

    Below is the code where i am trying the Job..  The get-civm is used in my function. Can you please check...

    #-----------------------------------------------------------------------------------------------------------------------------------------------

    $code = {

        param(

            [string]$Server,

            [string]$SessionId,

            [string]$VMName,

            [string]$org,

            [string]$vapp

        )

        Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

        Connect-VIServer -Server $Server -Session $SessionId

        . C:\Migration_Powershell_Scripts\New_functions\Get-vspherevmname.ps1

        get-vspherevmname -vmname $VMName -org $org -vapp $vapp -Server $Server -SessionId $SessionId

    }

    $vmName = 'R511-SRVA'

    $org = "ID14434-1-1231231"

    $vapp = "12345"

    $sJOb = @{

        ScriptBlock = $code

        ArgumentList = $global:DefaultVIServer.Name, $global:DefaultVIServer.SessionId, $vmName, $org, $vapp

    }

    $ja = Start-Job @sJob

    #-----------------------------------------------------------------------------------------------------------------------------------------------------------

    My Custom Function

    Function Get-vspherevmname{

        [CmdletBinding()]

        [OutputType([psobject])] # Select the output type of this function as "string/PSObject/Int"

    # Enter the list of Input Parameters for the function

    #The Parameter should have properties as "Mandatory", HelpMessage

    Param (

            [Parameter(Mandatory=$true, HelpMessage='VM name on VCD')] 

            [ValidateNotNull()]

            [ValidateNotNullOrEmpty()]

            [string]$vmname,

            [Parameter(Mandatory=$true, HelpMessage='Org name on VCD')] 

            [ValidateNotNull()]

            [ValidateNotNullOrEmpty()]

            [string]$org,

            [Parameter(Mandatory=$true, HelpMessage='Vapp name on VCD')] 

            [ValidateNotNull()]

            [ValidateNotNullOrEmpty()]

            [string]$vapp,

            [Parameter(Mandatory=$true, HelpMessage='VI Server Name')] 

            [ValidateNotNull()]

            [ValidateNotNullOrEmpty()]

            [string]$Server,

            [Parameter(Mandatory=$true, HelpMessage='VI connect - Session ID')] 

            [ValidateNotNull()]

            [ValidateNotNullOrEmpty()]

            [string]$SessionId

           )

    Begin{

    }

        Process {

            Try{

            #Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

            

                Connect-VIServer -Server $Server -Session $SessionId

                $vm = Get-CIVM -Org $org -VApp $vapp -Name $vmname

                $vmhref = $vm.ExtensionData.Href

                $vmstate = Invoke-vCloud -URI $vmhref -Method GET

                $vmnvramname = $vmstate.Vm.VirtualHardwareSection.ExtraConfig | Where-Object {$_.key -eq "nvram"}

                $vspherevmname = $vmnvramname.value.Split(".",6)[0]

                return $vspherevmname

            }

            Catch{Write-host "Catch Block"}

                }

        end     {

            Write-Verbose 'VM Name Read'

                }

    }



  • 10.  RE: Execute multiple Powershell tasks in parallel

    Posted Jul 18, 2019 09:20 AM

    You don't do a Connect-CIServer cmdlet in the background job, then it is normal I guess that you get that error on the Get-CIVm cmdlet.

    The Connect-CIServer also supports a SessionId parameter, by which you can reuse an existing connection in the calling script.



  • 11.  RE: Execute multiple Powershell tasks in parallel

    Posted Jul 19, 2019 10:54 AM

    Thanks a lot LucD.. I am able to get the output if i run it for one job, But have a problem while executing multiple jobs.. The output returned for is not correct. It always returns the output of the last VM queried.. Could you please find the mistake in my code...

    Below is the Job

    $code = {

        param(

            [string]$VIServer,

            [string]$VISessionId,

            [string]$VMName,

            [string]$org,

            [string]$vapp,

            [string]$CIServer,

            [string]$CISessionId

        )

         $rem1 = Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

        $rem2 = Connect-VIServer -Server $VIServer -Session $VISessionId

        $rem3 = Connect-CIServer -Server $CIServer -SessionId $CISessionId

        . C:\Migration_Powershell_Scripts\New_functions\Get-vspherevmname.ps1

        get-vspherevmname -vmname $VMName -org $org -vapp $vapp -CIServer $CIServer -CISessionId $CISessionId -VIserver $VIServer -VIsessionid $VISessionId

    }

    $VM = @("R511-SRVA", "R511-SRVB", "VEPMASTER")

    $results = @()

    $org = "ID14434-1-1231231"

    $vapp = "12345"

    $sJOb = @{

        ScriptBlock = $code

        ArgumentList = $global:DefaultVIServer.Name, $global:DefaultVIServer.SessionId, $vmName, $org, $vapp, $global:DefaultCIServers.Name, $global:DefaultCIServers.SessionId

    }

    $VM = @("R511-SRVA", "R511-SRVB", "VEPMASTER")

    $results = @()

    foreach($vmName in $VM){

    $j = Start-Job @sJob

    Wait-Job $j

    $row= new-object -TypeName PSObject -Property @{

        'VM_name' = $VMname

        'Job_ID' = $j.Id

        'vspherename' = Receive-Job -Id $j.Id -Keep

        }

    $results += $row

    }

    Output is

    Id     Name            PSJobTypeName   State         HasMoreData     Location             Command                 

    --     ----            -------------   -----         -----------     --------             -------                 

    107    Job107          BackgroundJob   Completed     True            localhost            ...                     

    VEPMASTER-ryQF

    109    Job109          BackgroundJob   Completed     True            localhost            ...                     

    VEPMASTER-ryQF

    111    Job111          BackgroundJob   Completed     True            localhost            ...                     

    VEPMASTER-ryQF

    Here i am expecting the return value to be different..

    My Custom Function is

    Function Get-vspherevmname{

        [CmdletBinding()]

        [OutputType([psobject])] # Select the output type of this function as "string/PSObject/Int"

    # Enter the list of Input Parameters for the function

    #The Parameter should have properties as "Mandatory", HelpMessage

    Param (

            [Parameter(Mandatory=$true, HelpMessage='VM name on VCD')] 

            [ValidateNotNull()]

            [ValidateNotNullOrEmpty()]

            [string]$vmname,

            [Parameter(Mandatory=$true, HelpMessage='VM name on VCD')] 

            [ValidateNotNull()]

            [ValidateNotNullOrEmpty()]

            [string]$org,

            [Parameter(Mandatory=$true, HelpMessage='VM name on VCD')] 

            [ValidateNotNull()]

            [ValidateNotNullOrEmpty()]

            [string]$vapp,

            [Parameter(Mandatory=$false, HelpMessage='Server Name')] 

            [ValidateNotNull()]

            [ValidateNotNullOrEmpty()]

            [string]$CIServer,

            [Parameter(Mandatory=$false, HelpMessage='VM name on VCD')] 

            [ValidateNotNull()]

            [ValidateNotNullOrEmpty()]

            [string]$CISessionId,

            [Parameter(Mandatory=$false, HelpMessage='VM name on VCD')] 

            [ValidateNotNull()]

            [ValidateNotNullOrEmpty()]

            [string]$VIserver,

            [Parameter(Mandatory=$false, HelpMessage='VM name on VCD')] 

            [ValidateNotNull()]

            [ValidateNotNullOrEmpty()]

            [string]$VIsessionid

           )

    Begin{

    }

        Process {

            Try{

            #Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

                $ignore = Set-PowerCLIConfiguration -DisplayDeprecationWarnings $false -Confirm:$false | Out-Null

                $ciconnect = Connect-CIServer -Server $ciserver -Session $cisessionid

                $viconnect =  Connect-VIServer -Server $viServer -Session $viSessionId

                $vm = Get-CIVM -Org $org -VApp $vapp -Name $vmname

                $vmhref = $vm.ExtensionData.Href

                #Write-Host $vmhref

                ##[string]$ApiVersion = '31.0'

                ##$Headers = @{ "x-vcloud-authorization" = $vcloudtoken; "Accept" = 'application/*+xml;version=' + $ApiVersion }

                ##$body = $null

                ##Invoke-RestMethod -Method GET -Uri $vmhref -Headers $Headers -Body $body

                . 'C:\Migration_Powershell_Scripts\New_functions\Invoke-vCloud.ps1'

                $vmstate = Invoke-vCloud -URI $vmhref -Method GET -vCloudToken $CISessionId

                $vmnvramname = $vmstate.Vm.VirtualHardwareSection.ExtraConfig | Where-Object {$_.key -eq "nvram"}

                $vspherevmname = $vmnvramname.value.Split(".",6)[0]

                #$Headers = @{ "x-vcloud-authorization" = $mySessionID; "Accept" = 'application/*+xml;version=' + $ApiVersion }

                return $vspherevmname

            }

            Catch{Write-host "Catch Block"}

                }

        end     {

            Write-Verbose 'VM Name Read'

                }

    }



  • 12.  RE: Execute multiple Powershell tasks in parallel

    Posted Jul 19, 2019 11:06 AM

    You should be filling the hash table ($sJob) inside the ForEach loop.

    Now you are starting each job with the same parameters,



  • 13.  RE: Execute multiple Powershell tasks in parallel

    Posted Jul 22, 2019 07:36 AM

    Thanks a lot LucD, you have been my saviour...

    i was able to make many of the tasks to run in parallel. But, i am facing issue in executing "Invoke-vmscript".

    I have a custom function to execute multiple tasks using invoke-vmscript. the problem here is,

    when i execute the custom script as a Job, only the 1st call of "invoke-vmscript" gets executed. Rest are not..



  • 14.  RE: Execute multiple Powershell tasks in parallel

    Posted Jul 22, 2019 08:09 AM

    This depends on what you are doing in the code submitted through Invoke-VMScript.

    I would need to see your code to understand what is happening.



  • 15.  RE: Execute multiple Powershell tasks in parallel

    Posted Jul 22, 2019 09:36 AM

    below are the scripts which are invoked..

                #Create a Scheduled task

                $Scheduletask = '

                $ST = New-ScheduledTaskAction -Execute "\\VEPMASTER\ESIS\setup.exe"

                $register = Register-ScheduledTask -Action $ST -TaskName "Experion-Installation" -Description "Task for invoking the experion Installation on this node"

                '

                $createscheduledtask = Invoke-VMScript -ScriptText $Scheduletask -VM $VSpherevm -GuestUser $username -GuestPassword $password -RunAsync

                Start-Sleep -Seconds 30

                do{

                Start-Sleep -Seconds 2

                }until($createscheduledtask.State -eq "Success")

                #Set-up Autologon to the VM

                $autologonscript = '

                $RegPath = "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon"

                $s1 = Set-ItemProperty $RegPath "AutoAdminLogon" -Value "1" -type String 

                $s2 = Set-ItemProperty $RegPath "DefaultUsername" -Value "administrator" -type String 

                $s3 = Set-ItemProperty $RegPath "DefaultPassword" -Value "Password1" -type String

                $s4 = Set-ItemProperty $RegPath "AutoLogonCount" -Value "2" -type DWord

                '

                $autologontask = Invoke-VMScript -ScriptText $autologonscript -VM $VSpherevm -GuestUser $username -GuestPassword $password -RunAsync

                do{

                Start-Sleep -Seconds 2

                }until($autologontask.State -eq "Success")

                #Invoke the Scheduled task on VM

                $installscript = '

                Start-ScheduledTask -TaskName "Experion-Installation"

                '

                $invokeinstalltask = Invoke-VMScript -ScriptText $installscript -VM $VSpherevm -GuestUser $username -GuestPassword $password -RunAsync



  • 16.  RE: Execute multiple Powershell tasks in parallel
    Best Answer

    Posted Jul 22, 2019 09:54 AM

    Why are you using the RunAsync switch on the Invoke-VMScript cmdlet?
    Since you are running this job in parallel for multiple VMs, there is no real need to use the RunAsync switch.

    That would also allow you to get rid of the loops after each Invoke-VMScript.

    ---------------------------------------------------------------------------------------------------------

    Was it helpful? Let us know by completing this short survey here.



  • 17.  RE: Execute multiple Powershell tasks in parallel

    Posted Jul 24, 2019 11:19 AM

    Thanks a lot LuCD.



  • 18.  RE: Execute multiple Powershell tasks in parallel

    Posted Dec 04, 2019 10:37 AM

    Hello NikhilPathak,

    ​I'm looking for means to automate VMs deployment process and as I can see you've already done it. Would you be so kind to share with your PowerShell scripts, functions, modules. Maybe you have your own repository on GitHub or website or something else...

    ​Thank you in advance.

    P.S. I'm writing you here because private messaging isn't working at the moment.