Azure Hybrid Worker Fails New-VM PowerCLI

[Praxters]

So I have installed a Azure Hybrid worker on my script server.

I can launch a script with Local System / Local Service or Network Service as a scheduled task or interactively.

But running it through Test Pane in fails on the Azure Runbook if fails with a rather non-informed error

New-VM : 6/27/2018 4:05:30 PM New-VM Could not find item C:\ProgramData.

At line:56 char:1

+ New-VM -Name $VMname -template $myTemplate -location $env -ResourcePo ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [New-VM], VimException

    + FullyQualifiedErrorId : Core_BaseCmdlet_UnknownError,VMware.VimAutomation.ViCore.Cmdlets.Commands.NewVM

Paired it down my script , to make it simple

Param (
[String]$VMNamesuffix = 'MyTestServerLocal',
[string]$Env = "Test01",
[String]$VCServiceAccount ="Domain\svc_UserAccount",
[String]$VCServicePassword = "SomePassword"
)
## Varaibles ##
$VMname = ($Env+"-"+$VMNameSuffix).ToUpper()
$sourcevc = "vcenter01.domain.test"
$TargetDataStore = "TFC_XIO_"+$env
$sourceDS = 'TFC_XIO_UTILITY'
$ResourcePool = 'GoldRP'
# List of modules to be loaded
$moduleList = @(
    "VMware.VimAutomation.Core",
    "VMware.VimAutomation.Vds",
    "VMware.VimAutomation.Cloud",
    "VMware.VimAutomation.PCloud",
    "VMware.VimAutomation.Cis.Core",
    "VMware.VimAutomation.Storage",
    "VMware.VimAutomation.HorizonView",
    "VMware.VimAutomation.HA",
    "VMware.VimAutomation.vROps",
    "VMware.VumAutomation",
    "VMware.DeployAutomation",
    "VMware.ImageBuilder",
    "VMware.VimAutomation.License"
    )
# Load modules
function LoadModules(){

   $loaded = Get-Module -Name $moduleList -ErrorAction Ignore | % {$_.Name}
   $registered = Get-Module -Name $moduleList -ListAvailable -ErrorAction Ignore | % {$_.Name}
   $notLoaded = $registered | ? {$loaded -notcontains $_}
 
   foreach ($module in $registered) {
      if ($loaded -notcontains $module) {
        
   Import-Module $module

      }
   }
}
#Run LoadModules function
LoadModules

$VCSecurePassword = Convertto-SecureString -String $VCServicePassword -AsPlainText -force
$VCCreds = New-object System.Management.Automation.PSCredential $VCServiceAccount ,$VCSecurePassword
Connect-VIServer -Server $sourcevc -Credential $VCCreds

$SourceDataStoreOB = Get-Datastore $SourceDataStore
$TargetDataStoreOB= Get-Datastore $TargetDataStore

$myTemplate = Get-Template -Name "WinTemplate" -Location Templates
New-VM -Name $VMname -template $myTemplate -location $env -ResourcePool $ResourcePool -Datastore $TargetDataStoreOB -Verbose

[LucD]

Can't really verify a run of your script through a Azure Hybrid worker, but some observations:

• you load the PowerCLI modules, but can you verify they are actually loaded (I guess yes, since Connect-VIServer, Get-Datastore, Get-Template seem to work)
• did you already try by passing a Folder object (returned by Get-Folder) instead of a string on the Location parameter?
• can you add the Verbose switch on the PowerCLI cmdlet?
• can you capture more info on the exception (perhaps use the Resolve-Error function from PowerCLI Support Breakdown)

[Praxters]

So actually it the azure edit run book has a nice "Test Pane" it returns the output in near real time.

Couple things, yes I removed the variables and hardcoded values, that didn't work for me and I did -verbose tag at the end and that didn't make a difference.

I know that cmdlet mostly work as I can call $Mytemplate,  $SourceDataStoreOB and $TargetDataStoreOB, and it returns information as expected.

I can even do get-vm and that works just not ... new-vm

Starting to feel as this I bug in the cmdlet .. as the response is a VMware exception..

Ill look at the resolve-error function next ... just I don't have enough information solve this.

[RemyW]

Is there any update or solution. We are running into the same issue. The new-vm cmdlet runs fine when we run it outside of the hybrid worker (even on the hybrid worker machine with the same credentials as the hbrid worker process). But run from a Azure automation account it fails with the error “New-VM Could not find item C:\ProgramData.”

It’s a weird issue, we have done several tests that show the script can read/write to the C:\ProgramData folder. Also process monitor shows the hybrid worker process reading files from the C:\ProgramData\VMware folder. Debug doesn't give any information beyond "VERBOSE: 9/14/2018 11:12:06 AM New-VM Started execution."  After that it fails.

Specs

PSVersion                      5.1.1439

PSEdition                      Desktop

PSCompatibleVersions           {1.0, 2.

BuildVersion                   10.0.143

CLRVersion                     4.0.3031

WSManStackVersion              3.0

PSRemotingProtocolVersion      2.3

SerializationVersion           1.1.0.1

Name                                Version

----                                -------

VMware.DeployAutomation             6.7.0.8250345

VMware.ImageBuilder                 6.7.0.8250345

VMware.PowerCLI                     10.2.0.9372002

VMware.Vim                          6.7.0.9288211

VMware.VimAutomation.Cis.Core       10.1.0.8377811

VMware.VimAutomation.Cloud          10.0.0.7893901

VMware.VimAutomation.Common         10.1.0.8342134

VMware.VimAutomation.Core           10.1.0.8344055

VMware.VimAutomation.HA             6.5.4.7567193

VMware.VimAutomation.HorizonView    7.5.0.8827468

VMware.VimAutomation.License        10.0.0.7893904

VMware.VimAutomation.Nsxt           10.2.0.9296602

VMware.VimAutomation.PCloud         10.0.0.7893924

VMware.VimAutomation.Sdk            10.1.0.8342078

VMware.VimAutomation.Srm            10.0.0.7893900

VMware.VimAutomation.Storage        10.1.0.8313015

VMware.VimAutomation.StorageUtility 1.2.0.0

VMware.VimAutomation.Vds            10.1.0.8344219

VMware.VimAutomation.Vmc            10.0.0.7893902

VMware.VimAutomation.vROps          10.0.0.7893921

VMware.VumAutomation                6.5.1.7862888

Thanks,

Remy

[VincentFl]

It's work if you change the working directory.
for exemple add this line before new-vm command: cd 'C:\temp'

[RemyW]

Thank you verry much!