VMware vSphere

 View Only
Expand all | Collapse all

Cannot communicate with ESXi through CIM (port 5989)

  • 1.  Cannot communicate with ESXi through CIM (port 5989)

    Posted Jun 16, 2016 09:03 AM

    We have recently started experiencing problems with communication with our ESXi devices through CIM, TCP port 5989. At first we thought this was related to our remote monitoring and maintenance tool, N-central by SolarWinds N-able, however we now have reason to believe that this is not the case.

    The reason for why I believe this is a Vmware is that I've attempted to reach one of the ESXi devices at 5989 by browsing to it with Internet Explorer. These attempts have all resulted in the HTTP error message 501/505 Not Implemented.

    Is anyone able to assist me in figuring out what is wrong, and fixing the error(s)? Please do let me know what information is required to best troubleshoot this issue.



  • 2.  RE: Cannot communicate with ESXi through CIM (port 5989)

    Posted Jun 16, 2016 09:41 AM

    HTTP error message 501/505 means the port is open, but you receive a binary stream that cannot be interpreted. So this is a good response and an indication you reached something. However I wouldnt use IE for testing if ports are open, and rather use Telnet from command line.



  • 3.  RE: Cannot communicate with ESXi through CIM (port 5989)

    Posted Jun 16, 2016 10:07 AM

    The reason for why I believe this is a Vmware is that I've attempted to reach one of the ESXi devices at 5989 by browsing to it with Internet Explorer. These attempts have all resulted in the HTTP error message 501/505 Not Implemented.

    That is normal. CIM is not supposed to work with generic browsers requests and mainly operates with HTTP POST instead of GET requests:

    #  curl -sv https://myesxi.local:5989

    * Connected to myesxi.local (10.1.1.99) port 5989 (#0)

    > GET / HTTP/1.1

    > User-Agent: curl/7.21.2 (x86_64-unknown-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8j zlib/1.2.5 libidn/1.10

    > Host: myesxi.local:5989

    > Accept: */*

    >

    < HTTP/1.1 501 Not Implemented

    < Server: sfcHttpd

    < Content-Length: 0

    <

    * Closing connection #0

    #  curl -sv https://myesxi.local:5989 --request POST

    * Connected to myesxi.local (10.1.1.99) port 5989 (#0)

    > POST / HTTP/1.1

    > User-Agent: curl/7.21.2 (x86_64-unknown-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8j zlib/1.2.5 libidn/1.10

    > Host: myesxi.local:5989

    > Accept: */*

    >

    < HTTP/1.1 401 Unauthorized

    < WWW-Authenticate: Basic realm="cimom"

    < Server: sfcHttpd

    < Content-Length: 0

    <

    * Closing connection #0

    Most likely your credentials for authentication on the hosts are wrong, or the user does not have sufficient permissions to access CIM info on the ESXi host. The user basically needs to have the builtin admin/root role on the host, see Configuring and securing local ESXi users for hardware monitoring via WBEM | alpacapowered

    You can check whether your user credentials and permissions are correct with the following, you should get a HTTP/200 response:

    # curl -vk 'https://myesxi.local:5989' --request POST --basic --user [MYUSERNAME]

    Enter host password for user '[MYUSERNAME]': *************

    * Connected to myesxi.local (10.1.1.99) port 5989 (#0)

    > POST / HTTP/1.1

    > Authorization: Basic [topsecret base64]

    > User-Agent: curl/7.21.2 (x86_64-unknown-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8j zlib/1.2.5 libidn/1.10

    > Host: myesxi.local:5989

    > Accept: */*

    >

    < HTTP/1.1 200 OK

    < Content-Type: application/xml; charset="utf-8"

    < Content-Length: 0

    < Cache-Control: no-cache

    < CIMOperation: MethodResponse

    <

    * Closing connection #0



  • 4.  RE: Cannot communicate with ESXi through CIM (port 5989)

    Posted Jun 16, 2016 10:14 AM

    Thanks for explaining this in such detail, I appreciate it! It would appear that I'm having some certificate problems, because this is the response I get from curl:

    C:\Users\ctadmin\Downloads\curl>curl -sv https://10.192.151.9:5989

    * Rebuilt URL to: https://10.192.151.9:5989/

    *   Trying 10.192.151.9...

    * Connected to 10.192.151.9 (10.192.151.9) port 5989 (#0)

    * schannel: SSL/TLS connection with 10.192.151.9 port 5989 (step 1/3)

    * schannel: checking server certificate revocation

    * schannel: using IP address, SNI is not supported by OS.

    * schannel: sending initial handshake data: sending 160 bytes...

    * schannel: sent initial handshake data: sent 160 bytes

    * schannel: SSL/TLS connection with 10.192.151.9 port 5989 (step 2/3)

    * schannel: encrypted data buffer: offset 1123 length 4096

    * schannel: next InitializeSecurityContext failed: SEC_E_UNTRUSTED_ROOT (0x80090

    325) - The certificate chain was issued by an authority that is not trusted.

    * Closing connection 0

    * schannel: shutting down SSL/TLS connection with 10.192.151.9 port 5989

    * schannel: clear security context handle

    * schannel: clear credential handle

    There has been absolutely no change to the certificate situation for the Vmware since the last time this worked. Do you have any idea why this all of a sudden has become a problem?



  • 5.  RE: Cannot communicate with ESXi through CIM (port 5989)

    Posted Jun 16, 2016 10:46 AM

    This is because curl does not trust your certificate. You can skip certificate checking with the "-k/--insecure" option.



  • 6.  RE: Cannot communicate with ESXi through CIM (port 5989)

    Posted Jun 16, 2016 11:00 AM

    Thank you so much MKguy! This is truly helpful :smileyhappy: Now I just need to figure out why N-central is sending a GET request.



  • 7.  RE: Cannot communicate with ESXi through CIM (port 5989)

    Posted Jun 16, 2016 11:20 AM

    You can also directly verify that the CIM application level data is being returned correctly with a request like this:

    # curl -vk 'myesxi.local:5989' --request POST --basic --user root --data '<?xml version="1.0" encoding="UTF-8"?>

      <CIM CIMVERSION="2.0" DTDVERSION="2.0">

        <MESSAGE ID="882670" PROTOCOLVERSION="1.0">

          <SIMPLEREQ>

            <IMETHODCALL NAME="EnumerateInstances">

              <LOCALNAMESPACEPATH>

                <NAMESPACE NAME="root"/>

                <NAMESPACE NAME="cimv2"/>

              </LOCALNAMESPACEPATH>

              <IPARAMVALUE NAME="ClassName">

                <CLASSNAME NAME="OMC_IPMIIPProtocolEndpoint"/>

              </IPARAMVALUE>

            </IMETHODCALL>

          </SIMPLEREQ>

        </MESSAGE>

      </CIM>'

    The Windows cmd may not work with the line breaks in the data string, remove all line breaks in that case or try < textfile.txt input redirection.

    Replace "OMC_IPMIIPProtocolEndpoint" from this example with the appropriate CIM Class you want to query. A list of Classes can be found here: http://pubs.vmware.com/vsphere-55/nav/7_0_2_1_1

    Or use this Powershell function I wrote to query CIM:

    https://github.com/alpacacode/Homebrewn-Scripts/blob/master/vmware-scripts/Get-CIMProperties.ps1



  • 8.  RE: Cannot communicate with ESXi through CIM (port 5989)

    Posted Jun 16, 2016 11:45 AM

    I attempted your PowerShell script, but this is the response I got:

    PS C:\Users\ctadmin\Documents> .\test.ps1

    cmdlet Get-CIMProperties at command pipeline position 1

    Supply values for the following parameters:

    Target: 10.192.151.9

    Enter the Username for a local ESXi user who is allowed to query the sfcb daemon CIM stack (e.g. root): root

    Using Username: root

    Enter the local ESXi User password: ********

    Error during CIM Request to https://10.192.151.9:5989/cimom with User root. Make sure the system is reachable, the username/password is correct and the user is

    allowed to query CIM information.

    Exception calling "GetResponse" with "0" argument(s): "The remote server returned an error: (501) Not Implemented.".Exception.Message

    At C:\Users\ctadmin\Documents\test.ps1:133 char:10

    +     Throw <<<<  "Error during CIM Request to $URI with User $Username. Make sure the system is reachable, the username/password is correct and the user is al

    lowed to query CIM information.`n$_.Exception.Message"

        + CategoryInfo          : OperationStopped: (Error during CI...ception.Message:String) [], RuntimeException

        + FullyQualifiedErrorId : Error during CIM Request to https://10.192.151.9:5989/cimom with User root. Make sure the system is reachable, the username/pass

       word is correct and the user is allowed to query CIM information.

    Exception calling "GetResponse" with "0" argument(s): "The remote server returned an error: (501) Not Implemented.".Exception.Message

    PS C:\Users\ctadmin\Documents>

    I'm pretty sure the root user has access to querying CIM :smileyhappy: Wouldn't you say?



  • 9.  RE: Cannot communicate with ESXi through CIM (port 5989)

    Posted Jun 16, 2016 12:51 PM

    Hm. works fine for me on various powershell versions to various ESXi hosts. Post the content of your test.ps1 script. You can also just paste the entire function definition into a powershell window and then interactively run the cmdlet with various parameters.

    Or maybe your cim is really somehow broken application-wise, but I doubt it.



  • 10.  RE: Cannot communicate with ESXi through CIM (port 5989)

    Posted Jun 16, 2016 12:55 PM

    This is the content of the PowerShell script, the only thing I added was the call to the actual function on the bottom line:

    function Get-CIMProperties {

    <#

    .SYNOPSIS

      Queries CIM management information for a specific CIM class from a host via WBEM.

    .DESCRIPTION

      This function can be used to query CIM information through a generic WBEM Web Request from an ESXi or any other compatible (especially for SFCB daemon based) host system.

      It returns an XML-styled response Object of all properties within the queried CIM class.

      An overview of the available CIM classes in ESXi and more information can be found here:

      http://pubs.vmware.com/vsphere-55/nav/7_0_2_1_1

      https://www.vmware.com/support/developer/cim-sdk/

    .PARAMETER Target

      DNS-Name or IP of the target host.

    .PARAMETER Username

      Username to use when authenticating against the target. If not supplied as a parameter, the function will query for user input. The user must have the appropriate permissions to query the CIM interface.

      On ESXi, only assigning the "Host - CIM - CIM Interaction" permission does NOT work, builtin administrator role or a workaround in /etc/security/access.conf is required. See here for more details:

      https://alpacapowered.wordpress.com/2013/09/27/configuring-and-securing-local-esxi-users-for-hardware-monitoring-via-wbem/

    .PARAMETER Password

      Password for the user. If not supplied as a parameter, the function will query for user input.

    .PARAMETER CIMClassName

      Name of the CIM class to query. By default the "OMC_IPMIIPProtocolEndpoint" class containing information about IPMI-based BMC/ILO/DRAC is queried.

    .PARAMETER CIMPort

      TCP port of the CIM daemon on the target host. Default is using port 5989 (sfcb-HTTPS-Daemon on ESXi).

    .PARAMETER Secure

      Whether to send the request via plain HTTP or SSL/TLS encrypted HTTPS. Default is encrypted HTTPS. Warning: Disabling will send the credentials in clear text.

    .OUTPUTS

      Powershell XML Object of the queried CIM properties.

    .EXAMPLE

      Get-CIMProperties -Target $Target -Username root

      Asks for a password and queries the host for IPMI-based BMC/ILO/DRAC information (default CIM class OMC_IPMIIPProtocolEndpoint) on default port https/5989.

    .EXAMPLE

      (Get-CIMProperties -Target $Target -Username root | ? {$_.NAME -eq 'IPv4Address'}).Value

      Asks for a password just returns the IPv4 address of the BMC/ILO/DRAC interface.

    .EXAMPLE

      Get-CIMProperties -Target $Target -Username admin -Password unhackable -CIMClassName OMC_PhysicalMemory -CIMPort 9001

      Returns the host's physical Memory CIM class properties on port https/9001 with username admin and an unhackable password.

    .LINK

      https://github.com/alpacacode/Homebrewn-Scripts

    .NOTES

      The latest version of this function can be found at Github:

      https://github.com/alpacacode/Homebrewn-Scripts

      Version History:

      1.0 20.03 2015   - Initial release

    #Requires -Version 2.0

    #>

      [CmdletBinding()]

      Param(

        [string][parameter(Position=0, Mandatory=$true, ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true)] $Target,

        [string]$Username = '',

        [string]$Password = '',

        [string]$CIMClassName = 'OMC_IPMIIPProtocolEndpoint',

        [int]$CIMPort = 5989,

        [bool]$Secure = 1

      )

      if ($Username -eq '') {

        $Username = Read-Host "Enter the Username for a local ESXi user who is allowed to query the sfcb daemon CIM stack (e.g. root)"

        Write "Using Username: $Username"

      }

      if ($Password -eq '') {

        [System.Security.SecureString]$Password = Read-Host "Enter the local ESXi User password" -AsSecureString

      }

      else {

        [System.Security.SecureString]$Password = ConvertTo-SecureString -String $Password -AsPlainText -force

      }

      #Build the target URI and the base64 [user:password] string to use for the HTTP basic Authentication header

      switch($Secure) {

        $true { $Protocol = 'https' }

        $false { $Protocol = 'http' }

      }

      $URI = [System.Uri]"${Protocol}://${Target}:$CIMPort/cimom"

      $BasicAuth = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes("${Username}:" + ([System.Runtime.InteropServices.Marshal]::PtrToStringAuto([System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($Password)))))

      #Custom HTTP headers for the Request

      $Headers = @{

        "Authorization" = "Basic $BasicAuth"

        "CIMMethod"= "EnumerateInstances"

        "CIMOperation" = "MethodCall"

        "CIMProtocolVersion" = "1.0"

        "CIMObject" = "root/cimv2"

      }

      #Content of the HTTP POST data

      $POSTBody = [byte[]][char[]]('<?xml version="1.0" encoding="UTF-8"?>

      <CIM CIMVERSION="2.0" DTDVERSION="2.0">

        <MESSAGE ID="882670" PROTOCOLVERSION="1.0">

          <SIMPLEREQ>

            <IMETHODCALL NAME="EnumerateInstances">

              <LOCALNAMESPACEPATH>

                <NAMESPACE NAME="root"/>

                <NAMESPACE NAME="cimv2"/>

              </LOCALNAMESPACEPATH>

              <IPARAMVALUE NAME="ClassName">

                <CLASSNAME NAME="' + $CIMClassName + '"/>

              </IPARAMVALUE>

            </IMETHODCALL>

          </SIMPLEREQ>

        </MESSAGE>

      </CIM>')

      #Deterministic pre-check to see if the target can be reached on this port. Only performed if the Test-NetConnection cmdlet is present (only available on Windows 8.1, Windows PowerShell 4.0, Windows Server 2012 R2)

      if((Get-Command Test-NetConnection -errorAction SilentlyContinue) -and (Test-NetConnection $Target -Port $CIMPort -InformationLevel Quiet) -eq $false) {

        Throw "Error: Could not establish a TCP connection to host $Target on port $CIMPort. Make sure the ESXi Firewall permits the connection and the sfcb daemon on the host is running.`n"

      }

      #Build the HTTP POST Request with all custom headers

      $Request = [System.Net.HttpWebRequest]::Create($URI)

      $Request.Method = 'POST'

      $Request.ContentType = 'application/xml; charset="utf-8"'

      $Headers.Keys | ForEach-Object { $Request.Headers.Add($_, $Headers[$_]) }

      #Disable sending the HTTP Expect header that would cause problems with the POST request and disable server certificate validation

      [System.Net.ServicePointManager]::Expect100Continue = $false

      [System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}

      #Send the Request and and ensure the response is valid

      Try {

        $Stream = $Request.GetRequestStream();

        $Stream.Write($POSTBody, 0, $POSTBody.Length)

        $Stream.Flush()

        $Stream.Close()

      

        #Read the Response

        $Response = $Request.GetResponse().GetResponseStream()

        $Stream = New-Object System.IO.StreamReader($Response)

        $ResponseTxt = $Stream.ReadToEnd()

        $Stream.Close()

        [xml]$XML = $ResponseTxt

      }

      Catch {

        Throw "Error during CIM Request to $URI with User $Username. Make sure the system is reachable, the username/password is correct and the user is allowed to query CIM information.`n$_.Exception.Message"

      }

      if($ResponseTxt -match '(<ERROR CODE=".+?/>)') {

        Throw "Error: The server CIM response contains an error:" + $matches[1] + "`nThe full server response was:`n$ResponseTxt"

      }

      Return $XML.CIM.MESSAGE.SIMPLERSP.IMETHODRESPONSE.IRETURNVALUE.'VALUE.NAMEDINSTANCE'.INSTANCE.PROPERTY

    }

    Get-CIMProperties



  • 11.  RE: Cannot communicate with ESXi through CIM (port 5989)

    Posted Mar 02, 2023 12:53 PM

    Hi, did you ever get this issue resolved? We're having the same problem with n-able.