An error occurred while communicating with the remote host

[Poestges]

Hi community,

I have a problem with my VC2.5 and ESX3.5.

50% of all actions results in "An error occurred while communicating with the remote host". The 2nd try mostly works, sometimes i have to do another try.

I start an action, the host disconnects and a few seconds later the host is back and i can give it another try. The error occures on every host i have.

Can anyone help me, please?

TIA,

Sascha

[jrenton]

Could be a DNS issue. Do you have a DNS server in the DNS domain your ESX servers reside in? I had a similar issue and I added both hosts in my cluster to /etc/hosts file on each ESX server. Problem was that each ESX server in the cluster could not communicate with the other host.

Hope this helps.

John

[Poestges]

Hi John,

the DNS was my first thought, too. I checked the hosts files on all ESX and VC. Everything is fine.

Sascha

[kjb007]

Make sure that your hostname matches what you have configured. Meaning, make sure when you run the commands 'hostname' and 'hostname -s', that you see what you are supposed to see. Also, make sure you add the hosts using their FQDN, and not their IPs.

-KjB

[admin]

Has this always been a problem?

Did you replace the VC ssl certificates with new ones?

Is the ESX server swapping heavily or low on disk space?

free -m

df -h

Have a look at the vpxa and hostd logs on the ESX server at the time of the disconnects for any errors/warnings:

/var/log/vmware/vpx/vpxa.log

/var/log/vmware/hostd.log

[Poestges]

The hostnames are all ok.

The VI was running fine for 18 month, the strange behavior is for 1 month now.

I just received the error during a VM migration.

- The migration starts

- I got the error message,

- The host got disconnected for a few seconds and reconnects

- I checked the VM. It was migrated succesfully.

So It seems to me that the connections between the hosts are runing fine, and that the problem is between the VirtualCenter and the hosts.

[admin]

So this happens with all your hosts and only for migration tasks?

Have you replaced the VirtualCenter ssl certs with new ones?

Have you tried restarting the VirtualCenter server service or re-creating the cluster?

Do they VC logs show anything unusual at the times of the disconnects?

Trigger the error, wait a few mins, then attach the latest vpxd log to a post. The logs are normally stored somewhere like C:\Documents and Settings\All Users\Application Data\VMware\VirtualCenter\Logs\

[Poestges]

- This happens to all my hosts. I added 3 new hosts last week and they show the same behavior.

Not only for migration tasks, every tasks (like working with Snapshots, add Ports to a vSwitch, edit VM settings, etc.) can result to the error.

Sometimes I can work for hours and everything is allright, and sometimes nearly every action leads to the error.

- The VirtualCenter certs are still the original

- I restarted the service and server several times - no luck

I wait for the next error now, and I'll post the log.

[Poestges]

I started a HBA rescan and the error occurred .. here is the log:

2008-06-24 13:56:28.098 'App' 1444 info] -- FINISH task-internal-122423 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

-- BEGIN task-internal-122424 -- -- vmodl.query.PropertyCollector.retrieveContents -- 4A939008-5249-45C5-ACCA-BB0B02E93E9D

-- FINISH task-internal-122424 -- -- vmodl.query.PropertyCollector.retrieveContents -- 4A939008-5249-45C5-ACCA-BB0B02E93E9D

-- BEGIN task-7339 -- host-2517 -- vim.host.StorageSystem.refresh -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122425 -- host-8 -- VpxdInvtHostSyncHostLRO.Synchronize

Synchronizing host: esx1.nospam.com (x.x.128.181)

Retrieved host update to 1039

Completed host synchronization

-- FINISH task-internal-122425 -- host-8 -- VpxdInvtHostSyncHostLRO.Synchronize

-- BEGIN task-internal-122426 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

Synchronizing host: esx7.nospam.com (x.x.128.171)

Retrieved host update to 481

Completed host synchronization

-- FINISH task-internal-122426 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

-- FINISH task-7339 -- host-2517 -- vim.host.StorageSystem.refresh -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122427 -- -- vim.ServiceInstance.retrieveContent -- 2DFA95CB-B3C1-4407-8281-6A43212F5DA2

-- FINISH task-internal-122427 -- -- vim.ServiceInstance.retrieveContent -- 2DFA95CB-B3C1-4407-8281-6A43212F5DA2

-- BEGIN task-internal-122428 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

Synchronizing host: esx7.nospam.com (x.x.128.171)

Retrieved host update to 483

Completed host synchronization

-- FINISH task-internal-122428 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

-- BEGIN task-internal-122429 -- -- vmodl.query.PropertyCollector.retrieveContents -- 481804DF-8C81-4E0A-A8E5-54CDEFCDF902

-- FINISH task-internal-122429 -- -- vmodl.query.PropertyCollector.retrieveContents -- 481804DF-8C81-4E0A-A8E5-54CDEFCDF902

ConfigInfo saved in db

-- BEGIN task-internal-122430 -- -- vmodl.query.PropertyCollector.Filter.destroy -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122430 -- -- vmodl.query.PropertyCollector.Filter.destroy -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122431 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122431 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122432 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122432 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122433 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122433 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122434 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122434 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122435 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122435 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122436 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122436 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122437 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122437 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122438 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122438 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122439 -- -- vmodl.query.PropertyCollector.Filter.destroy -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122439 -- -- vmodl.query.PropertyCollector.Filter.destroy -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122440 -- -- vmodl.query.PropertyCollector.Filter.destroy -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122440 -- -- vmodl.query.PropertyCollector.Filter.destroy -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-7340 -- host-2517 -- vim.host.StorageSystem.rescanAllHba -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122441 -- host-487 -- VpxdInvtHostSyncHostLRO.Synchronize

Synchronizing host: esx5.nospam.com (x.x.128.192)

Retrieved host update to 711

Completed host synchronization

-- FINISH task-internal-122441 -- host-487 -- VpxdInvtHostSyncHostLRO.Synchronize

-- BEGIN task-internal-122442 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

Synchronizing host: esx7.nospam.com (x.x.128.171)

SSLVerifyCertAgainstSystemStore: The remote host certificate has these problems:

• A certificate in the host's chain is based on an untrusted root.

SSLVerifyCertAgainstSystemStore: Certificate verification is disabled, so connection will proceed despite the error

Retrieved host update to 483

Completed host synchronization

Updating vpxa for esx7.nospam.com (spec. genno 694)

-- FINISH task-internal-122442 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

-- BEGIN task-internal-122443 -- host-476 -- VpxdInvtHostSyncHostLRO.Synchronize

Synchronizing host: esx6.nospam.com (x.x.128.193)

Retrieved host update to 497

Completed host synchronization

-- FINISH task-internal-122443 -- host-476 -- VpxdInvtHostSyncHostLRO.Synchronize

VpxdClientAdapter::InvokeOnSoap: (esx7.nospam.com, vim.host.StorageSystem.rescanAllHba) took 5546 ms

-- BEGIN task-internal-122444 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

Synchronizing host: esx7.nospam.com (x.x.128.171)

SSLVerifyCertAgainstSystemStore: The remote host certificate has these problems:

• A certificate in the host's chain is based on an untrusted root.

SSLVerifyCertAgainstSystemStore: Certificate verification is disabled, so connection will proceed despite the error

Got vmacore exception: SSL Exception: error:00000001:lib(0):func(0):reason(1)

Backtrace:

backtrace[00] eip 0x0122b526

backtrace[01] eip 0x01176caa

backtrace[02] eip 0x0115192e

backtrace[03] eip 0x01141b38

backtrace[04] eip 0x0117e254

backtrace[05] eip 0x011d7e1f

backtrace[06] eip 0x011d7f3b

backtrace[07] eip 0x011d8364

backtrace[08] eip 0x011d55c6

backtrace[09] eip 0x0118ce06

backtrace[10] eip 0x0115bbff

backtrace[11] eip 0x013a618a

backtrace[12] eip 0x013a6332

backtrace[13] eip 0x013a81ff

backtrace[14] eip 0x013a9051

backtrace[15] eip 0x00510d11 (no symbol)

backtrace[16] eip 0x0051169f (no symbol)

backtrace[17] eip 0x00511af1 (no symbol)

backtrace[18] eip 0x013779ce

backtrace[19] eip 0x008630a2 (no symbol)

backtrace[20] eip 0x0053b0e5 (no symbol)

backtrace[21] eip 0x0053bad8 (no symbol)

backtrace[22] eip 0x006d5df6 (no symbol)

backtrace[23] eip 0x006dad6e (no symbol)

backtrace[24] eip 0x006e0e04 (no symbol)

backtrace[25] eip 0x00a1c06a (no symbol)

backtrace[26] eip 0x7c824829 GetModuleHandleA

Enter DAS_PROFILE UpdateDasStatus

Exit DAS_PROFILE UpdateDasStatus (0 ms)

-- FINISH task-internal-122444 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

Awaited sync failed for esx7.nospam.com

-- FINISH task-7340 -- host-2517 -- vim.host.StorageSystem.rescanAllHba -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- ERROR task-7340 -- host-2517 -- vim.host.StorageSystem.rescanAllHba: vmodl.fault.HostCommunication:

(vmodl.fault.HostCommunication) {

dynamicType = <unset>,

msg = ""

}

-- BEGIN task-internal-122445 -- -- vmodl.query.PropertyCollector.retrieveContents -- 4A939008-5249-45C5-ACCA-BB0B02E93E9D

-- FINISH task-internal-122445 -- -- vmodl.query.PropertyCollector.retrieveContents -- 4A939008-5249-45C5-ACCA-BB0B02E93E9D

-- BEGIN task-internal-122446 -- -- vmodl.query.PropertyCollector.createFilter -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122446 -- -- vmodl.query.PropertyCollector.createFilter -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122447 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122447 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122448 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122448 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122449 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122449 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122450 -- -- vim.LicenseManager.queryUsage -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122450 -- -- vim.LicenseManager.queryUsage -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- ERROR task-internal-122450 -- -- vim.LicenseManager.queryUsage: vmodl.fault.InvalidType:

(vmodl.fault.InvalidType) {

dynamicType = <unset>,

argument = "Non-optional data field is unset: source, vim.LicenseManager.LicenseUsageInfo",

msg = ""

}

-- BEGIN task-internal-122451 -- -- vmodl.query.PropertyCollector.Filter.destroy -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122451 -- -- vmodl.query.PropertyCollector.Filter.destroy -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122452 -- -- vim.LicenseManager.queryUsage -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122452 -- -- vim.LicenseManager.queryUsage -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- ERROR task-internal-122452 -- -- vim.LicenseManager.queryUsage: vmodl.fault.InvalidType:

(vmodl.fault.InvalidType) {

dynamicType = <unset>,

argument = "Non-optional data field is unset: source, vim.LicenseManager.LicenseUsageInfo",

msg = ""

}

-- BEGIN task-internal-122453 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

Synchronizing host: esx7.nospam.com (x.x.128.171)

SSLVerifyCertAgainstSystemStore: The remote host certificate has these problems:

• A certificate in the host's chain is based on an untrusted root.

SSLVerifyCertAgainstSystemStore: Certificate verification is disabled, so connection will proceed despite the error

-- BEGIN task-internal-122454 -- -- vmodl.query.PropertyCollector.retrieveContents -- 481804DF-8C81-4E0A-A8E5-54CDEFCDF902

-- FINISH task-internal-122454 -- -- vmodl.query.PropertyCollector.retrieveContents -- 481804DF-8C81-4E0A-A8E5-54CDEFCDF902

Retrieved host update to 489

Enter DAS_PROFILE UpdateDasStatus

Exit DAS_PROFILE UpdateDasStatus (15 ms)

VM vim.VirtualMachine:vm-98: config changed, invalidating cache...

VM vim.VirtualMachine:vm-179: config changed, invalidating cache...

VM vim.VirtualMachine:vm-1839: config changed, invalidating cache...

VM vim.VirtualMachine:vm-167: config changed, invalidating cache...

VM vim.VirtualMachine:vm-3296: config changed, invalidating cache...

VM vim.VirtualMachine:vm-132: config changed, invalidating cache...

VM vim.VirtualMachine:vm-2576: config changed, invalidating cache...

-- BEGIN task-internal-122455 -- -- vmodl.query.PropertyCollector.retrieveContents -- 4A939008-5249-45C5-ACCA-BB0B02E93E9D

VM vim.VirtualMachine:vm-3251: config changed, invalidating cache...

-- FINISH task-internal-122455 -- -- vmodl.query.PropertyCollector.retrieveContents -- 4A939008-5249-45C5-ACCA-BB0B02E93E9D

-- BEGIN task-internal-122456 -- -- vim.LicenseManager.queryUsage -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122456 -- -- vim.LicenseManager.queryUsage -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- ERROR task-internal-122456 -- -- vim.LicenseManager.queryUsage: vmodl.fault.InvalidType:

(vmodl.fault.InvalidType) {

dynamicType = <unset>,

argument = "Non-optional data field is unset: source, vim.LicenseManager.LicenseUsageInfo",

msg = ""

}

Completed host synchronization

Updating vpxa for esx7.nospam.com (spec. genno 694)

Enter DAS_PROFILE UpdateDasStatus

Exit DAS_PROFILE UpdateDasStatus (0 ms)

InvtHostSyncLRO::StartWork took 4078 ms

-- FINISH task-internal-122453 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

-- BEGIN task-internal-122457 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122457 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122458 -- -- vmodl.query.PropertyCollector.retrieveContents -- 4A939008-5249-45C5-ACCA-BB0B02E93E9D

-- FINISH task-internal-122458 -- -- vmodl.query.PropertyCollector.retrieveContents -- 4A939008-5249-45C5-ACCA-BB0B02E93E9D

-- BEGIN task-internal-122459 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122459 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122460 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122460 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122461 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122461 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122462 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122462 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122463 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122463 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122464 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122465 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122465 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122466 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122466 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

ConfigInfo saved in db

-- FINISH task-internal-122464 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122467 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122467 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

ConfigInfo saved in db

ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow group-d1 childEntity.length

ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow group-d1 childType

ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow group-d1 disabledMethod

ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow group-d1 effectiveRole

ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow group-d1 name

ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow group-d1 overallStatus

ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow group-d1 parent

-- BEGIN task-internal-122468 -- -- vim.option.OptionManager.queryView -- 4A939008-5249-45C5-ACCA-BB0B02E93E9D

-- FINISH task-internal-122468 -- -- vim.option.OptionManager.queryView -- 4A939008-5249-45C5-ACCA-BB0B02E93E9D

-- BEGIN task-internal-122469 -- vm-132 -- vim.VirtualMachine.acquireMksTicket -- 4A939008-5249-45C5-ACCA-BB0B02E93E9D

SSLVerifyCertAgainstSystemStore: The remote host certificate has these problems:

• A certificate in the host's chain is based on an untrusted root.

SSLVerifyCertAgainstSystemStore: Certificate verification is disabled, so connection will proceed despite the error

-- FINISH task-internal-122469 -- vm-132 -- vim.VirtualMachine.acquireMksTicket -- 4A939008-5249-45C5-ACCA-BB0B02E93E9D

-- BEGIN task-internal-122470 -- -- vim.LicenseManager.queryUsage -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

SSLVerifyCertAgainstSystemStore: The remote host certificate has these problems:

• A certificate in the host's chain is based on an untrusted root.

SSLVerifyCertAgainstSystemStore: Certificate verification is disabled, so connection will proceed despite the error

ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow group-d1 childEntity.length

ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow group-d1 childType

ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow group-d1 disabledMethod

ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow group-d1 effectiveRole

ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow group-d1 name

ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow group-d1 overallStatus

ERProviderImpl<BaseT>::_GetChanges: Aggregate version Overflow group-d1 parent

-- FINISH task-internal-122470 -- -- vim.LicenseManager.queryUsage -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122471 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

Synchronizing host: esx7.nospam.com (x.x.128.171)

Retrieved host update to 493

-- BEGIN task-internal-122472 -- -- vmodl.query.PropertyCollector.retrieveContents -- 4A939008-5249-45C5-ACCA-BB0B02E93E9D

Completed host synchronization

-- FINISH task-internal-122472 -- -- vmodl.query.PropertyCollector.retrieveContents -- 4A939008-5249-45C5-ACCA-BB0B02E93E9D

Updating vpxa for esx7.nospam.com (spec. genno 696)

-- FINISH task-internal-122471 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

ConfigInfo saved in db

-- BEGIN task-internal-122473 -- host-8 -- VpxdInvtHostSyncHostLRO.Synchronize

Synchronizing host: esx1.nospam.com (x.x.128.181)

Retrieved host update to 1040

Completed host synchronization

-- FINISH task-internal-122473 -- host-8 -- VpxdInvtHostSyncHostLRO.Synchronize

-- BEGIN task-7341 -- host-2517 -- vim.host.StorageSystem.rescanVmfs -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

SSLVerifyCertAgainstSystemStore: The remote host certificate has these problems:

• A certificate in the host's chain is based on an untrusted root.

SSLVerifyCertAgainstSystemStore: Certificate verification is disabled, so connection will proceed despite the error

-- BEGIN task-internal-122474 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

Synchronizing host: esx7.nospam.com (x.x.128.171)

SSLVerifyCertAgainstSystemStore: The remote host certificate has these problems:

• A certificate in the host's chain is based on an untrusted root.

SSLVerifyCertAgainstSystemStore: Certificate verification is disabled, so connection will proceed despite the error

Retrieved host update to 497

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

Completed host synchronization

Updating vpxa for esx7.nospam.com (spec. genno 698)

-- FINISH task-internal-122474 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

-- FINISH task-7341 -- host-2517 -- vim.host.StorageSystem.rescanVmfs -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- BEGIN task-internal-122475 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

-- FINISH task-internal-122475 -- -- vmodl.query.PropertyCollector.retrieveContents -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

ConfigInfo saved in db

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

PropertyJournalImpl::HasChanged: Overflow summary.accessible

PropertyJournalImpl::HasChanged: Overflow summary.name

PropertyJournalImpl::GetChangeMap: Overflow , summary.accessible

PropertyJournalImpl::GetChangeMap: Overflow , summary.name

-- BEGIN task-internal-122476 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

Synchronizing host: esx7.nospam.com (x.x.128.171)

Retrieved host update to 497

Completed host synchronization

Updating vpxa for esx7.nospam.com (spec. genno 700)

-- FINISH task-internal-122476 -- host-2517 -- VpxdInvtHostSyncHostLRO.Synchronize

-- BEGIN task-internal-122477 -- -- vmodl.query.PropertyCollector.retrieveContents -- 481804DF-8C81-4E0A-A8E5-54CDEFCDF902

-- FINISH task-internal-122477 -- -- vmodl.query.PropertyCollector.retrieveContents -- 481804DF-8C81-4E0A-A8E5-54CDEFCDF902

[admin]

Okay so here's a summary of the relevant errors I found:

2008-06-24 13:56:51.411 'App' 2088 error VpxdVmomi Got vmacore exception: SSL Exception: error:00000001:lib(0):func(0):reason(1)

2008-06-24 13:56:51.426 'App' 2088 error VpxdVmomi Backtrace:

backtrace00 eip 0x0122b526 mailto:?GenerateCoreDump@System

....

2008-06-24 13:56:51.723 'App' 2748 warning VpxdInvtHost Awaited sync failed for esx7.nospam.com
2008-06-24 13:56:51.723 'App' 2748 info VpxLRO -- FINISH task-7340 -- host-2517 -- vim.host.StorageSystem.rescanAllHba -- 44D70674-39D3-4A4D-A25A-C4F27E6DC206
2008-06-24 13:56:51.723 'App' 2748 info VpxLRO -- ERROR task-7340 -- host-2517 -- vim.host.StorageSystem.rescanAllHba: vmodl.fault.HostCommunication:
(vmodl.fault.HostCommunication) {
dynamicType = , 
argument = "Non-optional data field is unset: source, vim.LicenseManager.LicenseUsageInfo", 
msg = ""
}

......

PropertyJournalImpl::HasChanged: Overflow

[admin]

It appears to be an SSL cert issue. Are you sure no-one changed the SSL certs on either VC or the ESX servers?

Try to re-create the ESX SSL certs:

1. If enabled, disable Virtual Machine Auto-start through the VI Client by clicking the host --> Configuration --> Virtual Machine Startup/Shutdown --> Properties

2. Disconnect and remove the host from VC by right-clicking it in the VI Client and choosing Disconnect then Remove

3. Login to the ESX server and stop hostd: service mgmt-vmware stop

4. Backup the old certs:

mkdir /root/ssl-old

cp /etc/vmware/ssl/* /root/ssl-old

5. Delete the certs: rm -rf /etc/vmware/ssl/*

6. Start hostd: service mgmt-vmware start

7. Reconnect the host to VC through the VI Client

8. Re-enable Virtual Machine auto-start if it was enabled.

[Poestges]

@appk: tried it, but no luck :smileysad:

@kjb: licences are fine, don't know what this error means

i think i have to start a SR with VMware, too

Thanks so far,

Sascha

[admin]

Try to copy over the rui.crt file from the ESX server to the VC server and install it by right-clicking it and clicking Install.

Restart the VC service after its installed.

[admin]

Have you made any progress on this?

Did you install the ESX cert on the VC server?

Did VMware support find anything?

[Poestges]

i copied and installed the certs to server, but still no luck :smileysad:

was waiting for results overnight, so i haven't contacted VMware support yet. I'm going to do so now.

[admin]

It's a strange one.

I'm wondering if the certs have expired - are both the VC and ESX cert expiry dates okay?

Do you happen to have another set of the openssl libraries installed?

Do a search for ssleay32.dll and libeay32.dll. They should be in the VC install directory.

Have you changed the security policy of the server regarding certs/encryption?

Have you installed any other applications or service packs before it broke?

[Poestges]

The expiry dates are oaky.

I haven't changed any policies or installed any other applications on the VC.

ssleay32.dll and libeay32.dll are in the install folder.

The strange thing is, that the error appeares sporadicly.

[kjb007]

I would open an SR with vmware. Are you up-to-date on patches?

-KjB

[Poestges]

I have opened a SR today. Patches are up-to-date

[admin]

For reference: http://communities.vmware.com/thread/149438

2 people in the above thread appear to have the same error.

One fixed it by installing the ESX certs on the VC server, the other still has the problem.

[silicoon]

I am one of the people in the other thread seeing this issue

This issue is still happening - We have resinstalled an esx server and - i have regenerated / re'added certs to VCS and restarted everything - the ssl error and stream ended error are still occuring - it seems the certs are not being picked up for some reason

if possible - what are the specs of your servers and switches etc? are you using blades? im trying to narrow down why this happens on only a small portion of installations

i have an SR open with vmware aswell

[Poestges]

Hi silicoon,

welcome to the club :smileywink:

I am using HP blades (ProLiant BL460c G1) and Brocade SAN switches. But I don't think this is a hardware problem. Everything was running fine before, and I haven't changed anything at the hardware.

[moberle]

Make that a big me 5 or 6 or whatever. same symptoms .........

Dell blades, Cisco, EMC Broadcom NIC'w

I have two environments. My new data center running 3.5/2.5 has the problem. My old data center running 3.0.x/2.0.x doesnt have the problem. Am currently rebuilding the environment in the old data center on 3.5/2.5 havent seen the problem there yet but i suspect i will. Im betting a patch broke this.

[RegNullify]

Greetings,

This is the exact same issue that I am having. It started happening last Tuesday. I am running ESX 3i 3.5.0 B94430 and VC 2.5. B84767. In VC I have an HA cluster with 2 esx hosts in it. I have been working on the issue pretty much non stop and haven't found a solution yet. One thing that I did pickup on was, I used a protocol analyzer (WireShark) and created a filter to monitor traffic between my esx hosts and vc. I notice that when esx hosts are grayed out and showing "disconnected" or "not responding" that there is a large number SSLv3 checksum errors that are happening on the VC server when an esx host is communicating with it. It looks like the problem is a tcp windowing issue. I don't know how to fix it but just thought I would shed some light on it. It could even be nothing but seems kind of weird to me. I have ensured that all NICS are supported and have the latest firmware and drivers. To me the issue is with VC and HA. Somehow VC is not receiving a heartbeat from the ESX hosts. To limit out hardware and network issues I am going to install ESX3.5 update1 and test it. The next test I am going to do is setup another network switch to ensure that it isn't a network issue. Anyway hope we get this resolved. Thanks for your support.

-Reg

[admin]

According to Windows Process Explorer, the VC process vpxd also uses the dll Crypt32.dll

Are there any errors related to its use in the windows servers logs? (application, system or security)

[Poestges]

logs are fine

[RegNullify]

Greetings,

After carefully examining Application, Security, and System logs I do not see any errors with the crypt32.dll in my VC server's event logs. In addition since my logs are permanently archived I looked at all of the logs dating back to the time when the server was first put online and they too show nothing in regards to crypt32.dll in the event logs.Thanks for your support!!

[silicoon]

Hi guys

VMware support havnt exactly helped me with this issue - keep getting moved around to different support engineers... joyous

Have you had any luck with them?

[RegNullify]

Greetings,

No luck yet!! I have an open case that is outstanding for about a week and a half now. I haven't been shifted around too much, but the VMware folks are clueless so far. They haven't been helpful, but I know they are working on it. My belief at this point is SSL cert's or possibly a DB connectivity problem, maybe even both who knows. I can say that I am about to lose my patience very soon if I don't get any answers. Hopefully we can just keep working the problem it will be figured out, so hang in there.

-Reg

[Poestges]

No luck here, too :smileysad:

Case is open for 1,5 weeks now. Actual state of affairs:

"From the logs, the issue is matching VMware Problem Report#268505 which is currently unresolved"

Can't find the Problem Report, anyone knows where it is?

[Poestges]

I just phones with VMware Support, and we tried a workaround:

- Deinstall "Update Manager" and "Converter for VirtualCenter"

- Reboot VC Server

I checked some actions after that and everything seems fine.

Now i wait 24h and I'll contact VMware again to get further instructions.

I'll post further improvements.

[RegNullify]

Greetings,

Hey thanks for the info. I have uninstalled "Converter" and "Update Manager" for VC and the client. I am rebooting the VC server. I will keep you posted with results. Good work!!

-Reg

[RegNullify]

Greetings,

So far so good. That seems to be doing the trick, at least until VMware can work out the bugs. I am actually going to clone my existing VC server, then blow it away and start fresh. Hopefully I can find out were it all goes bad.

-Reg

[Poestges]

After nearly 24h everything is running fine, no errors at all.

I have installed the Update Manager on the Virtual Center Server again. Everything is still running fine, so far. I'll have a look at.

It seems to be the Converter, which is causing the problems.

[RegNullify]

Greetings,

Good work!! Everything is working perfectly at this point without "Update Manager" or "Converter". I am going reinstall "Update Manager" again and confirm. Once I have confirmed that everything is ok with "Update Manager" installed I will reinstall "Converter" and attempt a p2v conversion of a host and see what happens. Now that my memory serves me correct I recall doing a conversion on a physical host the night before the problem started to occur. Right before I went home for the day I started a conversion process of a host in VC and then left because It was going to take quit a bit of time. The next morning everything in VC was all horked up.

-Reg

[Poestges]

Another 24h later:

Everything is still running fine, including "Update Manager".

I'm not going to reinstall "Converter" now, cause I don't need it right now and I'm glad to have a stable VI again. :smileywink:

Reg is doing this right now (Thanks!), so he can give us a report.

I'm still in contact with VMware and I'll post any news.

[RegNullify]

Greetings,

I have reinstalled "Converter" and the "Converter Agent" on my VC host and have successfully duplicated the problem and at this point have confirmed that something in "Converter" is the cause. The work around is of course to remove converter and run it on another host.

Course of actions

• Reinstalled converter and rebooted the host everything seemed to be fine, but I noticed a much slower starting and running "VMware Virtual Infrastructure Client" on my remote hosts. In addition VIC would start up sluggish on the VC server, however the hosts did not disconnect or show a status of (Not Responding), perhaps I didnt give it enough time.

• Attempted a conversion of a Windows XP physical host to a VM while connected to the VC server through "Virtual Infrastructure Client" on my workstation. The conversion process failed. Soon after that the ESX hosts were showing a status of (not responding) at random intervals. Once again I removed "Converter" and the "Converter agent", rebooted my VC server and everything is working fine again.:D

Conclusion

• Something in "Converter" is clobbering the communication between VC and the ESX hosts.

-Reg

[Poestges]

Thank you Reg for the test and the final confirmation, that Converter causes the problem.

I'll mark this thread as "question answered" now, cause we have a working workaround.

Now we have to wait for VMware to fix it.

If there are any news, I'll post them here.

[silicoon]

One question

When you got rid of the converter - did you just uninstall the package through add remove programs to get rid of the converter and the agent?

Thanks in advance

[RegNullify]

Greetings,

Yes, I did remove "Converter" through add/remove programs. I removed both the "Converter" and the "Converter Agent". I don't recall if I rebooted the machine or not. I don't think you need to. Anyway I hope this helps resolves your issue. Please let us know your results. Thanks for your support.

-Reg

[Ph_Seifert]

Hi @all,

thanks for the information in this thread. I wrote to the vmware support but without solving this problem. I hope to fix the problem with the uninstallation of Converter and Update Manager. At the moment everything works fine. I'll report too.

kind regards

Philipp

[Ph_Seifert]

I have bad news. The Problem isn't fixed. When I want to migrate on VM to another host the action stopped with teh An error occurred ... message. When I then restart the Virtual Center Service on the VC host everything works fine. How long it will work? Its sporadically.

[Travis_Wood]

This has been fixed in the VC Update 5

http://www.vmware.com/support/vi3/doc/vi3_vc25u5_rel_notes.html#resolvedissues

[GPavan]

Hi All,

I've received same error "An error occurred while communicating with the remote host". Here What i want to know is does it records any event errors on VC center concole? Please advice. Thanks in advance.

[Evilnerd]

I was having this issue as well and here is ths steps I had to do to get it working.

1.logged into Vcenter and disconnected each of my cluster nodes

2. Removed each node after the disconnect completed

3. Logged into each cluster node seperately and ran to validate the vcenter agent was running service vmware-vpxa status

4. Stop the vcenter agent service vmware-vpxa stop

5. locate the vcenter agetn version and package to remove rpm -qa | grep -i vpxa

6. Remove teh vcenter agent rpm -e VMware-vpxa-4.0.0-208111

7. Restart the vmware management service service mgmt-vmware restart

8. Log into each ESX host by IP just to validate the management service came up and your able to view your VMs

9. Log into the Vcenter server and restart the VMware VirtualCenter Server windows service

10. Log into Vcenter from the Vsphere client and readd your nodes to the cluster

11. I had to renable HA on each node in order to clear all alarms after they were initially added

These steps fixed the issue for me

[the_norris]

hi Evilnerd.

We are having the same issue here - vsphere 4.1 esxi hosts and vc4. All patches as of last week applied. System was upgraded from 3.x.

Problem include: connect host in to VC, power operations on VMs, and VM migrations.

Now not sure what triggered the issues - but two things happen 1) New version of Trend AV client installed onto the VC, and the SQL server hosting the VCDB when down (not clean shutdown).

I thought the problems was only limited to 4 hosts that were powered down for datacenter power work been done. All 4 hosts and hosts VMs powered down cleanly.

Then I found this is is effecting all hosts connected to the VC.

I can connect the VI client to the 4 hosts that was downed directly to start the critical VMs needed for Monday start of business.

I've opened a SR with VMWare yesterday - but need to wait until Monday as only have Basic support.

Phil

[stripe]

This happened me on Dell servers with Broadcom onboard NICs, where if there was an signle VLAN adapter without trunk made ping responses to vmkernel within data transfer in hundreds of ms, in some whiles complete packet loss. So vcenter killed the session and the host reconnocted successfully after cleanup of the network.

Resolution was to do some activity in console, use a wider connection to switch or use of another NIC brand like Intel.

[kjb007]

Have you looked at our licensing? There are errors related to that in your log below.

I would also open an SR with vmware.

-KjB

[harishraman01]

It is DNS Issue , verify the DNS entry to resolve the host on network is right !

in my case the host had wrong DNS entry , thereby the hostname was not getting resolved and had this error, after keying in the correct DNS ip , it worked ..