vCenter

 View Only
  • 1.  Migrating vmnics to DVSwitch : Operation Timed Out

    Posted May 05, 2021 05:59 AM

    Hi,

    This setup is in Workstation version 16.1.0 build-17198959

    vCenter version VMware-vCenter-Server-Appliance-6.7.0.45000-16708996_OVF10 (6.7 U3 J)

    ESXi version VMware-VMvisor-Installer-201912001-15160138.x86_64

    I'm facing an issue with vCenter timing out when migrating vmnics to DVSwitch. The vmnics are not assigned to anything. I create VMKernel adapters in DVswitch, then attach the vmnics to DVSwitch. I did the same for Management ports, and they worked fine. The moment I start the migration for the remaining vmnics, after sometime has passed vCenter shows as reconnecting https://i.ibb.co/19cXJgV/v-Center-Reconnecting.png.

    The vpxd.log shows the following.

    2021-05-05T13:42:45.149Z info vpxd[05049] [Originator@6876 sub=vpxLro opID=q-298:01-b8] [VpxLRO] -- BEGIN lro-3207 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 52de6b05-ff57-5212-8e2f-2e5c30351224(520a6ec9-121f-c4c8-7aaa-cb0b0aa54c06)
    2021-05-05T13:42:45.149Z info vpxd[05049] [Originator@6876 sub=vpxLro opID=q-298:01-b8] [VpxLRO] -- FINISH lro-3207
    2021-05-05T13:42:45.150Z info vpxd[05049] [Originator@6876 sub=vpxLro opID=q-297:01-d0] [VpxLRO] -- BEGIN lro-3208 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 52de6b05-ff57-5212-8e2f-2e5c30351224(520a6ec9-121f-c4c8-7aaa-cb0b0aa54c06)
    2021-05-05T13:42:45.150Z info vpxd[05049] [Originator@6876 sub=vpxLro opID=q-297:01-d0] [VpxLRO] -- FINISH lro-3208
    2021-05-05T13:42:45.160Z info vpxd[05003] [Originator@6876 sub=vpxLro opID=q-299:01-a1] [VpxLRO] -- BEGIN lro-3209 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 52de6b05-ff57-5212-8e2f-2e5c30351224(520a6ec9-121f-c4c8-7aaa-cb0b0aa54c06)
    2021-05-05T13:42:45.160Z info vpxd[05003] [Originator@6876 sub=vpxLro opID=q-299:01-a1] [VpxLRO] -- FINISH lro-3209
    2021-05-05T13:42:45.594Z error vpxd[05738] [Originator@6876 sub=hostMethod] Commit call for method [updateNetworkConfig] transaction Id [17] failed on host [[vim.HostSystem:host-44,w-esx-vsn-01.vlab.lab]] with exception:[(vmodl.fault.SystemError) {
    --> faultCause = (vmodl.MethodFault) null,
    --> faultMessage = <unset>,
    --> reason = "Transaction has rolled back on the host."
    --> msg = "Received SOAP response fault from [<cs p:00007fb83c3d4020, TCP:w-esx-vsn-01.vlab.lab:443>]: commitTransaction
    --> Received SOAP response fault from [<cs p:000000126b0b8950, TCP:localhost:8307>]: commitTransaction
    --> A general system error occurred: Transaction has rolled back on the host."
    --> }]
    2021-05-05T13:42:45.595Z error vpxd[05734] [Originator@6876 sub=hostMethod] Commit call for method [updateNetworkConfig] transaction Id [15] failed on host [[vim.HostSystem:host-47,w-esx-vsn-02.vlab.lab]] with exception:[(vmodl.fault.SystemError) {
    --> faultCause = (vmodl.MethodFault) null,
    --> faultMessage = <unset>,
    --> reason = "Transaction has rolled back on the host."
    --> msg = "Received SOAP response fault from [<cs p:000055b41bcbab90, TCP:w-esx-vsn-02.vlab.lab:443>]: commitTransaction
    --> Received SOAP response fault from [<cs p:0000002102452d90, TCP:localhost:8307>]: commitTransaction
    --> A general system error occurred: Transaction has rolled back on the host."
    --> }]
    2021-05-05T13:42:45.605Z error vpxd[05077] [Originator@6876 sub=hostMethod] Commit call for method [updateNetworkConfig] transaction Id [16] failed on host [[vim.HostSystem:host-50,w-esx-vsn-03.vlab.lab]] with exception:[(vmodl.fault.SystemError) {
    --> faultCause = (vmodl.MethodFault) null,
    --> faultMessage = <unset>,
    --> reason = "Transaction has rolled back on the host."
    --> msg = "Received SOAP response fault from [<cs p:00007fb8344acf50, TCP:w-esx-vsn-03.vlab.lab:443>]: commitTransaction
    --> Received SOAP response fault from [<cs p:0000003ff197bfb0, TCP:localhost:8307>]: commitTransaction
    --> A general system error occurred: Transaction has rolled back on the host."
    --> }]
    2021-05-05T13:42:45.682Z info vpxd[05739] [Originator@6876 sub=vpxLro opID=q-301:h5ui-getProperties:urn:vmomi:DistributedVirtualPortgroup:dvportgroup-88:ddefdc78-bdea-43c9-8e4d-670bab2c3978:1364190193:01-4c] [VpxLRO] -- BEGIN lro-3210 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 52de6b05-ff57-5212-8e2f-2e5c30351224(520a6ec9-121f-c4c8-7aaa-cb0b0aa54c06)
    2021-05-05T13:42:45.683Z info vpxd[05739] [Originator@6876 sub=vpxLro opID=q-301:h5ui-getProperties:urn:vmomi:DistributedVirtualPortgroup:dvportgroup-88:ddefdc78-bdea-43c9-8e4d-670bab2c3978:1364190193:01-4c] [VpxLRO] -- FINISH lro-3210

    Firewall has nothing blocked, all ports and protocols are open. I don't know if this is related to the NTP time sync issue I'm facing https://communities.vmware.com/t5/VMware-vCenter-Discussions/NTP-Syncing-6-hours-ahead-ESXi-from-same-source-show-proper-time/m-p/2845477#M42912

    Complete vxpd.log can be found here : https://mega.nz/folder/hpwz2ISQ#lEuR4AbcECRxBkhqmK7HqA



  • 2.  RE: Migrating vmnics to DVSwitch : Operation Timed Out

    Posted May 07, 2021 06:47 PM

    Anyone knows about this error

    A general system error occurred: Connection reset by peer: The connection is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem, timeout, or service overload.



  • 3.  RE: Migrating vmnics to DVSwitch : Operation Timed Out

    Posted May 07, 2021 07:07 PM

    Found this, going through it..

    https://kb.vmware.com/s/article/76259



  • 4.  RE: Migrating vmnics to DVSwitch : Operation Timed Out

    Posted May 08, 2021 06:34 PM

    Hello,

    Enter via ssh to vcsa, enter the following commands. and then retry the migration from the vmnic to the DVswitch.

    Run the following commands:

    service-control --stop -all
    Service-control --start -all

    and you share the results with us.



  • 5.  RE: Migrating vmnics to DVSwitch : Operation Timed Out

    Posted May 08, 2021 07:25 PM

    Hi,

    Thanks, I ran the command with the results below.

    Command> service-control --stop --all
    Operation not cancellable. Please wait for it to finish...
    Performing stop operation on service vmware-pod...
    Successfully stopped service vmware-pod
    Performing stop operation on profile: ALL...
    Successfully stopped service vmware-vmon
    Successfully stopped profile: ALL.
    Performing stop operation on service vmdnsd...
    Successfully stopped service vmdnsd
    Performing stop operation on service vmware-stsd...
    Successfully stopped service vmware-stsd
    Performing stop operation on service vmware-sts-idmd...
    Successfully stopped service vmware-sts-idmd
    Performing stop operation on service vmcad...
    Successfully stopped service vmcad
    Performing stop operation on service vmdird...
    Successfully stopped service vmdird
    Performing stop operation on service vmafdd...
    Successfully stopped service vmafdd
    Performing stop operation on service lwsmd...
    Successfully stopped service lwsmd
    Command> service-control --start --all
    Operation not cancellable. Please wait for it to finish...
    Performing start operation on service lwsmd...
    Successfully started service lwsmd
    Performing start operation on service vmafdd...
    Successfully started service vmafdd
    Performing start operation on service vmdird...
    Successfully started service vmdird
    Performing start operation on service vmcad...
    Successfully started service vmcad
    Performing start operation on service vmware-sts-idmd...
    Successfully started service vmware-sts-idmd
    Performing start operation on service vmware-stsd...
    Successfully started service vmware-stsd
    Performing start operation on service vmdnsd...
    Successfully started service vmdnsd
    Performing start operation on profile: ALL...
    Successfully started service vmware-vmon
    Successfully started profile: ALL.
    Performing start operation on service vmware-pod...
    Successfully started service vmware-pod
    Command>

    Then I performed vmnics assignment, as usual the 1st set assigns successfully, no issues none whatsoever.

    https://i.ibb.co/hYHyNPN/W-WSRV-ADDC-01-2021-05-08-20-13-48.png

    The next assignment fails as usual, take note of the time between 1st and second assignment, merely 1 minute.

    https://i.ibb.co/MVGkkZQ/W-WSRV-ADDC-01-2021-05-08-20-14-32.png

    https://i.ibb.co/Krjs2fk/W-WSRV-ADDC-01-2021-05-08-20-18-20.png

    Thank You

     



  • 6.  RE: Migrating vmnics to DVSwitch : Operation Timed Out

    Posted May 08, 2021 07:31 PM

    Just something of note, I have observed vCenter keeps logging me out very very often, I had to log out just a few minutes after assigning the vmnics, and then again after a few minutes.

    https://i.ibb.co/TRs1d8v/W-WSRV-ADDC-01-2021-05-08-20-12-06.png

    I don't know if this is the reason I can't assign more vmnics because its times out and then logs out as well.



  • 7.  RE: Migrating vmnics to DVSwitch : Operation Timed Out

    Posted May 08, 2021 11:49 PM

    Hello,

    Disable the firewall completely in the ESXi, I indicate the command:
    esxcli network firewall set --enabled false.

    Restart the management agents of the esxi, I indicate the command:
    /etc/init.d/hostd restart
    /etc/init.d/vpxa restart

    Try again to migrate the vmnic to the distributed switch.

    Another test option is to put a host in maintenance mode and then run the migration of that host's vmnic to rule out getting the same error.

    I remain attentive to your comments.

    Regards.



  • 8.  RE: Migrating vmnics to DVSwitch : Operation Timed Out

    Posted Jun 06, 2021 01:18 PM

    Sorry the steps you mentioned did not resolve the issue.

    Not able to find the source of the problem.

    Tried removing and re-adding the hosts to vCenter again the same issue and no information in vCenter logs except Operation Timed Out. I have disabled RollBack still the issue exists.

    Anything else I can do ?