vSphere Replication

 View Only
Expand all | Collapse all

Unexpected vib installation on hosts

  • 1.  Unexpected vib installation on hosts

    Posted Sep 30, 2014 05:50 PM

    Today I deployed vSphere Replication 5.8 on a vSphere 5.5 U1 cluster (single site installation). Installation and configuration went fine, and I was able to configure replications as expected. However, shortly after recovering a replicated VM, an installation task - initiated by "com.vmware.vcHms" - started on all ESXi hosts. After this, all host's had event entries like "Installation started" and "... 0 vibs installed, 0 vibs removed ...", and all hosts are flagged with "Reboot required" now. I didn't find anything related to this in the documentation, the Release Notes nor online.

    Did someone else see this too, or have an explanation for this behavior? I didn't open a support request yet, but may do this later.

    André



  • 2.  RE: Unexpected vib installation on hosts

    Posted Sep 30, 2014 06:36 PM

    Hi,

    VR 5.6 (available to subscribers of vCHS DR service), VR 5.8 (publicly available) and next VR releases, do deploy a VIB package with firewall rules to allow outgoing traffic from source ESXi to the VR appliance at ports 10000-10010 for tunneling replication traffic to the cloud site.

    This is done on each host in the vCenter inventory, even before replication is configured, to allow vMotion of the replicated VM to another source ESXi.

    The 'reboot required' is not intended behavior.

    Please file an SR and if possible post the SR number here.

    Regards,

    Martin



  • 3.  RE: Unexpected vib installation on hosts

    Posted Sep 30, 2014 06:43 PM

    Thanks for the quick feedback. Tomorrow, I will be at the customer's site again and will try to collect the log files.

    André



  • 4.  RE: Unexpected vib installation on hosts

    Posted Apr 07, 2015 02:57 PM

    Martin,

    I am running vSphere Replication 5.8 on (2) ESXi hosts in a single vCenter which contains about 25 ESXi hosts world-wide.  Even though vR is only in use on (2) ESXi hosts, the VRA seems to kick off an "Install" task on a regular basis, pushing the VIB to all 25 hosts world-wide repeatedly.  Just today it pushed the VIBs at 4:49am, then at 5:38am, then at 8:43am, then at 9:50am.  This seems a little excessive, especially for hosts that are not even involved in vSphere Replication.  Can you discuss with the product team ways in which a future version of vSphere Replication can be a little more selective with where it pushes out the VIBs, and how often they get pushed out?

    Thanks,

    Bill



  • 5.  RE: Unexpected vib installation on hosts

    Posted Oct 27, 2014 12:12 PM

    Hi

    The same has happened to me.  I see a reboot required on a lot, but not all of the hosts in the site.

    There are also a lot of repeat VIB install events: Successfully installed 0 VIBS(s), removed 0 VIBS(s)

    I thought I would see if this was resolved by you before I open an SR.

    Tony



  • 6.  RE: Unexpected vib installation on hosts

    Posted Oct 27, 2014 06:11 PM

    You remind me of something...

    I didn't open an SR yet, but I looked at the log files and found some entries that I'm not sure how to interpret. Anyway - out of curiosity - how did you install the ESXi hosts? In my case these hosts were all reinstalled using the "VMware-VMvisor-Installer-5.5.0.update01-1746018.x86_64-Dell_Customized-A00.iso" and then patched using Update Manager.

    André



  • 7.  RE: Unexpected vib installation on hosts

    Posted Oct 30, 2014 02:25 PM

    These were installed with the Customised HP Image - upgraded from 5.1 via Update Manager. The install task occurs a lot and also after a host is rebooted.  I will raise an SR with VMware to find out why this keeps occurring.



  • 8.  RE: Unexpected vib installation on hosts

    Posted Nov 20, 2014 08:52 PM

    Has anyone found a solution for this problem yet?

    I see the install task on every host hundreds off times per day. It started after installing vsphere replication 5.8.



  • 9.  RE: Unexpected vib installation on hosts

    Posted Nov 21, 2014 08:55 AM

    I have an SR open with VMWare so will hopefully have an update soon.  Perhaps you could raise one too if possible as I don't think they have seen many occurrences of this problem.



  • 10.  RE: Unexpected vib installation on hosts

    Posted Nov 21, 2014 01:57 PM

    perhaps manual installation of the firewall VIB could stop this from occuring  ...:/

    VMware KB: Replications to cloud remain in Not Active status



  • 11.  RE: Unexpected vib installation on hosts

    Posted Nov 21, 2014 03:40 PM

    The package you mentioned is installed perfect:

    # esxcli software vib list | grep vr2c

    vr2c-firewall                  5.8.0.1-2170514                        VMware           VMwareCertified   2014-11-11



  • 12.  RE: Unexpected vib installation on hosts

    Posted Nov 22, 2014 03:29 PM

    have you tried to uninstall vr2 VIB from the host delete all the VIBs (or just vr2 VIB) from the VUM patch repository redownload all the VIBs/VIB and initiate stage and remediation process again?



  • 13.  RE: Unexpected vib installation on hosts

    Posted Feb 10, 2015 02:11 PM

    I've removed the vib's from all servers:

    ~ # esxcli software vib remove -n vr2c-firewall

    Removal Result

       Message: Operation finished successfully.

       Reboot Required: false

       VIBs Installed:

       VIBs Removed: VMware_bootbank_vr2c-firewall_5.8.0.1-2170514

       VIBs Skipped:

    But as soon as a new task runs (for example creating or removing a snapshot) it's installed again.

    And again, and again, and again.

    2015-02-10T13:44:32Z esxupdate: downloader: DEBUG: Downloading from https://10.1.137.31:8043/vib/vr2c-firewall.vib...

    2015-02-10T13:44:32Z esxupdate: Transaction: INFO: Skipping installed VIBs VMware_bootbank_vr2c-firewall_5.8.0.1-2170514

    2015-02-10T13:44:32Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

    Actions for the last few hours:

    ~ # grep "Final list of VIBs being installed" /var/log/esxupdate.log

    2015-02-10T11:23:50Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

    2015-02-10T11:24:50Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

    2015-02-10T11:25:49Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

    2015-02-10T11:52:41Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

    2015-02-10T11:53:41Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

    2015-02-10T11:54:51Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

    2015-02-10T11:55:50Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

    2015-02-10T11:56:50Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

    2015-02-10T12:01:41Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

    2015-02-10T12:40:47Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

    2015-02-10T13:12:31Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

    2015-02-10T13:13:32Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

    2015-02-10T13:40:42Z esxupdate: Transaction: INFO: Final list of VIBs being installed: VMware_bootbank_vr2c-firewall_5.8.0.1-2170514

    2015-02-10T13:44:32Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

    2015-02-10T13:45:41Z esxupdate: Transaction: INFO: Final list of VIBs being installed:

    All the install tasks are obfuscating 'normal' tasks in the task history.



  • 14.  RE: Unexpected vib installation on hosts

    Posted Feb 10, 2015 02:31 PM

    Hi,

    Do you know what changes are done at ESXi configuration each and every minute? VRMS will only attempt to re-install the VIB, if it detects the ESXi coming out of maintenance mode, or network configuration change or something similar.

    Please open SR and mention that this issue might be related to SR 15583212901. Engineering team is already looking into the issue with unnecessary VIB install attempt, if the VIB is already present.

    Regards,

    Martin



  • 15.  RE: Unexpected vib installation on hosts

    Posted Feb 10, 2015 03:52 PM

    I've created a Service Request with number 15602397302



  • 16.  RE: Unexpected vib installation on hosts

    Posted Feb 16, 2015 08:16 PM

    I have the SR with VMware support for more than a month now, yet to get anything concrete. In my case we get the VIB install messages and immediately the hosts will get disconnected and reconnected automatically. I have around 150+ hosts including remote site hosts. All remote site hosts are ESXi 5.0 hosts and local site hosts (around 50+) are ESXi 5.5 Build 2302651. It happens only between 1:00 am and 2:00 am) every night.

    This vCenter is part of a SRM site and the VIB install is initiated by VRMS server. Last week I shut down the VRMS form 12:30 am to 2:30 and didn't have any host disconnects at all. I passed the info to VMware support waiting on their feedback. Shall post once i hear anything from them.

    I have 3 other vCenters with SRM but not seeing this behavior on them.



  • 17.  RE: Unexpected vib installation on hosts

    Posted Mar 11, 2015 12:43 PM

    Hi guys

    +1 from a Belgian engineer here.

    I've got the same symptoms: Dell PowerEdge's installed with the 5.1 Dell customized image and the install events tasks keep piling up...

    Rebooting the hosts now to see if that changes anything.

    I must say this is an "inherited" environment with absolutely no patching done in the last years...

    Environment:

    vCenter 5.1 build 880146

    Cluster with

    ESXi 5.1.0 build 799733

    ESXi 5.1.0 build 799733


    Stand alone host

    ESXi 5.1.0 build 799733



    In the meantime I already logged a SR: 15621359303



  • 18.  RE: Unexpected vib installation on hosts

    Posted Mar 17, 2015 03:34 PM

    Hi

    I was informed by VMware this is a bug and that it would be fixed in the new release.  I've not installed 6.0 yet so cannot confirm if the problem goes away.

    Tony



  • 19.  RE: Unexpected vib installation on hosts

    Posted Mar 17, 2015 03:39 PM

    VMware vSphere Replication 6.0 is compatible with VMware vCenter Server 5.5 / 5.5U1 / 5.5U2 according to the VMware Product Interoperability Matrixes

    So it's worth a try.



  • 20.  RE: Unexpected vib installation on hosts

    Posted Mar 17, 2015 03:45 PM

    Do you know version 6 is compatible with Site Recovery Manager 5.8?  It wasn't clear on the matrix.  I would upgrade to SRM 6.0 but it requires vSphere 6.



  • 21.  RE: Unexpected vib installation on hosts

    Posted Mar 17, 2015 04:03 PM

    Hi,

    VR 6.0 is compatible only with SRM 6.0. VR 6.0 is not compatible with SRM 5.8.

    Regards,

    Martin



  • 22.  RE: Unexpected vib installation on hosts

    Posted Mar 17, 2015 06:57 PM

    Hi Tony,

    Thanks for reporting the omission.

    The interoperability matrices have been corrected.

    Best,
    Hristo



  • 23.  RE: Unexpected vib installation on hosts

    Posted Apr 06, 2015 09:25 PM

    Hristo,

    When you say the interoperability matrices have been updated, what exactly did you mean there?

    This link says that "vSphere Replication 6.0 requires vCenter Server 5.5 or higher" - http://www.vmware.com/files/pdf/vsphere/VMware-vSphere-Replication-Overview.pdf

    But the interoperability matrices seem to say that vSphere Replication requires vCenter 6.0 - VMware Product Interoperability Matrixes

    Can someone confirm for me?

    Thanks,

    Bob



  • 24.  RE: Unexpected vib installation on hosts

    Posted Mar 17, 2015 03:55 PM

    Hi Tony,

    VRMS attempts to re-deploy its VIB at certain configuration/state changes at ESXi. This might result in unnecessary download and log messages.

    VRMS does not first attempt to check the metadata at ESXi before attempting a (re-)deploy. This is not fixed in VR 6.0, but fix is expected for a 5.8.x and 6.0.x maintenance release.


    Regards,

    Martin



  • 25.  RE: Unexpected vib installation on hosts

    Posted Mar 17, 2015 04:00 PM

    Thanks Martin, do you know when we can expect an update to 5.8?



  • 26.  RE: Unexpected vib installation on hosts

    Posted Mar 17, 2015 04:12 PM

    This is what i got from VMware support an hour ago.

    This is the answer from the escalation

    --- Summary
    There is currently a known issue in HMS (VR) whereby the HMS component on ESXi is triggering ESXUPDATE to attempt to download the package "vr2c-firewall.vib" from the VR appliance (i.e VRM/HMS server) even though that package may already be installed. This download attempt is retried each time ESXi connects with VC.

    The ESXUPDATE logs show the download activity but also show that the installation of this package will be skipped since it is already installed.

    ---
    2015-01-12T07:21:51Z esxupdate: downloader: DEBUG: Downloading from https://10.x.x.x:8043/vib/vr2c-firewall.vib
    ...
    2015-01-12T07:21:51Z esxupdate: Transaction: INFO: Skipping installed VIBs VMware_bootbank_vr2c-firewall_5.8.0.1-2170514
    ---

    So to stop this behavior, the HMS (VR) agent on ESXi needs to check if the package "vr2c-firewall.vib" is already installed before attempting to download it. If it is installed, there won't be a need to download it.

    This is exactly the fix that engineering is working on


    Workaround:
    -----------

      For the HMS continuous download of the 'vr2c-firewall.vib' package, there is no practical workaround. Shutting down the VR appliance is the only one we know of as the customer has already verified.

    Thanks again for all your patience



  • 27.  RE: Unexpected vib installation on hosts

    Posted Aug 07, 2015 04:46 PM

    Looks like there might be a workaround:

    kb.vmware.com/kb/2110304

    Bob



  • 28.  RE: Unexpected vib installation on hosts

    Posted Sep 22, 2015 09:18 AM

    Hi

    This is from the release notes of the latest version compatible with vSphere 5.5

    VMware vSphere Replication 5.8.1 | 16 SEP 2015 | Build 2915556

    NEW After upgrade to vSphere Replication 5.8.x all hosts in both sites repeatedly display VIB install message

    The repeated VIB install messages reappear each time a host reconnects to vCenter or changes certificate.

    Workaround: You must disable the automatic DR2C firewall rules of VIB by using the following procedure.

    1. Connect to the vSphere Replication appliance by using an SSH connection.
    2. On the console, run the /opt/vmware/hms/bin/hms-configtool -cmd reconfig -property hms-auto-install-vr2c-vib=false command.

    I deleted my old appliances and re-deployed using this version.  I carried out the above steps BEFORE connecting the network on the appliances, preventing it from pushing out the VIB when it started up for the first time, as it will auto-register to vCenter and I assume then push out the VIB if the network is connected.  This meant I had to connect direct to the console instead of SSH but the process is the same.   I have not seen a single vib install message. 

    Tony