VMware NSX

 View Only
Expand all | Collapse all

NSX-T REPO SYNC FAILED

  • 1.  NSX-T REPO SYNC FAILED

    Posted Oct 11, 2022 07:24 AM

    Hi,

    I have 3 NSX-T Managers and I have a  repo sync problem on two of them. How to resolve this problem?

    ggovek_0-1665472935122.png

     

    I try to reboot all three appliances and this is not help. Also I try with Resolve button.



  • 2.  RE: NSX-T REPO SYNC FAILED

    Posted Nov 16, 2022 06:10 AM

    I have this problem too.

    Did you find a fix for this issue?



  • 3.  RE: NSX-T REPO SYNC FAILED

    Posted Jun 02, 2023 04:45 PM

    Hi,

    I have the same issue. Running 4.1.0.0 by the way.

    Searching KB, internet - no hits whatsoever. 

    Does anyone know how to troubleshoot this issue? Which logs and services to check? 

     

    I had a look through bunch of them - but I can't get this to work. One strange thing I noticed - it seems NSX database has a fqdn saved in the IPv4 field of NSX Manager nodes under get nodes verbose:

    UUID : xx
    Label : Manager
    IPv4 Addresses : ['nsx-manager3.lab.local']
    IPv6 Addresses : N/A
    Hostname/FQDN : nsx-manager3.lab.local
    Display Name : nsx-manager3

    UUID : xx
    Label : Manager
    IPv4 Addresses : ['nsx-manager2.lab.local']
    IPv6 Addresses : N/A
    Hostname/FQDN : nsx-manager2.lab.local
    Display Name : nsx-manager2

    UUID : xx
    Label : Manager
    IPv4 Addresses : ['nsx-manager1.lab.local']
    IPv6 Addresses : N/A
    Hostname/FQDN : nsx-manager1.lab.local
    Display Name : nsx-manager1.lab.local

     

    There is a reference to invalid IP in proton logs:

    2023-06-02T13:51:36.516Z ERROR http-nio-127.0.0.1-7440-exec-3 IPv4Address 4407 - [nsx@6876 comp="nsx-manager" errorCode="MP260" level="ERROR" reqId="d385ced2-52e5-4be0-808e-049e646a00df" subcomp="manager" username="uproton"] Invalid ip string 'nsx-manager1.lab.local'. Error parsing 'nsx-manager1'.

    Also - there is a reference to missing IP: 

    2023-06-02T15:09:44.833Z WARN http-nio-127.0.0.1-7440-exec-1 FacadeInterceptorHelperImpl 4452 - [nsx@6876 comp="nsx-manager" level="WARNING" reqId="df7a2453-7ffd-45c1-addd-6b4644d994c0" subcomp="manager" username="UC"] API result did not pass validationField level validation errors: {required property results[3].controller_role.control_cluster_listen_addr.ip_address is missing}, {required property results[4].controller_role.control_cluster_listen_addr.ip_address is missing}

     



  • 4.  RE: NSX-T REPO SYNC FAILED

    Posted Jun 09, 2023 09:56 AM

    Good afternoon, encountered the same problem on NSX 4.1.0, could you please tell me if you were able to solve this problem?



  • 5.  RE: NSX-T REPO SYNC FAILED

    Posted Jun 13, 2023 09:53 AM

    I have the same problem. Is there any solution to the problem?



  • 6.  RE: NSX-T REPO SYNC FAILED

    Posted Jun 13, 2023 09:59 AM

    I got it fixed - worked on it together with VMware support.

    It seems there are potentially two problems:

    1./ FQDN in IPv4Address field on NSX Manager "get node" output. If this happens - I suspect RepoSync process can't work.

    2./ RepoSync failing for another reason - this reason can be discovered in nsxapi.log.

    Login to NSX Manager CLI as admin user - then check that you have FQDN in IPv4Address field using "get node" command?



  • 7.  RE: NSX-T REPO SYNC FAILED

    Posted Jun 13, 2023 12:44 PM

    There is no FQDN in the IPv4Address field. There are errors in nsxapi.log that clearly show that the repository is going to synchronize files that are not there



  • 8.  RE: NSX-T REPO SYNC FAILED

    Posted Jun 13, 2023 01:12 PM

    The question is why? 

    The correct way to fix it would be engaging support. 

    I have manually uploaded the bundle to my repository server. This however is not supported - while I can't see anything wrong with it, support will probably not like it. 

    Example - RepoSync is complaining about missing files for "4.1.0.0.0.21332672" version:

    I downloaded the 4.1.0.0 MUB (upgrade bundle), then extracted all the files with 7z (gunzip should work). Then copied the entire extracted package to my repository node /repository/4.1.0.0.0.21332672/ with scp.

    Last step was to change owner on the files I copied - in this example: chown -R uuc:grepodir 4.1.0.0.0.21332672\

    After that, issue a manual repo-sync with API and it should make progress. 

    Let us know if that helps.



  • 9.  RE: NSX-T REPO SYNC FAILED

    Posted Sep 25, 2023 05:53 PM

     

    I'm facing the REPO_SYNC issue, and I have tried all the mentioned methods and now I don't know how to solve it currently I'm running version  4.1.0.2.0.21761695. I have ALB running on Manager. 

     
     


  • 10.  RE: NSX-T REPO SYNC FAILED

    Posted Oct 18, 2023 04:57 AM

     Did you by any chance upgrade from NSX-T 3.x?

    If so, we did the same, ran into massive problems, in the end had to replace all 3 manager nodes with fresh ones and ran into the REPO_SYNC problem.

    Check /var/log/proton/nsxapi.log for FAILED repo sync messages. We got something like "Unable to connect to File /repository/4.1.1.0.0.22224312/Manager/vmware-mount/libvixMntapi.so.1 on source xxx.xxx.xxx.xxx".

    The IP was the one of the appliance which was unable to sync the repo.

    We then ran the nsxcli command "get service install-upgrade" and the output showed, that this service was running on the node with the missing files/folders.

    I connected to another manager node where there were no REPO_SYNC problems, checked if the files/folders were there, which they were and ran the nsxcli command "set repository-ip" which changed the repository to that node.

    After a minute everything from that repository was synced to the node that had the error and the error was gone.



  • 11.  RE: NSX-T REPO SYNC FAILED

    Posted Oct 29, 2023 06:01 AM

    Good to hear that your issue got resolved. I upgraded from 3.x to 4.x and I tried all those things but sync was not working seems that some files were missing from the upgrade on one of the managers however, I extracted .mib and copy pasted manually on all three repos and issue got resolved. 



  • 12.  RE: NSX-T REPO SYNC FAILED

    Posted Jan 13, 2024 02:38 PM

    Hello,

    I am facing this message when i am trying to get de logs 

    "-bash: /var/log/proton/nsxapi.log: Permission denied"

    ANyone has any ideea what this permission is? Who is supposed to give it to me keeping in mind that I deployed the whole solution?

     

    Thank you,

    Liviu



  • 13.  RE: NSX-T REPO SYNC FAILED

    Posted Jan 17, 2024 07:48 PM

     wrote:

     Did you by any chance upgrade from NSX-T 3.x?

    If so, we did the same, ran into massive problems, in the end had to replace all 3 manager nodes with fresh ones and ran into the REPO_SYNC problem.

    Check /var/log/proton/nsxapi.log for FAILED repo sync messages. We got something like "Unable to connect to File /repository/4.1.1.0.0.22224312/Manager/vmware-mount/libvixMntapi.so.1 on source xxx.xxx.xxx.xxx".

    The IP was the one of the appliance which was unable to sync the repo.

    We then ran the nsxcli command "get service install-upgrade" and the output showed, that this service was running on the node with the missing files/folders.

    I connected to another manager node where there were no REPO_SYNC problems, checked if the files/folders were there, which they were and ran the nsxcli command "set repository-ip" which changed the repository to that node.

    After a minute everything from that repository was synced to the node that had the error and the error was gone.


    This worked!!  Simplest solution is the best!



  • 14.  RE: NSX-T REPO SYNC FAILED

    Posted Nov 10, 2023 09:54 AM

    Did you tried to delete other 2 nodes and reAdd new appliance?