VMware Cloud Foundation

 View Only
  • 1.  Backup Server for SDDC Manager and NSX Manager

    Posted Oct 07, 2022 11:40 AM

    For File-Based-Backup of our VCF (SDDC manager and NSX Manager) I am trying to configure an external SFTP server (Ubuntu with OpenSSH). The 'Validate Backup Location Details' keeps failing with error: Validation failed for directory path /mnt/backups on server xx.xx.yy.yy. Please make sure backup directory is intact and sftp server has write permissions on backup path.

    SFTP and SSH from command line of SDDC manager works and I am able to create and delete folders. Also am I using the same SFTP server for File-Based-Backup of our vCenter Server without these issues.

    Any ideas?

    Thanks in advance,

    Tom



  • 2.  RE: Backup Server for SDDC Manager and NSX Manager

    Posted Oct 07, 2022 07:16 PM

    Hi Tom


    **Possible Cause**

    The SFTP account was expired on the backup server.


    ** Resolution **

    To resolve this issue, refresh the expired account on the backup server.


    ** Other checks list **

    [] - Verify that the folder exists on the SFTP backup server.

    [] - Verify that the SFTP permissions on the folder allow you to write to it.

    [] - Check the user account being used to connect to the Backup server is valid and working.

    [] - Try manually connecting to the same SFTP using either an FTP utility or CLI.

    Example: sftp username@IPAddress

    [] - Manually clear and readd all the Backup parameters and save the configuration again in SDDC Manager.

    [] - To obtain the SSH Fingerprint of the target system to verify, connect to the SDDC Manager Appliance over ssh and run the following command:

    ssh-keygen -lf <(ssh-keyscan -p 22 -t rsa sftp_server_fqdn 2> /dev/null) | cut -d' ' -f2

    [] - Testing connectivity between NSX-T manager and the SFTP server

    Login to an NSX-T manager appliance via root or if you do not have root login enabled you can login via the ‘admin’ account and then type ‘st en’ and then provide your root password.

    nsxt-mgr> st en
    Password:
    ***************************************************************************
    NOTICE TO USERS

    WARNING! Changes made to NSX Data Center while logged in as the root user
    can cause system failure and potentially impact your network. Please be
    advised that changes made to the system as the root user must only be made
    under the guidance of VMware.
    ***************************************************************************
    root@nsxt-mgr:~#

    From the root CLI of an NSX-T manager appliance we will run the following command to try and connect to our SFTP server and login using the built in OpenSSH package on the NSX-T appliance.

    sftp svc_backup-nsxt@sftp-backup
    The authenticity of host 'sftp-backup (10.0.1.27)' can't be established.
    ECDSA key fingerprint is SHA256:tm1qkfHb19sx5qzwJABAtojOd4cVywRCjmkoGeHeR6E.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'sftp-backup' (ECDSA) to the list of known hosts.

    Authorized Personnel Only
    --------------------------------------------------------
    WARNING: Unauthorized access to this system is forbidden
    and will be prosecuted to the fullest extent of the law.
    --------------------------------------------------------

    svc_backup-nsxt@sftp-backup's password:
    Connected to sftp-backup.


    Now that we are connected to our SFTP server we have successfully validated

    We have network connectivity from the NSX-T manager to the SFTP server
    The SFTP server service is running on the SFTP server listening for connections
    The service account created has permission to connect via the SFTP protocol

    Now we need to verify the service account has permission to navigate to the destination backup folder structure we created earlier and create a folder to verify we have ‘write’ permissions

    cd /mnt/backups
    mkdir test
    ls
    test
    rmdir test
    ls
    bye

    We have now successfully deployed a backup SFTP server that meets the security requirements of NSX-T and validated end to end network connectivity, the SFTP service is running/listening and we have the necessary permissions.

    HTH
    Michele V.



  • 3.  RE: Backup Server for SDDC Manager and NSX Manager

    Posted Oct 11, 2022 08:13 AM

    Hi MIchele,

    As stated in my post: 

    "SFTP and SSH from command line of SDDC manager works and I am able to create and delete folders. Also am I using the same SFTP server for File-Based-Backup of our vCenter Server without these issues."

    I even used the same sftp-account and directory as i have used for the vCenter Server backup.



  • 4.  RE: Backup Server for SDDC Manager and NSX Manager

    Posted Oct 11, 2022 09:32 PM

    Tom, please check to see if your OpenSSH on your SFTP server is at or above version 8.8. I was having the exact same issue with the exact same symptoms that you describe with both a fresh Windows and Linux SFTP server. In the /var/log/vmware/vcf/operationsmanager/operationsmanager.log, I kept seeing errors regarding ssh-rsa keys. If you check out https://www.openssh.com/txt/release-8.8 and https://ikarus.sg/rsa-is-not-dead/, you'll see that ssh-rsa has been deprecated in OpenSSH 8.8 and above this could possibly cause errors like we're seeing.

    I tried building a Windows SFTP server with OpenSSH 8.6 using (https://github.com/PowerShell/Win32-OpenSSH/releases/tag/V8.6.0.0p1-Beta), and set it up exactly the way I had set my other Windows SFTP server, and it connected instantly. So it seems like there may be a bug in SDDC manager where it's still looking for ssh-rsa keys instead of bouncing to the ecdsa-sha2-nistp256 keys. Will have to follow up with VMWare to see how we can use the latest version of OpenSSH with SDDC manager as I don't want to use the old standards.



  • 5.  RE: Backup Server for SDDC Manager and NSX Manager
    Best Answer

    Posted Oct 12, 2022 06:56 AM

    Hi ksagona, thank you for your reply. The openSSH version is indeed above 8.8. But as I had also opened a case with VMware and as they were able to solve the issue by temporary lowering the security settings, I did not make any changes to the openSSH version.



  • 6.  RE: Backup Server for SDDC Manager and NSX Manager

    Broadcom Employee
    Posted Mar 23, 2023 04:37 AM

    For those who arrive here and want to know more, the required "lowering" of security is permitting SHA-1 RSA keys on your SSH/SFTP server. This was deprecated as of OpenSSH v8.8, as called out in the release notes - https://www.openssh.com/txt/release-8.8 

    This release disables RSA signatures using the SHA-1 hash algorithm
    by default. This change has been made as the SHA-1 hash algorithm is
    cryptographically broken

    The changes I made to my /etc/ssh/sshd_config file to resolve the issue (VCF 4.5.0) were the addition of two lines:

    HostKeyAlgorithms=+ssh-rsa
    PubkeyAcceptedAlgorithms=+ssh-rsa

    followed by a restart of the sshd service.



  • 7.  RE: Backup Server for SDDC Manager and NSX Manager

    Posted Mar 29, 2023 12:17 PM

    Thanks, I tried adding those two lines 

    HostKeyAlgorithms=+ssh-rsa
    PubkeyAcceptedAlgorithms=+ssh-rsa

    to the sshd_config on my Windows server and a quick restart of the service and it worked perfectly. Hopefully it will get addressed in a future update, but for now, that works. Thank you!



  • 8.  RE: Backup Server for SDDC Manager and NSX Manager

    Posted Nov 29, 2023 09:29 PM

    I've just run into the same issue on a fresh build of vCF 5.1.0 backing up to a Photon OS VM.

    Thanks for posting the issue  and to  for the solution

    Cheers,
    Greg



  • 9.  RE: Backup Server for SDDC Manager and NSX Manager

    Posted Mar 23, 2023 10:45 AM

    You need to change permission on sddc of backup user

     

    go inside cd  /nfs/vmware/vcf/nfs-mount/

    ls -lrth 

    check is backup folder is there 

    if yes

    run this command 

     chmod g+w backup

    then try