vCenter Server Appliance Backup Process

[insearchof]

vCenter VCSA 6.7

I have an FTP site where I want to place the backup files

I setup a schedule for backup of the VCSA

Backup job failed to complete 

Where can I find out why this failed?

Thank you

Tom

[b.w]

ssh to the vcsa  - in the folder /var/log/vmware/applmgmt you see backup.log, backupscheduler.log, backupschedulercron.log.

also see /var/log/vmware/messages

[insearchof]

Bewe

Thanks for the info.

The backup.log is a very large file   what can I search for in that log?

Not I Linux person but I did a cat of the log and it was very long too long to scan thru.

Thank you,

[b.w]

do you know "grep" for filtering ? search for error, fail etc. or start a new backup job then you have only a short time span to search

[insearchof]

I have some notes on GREP 

root@tgcsnet-vcsa-2 [ /var/log/vmware/applmgmt ]# grep -w "ERROR" backup.log

2020-05-13T10:31:03.443 [StatsMonitorDBBackup:PID-23322] ERROR: Process returncode is -13, but expected exit codes are [0].

2020-05-13T10:31:03.444 [StatsMonitorDBBackup:PID-23322] ERROR: rc: 1, stderr: Traceback (most recent call last):

2020-05-13T10:31:03.445 [StatsMonitorDBBackup:PID-23322] ERROR: Process returncode is 1, but expected exit codes are [0].

2020-05-13T10:31:03.446 [StatsMonitorDBBackup:PID-23322] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-13T10:31:03.446 [StatsMonitorDBBackup:PID-23322] ERROR: Failed to dispatch dump image of Appliance Stats Monitor database.

2020-05-13T10:31:03.858 [ComponentScriptsBackup:PID-23321] ERROR: Process returncode is -13, but expected exit codes are [0].

2020-05-13T10:31:03.859 [ComponentScriptsBackup:PID-23321] ERROR: rc: 1, stderr: Traceback (most recent call last):

2020-05-13T10:31:03.859 [ComponentScriptsBackup:PID-23321] ERROR: Process returncode is 1, but expected exit codes are [0].

2020-05-13T10:31:03.860 [ComponentScriptsBackup:PID-23321] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-13T10:31:03.864 [ComponentScriptsBackup:PID-23321] ERROR: Unable to perform backup: [Errno 32] Broken pipeException ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>

2020-05-13T10:31:03.864 [ComponentScriptsBackup:PID-23321] ERROR: BrokenPipeError: [Errno 32] Broken pipe

2020-05-13T10:31:03.890 [ComponentScriptsBackup:PID-23321] ERROR: Component backup command "/etc/vmware/backup/component-scripts/vum/backup_restore.py --startBackup" failed 1.

2020-05-13T10:31:03.892 [ComponentScriptsBackup:PID-23321] ERROR: Error during component vum backup

2020-05-13T10:31:04.259 [MainProcess:PID-23044] ERROR: Error at process ComponentScriptsBackup; rc:1.

2020-05-13T10:31:04.260 [MainProcess:PID-23044] ERROR: stderr:Error during component vum backup

2020-05-13T10:31:04.261 [MainProcess:PID-23044] ERROR: BackupManager encountered an exception: Hit exception inside process ComponentScriptsBackup:

2020-05-13T10:31:04.318 [LotusBackup:PID-23327] ERROR: rc: 2, stderr: /usr/bin/tar: -: Wrote only 4096 of 10240 bytes

2020-05-13T10:31:04.321 [LotusBackup:PID-23327] ERROR: Process returncode is 2, but expected exit codes are [0].

2020-05-13T10:31:04.321 [LotusBackup:PID-23327] ERROR: rc: 1, stderr: Traceback (most recent call last):

2020-05-13T10:31:04.321 [LotusBackup:PID-23327] ERROR: Process returncode is 1, but expected exit codes are [0].

2020-05-13T10:31:04.322 [LotusBackup:PID-23327] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-13T10:31:04.322 [LotusBackup:PID-23327] ERROR: Failed to backup Lotus DB

2020-05-13T10:31:04.323 [LotusBackup:PID-23327] ERROR: Failed to backup Lotus DB

2020-05-13T10:31:04.667 [MainProcess:PID-23044] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-13T10:31:05.738 [MainProcess:PID-23044] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-13T10:31:05.746 [MainProcess:PID-23044] ERROR: Failed to perform on-exit cleanup

2020-05-13T14:30:55.541 [StatsMonitorDBBackup:PID-62440] ERROR: Process returncode is -13, but expected exit codes are [0].

2020-05-13T14:30:55.542 [StatsMonitorDBBackup:PID-62440] ERROR: rc: 1, stderr: Traceback (most recent call last):

2020-05-13T14:30:55.542 [StatsMonitorDBBackup:PID-62440] ERROR: Process returncode is 1, but expected exit codes are [0].

2020-05-13T14:30:55.542 [StatsMonitorDBBackup:PID-62440] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-13T14:30:55.543 [StatsMonitorDBBackup:PID-62440] ERROR: Failed to dispatch dump image of Appliance Stats Monitor database.

2020-05-13T14:30:55.792 [ComponentScriptsBackup:PID-62442] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-13T14:30:55.792 [ComponentScriptsBackup:PID-62442] ERROR: Component rbd backup dispatch failed.

2020-05-13T14:30:55.793 [ComponentScriptsBackup:PID-62442] ERROR: Failed to dispatch component rbd backup

2020-05-13T14:30:56.294 [MainProcess:PID-62063] ERROR: Error at process StatsMonitorDBBackup; rc:-13.

2020-05-13T14:30:56.294 [MainProcess:PID-62063] ERROR: stderr:Failed to dispatch dump image of Appliance Stats Monitor database.

2020-05-13T14:30:56.294 [MainProcess:PID-62063] ERROR: BackupManager encountered an exception: Hit exception inside process StatsMonitorDBBackup:

2020-05-13T14:30:56.512 [MainProcess:PID-62063] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-13T14:30:57.442 [MainProcess:PID-62063] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-13T14:30:57.451 [MainProcess:PID-62063] ERROR: Failed to perform on-exit cleanup

Looks like it had a problem cleaning up the data in the folder should I clear the folder and try again?

Thoughts.

Thank you

Tom

[b.w]

no there are to many errors before cleaning up - the main error will sit on the beginning of the job.

better rename the backup.log and start a new backup - then you have a small backup.log to read

[insearchof]

renamed the backup.log file and restarted the backup job and it is just as large as before.

I got this on the console

Structure com.vmware.appliance.recovery.backup.job.details.info has a union with a field not required for this case = end_time

On the FTP Server in the folder I see only two files    config_files.tar.gz  and lotus_backup.tar.gz

On the Appliance the backup

It is just sitting there no % moving

Any ideas?

Thanks

Tom

[insearchof]

It finally finished and it failed again

this time not so many errors

root@tgcsnet-vcsa-2 [ /var/log/vmware/applmgmt ]# grep -w "ERROR" backup.log

2020-05-13T17:27:05.280 [StatsMonitorDBBackup:PID-12644] ERROR: Process returncode is -13, but expected exit codes are [0].

2020-05-13T17:27:05.281 [StatsMonitorDBBackup:PID-12644] ERROR: rc: 1, stderr: Traceback (most recent call last):

2020-05-13T17:27:05.281 [StatsMonitorDBBackup:PID-12644] ERROR: Process returncode is 1, but expected exit codes are [0].

2020-05-13T17:27:05.282 [StatsMonitorDBBackup:PID-12644] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-13T17:27:05.283 [StatsMonitorDBBackup:PID-12644] ERROR: Failed to dispatch dump image of Appliance Stats Monitor database.

2020-05-13T17:27:05.475 [ComponentScriptsBackup:PID-12645] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-13T17:27:05.476 [ComponentScriptsBackup:PID-12645] ERROR: Component rbd backup dispatch failed.

2020-05-13T17:27:05.476 [ComponentScriptsBackup:PID-12645] ERROR: Failed to dispatch component rbd backup

2020-05-13T17:27:05.772 [MainProcess:PID-12343] ERROR: Error at process StatsMonitorDBBackup; rc:-13.

2020-05-13T17:27:05.773 [MainProcess:PID-12343] ERROR: stderr:Failed to dispatch dump image of Appliance Stats Monitor database.

2020-05-13T17:27:05.773 [MainProcess:PID-12343] ERROR: BackupManager encountered an exception: Hit exception inside process StatsMonitorDBBackup:

2020-05-13T17:27:05.995 [MainProcess:PID-12343] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-13T17:27:06.934 [MainProcess:PID-12343] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-13T17:27:06.943 [MainProcess:PID-12343] ERROR: Failed to perform on-exit cleanup

Any ideas?

Thank you

Tom

[b.w]

maybe this can help ?

https://www.techcrumble.net/2018/06/vcsa-6-7-backup-failed/

VCSA 6.7 Appliance Native Backup Failing at BackupLotus

or you can post the backup.log here ? it should not contain any confidential information. but please do not post the content here - better attach as .txt or .zip

[insearchof]

Bewe

I was thinking about the permissions issue on my SFTP server I checked and the user account I am using has full control of the folder in use.  It does place several folders and two file there so it can not be a permissions problem.

I would be glad to post the backup log file but I am unable to copy it off the VCSA appliance.

I have WINSCP on my windows 10 computer   but unable to scp to the appliance.

Any ideas on how I can get the file off?

Thank you

Tom

[b.w]

here is described how to scp to the appliance : VMware Knowledge Base

but do a test:

- ssh to the appliance

- from there open a ftp session to your target

- upload the backup.log to the ftp

the side effect is that you test with this upload that the permissions to the ftp are correct

[insearchof]

That worked see attached log

[b.w]

i will check the log tomorrow.

could be an firewall problem because vcsa uses passive ftp. what os is your ftp-server ?

[insearchof]

My FTP server is Windows 2012 R2  Using IIS  for FTP

The firewall in Windows Firewall I will look into the firewall setting too.

Thanks

Tom

[b.w]

please doublecheck your permissions on ftp: open a ftp session from vcsa and create, modify and delete files and folders.

also for a test disable the firewall on your ftp

the log reports trouble with statsmonitordbbackup and componentsscriptbackup, also with modifying files on the backup target

[insearchof]

bewe

Ok

1  How can I open an FTP session on the VCSA from the command prompt I type shell    then type FTP and get unknown command

Command> shell

Shell access is granted to root

root [ ~ ]# ftp

bash: ftp: command not found

root [ ~ ]#

.

I then disabled the firewall on the FTP Server and ran the job the same thing happened job failed only copied the two files and stops.

root@tgcsnet-vcsa-2 [ /var/log/vmware/applmgmt ]# grep -w "ERROR" backup.log

2020-05-14T12:23:37.991 [StatsMonitorDBBackup:PID-37802] ERROR: Process returncode is -13, but expected exit codes are [0].

2020-05-14T12:23:37.992 [StatsMonitorDBBackup:PID-37802] ERROR: rc: 1, stderr: Traceback (most recent call last):

2020-05-14T12:23:37.993 [StatsMonitorDBBackup:PID-37802] ERROR: Process returncode is 1, but expected exit codes are [0].

2020-05-14T12:23:37.993 [StatsMonitorDBBackup:PID-37802] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-14T12:23:37.994 [StatsMonitorDBBackup:PID-37802] ERROR: Failed to dispatch dump image of Appliance Stats Monitor database.

2020-05-14T12:23:38.307 [ComponentScriptsBackup:PID-37807] ERROR: rc: 9, stderr: b'curl: (9) Server denied you to change to the given directory\n'

2020-05-14T12:23:38.307 [ComponentScriptsBackup:PID-37807] ERROR: Component rbd backup dispatch failed.

2020-05-14T12:23:38.308 [ComponentScriptsBackup:PID-37807] ERROR: Failed to dispatch component rbd backup

2020-05-14T12:23:38.691 [MainProcess:PID-37647] ERROR: Error at process StatsMonitorDBBackup; rc:-13.

2020-05-14T12:23:38.691 [MainProcess:PID-37647] ERROR: stderr:Failed to dispatch dump image of Appliance Stats Monitor database.

2020-05-14T12:23:38.692 [MainProcess:PID-37647] ERROR: BackupManager encountered an exception: Hit exception inside process StatsMonitorDBBackup:

2020-05-14T12:23:38.876 [MainProcess:PID-37647] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-14T12:23:39.748 [MainProcess:PID-37647] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.

2020-05-14T12:23:39.763 [MainProcess:PID-37647] ERROR: Failed to perform on-exit cleanup

root@tgcsnet-vcsa-2 [ /var/log/vmware/applmgmt ]#

Any ideas?

Thank you

Tom

[b.w]

ah sorry - vcsa has no ftp client. but you can test from windows if your account can create, delete files and folders

because of the errors i will look further

[insearchof]

Using the account in filzilla  FTP app I was unable to connect So I chose another account and I was able to connect to the FTP folder    created files folders deleted files and folders no problem

I tried that account on the backup job but I get the same results

I have the firewall still off too

[b.w]

hi,

do you have another status for backup ?

with deactivated firewall and proper permissions to ftp it should run

[insearchof]

Bewe,

sorry no change same error ever time I try

Not sure what to try now

[insearchof]

Bewe

I am trying something different now

Trying to use SMB

The server is Windows 2012 R2 has SMB 1.0 installed

I setup a backup to point to the SMB share and it will not run either.

SMB is supported method on 6.7   the error makes no sense.

Any ideas?

[b.w]

yes - vmware has problems with smb. you have to use an administrative share like smb://<server>/c$/vcenterbackup/...

but in my experience backup to smb killed my vcsa twice. dont know if it was the reason but 2 times vcsa got corrupted during the timespan backup to smb was running.

do you have the option to backup to scp ? just for a test  - scp worked fine in a customer environment for a long time, now ftp is running w/o problems there

[insearchof]

BEWE

what do I need to setup the SCP method

Never Mind I need a Linux machine to use SCP not an option here.

The Error does not help at all  Location type 'SMB' is not supported.

I greped the log and no errors  found

Location type SMB is not supported          where ? on the vcsa or the windows server?

Any ideas

Another thought

Is there a way to back this up using Powercli?

[insearchof]

Bewe,

Success...……..

I tried SCP method  I have a cent OS machine as a VM I used in conjunction with Nagios which runs on a separate Linux OS

I ssh to the cent os server and then created a folder

went to the vcsa appliance  interface and setup a backup to use SCP:\\10.10.10.10\vcsabackup       account and password and the job ran successfully.

My Linux server have a very small foot print as far as disk space goes so backing up to them is ok but not a long term solution.

Looking into getting this to work with a windows server  not sure which protocol to try on this.

Thanks

Tom

[insearchof]

Bewe,

I then tried to SCP to my Windows server and it failed

First thing

From the Linux server that I ran the backup to I used scp to copy to the windows server and it worked

So if I can SCP from Linux to my windows then why can't I from the Appliance?

I get this error

  

Location 'scp://10.2.8.74/vcsabackup' does not have write permission.

the account I am using has full permission to the folder.

from the Linux

scp -r vcsabackup useradmin@10.2.8.120:e:/vcsabackup

Any ideas

Tom

[b.w]

but now you can be sure that your vcsa is healthy and can perform backups. all other problems are related to your backup targets

[insearchof]

Bewe

yes that sounds about right

I have not found any good article that show how to prepare the targets.

Only articles on how to run the backup process.   That part is simple

I would like to get the FTP function working

I have a Kemp here and I use FTP to back that up on a daily basis to the same FTP server I am trying with the appliance.

Just do not know what the appliance is looking for

I have yet to find any requirements for this setup.

Thanks

Tom

[b.w]

https://www.altaro.com/vmware/backing-up-vcsa-6-5-natively-using-ftps/

note that the target folder has to be empty when creating an ftp-job

[insearchof]

BEWE

Good NEWS...……..

I decided to build a New VM  Windows 2019 Server

First I installed IIS and FTP and that did not work either.

Then I downloaded Filezilla Server and had the article to show how to set up the information for the VCSA backup   and now both vcenters are backing up to my FTP site.

One other thing I think the problem with FTP all along was my command line string.

I was always doing this ftp://10.10.10.123/ftpsvcsa

when you only need this

ftp://10.10.10.123/                 di not use the name of the folder FTP site knows where to place the incoming files

Now have it setup for a weekly backup schedule.   will keep eye on this

But I think we are now good.

Thanks  for all your help we can close this out now as resolved

[btechit]

good gawd ftp is so insecure. 

[btechit]

you probably need to backup to an insecure location...like smb 1.0 or ftp