Agent responded with error: Action Failed get_task: Task c02e36fe-33c0-4a10-4bcd-da5122839ba2 result: 1 of 11 pre-start scripts failed. Failed Jobs: uaa. Successful Jobs: monit, registry, blobstore, nats, postgres-10, bpm, credhub, user_add, ca_certs, director.

22:56:00

Exit code 1

22:56:00

===== 2020-01-30 17:25:53 UTC Finished "/usr/local/bin/bosh --no-color --non-interactive --tty create-env /var/tempest/workspaces/default/deployments/bosh.yml"; Duration: 829s; Exit Status: 1

22:56:00

{"type":"step_finished","id":"bosh_product.deploying","description":"Installing BOSH"}

22:56:00

Exited with 1.could not execute "apply-changes": installation was unsuccessful

What i tried:

1) Tried to go into bosh and see the logs for UAA instance. But unable to log into BOSH. It was accessible yesterday before triggering the BOSH update.

Is it that the deployment failed i unable to log into BOSH ?

Fetching info:
  Performing request GET 'https://xx.xx.xx.xx:25555/info':
    Performing GET request:
      Retry: Get https://xx.xx.xx.xx:25555/info: dial tcp xx.xx.xx.xx:25555: connect: connection refused
 
Exit code 1

Please suggest how to deal with it.

Thanks,

Sujith K

Daniel Mikusa posted Jan 31, 2020 10:12 AM

You need to get into Bosh so you can see why UAA is failing. See https://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html for help getting signed in.

Otherwise you can use Ops Manager to fetch the logs as well. Hope that helps!

Sujith Kammila posted Feb 03, 2020 02:22 PM

@Daniel Mikusa - Tanzu Support Thank you for the response.

Well, i am able to get into BOSH via bbr key to see what the /var/vcap/sys/log/uaa/pre-start.stdout.log states as error looks to be around "pre-start scripts failed" UAA job in the concourse logs.

What can be inferred from below logs? and guide me if the direction of solving the error is not correct.

bosh/0:/var/vcap/sys/log/uaa# cat pre-start.stdout.log
uaa-pre-start - starting at Fri Jan 31 18:31:40 UTC 2020
Adding certificate from manifest to OS certs /usr/local/share/ca-certificates/uaa_0.crt
trying to run update-ca-certificates...
Clearing symlinks in /etc/ssl/certs...
done.
Updating certificates in /etc/ssl/certs...
Doing .
119 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Processing certificates for Java cacerts file
[uaa-ctl]   Processed certificate 1 of 204
[uaa-ctl]   Processed certificate 2 of 204
[uaa-ctl]   Processed certificate 3 of 204
[uaa-ctl]   Processed certificate 4 of 204
[uaa-ctl]   Processed certificate 5 of 204
[uaa-ctl]   Processed certificate 6 of 204
[uaa-ctl]   Processed certificate 7 of 204
[uaa-ctl]   Processed certificate 8 of 204
[uaa-ctl]   Processed certificate 9 of 204
[uaa-ctl]   Processed certificate 10 of 204
[uaa-ctl]   Processed certificate 11 of 204
[uaa-ctl]   Processed certificate 12 of 204
[uaa-ctl]   Processed certificate 13 of 204
[uaa-ctl]   Processed certificate 14 of 204
[uaa-ctl]   Processed certificate 15 of 204
[uaa-ctl]   Processed certificate 16 of 204
[uaa-ctl]   Processed certificate 17 of 204
[uaa-ctl]   Processed certificate 18 of 204
[uaa-ctl]   Processed certificate 19 of 204
[uaa-ctl]   Processed certificate 20 of 204
[uaa-ctl]   Processed certificate 21 of 204
[uaa-ctl]   Processed certificate 22 of 204
[uaa-ctl]   Processed certificate 23 of 204
[uaa-ctl]   Processed certificate 24 of 204
[uaa-ctl]   Processed certificate 25 of 204
[uaa-ctl]   Processed certificate 26 of 204
[uaa-ctl]   Processed certificate 27 of 204
[uaa-ctl]   Processed certificate 28 of 204
[uaa-ctl]   Processed certificate 29 of 204
[uaa-ctl]   Processed certificate 30 of 204
[uaa-ctl]   Processed certificate 31 of 204
[uaa-ctl]   Processed certificate 32 of 204
[uaa-ctl]   Processed certificate 33 of 204
[uaa-ctl]   Processed certificate 34 of 204
[uaa-ctl]   Processed certificate 35 of 204
[uaa-ctl]   Processed certificate 36 of 204
[uaa-ctl]   Processed certificate 37 of 204
[uaa-ctl]   Processed certificate 38 of 204
[uaa-ctl]   Processed certificate 39 of 204
[uaa-ctl]   Processed certificate 40 of 204
[uaa-ctl]   Processed certificate 41 of 204
[uaa-ctl]   Processed certificate 42 of 204
[uaa-ctl]   Processed certificate 43 of 204
[uaa-ctl]   Processed certificate 44 of 204
[uaa-ctl]   Processed certificate 45 of 204
[uaa-ctl]   Processed certificate 46 of 204
[uaa-ctl]   Processed certificate 47 of 204
[uaa-ctl]   Processed certificate 48 of 204
[uaa-ctl]   Processed certificate 49 of 204
[uaa-ctl]   Processed certificate 50 of 204
[uaa-ctl]   Processed certificate 51 of 204
[uaa-ctl]   Processed certificate 52 of 204
[uaa-ctl]   Processed certificate 53 of 204
[uaa-ctl]   Processed certificate 54 of 204
[uaa-ctl]   Processed certificate 55 of 204
keytool error: java.lang.Exception: Input not an X.509 certificate

Thanks,

Sujith K

Daniel Mikusa posted Feb 03, 2020 11:31 PM

Ah, interesting. Looks like there is an invalid certificate. It's not saying which one though. Do you have any Bosh Trusted Certificates configured? If so, I would suggest reviewing them and making sure they are all valid. You can use `openssl x509 -text -noout -in <file>` to print the cert, which should tell you if one is bad.

Aside from that, you might need to look at the pre-start script and add an `echo` line or something to print the specific cert file name before it's processed so that when it fails, you can see which one failed.

Hope that helps!