VMware Tanzu Application Service for VMs

View Only

I deployed PKS 1.2 : BOSH DIRECTOR: 2.4b152, PKS: 1.2.10b2 , NSX-T: 2.3b1, BOSH CLI: 2.0.48 I'm attempting to deploy a new PKS cluster into and the task fails, logging the following: result: 3 of 8 pre-start scripts failed. Failed Jobs: pks-n.....

Assaf Sauer posted Mar 07, 2019 01:08 PM

Assaf Sauer posted Mar 07, 2019 01:10 PM

Pls find furhter logs:

CPI DEBUG:

W, [2019-03-07T07:29:21.564111 #25251] WARN -- [req_id cpi-731536]: Error running method 'AddCustomFieldDef'. Failed with message 'The name 'director' already exists.'.

TASK (Grepped by error):

D, [2019-03-06T14:04:57.834878 #28477] [task:362] DEBUG -- DirectorJobRunner: (0.000369s) (conn: 70038073421660) INSERT INTO "events" ("parent_id", "timestamp", "user", "action", "object_type", "object_name", "error", "task", "deployment", "instance", "context_json") VALUES (NULL, '2019-03-06 14:04:57.833083+0000', 'pivotal-container-service-e09c978f57421312c808', 'release', 'lock', 'lock:deployment:service-instance_a014509c-faf9-44d2-af23-97046618904a', NULL, '362', 'service-instance_a014509c-faf9-44d2-af23-97046618904a', NULL, '{}') RETURNING *

I, [2019-03-06T14:04:57.836615 #28477] [task:362] INFO -- DirectorJobRunner: sending update deployment error event

D, [2019-03-06T14:04:57.836733 #28477] [task:362] DEBUG -- DirectorJobRunner: SENT: hm.director.alert {"id":"b7e41e4b-a617-4b6e-a30a-d07af26afd5e","severity":3,"source":"director","title":"director - error during update deployment","summary":"Error during update deployment for 'service-instance_a014509c-faf9-44d2-af23-97046618904a' against Director '74b82b0c-25e6-4cc9-b6cd-d478f9e27ee5': #<Bosh::Director::RpcRemoteException: Action Failed get_task: Task e602c2b6-8c2a-4407-54cc-dfad8a1fb49f result: 3 of 8 pre-start scripts failed. Failed Jobs: pks-nsx-t-prepare-master-vm, pks-nsx-t-ncp, ncp. Successful Jobs: etcd, bosh-dns-enable, bpm, bosh-dns, syslog_forwarder.>","created_at":1551881097,"deployment":"service-instance_a014509c-faf9-44d2-af23-97046618904a"}

D, [2019-03-06T14:04:57.843145 #28477] [task:362] DEBUG -- DirectorJobRunner: (0.000719s) (conn: 47379506128520) INSERT INTO "events" ("parent_id", "timestamp", "user", "action", "object_type", "object_name", "error", "task", "deployment", "instance", "context_json") VALUES (2750, '2019-03-06 14:04:57.836846+0000', 'pivotal-container-service-e09c978f57421312c808', 'create', 'deployment', 'service-instance_a014509c-faf9-44d2-af23-97046618904a', 'Action Failed get_task: Task e602c2b6-8c2a-4407-54cc-dfad8a1fb49f result: 3 of 8 pre-start scripts failed. Failed Jobs: pks-nsx-t-prepare-master-vm, pks-nsx-t-ncp, ncp. Successful Jobs: etcd, bosh-dns-enable, bpm, bosh-dns, syslog_forwarder.', '362', 'service-instance_a014509c-faf9-44d2-af23-97046618904a', NULL, '{"before":{},"after":{"releases":["kubo/0.21.13","bosh-dns/1.10.0","pks-helpers/50.0.0","cfcr-etcd/1.4.1","docker/35.0.0","pks-nsx-t/1.19.0","nsx-cf-cni/2.3.1.10693410","pks-vrli/0.6.0","pks-telemetry/0.9.4","syslog/11.4.0","bpm/0.6.0","wavefront-proxy/0.9.0","sink-resources-release/0.1.13"],"stemcells":["bosh-vsphere-esxi-ubuntu-xenial-go_agent/97.57"]}}') RETURNING *

D, [2019-03-06T14:04:57.858430 #28477] [task:362] DEBUG -- DirectorJobRunner: (0.000562s) (conn: 70038073421660) UPDATE "tasks" SET "event_output" = ("event_output" || '{"time":1551881097,"error":{"code":450001,"message":"Action Failed get_task: Task e602c2b6-8c2a-4407-54cc-dfad8a1fb49f result: 3 of 8 pre-start scripts failed. Failed Jobs: pks-nsx-t-prepare-master-vm, pks-nsx-t-ncp, ncp. Successful Jobs: etcd, bosh-dns-enable, bpm, bosh-dns, syslog_forwarder."}}

E, [2019-03-06T14:04:57.859460 #28477] [task:362] ERROR -- DirectorJobRunner: Action Failed get_task: Task e602c2b6-8c2a-4407-54cc-dfad8a1fb49f result: 3 of 8 pre-start scripts failed. Failed Jobs: pks-nsx-t-prepare-master-vm, pks-nsx-t-ncp, ncp. Successful Jobs: etcd, bosh-dns-enable, bpm, bosh-dns, syslog_forwarder.

D, [2019-03-06T14:04:57.863431 #28477] [task:362] DEBUG -- DirectorJobRunner: (0.000701s) (conn: 70038073421660) UPDATE "tasks" SET "state" = 'error', "timestamp" = '2019-03-06 14:04:57.861433+0000', "description" = 'create deployment', "result" = 'Action Failed get_task: Task e602c2b6-8c2a-4407-54cc-dfad8a1fb49f result: 3 of 8 pre-start scripts failed. Failed Jobs:...', "output" = '/var/vcap/store/director/tasks/362', "checkpoint_time" = '2019-03-06 14:04:57.259074+0000', "type" = 'update_deployment', "username" = 'pivotal-container-service-e09c978f57421312c808', "deployment_name" = 'service-instance_a014509c-faf9-44d2-af23-97046618904a', "started_at" = '2019-03-06 13:30:26.974006+0000', "event_output" = '{"time":1551879027,"stage":"Preparing deployment","tags":[],"total":1,"task":"Preparing deployment","index":1,"state":"started","progress":0}

{"time":1551881097,"stage":"Updating instance","tags":["master"],"total":1,"task":"master/fc9dbccd-0353-4e48-b4a7-41acfa60e36b (0) (canary)","index":1,"state":"failed","progress":100,"data":{"error":"Action Failed get_task: Task e602c2b6-8c2a-4407-54cc-dfad8a1fb49f result: 3 of 8 pre-start scripts failed. Failed Jobs: pks-nsx-t-prepare-master-vm, pks-nsx-t-ncp, ncp. Successful Jobs: etcd, bosh-dns-enable, bpm, bosh-dns, syslog_forwarder."}}

{"time":1551881097,"error":{"code":450001,"message":"Action Failed get_task: Task e602c2b6-8c2a-4407-54cc-dfad8a1fb49f result: 3 of 8 pre-start scripts failed. Failed Jobs: pks-nsx-t-prepare-master-vm, pks-nsx-t-ncp, ncp. Successful Jobs: etcd, bosh-dns-enable, bpm, bosh-dns, syslog_forwarder."}}

Attachment View in library

task362.rtf 14 KB

Broadcom Employee Mark Nagle posted Apr 04, 2019 04:59 PM

If you bosh ssh into the master vm

bosh -d service-instance_a014509c-faf9-44d2-af23-97046618904a ssh master/fc9dbccd-0353-4e48-b4a7-41acfa60e36b

Then cd into the job directory and check the pre-start logs to determine more details about the failure.

cd /var/vcap/sys/log/pks-nsx-t-prepare-master-vm/
cd /var/vcap/sys/log/pks-nsx-t-ncp/
cd /var/vcap/sys/log/ncp/