RabbitMQ

 View Only

 Upgrade to RabbitMQ 1.14.4 fails

Sri Timmaraju's profile image
Sri Timmaraju posted Dec 04, 2018 11:13 PM

We are upgrading RabbitMQ tile from 1.13.x to 1.14.4 and apply changes failed with the following error

 

Task 281921 | 18:22:19 | Preparing deployment: Preparing deployment (00:00:03) Task 281921 | 18:22:38 | Preparing package compilation: Finding packages to compile (00:00:00) Task 281921 | 18:22:39 | Updating instance rabbitmq-server: rabbitmq-server/b4fe0c68-5343-484f-bb6c-14e5544c3335 (0) (canary) (00:05:38) L Error: 'rabbitmq-server/b4fe0c68-5343-484f-bb6c-14e5544c3335 (0)' is not running after update. Review logs for failed jobs: service-metrics Task 281921 | 18:28:17 | Error: 'rabbitmq-server/b4fe0c68-5343-484f-bb6c-14e5544c3335 (0)' is not running after update. Review logs for failed jobs: service-metrics

 

And upon reviewing logs for failed service-metrics job, this is what I found

 

{"timestamp":"1543954269.516544580","source":"service-metrics","message":"service-metrics.executing-metrics-cmd","log_level":2,"data":{"error":"exit status 1","event":"failed","output":"2018/12/04 20:11:09 Missing argument -rabbitmqUsername. Usage: Username of RabbitMQ API Endpoint\n"}}

And, this is what I believe is the reason for error (username and password fields are empty)

 

exec chpst -u vcap:vcap ${pkg_dir}/bin/service-metrics \ --origin p-rabbitmq \ --agent-addr localhost:3458 \ --ca $job_dir/config/certs/ca.crt \ --cert $job_dir/config/certs/client.crt \ --key $job_dir/config/certs/client.key \ --metrics-cmd /var/vcap/packages/rabbitmq-server-metrics/bin/rabbitmq-server-metrics \ --metrics-cmd-arg '-erlangBinPath=/var/vcap/packages/erlang/bin/' --metrics-cmd-arg '-rabbitmqCtlPath=/var/vcap/packages/rabbitmq-server/bin/rabbitmqctl' --metrics-cmd-arg '-logPath=/var/vcap/sys/log/service-metrics/rabbitmq-server-metrics.log' --metrics-cmd-arg '-rabbitmqUsername=' --metrics-cmd-arg '-rabbitmqPassword=' --metrics-cmd-arg '-rabbitmqApiEndpoint=http://127.0.0.1:15672' \ \ --metrics-interval 30s \ 2>&1 >> $metrics_log_file

 

Sri Timmaraju's profile image
Sri Timmaraju

Issue is because of pre-provisioned services, resolved after following steps mentioned in the below link

https://docs.pivotal.io/rabbitmq-cf/1-14/turn-pre-provisioned-off.html

Luciano Silva's profile image
Luciano Silva

Hello Srinivas,

 

If you are using CI/CD pipelines to install/upgrade the RabbitMQ 1.14.4 tile, there is a known potential issue you may run into that looks very similar to the one you described here.

Here is more information about it.

https://community.pivotal.io/s/article/rabbitmq-1-14-4-potential-automated-deployment-issue

 

Regards,