Why Tenant Provisioning Won't Start

[Legacy User]

Hello guys,

 

I am new to the community and I would like to say that I am really sorry if I am asking this question in the wrong place.

Melanie_Giuliani, please fell free to delete my post at any moment.

 

The thing is that I am trying to install the Portal 4.2 in a AWS instance, a c5.4xlarge by the way. It has 16 cores and 32gb of RAM.

 

Everything goes fine, but the portal_tenant-provisioning service never starts. I have tried both online and offline installation and followed the great cotal01 guide.

 

I have followed the installation troubleshoot, but there was not much to do.

 

I leave a few screenshoot from the logs, but I don't have a good clue of what the problem may be. I am guessing that it's somehow related to the Portal trying to reach the Gateway, but as the gateway is in another machine and there is no configuration telling the portal where to find the gateway, it will never happen.

 

None of the installation guides that i have followed tells anything about the gateway instance, so I am not sure that the gateway is really necessary to the Portal.

 

I appreciate your time and any help will be welcome.

 

Thank you and have a good day.

 

Regards,

Herbert

[Alan Cota]

Hello herbert.silva ! I'm glad that the tutorial has been helpful. I have to say that I need to update it. So many improvements have been made since I first wrote this tutorial. Now, most of the steps are not required. But I don't think that has anything to do with the issue you're facing now.

 

Let me ask you something. How's your DNS configuration? Is this a 1st installation? If this is your first install, then I would just start the configuration from scratch. But only if you can erase all the volumes and persisted data, ok? To clean up and start from scratch, run the following commands:

 

$ docker system prune -a --volumes

 

Again, bu running the mentioned command you will be losing all the portal data and other docker volumes you may have. If you need the data, make sure you have a backup. After running the command, run the config.sh and then the portal.sh again. All the images will be downloaded. The first start of the CA API Portal can take between 15 to 20 minutes depending on the network.

 

Best regards,

Alan Cota.

[Legacy User]

Hello cotal01! So glad to see that you replied. Your tutorial was really awesome.

 

It's a fresh installation, so I can erase as many times as it takes!

The only DNS configuration that I made was in my Windows machine, where I added the following line in my hosts file:

 

<IP> apim.example.com apim-ssg.example.com analytics.example.com broker.example.com sso.example.com sync.example.com enroll.example.com

 

Replacing <IP> with the CentOS Server IP and example.com with the same domain used in the config.sh.

 

Also, when trying the offline installation, I had to add 

jarvis.packages.ca.com apim-portal.packages.ca.com

to the localhost line of the CentOS Server hosts file. Besides that, no other DNS configuration was done.

 

Do you think I should clean up again and start a new installation from scratch?

 

Thank you for your help!

Best regards,

Herbert

[Alan Cota]

Hello Herbert. I would start it from scratch. Also, make sure that you're using the latest version from here. When you do the docker system prune -a --volumes make sure everything has been erased afterward by checking if no volume is available. Let us know how it goes.

[DustinDauncey]

I'll leave this to you, Alan, as you are surely the most knowledgeable with this. I just wanted to add that depending on how the instance doesn't start (e.g. the container is in a restart loop or never actually even attempts to start), I wonder if memory needs to be bumped up a bit? I know in a few support cases, we've been able to get past some problems by simply throwing a bit more memory at it, upwards of 40 GB or so. Just something to ponder, in case a component is requiring more memory than the 32 GB available at boot.

 

From the screenshots attached though, I saw a lot of "Connection refused" errors, with routes seeming to go over localhost rather than an outside address. If that's the main issue, then memory of course won't be at play, and this would definitely sound like a network-related issue, most likely the way the node's network / DNS is configured.

[Alan Cota]

Very good points DustinDauncey! Thanks for sharing!

[Legacy User]

Hi All,

 

Seen before where a lock was set in the tenant_provisioning and not in the Portal and that caused similar issues.

 

So one can try something like:

 

docker exec -it $(docker ps --format "{{.ID}}" --filter name=portal_portaldb.1) psql -U tenant_provisioning admin --command "select * from databasechangeloglock;"

 

To see if there is a lock on the 'tenant_provisioning' and not just the 'portal' database.

 

If so, reset it similar to the troubleshooting guide, just with the correct database name.

 

Hope this helps,

 

Regards,

 

Amit.

[Legacy User]

cotal01 Hey Allan. I will try a new installation today. I just have to decide if online or offline.

 

DustinDauncey Hello Dustin. Thanks for your reply. I have talked to the administrators and unfortunately increasing memory is not a option for this project as this instance already corresponds to more than half of the budget. 

I also noticed the 'connection refused' from the logs and I think it's somehow related to the DNS configuration and I may have missed something.

 

Amit_Aharon Hey Amit. Thanks for replying. I have tried to run the command, but it asks me for a password. I entered "layer7", but it's not correct. Any idea on which the correct password may be?

 

Thank you all guys for trying to help. I really appreciate.

 

Regards,

Herbert

[Legacy User]

Hi Herbert,

 

Try password '7layer' and let us know f helps.

 

Regards,

 

Amit.

[Legacy User]

I am sorry about the "layer7". I mistyped it when replying to your message.

Just tried 7layer and got the same "wrong password".

 

I think this is the only default password that I know.

 

Best regards,

Herbert

[Legacy User]

Hello Alan!

 

Just did a fresh installation. I followed this instructions and also ran your script to clean the volumes.

Same result. I am starting to think that this may be a memory issue... despite the fact that I also tried an installation with no Analytics and I got the same result.

 

Guys, I appreciate your help and if you have no more ideas this is fine.

 

Thank you.

 

Best regards,

Herbert

[Ido Neamani]

Hi, Please make sure That the RAM memory which was allocated to the portal server is "Reserved" , meaning that it will no be shared among other VMs.

I faced issues with post portal installation that some containers kept rebooting and eventually it was a memory issue which was not reserved to the portal VM in or VMWARE environment.

[Legacy User]

Hello,

 

I just would like to let you know that increasing memory solved the problem.

We are now running a r4.4xlarge and it has 122Gb of RAM. Probably 40Gb would be more than enough, but it was not my call.

 

Thank you all for helping!

 

Best regards,

Herbert