AAKE install operator

[Michael A. Lowry]

The Automic Automation Install Operator is not well documented in the publicly-available documentation. Would someone from Broadcom be willing to address this gap?

Here's my high-level outline of how the install operator works in v24.4.4, and the order in which it starts other programs/pods.

•  install-operator
  • initial-data-24-4-4-0 (load-initialdata)
  • cust-aa-24-4-4-0-data (install-customization-aa)
  • cust-aa-24-4-4-0-ready (install-customization-aa)

What does each program/pod do? What are the sub-steps?

-------------------------------------------

[Oana Botez]

Hi,

• the initial data job is using DBLoad to upgrade to the latest version of the initial data (same as for on-prem)
• the install customization jobs are used to create the default client and users, load CAU packages and update settings like REST or JCP endpoints in UC_SYSTEM_SETTINGS

These changes are also included in the pods log files.

BR,
Oana 

-------------------------------------------

[B Bharath Kumar Reddy]

Note: We were able to gather this info purely based on our investigation and past experience. Please feel free to provide your feedback if any.

Below are the high-level steps performed by the install-operator during a typical upgrade/installation.

1. Generate Keystores & Create users

The install-operator initializes the TLS regeneration controller, which is responsible for generating the required keystores and updating the corresponding Kubernetes secrets whenever changes to Automation Engine (AE) certificates are detected. Parallelly, install-operator initiates the install/upgrade pipeline by setting the relevant values in the automic-automation ConfigMap as shown below. Additionally, it also validates the presence of secrets for database connectivity (database & ae) and the dedicated client users secrets client0-user and client100-user. If these secrets already exist, their creation is skipped and the existing secrets are reused in subsequent installation steps.

• Impacted Certs, Keystores & Secrets
  • awi-java-keystore
  • awi-cert
  • jcp-rest-java-keystore
  • jcp-rest-cert
  • jcp-ws-java-keystore
  • jcp-ws-cert
  • client0-user
  • client100-user
  • database secret
  • ae

• ConfigMap Update

statusVersion: none
statusStage: provisioning

2. Initial data Load

This job runs the ucybdbld utility by passing the UC_UPD.txt file as the input file with the below start parameters to update the DB Schema to new version. The ini file references the DB connection from ae secret created in the above step.

Start parameters: '-B -SNUC4_EXP2 -EREPLACE -X/opt/automic/ae-utils/db/general/24.4/UC_UPD.TXT -ZTS_DATA=SR_UC4 -WaitForSystemShutDown=30 -ZTS_INDEX=SR_UC4'

B	Batch Mode
SN	System Name
E	Mode
X	Path and File Name
WaitForSystemShutDown	Seconds to wait to make sure that the system is down
Z	Tablespace Name

3. cust-aa-<version>-data job

This is just a placeholder for time being without doing any actual work.

4. AE Component deployments

The install-operator creates necessary deployments & services for the AE Server processes WP, CP, JWP, JCP & REST. The desired replica count, resource requests and limits (CPU & Memory) are referenced from the values.yaml file.

5. cust-aa-<version>-ready job

Once all the core AE Components (WP, JCP, REST & JWP) are available, install-operator starts the cust-aa-ready-job. Starting from 24.4.4+hf1, this job contains two steps. 

5.1.  wait-for-jcp-ws

This step uses a simple busybox image to wait for jcp-ws-0 service to be ready on port 8443. The check happens for every 3 seconds repeatedly.

until nc -vz jcp-ws-0 8443;
do   echo "Waiting for jcp-ws..."
  sleep 3

done

Note: This step might fail if the cluster doesn't have access to public docker hub to fetch the busybox image. The workaround for this issue can be found here.

5.2. install-customization-aa

This step takes care of user creation and package installation based on the installation type (On-Provision/On-Update)

5.2.1. On-Provision 

• 000_add_client0_user.groovy

Creates new ADMIN/ADMIN user if not already present and locks the default UC/UC user.

• 001_configure_uc_system_settings.groovy 

Updates the below UC_SYSTEM_SETTINGS values based on the ENV values provided in the values.yaml file.

• • JCP_ENDPOINT
  • REST_ENDPOINT
  • CP_ENDPOINT
  • AUTOMATION_AI_ENDPOINT
• 002_add_client100_user.groovy

Creates a new user AUTOMIC/AUTOMIC for client 100 if not already present. If the user is already available, the job resets the password as defined in the secret.

• 004_install_all_packages.sh

Installs all the default packages along with CAU Packages

• • Package.ITPA.Shared_PCK.ITPA_SHARED_1_3_4+build.1771733760728.zip
  • Package.Analytics_PCK.AUTOMIC_ANALYTICS_2_0_9+build.1771596711955.zip
  • Package.RA.REST_PCK.AUTOMIC_RA_REST_1_0_1+build.7622.zip

• • Automation.Engine_PCK.AUTOMIC_CAU_AGENT_SAP_24_4_4+build.1771583105288.zip
  • Automation.Engine_PCK.AUTOMIC_CAU_AGENT_PS_24_4_2+build.1758533302027.zip
  • Automation.Engine_PCK.AUTOMIC_CAU_AGENT_WINDOWS_24_4_4+build.1771583108722.zip
  • Automation.Engine_PCK.AUTOMIC_CAU_JRE_UNIX_24_4_4+build.1771583121562.zip
  • Automation.Engine_PCK.AUTOMIC_CAU_JRE_WINDOWS_24_4_4+build.1771583121562.zip
  • Automation.Engine_PCK.AUTOMIC_CAU_AGENT_TLS_GTW_24_4_4+build.1771583106896.zip
  • Automation.Engine_PCK.AUTOMIC_CAU_AGENT_RA_24_4_4+build.1771583103881.zip
  • Automation.Engine_PCK.AUTOMIC_CAU_AGENT_UNIX_24_4_4+build.1771583107558.zip
  • Automation.Engine_PCK.AUTOMIC_CAU_AGENT_SQL_24_4_3+build.1764325844613.zip

5.2.2. On-Update

• 001_configure_uc_system_settings.groovy

<Same as above>

• 002_install_cau_packages.sh

Installs the CAU Packages.

6. CP deployment

After successful completion of the cust-aa-ready job, install-operator creates the necessary CP Deployments & services and scale the replicas based on the configuration provided in values.yaml file.

7. configMap update

Once done, the install-operator updates the automic-automation configMap values as shown below, thereby completing the installation process.

statusVersion: <version> E.g. 24.4.4
statusStage: provisioned

------------------------------
Thanks & Regards,
Bharath B.
------------------------------

[Michael A. Lowry]

The Helm chart for Automic Automation includes this conditional:

{{- $labels := .Values.labels | default dict }}
{{ if index $labels "features.aa-install-operator/force-operator-update-timestamp" }}
- name: FORCE_INSTALLOPERATOR_UPDATE
  value: {{ index $labels "features.aa-install-operator/force-operator-update-timestamp" | quote }}
{{- end }}

It works like this: when the top-level Helm value labels.features.aa-install-operator/force-operator-update-timestamp is set to a non-empty value, the chart renders this value as the FORCE_INSTALLOPERATOR_UPDATE environment variable in the install-operator container.

I could not find any mention of this in the documentation.

What are the valid values, and how do different values change the behavior of the install operator?