Automic Workload Automation

 View Only

  • 1.  Fetch list of Docker images for an AAKE release

    Posted Jan 28, 2026 05:14 PM

    When we moved our Automic Automation systems from VMs to Kubernetes clusters, we decided that we would keep our own local copies of all of the required Docker images, rather than relying on Broadcom's image repositories in Google Container Registry (GCR). To this end, we have relied on the AAKE quick start web page to obtain the list of the Docker images we need to copy. Until recently, fetching this list and the images has always involved some manual steps.

    Today we finished work on a pipeline that completely automates the process. In broad strokes, the new pipeline performs these steps:

    1. Fetch the "HELM" component ZIP archive for a new AAKE release from Broadcom using the Automic Downloads API.
    2. Extract the automic-automation Helm chart from this ZIP archive and extract the TGZ file.
    3. Run helm template on the chart and parse the output to generate the list of (15) Docker images (from the operator-config ConfigMap).
    4. Pull the images from GCR using docker pull.
    5. Tag the images using docker tag.
    6. Push the Helm chart and images to our private image repository using docker push.

    If anyone is interested, I'd be happy to share the details.



    -------------------------------------------


  • 2.  RE: Fetch list of Docker images for an AAKE release

    Posted Mar 26, 2026 06:30 AM
    Edited by Michael A. Lowry Mar 26, 2026 06:32 AM

    Here is the Azure DevOps pipeline we're using. Yesterday I received confirmation from Broadcom that this is a viable approach.

    # Download AAKE Helm chart from Broadcom.
    # Helm chart versions lack the leading digit. E.g., the version AE '24.4.2.3' is HELM 'v'4.4.2.3'.
    # Version numbers with a hotfix must be in the format '4.4.2 HF3' not '4.4.2.3'.
    pool:
      name: "Ubuntu-Default"
    
    parameters:
    - name: broadcom_helm_chart_version
      type: string
      default: '4.4.4'
      
    - name: containerRegistryUrl
      type: string
      displayName: Docker Repository
      default: docker-uc4.artifact.example.com
    
    variables:
    - group: BroadcomSupportAccount
    - group: DockerReposAccount
    
    steps:
    
    # Download pull secret
    - task: DownloadSecureFile@1
      name: imagePullSecret
      displayName: 'Download Automic Image Pull Secret'
      inputs:
        secureFile: 'automic-image-pull-secret.json'
    
    # Download the AAKE Helm chart from Broadcom using the Automic Downloads REST API -- https://downloads.automic.com/downloads/download_api_documentation
    - bash: |
        shopt -s extglob
        cd $STAGING_DIR
        echo "Current working directory: $(pwd)"
        ls -l $filename
        printf '%s\n' "${s// /%20}"
        echo
        download_category_id="1570005424185"                      # Components
        component_id="1601569166697"                              # HELM
        version="${{ parameters.broadcom_helm_chart_version }}"   # Version
        version_match=$(printf '%s\n' "${version// /%20}")        # Convert any spaces to '%20'
        base_url="https://downloads.automic.com/api/downloads/v1"
        header=Authorization:\ Basic\ $(basic_credentials)
        location="${base_url}?download_category_id=${download_category_id}&component_id=${component_id}&delivery_filter=${delivery_filter}&version_match=${version_match}"
        echo "Fetching location: $location"
        curl --show-error -v --location "$location" --header "$header" > download.json
        if [[ ! -s download.json ]]; then echo "File download.json does not exist or is empty"; exit; fi
        echo "Curl output:"
        jq --color-output < download.json
        echo
        # An efficient way of parsing the JSON using read, process substitution, and a single jq command.
        IFS=$'\t'
        read filename component_version download_file_id download_link filehash < <(jq -r ".download_category[].component[].download_file[] | [.filename, .component_version_name, .download_file_id, .download_link, .filehash] | @tsv" < download.json )
        filehash=$(echo $filehash | cut -c3- )
        echo "File name     : $filename"
        echo "Version       : $component_version"
        echo "DL file ID    : $download_file_id"
        echo "Download link : $download_link"
        echo "File hash     : $filehash"
        echo
        echo "Downloading file..."
        location="${base_url}?action=get&download_file_id=${download_file_id}"
        echo "Fetching location: $location"
        curl --show-error -v --header "$header" --request GET "$location" --remote-name --remote-header-name
        if [[ ! -f "$filename" ]]; then echo "ERROR: Download file not found."; exit 1; fi
        echo "Current working directory: $(pwd)"
        ls -l $filename
        md5sum=$( md5sum $filename | cut -d' ' -f1 | tr '[:lower:]' '[:upper:]' )
        echo "DL file hash  : $md5sum"
        if [[ "$md5sum" == "$filehash" ]]; then
        echo "File checksum OK."
        else
        echo "ERROR: Checksum mismatch."
        filetype=$(file $filename | cut -d ':' -f 2)
        echo "File type: $filetype"
        if [[ $(echo $filetype | grep 'text' | wc -l ) -gt 0 ]] || [[ $(echo $filetype | grep 'JSON' | wc -l ) -gt 0 ]] || [[ $(echo $filetype | grep 'HTML' | wc -l ) -gt 0 ]]; then
        echo "Text response:"
        cat $filename
        fi
        exit 1
        fi
        echo "Extracting ZIP archive."
        unzip $filename
        echo "Removing Helm pluging file."
        rm -v automic-automation-plugin-*
        echo "Contents of $STAGING_DIR:"
        ls -l $STAGING_DIR
      env:
        STAGING_DIR: $(Build.ArtifactStagingDirectory)
      displayName: 'Download AAKE Helm chart from Broadcom'
      condition: true
    
    # Push Helm chart directly to OCI registry
    - bash: |
        # Make sure there's only one file.
        cd $STAGING_DIR
        mapfile -t files < <(find . -maxdepth 1 -type f -name 'automic-automation-*.tgz' ! -name 'automic-automation-plugin-*.tgz')
        if (( ${#files[@]} != 1 )); then
          echo "Expected exactly one Helm chart .tgz file, found ${#files[@]}: ${files[*]}" >&2
          exit 1
        fi
        # Login securely
        printf '%s' "$(DockerReposPassword)" | helm registry login ${{ parameters.containerRegistryUrl }} --username $(DockerReposUser) --password-stdin
        # Push the file to container registry
        base="$(basename "${files[0]}")"
        helm push "$base" oci://${{ parameters.containerRegistryUrl }}/helm
      env:
        STAGING_DIR: $(Build.ArtifactStagingDirectory)
      displayName: 'Push AAKE Helm Helm chart to container registry'
      condition: succeeded()
    
    # Install Helm if it's not already installed.
    - task: HelmInstaller@1
      inputs:
        helmVersionToInstall: 'latest'
      displayName: Install Helm
    
    # Use 'helm template' to parse the AAKE Helm chart, and generate a list of required Docker images
    - bash: |
        set -euo pipefail
        shopt -s extglob
        cd "$STAGING_DIR"
        # Extract just automic-automation/values.yaml from the automic-automation-*.tgz into the current directory
        tar -xzf automic-automation-*.tgz # --strip-components=1 automic-automation/values.yaml
        cd automic-automation
        ls -l
        # Fill an array with a list of Docker images from the operator-config ConfigMap in the Helm chart.
    
          helm template automic-automation . \
            --set-string environment.AUTOMIC_SYSTEM_NAME=DUMMY \
            --set-string environment.AUTOMIC_GLOBAL_SYSTEM=DUMMY \
            --include-crds \
          | yq -r '
              select(.kind == "ConfigMap" and .metadata.name == "operator-config")
              | .data["image.properties"]
            ' \
          | awk -F= '
              /^images\./ && /\.repository=/ { key=$1; sub(/^images\./,"",key); sub(/\.repository$/,"",key); repo[key]=$2 }
              /^images\./ && /\.tag=/        { key=$1; sub(/^images\./,"",key); sub(/\.tag$/,"",key); tag[key]=$2 }
              END {
                for (k in tag) if (tag[k] != "") {
                  r = (k in repo ? repo[k] : "")
                  if (r != "") print r k ":" tag[k]
                }
              }
            ' \
          | sort -u \
          | grep -v 'psql' | tee images.txt
        echo "Wrote $(wc -l < images.txt) images to images.txt"
      displayName: 'Generate image list from operator-config'
      condition: succeeded()
    
    # Publish image list
    - publish: images.txt
      artifact: aake-image-list
      displayName: 'Publish image list'
      condition: succeeded()
    
    # Docker login
    - bash: |
        set -euo pipefail
        echo "Logging into Broadcom's GCR repo using the pull secret."
        docker login -u _json_key --password-stdin https://gcr.io < $(imagePullSecret.secureFilePath)
      displayName: 'Docker login (GCR)'
      condition: succeeded()
    
    # Docker pull
    - bash: |
        while IFS= read -r src; do
          echo "Processing: $src"
          docker pull "$src"
        done < images.txt
      displayName: 'Docker pull (GCR)'
      condition: succeeded()
    
    # Docker tag
    - bash: |
        set -euo pipefail
        echo "Tagging images"
        while IFS= read -r src; do
          echo "Processing: $src"
          name_and_tag="${src##*/}"            # e.g., awi:24.4.3.1
          dst="${TARGET_REPO}/${name_and_tag}"
          docker tag  "$src" "$dst"
        done < images.txt
      displayName: 'Docker tag'
      condition: succeeded()
    
    # Docker push
    - bash: |
        set -euo pipefail
        echo "Pushing images to container registry"
        docker login ${TARGET_REPO} --username $(DockerReposUser) --password $(DockerReposPassword)
        while IFS= read -r src; do
          echo "Processing: $src"
          name_and_tag="${src##*/}"            # e.g., awi:24.4.3.1
          dst="${TARGET_REPO}/${name_and_tag}"
          docker push "$dst"
        done < images.txt
      env:
        TARGET_REPO: ${{ parameters.containerRegistryUrl }}
      displayName: 'Docker push'
      condition: succeeded()

    Notes:

    1. Passwords are stored in variable groups.
    2. The GCR pull secret is stored in a secure file.
    3. The grep -v 'psql' part is required because the operator-config ConfigMap in the AAKE Helm chart includes psql, an image that is not needed and not present in GCR. Broadcom informed me that they will fix this in v26.1.


  • 3.  RE: Fetch list of Docker images for an AAKE release

    Posted 17 days ago
    Edited by Michael A. Lowry 17 days ago

    Here is an Azure DevOps pipeline updated to use broadcom.com servers:

    • The pipeline fetches the Helm chart from Broadcom's Artifactory server instead of from downloads.automic.com.
    • The pipeline pulls AAKE images from Broadcom's Artifactory server instead of from Google Container Registry.
    • The pipeline includes a new Bash function normalize_version() that automatically adapts version numbers like 4.4.5.1 to the style needed for the AQL query, e.g., 4.4.5+hf.1.
    # Download AAKE Helm chart from Broadcom.
    # Helm chart versions lack the leading digit. E.g., the version AE '24.4.5.1' is HELM 'v'4.4.5.1'.
    
    pool:
      name: "Ubuntu-Default"
    
    parameters:
    - name: broadcom_helm_chart_version
      type: string
      default: '4.4.5.1'
    - name: broadcomArtifactory
      type: string
      displayName: Broadcom Artifactory REST API server
      default: "https://packages.broadcom.com/artifactory"
    - name: broadcomContainerRegistry
      type: string
      displayName: Broadcom Docker Registry
      default: automic-docker.packages.broadcom.com
    - name: containerRegistry
      type: string
      displayName: Local Docker Registry
      default: docker-uc4.artifact.example.com
    
    variables:
    - group: BroadcomSupportAccount
    - group: DockerReposAccount
    
    steps:
    
      # Download the AAKE Helm chart using Broadcom's Artifactory REST API server
      - bash: |
          set -euo pipefail
          shopt -s extglob
          normalize_version() {
          local v="$1"
          # Format: X.Y.Z.N  → X.Y.Z+hf.N
          if [[ "$v" =~ ^([0-9]+)\.([0-9]+)\.([0-9]+)\.([0-9]+)$ ]]; then
            echo "${BASH_REMATCH[1]}.${BASH_REMATCH[2]}.${BASH_REMATCH[3]}+hf.${BASH_REMATCH[4]}"
            return 0
          fi
          # Format: X.Y.Z HFN → X.Y.Z+hf.N
          if [[ "$v" =~ ^([0-9]+)\.([0-9]+)\.([0-9]+)[[:space:]]+HF([0-9]+)$ ]]; then
            echo "${BASH_REMATCH[1]}.${BASH_REMATCH[2]}.${BASH_REMATCH[3]}+hf.${BASH_REMATCH[4]}"
            return 0
          fi
          # Format: X.Y.Z (no hotfix)
          if [[ "$v" =~ ^([0-9]+)\.([0-9]+)\.([0-9]+)$ ]]; then
            echo "${BASH_REMATCH[1]}.${BASH_REMATCH[2]}.${BASH_REMATCH[3]}"
            return 0
          fi
          echo "ERROR: Invalid version format: '$v'" >&2
          echo "Expected formats:" >&2
          echo "  X.Y.Z" >&2
          echo "  X.Y.Z.N" >&2
          echo "  X.Y.Z HFN" >&2
          return 1
          }
          cd "$STAGING_DIR"
          version=$(normalize_version "$AAKE_HELM_CHART_VERSION")   # Version
          echo "Searching Broadcom Artifactory for AAKE Helm chart with version ${version}..."
    
          # Broadcom Artifactory
          base_url="${BROADCOM_ARTIFACTORY}"
          location="${base_url}/api/search/aql"
    
          echo "Token defined? $( [ -n "${BROADCOM_TOKEN:-}" ] && echo yes || echo no )"
          echo "Token length     : ${#BROADCOM_TOKEN}"
    
          # Perform AQL query to find Helm chart
          echo "Fetching location: $location"
          
          request_body=$(cat <<EOF
          items.find({
            "repo": "automic-helm",
            "path": "automic/component/release/HELM",
            "type": "file",
            "@chart.version": "${version}"
          }).include("*", "property")
          EOF
          )
    
          echo "Request body:"
          echo "$request_body"
    
          printf '%s\n' "$request_body" | \
          curl \
            --user "${BROADCOM_USERNAME}:${BROADCOM_TOKEN}" \
            --header 'content-type: text/plain' \
            --location "$location" \
            --show-error \
            --fail-with-body \
            -o response.json \
            --data @-
    
          if [[ ! -s response.json ]]; then echo "File response.json does not exist or is empty"; exit 1; fi
          echo "Curl output:"
          jq --color-output < response.json
    
          # Fail if the JSON is exactly {} (ignoring whitespace). Also treat [] as null/empty.
          if jq -e '((type=="object" or type=="array") and length==0)' response.json >/dev/null; then
            echo "File response.json is empty JSON ({} or [])"
            exit 1
          fi
    
          count=$(jq '.results | length' response.json)
          if [[ "$count" -ne 1 ]]; then
            echo "Expected exactly one result, got $count"
            exit 1
          fi
    
          IFS=$'\t'
          tmpfile=$(mktemp)
          jq -r '.results[] |
            [.repo, .path, .name, .size, .actual_md5] |
            @tsv' response.json > "$tmpfile"
          read -r repo path filename size actual_md5 < "$tmpfile"
    
          echo "Repoisitory   : $repo"
          echo "Path          : $path"
          echo "File name     : $filename"
          echo "Size          : $size"
          echo "MD5 hash      : $actual_md5"
    
          location="${base_url}/${repo}/${path}/${filename}"
    
          echo "Token defined? $( [ -n "${BROADCOM_TOKEN:-}" ] && echo yes || echo no )"
          echo "Token length     : ${#BROADCOM_TOKEN}"
    
          echo "Fetching location: $location"
          curl --user "${BROADCOM_USERNAME}:${BROADCOM_TOKEN}" --request GET "$location" --remote-name --remote-header-name --show-error
    
          if [[ ! -f "$filename" ]]; then echo "ERROR: Download file not found."; exit 1; fi
          echo "Current working directory: $(pwd)"
          ls -l $filename
          md5sum="$(md5sum "$filename" | cut -d' ' -f1)"
          echo "DL file hash  : $md5sum"
          if [[ "$md5sum" == "$actual_md5" ]]; then
            echo "File checksum OK."
          else
            echo "ERROR: Checksum mismatch."
            filetype=$(file $filename | cut -d ':' -f 2)
            echo "File type: $filetype"
            if [[ $(echo $filetype | grep 'text' | wc -l ) -gt 0 ]] || [[ $(echo $filetype | grep 'JSON' | wc -l ) -gt 0 ]] || [[ $(echo $filetype | grep 'HTML' | wc -l ) -gt 0 ]]; then
              echo "Text response:"
              cat $filename
          fi
            exit 1
          fi
    
          echo "Extracting TGZ archive."
          tar -xzvf "$filename"
          echo "Contents of $STAGING_DIR:"
          ls -la "$STAGING_DIR"
          echo "Contents of automic-automation"
          ls -la automic-automation
    
        env:
          STAGING_DIR: $(Build.ArtifactStagingDirectory)
          BROADCOM_ARTIFACTORY: ${{ parameters.broadcomArtifactory }}
          BROADCOM_USERNAME: 'MyBroadcomAccount@exmaple.com'
          BROADCOM_TOKEN: '$(registry_token)'
          AAKE_HELM_CHART_VERSION: ${{ parameters.broadcom_helm_chart_version }}
        displayName: 'Download AAKE Helm chart from Broadcom'
    
    # Push Helm chart directly to OCI registry
    - bash: |
        # Make sure there's only one file.
        cd $STAGING_DIR
        mapfile -t files < <(find . -maxdepth 1 -type f -name 'automic-automation-*.tgz' ! -name 'automic-automation-plugin-*.tgz')
        if (( ${#files[@]} != 1 )); then
          echo "Expected exactly one Helm chart .tgz file, found ${#files[@]}: ${files[*]}" >&2
          exit 1
        fi
        # Login securely
        printf '%s' "$(DockerReposPassword)" | helm registry login ${{ parameters.containerRegistryUrl }} --username $(DockerReposUser) --password-stdin
        # Push the file to container registry
        base="$(basename "${files[0]}")"
        helm push "$base" oci://${{ parameters.containerRegistryUrl }}/helm
      env:
        STAGING_DIR: $(Build.ArtifactStagingDirectory)
      displayName: 'Push AAKE Helm Helm chart to container registry'
      condition: succeeded()
    
    # Install Helm if it's not already installed.
    - task: HelmInstaller@1
      inputs:
        helmVersionToInstall: 'latest'
      displayName: Install Helm
    
    # Use 'helm template' to parse the AAKE Helm chart, and generate a list of required Docker images
    - bash: |
        set -euo pipefail
        shopt -s extglob
        cd "$STAGING_DIR"
        # Extract just automic-automation/values.yaml from the automic-automation-*.tgz into the current directory
        tar -xzf automic-automation-*.tgz # --strip-components=1 automic-automation/values.yaml
        cd automic-automation
        ls -l
        # Fill an array with a list of Docker images from the operator-config ConfigMap in the Helm chart.
    
          helm template automic-automation . \
            --set-string environment.AUTOMIC_SYSTEM_NAME=DUMMY \
            --set-string environment.AUTOMIC_GLOBAL_SYSTEM=DUMMY \
            --include-crds \
          | yq -r '
              select(.kind == "ConfigMap" and .metadata.name == "operator-config")
              | .data["image.properties"]
            ' \
          | awk -F= '
              /^images\./ && /\.repository=/ { key=$1; sub(/^images\./,"",key); sub(/\.repository$/,"",key); repo[key]=$2 }
              /^images\./ && /\.tag=/        { key=$1; sub(/^images\./,"",key); sub(/\.tag$/,"",key); tag[key]=$2 }
              END {
                for (k in tag) if (tag[k] != "") {
                  r = (k in repo ? repo[k] : "")
                  if (r != "") print r k ":" tag[k]
                }
              }
            ' \
          | sort -u \
          | grep -Ev 'psql|ae-postgres' | tee images.txt # Exclude unneeded/unavailable images.
        echo "Wrote $(wc -l < images.txt) images to images.txt"
      displayName: 'Generate image list from operator-config'
      condition: succeeded()
    
    # Publish image list
    - publish: images.txt
      artifact: aake-image-list
      displayName: 'Publish image list'
      condition: succeeded()
    
      # Login & pull from Broadcom Artifactory with retry/backoff
      - bash: |
          retry() {
            local max=5 delay=5 n=1
            while true; do
              "$@" && break || {
                if (( n >= max )); then
                  echo "ERROR: failed after $n attempts: $*" >&2
                  return 1
                fi
                echo "WARN: attempt $n/$max failed: $* ; retrying in ${delay}s..." >&2
                sleep "$delay"
                n=$((n+1))
                delay=$((delay*2))
              }
            done
          }
    
          LIST="$STAGING_DIR/images.txt"
          echo "Using image list: $LIST"
          test -f "$LIST" || (echo "Missing $LIST" && exit 1)
    
          echo "Token defined? $( [ -n "${BROADCOM_TOKEN:-}" ] && echo yes || echo no )"
          echo "Token length     : ${#BROADCOM_TOKEN}"
    
          # Docker login
          echo "Running Docker login."
          if ! printf '%s' "$BROADCOM_TOKEN" | docker login "$BROADCOM_REPO" \
                  --username "$BROADCOM_USERNAME" \
                  --password-stdin; then
              echo "Docker login failed." >&2
              exit 1
          fi
    
          set -euo pipefail
          echo "Pulling images."
          while IFS= read -r src; do
            [[ -z "$src" ]] && continue
            echo "Pulling: $src"
            retry docker pull "$src"
          done < "$LIST"
        env:
          STAGING_DIR: $(Build.ArtifactStagingDirectory)
          BROADCOM_REPO: ${{ parameters.broadcomContainerRegistry }}
          BROADCOM_USERNAME: 'MyBroadcomAccount@example.com'
          BROADCOM_TOKEN: '$(registry_token)'
        displayName: 'Docker login & pull (Broadcom Artifactory)'
    
      # Docker login, tag, & push
      - bash: |
          set -euo pipefail
    
          # Docker login
          echo "Logging into container registry as ${USER}."
          docker login ${TARGET_REPO} --username ${USER} --password ${PASSWORD}
          echo "Login successful."
    
          # Docker tag
          echo "Tagging images"
          while IFS= read -r src; do
            echo "Processing: $src"
            name_and_tag="${src##*/}"            # e.g., awi:24.4.3.1
            dst="${TARGET_REPO}/${name_and_tag}"
            docker tag  "$src" "$dst"
          done < $(Build.ArtifactStagingDirectory)/images.txt
    
          # Docker push
          echo "Pushing images to local container registry."
          while IFS= read -r src; do
            echo "Processing: $src"
            name_and_tag="${src##*/}"            # e.g., awi:24.4.3.1
            dst="${TARGET_REPO}/${name_and_tag}"
            docker push "$dst"
          done < $(Build.ArtifactStagingDirectory)/images.txt
        env:
          USER_NAME: myuser
          PASSWORD: $(password)
          TARGET_REPO: ${{ parameters.containerRegistry }}
        displayName: 'Docker login, tag, & push'
        condition: succeeded()