In AAKE v26, the install operator periodically polls the AE server using the the system/processes REST API endpoint, and fetches details about running AE server processes. Is uses these data to update pod labels for things like process name, type, role(s), and number of connections.
However, these labels are not perfect.
- There is no label for pod state.
- There are no labels for the B01, B10, and B60 utilization statistics.
- There is no label displaying the aggregated roles of an AE server process.
- The labels are completely unavailable in v24.
I have been working on a solution that addresses all of these limitations.
Here is a sneak preview of what the new labels look like in K9s for one of our v24 systems.
And here is what the script looks like in action.
$ ./ae_pod_labeler.py
Namespace: uc4
AE process info provider returned 94 AE processes
Recovered identity of AE_EXP#CP020 from CPsrv_log_020_00.txt: hostname='jcp-rest-0-8466d7cd74-cx6ln', ip='11.32.2.209'
Recovered identity of AE_EXP#CP019 from CPsrv_log_019_00.txt: hostname='jcp-rest-0-5bd4f8db8f-ff98l', ip='11.32.2.24'
Recovered identity of AE_EXP#CP018 from CPsrv_log_018_00.txt: hostname='jcp-rest-0-5bd4f8db8f-nk4r6', ip='11.32.2.54'
Recovered identity of AE_EXP#CP017 from CPsrv_log_017_00.txt: hostname='jcp-ws-0-b5c587866-qfnjs', ip='11.32.2.199'
Kubernetes returned 90 AE pods
PROCESS TYPE ROLES HOSTNAME AE IP POD IP CONN B01 B10 B60 STATE RECONCILIATION
---------------------------------------------------------------------------------------------------------------------------------------------------------
CP001 JCP - jcp-ws-0-7cbbfc5cf5-sdvfp 11.32.2.167 11.32.2.167 0 0 0 0 active MATCHED
CP002 JCP - jcp-ws-0-7cbbfc5cf5-n5bqx 11.32.2.176 11.32.2.176 3 0 0 0 active MATCHED
CP003 JCP - jcp-ws-0-7cbbfc5cf5-q29q9 11.32.2.185 11.32.2.185 0 0 0 0 active MATCHED
CP004 JCP - jcp-ws-0-7cbbfc5cf5-fb8qb 11.32.2.171 11.32.2.171 2 0 0 0 active MATCHED
CP005 JCP - jcp-ws-0-7cbbfc5cf5-5xrrd 11.32.2.184 11.32.2.184 5 0 0 0 active MATCHED
CP006 JCP - jcp-ws-0-7cbbfc5cf5-s27w6 11.32.2.179 11.32.2.179 2 0 0 0 active MATCHED
CP007 JCP - jcp-ws-0-7cbbfc5cf5-4bpc9 11.32.2.178 11.32.2.178 0 0 0 0 active MATCHED
CP008 JCP - jcp-ws-0-7cbbfc5cf5-944nn 11.32.2.183 11.32.2.183 2 0 0 0 active MATCHED
CP009 JCP - jcp-ws-0-7cbbfc5cf5-kdcpn 11.32.2.177 11.32.2.177 4 0 0 0 active MATCHED
CP010 JCP - jcp-ws-0-7cbbfc5cf5-dwvmc 11.32.2.170 11.32.2.170 5 0 0 0 active MATCHED
CP011 REST - jcp-rest-0-68f7794467-7rt8h 11.32.2.188 11.32.2.188 0 0 0 0 active MATCHED
CP012 REST - jcp-rest-0-68f7794467-l9wxn 11.32.2.187 11.32.2.187 0 0 0 0 active MATCHED
CP013 REST - jcp-rest-0-68f7794467-274rh 11.32.2.189 11.32.2.189 0 0 0 0 active MATCHED
CP014 REST - jcp-rest-0-68f7794467-cgbw7 11.32.2.186 11.32.2.186 0 0 0 0 active MATCHED
CP015 CP - cp-0-57f87ddbff-wbds7 11.32.2.4 11.32.2.4 1 0 0 0 active MATCHED
CP016 CP - cp-0-57f87ddbff-jtvcm 11.32.2.5 11.32.2.5 0 0 0 0 active MATCHED
CP017 CP - jcp-ws-0-b5c587866-qfnjs 11.32.2.199 - 0 0 0 0 unknown ORPHANED_AE
CP018 CP - jcp-rest-0-5bd4f8db8f-nk4r6 11.32.2.54 - 0 0 0 0 unknown ORPHANED_AE
CP019 CP - jcp-rest-0-5bd4f8db8f-ff98l 11.32.2.24 - 0 0 0 0 unknown ORPHANED_AE
CP020 CP - jcp-rest-0-8466d7cd74-cx6ln 11.32.2.209 - 0 0 0 0 unknown ORPHANED_AE
WP001 JWP PER jwp-0-7dc885cb9f-nhblb 11.32.2.172 11.32.2.172 0 0 0 0 active MATCHED
WP002 JWP UTL jwp-0-7dc885cb9f-wfk65 11.32.2.165 11.32.2.165 0 0 0 0 active MATCHED
WP003 JWP - jwp-0-7dc885cb9f-5mn8t 11.32.2.169 11.32.2.169 0 0 0 0 active MATCHED
WP004 JWP - jwp-0-7dc885cb9f-xhpm5 11.32.2.174 11.32.2.174 0 0 0 0 active MATCHED
WP005 JWP - jwp-0-7dc885cb9f-2fn7s 11.32.2.175 11.32.2.175 0 0 0 0 active MATCHED
WP006 JWP - jwp-0-7dc885cb9f-qflt9 11.32.2.181 11.32.2.181 0 0 0 0 active MATCHED
WP007 JWP - jwp-0-7dc885cb9f-qj7cl 11.32.2.168 11.32.2.168 0 0 0 0 active MATCHED
WP008 JWP - jwp-0-7dc885cb9f-j847d 11.32.2.173 11.32.2.173 0 0 0 0 active MATCHED
WP009 JWP AUT jwp-0-7dc885cb9f-s27wf 11.32.2.180 11.32.2.180 0 0 0 0 active MATCHED
WP010 JWP IDX jwp-0-7dc885cb9f-pfkcf 11.32.2.182 11.32.2.182 0 0 0 0 active MATCHED
WP011 PWP - wp-0-5689bb5969-mkmbd 11.32.2.104 11.32.2.104 0 7 5 5 active MATCHED
WP012 WP R wp-0-5689bb5969-n48c9 11.32.2.245 11.32.2.245 0 1 1 1 active MATCHED
WP013 WP O wp-0-5689bb5969-7hk8l 11.32.2.248 11.32.2.248 0 1 1 1 active MATCHED
WP014 DWP - wp-0-5689bb5969-mbh9r 11.32.2.198 11.32.2.198 0 1 1 1 active MATCHED
WP015 DWP - wp-0-5689bb5969-w5clj 11.32.2.254 11.32.2.254 0 1 1 1 active MATCHED
WP016 DWP - wp-0-5689bb5969-42hfj 11.32.2.251 11.32.2.251 0 1 1 1 active MATCHED
WP017 DWP - wp-0-5689bb5969-28vnb 11.32.2.206 11.32.2.206 0 1 1 1 active MATCHED
WP018 DWP - wp-0-5689bb5969-wkx8n 11.32.2.209 11.32.2.209 0 1 1 1 active MATCHED
WP019 DWP - wp-0-5689bb5969-9zbt2 11.32.2.239 11.32.2.239 0 1 1 1 active MATCHED
WP020 DWP - wp-0-5689bb5969-ng7v6 11.32.2.242 11.32.2.242 0 1 1 1 active MATCHED
WP021 DWP - wp-0-5689bb5969-tq7m5 11.32.2.216 11.32.2.216 0 0 0 0 active MATCHED
WP022 DWP - wp-0-5689bb5969-7qx7g 11.32.2.200 11.32.2.200 0 1 1 1 active MATCHED
WP023 DWP - wp-0-5689bb5969-m7kzh 11.32.2.201 11.32.2.201 0 0 0 0 active MATCHED
WP024 DWP - wp-0-5689bb5969-4fxv8 11.32.2.230 11.32.2.230 0 1 1 1 active MATCHED
WP025 DWP - wp-0-5689bb5969-s8spp 11.32.2.236 11.32.2.236 0 1 1 1 active MATCHED
WP026 DWP - wp-0-5689bb5969-7zz8d 11.32.2.194 11.32.2.194 0 0 0 0 active MATCHED
WP027 DWP - wp-0-5689bb5969-tsvkd 11.32.2.197 11.32.2.197 0 0 0 0 active MATCHED
WP028 DWP - wp-0-5689bb5969-gsvp6 11.32.2.193 11.32.2.193 0 0 0 0 active MATCHED
WP029 WP - wp-0-5689bb5969-qsljh 11.32.2.202 11.32.2.202 0 10 1 0 active MATCHED
WP030 WP - wp-0-5689bb5969-8mr5x 11.32.2.211 11.32.2.211 0 0 0 0 active MATCHED
WP031 WP - wp-0-5689bb5969-fz72f 11.32.2.213 11.32.2.213 0 0 0 0 active MATCHED
WP032 WP - wp-0-5689bb5969-rjjj4 11.32.2.244 11.32.2.244 0 0 0 0 active MATCHED
WP033 WP - wp-0-5689bb5969-bthk5 11.32.2.208 11.32.2.208 0 0 0 0 active MATCHED
WP034 WP - wp-0-5689bb5969-lhk6q 11.32.2.2 11.32.2.2 0 0 0 0 active MATCHED
WP035 WP - wp-0-5689bb5969-xzjw6 11.32.2.220 11.32.2.220 0 0 0 0 active MATCHED
WP036 WP - wp-0-5689bb5969-xjnfq 11.32.2.238 11.32.2.238 0 0 0 0 active MATCHED
WP037 WP - wp-0-5689bb5969-fvpl7 11.32.2.214 11.32.2.214 0 0 0 0 active MATCHED
WP038 WP - wp-0-5689bb5969-629q4 11.32.2.210 11.32.2.210 0 0 0 0 active MATCHED
WP039 WP - wp-0-5689bb5969-q7zpq 11.32.2.250 11.32.2.250 0 0 0 0 active MATCHED
WP040 WP - wp-0-5689bb5969-4fm2n 11.32.2.196 11.32.2.196 0 0 0 0 active MATCHED
WP041 WP - wp-0-5689bb5969-csvdp 11.32.2.249 11.32.2.249 0 0 0 0 active MATCHED
WP042 WP - wp-0-5689bb5969-5gxjz 11.32.2.253 11.32.2.253 0 0 0 0 active MATCHED
WP043 WP - wp-0-5689bb5969-rpfvr 11.32.2.247 11.32.2.247 0 0 0 0 active MATCHED
WP044 WP - wp-0-5689bb5969-5x8q6 11.32.2.223 11.32.2.223 0 0 0 0 active MATCHED
WP045 WP - wp-0-5689bb5969-69tzz 11.32.2.246 11.32.2.246 0 0 0 0 active MATCHED
WP046 WP - wp-0-5689bb5969-n75zk 11.32.2.204 11.32.2.204 0 0 0 0 active MATCHED
WP047 WP - wp-0-5689bb5969-6wvcn 11.32.2.240 11.32.2.240 0 0 0 0 active MATCHED
WP048 WP - wp-0-5689bb5969-l6kxj 11.32.2.212 11.32.2.212 0 1 1 1 active MATCHED
WP049 WP - wp-0-5689bb5969-hxj9z 11.32.2.207 11.32.2.207 0 0 0 0 active MATCHED
WP050 WP - wp-0-5689bb5969-w4lbt 11.32.2.221 11.32.2.221 0 1 1 1 active MATCHED
WP051 WP - wp-0-5689bb5969-4hzct 11.32.2.195 11.32.2.195 0 0 0 0 active MATCHED
WP052 WP - wp-0-5689bb5969-jfnp2 11.32.2.219 11.32.2.219 0 0 0 0 active MATCHED
WP053 WP - wp-0-5689bb5969-b6bs5 11.32.2.243 11.32.2.243 0 0 0 0 active MATCHED
WP054 WP - wp-0-5689bb5969-82zsh 11.32.2.225 11.32.2.225 0 0 0 0 active MATCHED
WP055 WP - wp-0-5689bb5969-jqkz6 11.32.2.199 11.32.2.199 0 0 0 0 active MATCHED
WP056 WP - wp-0-5689bb5969-mx7lc 11.32.2.228 11.32.2.228 0 0 0 0 active MATCHED
WP057 WP - wp-0-5689bb5969-hztnb 11.32.2.226 11.32.2.226 0 0 0 0 active MATCHED
WP058 WP - wp-0-5689bb5969-njqr6 11.32.2.218 11.32.2.218 0 1 1 1 active MATCHED
WP059 WP - wp-0-5689bb5969-7hcrm 11.32.2.224 11.32.2.224 0 1 1 1 active MATCHED
WP060 WP - wp-0-5689bb5969-wr8j2 11.32.2.227 11.32.2.227 0 1 1 1 active MATCHED
WP061 WP - wp-0-5689bb5969-9g4q2 11.32.2.217 11.32.2.217 0 0 0 0 active MATCHED
WP062 WP - wp-0-5689bb5969-szjcj 11.32.2.215 11.32.2.215 0 1 1 1 active MATCHED
WP063 WP - wp-0-5689bb5969-hlx2c 11.32.2.205 11.32.2.205 0 0 0 0 active MATCHED
WP064 WP - wp-0-5689bb5969-fh57k 11.32.2.237 11.32.2.237 0 0 0 0 active MATCHED
WP065 WP - wp-0-5689bb5969-cfv2g 11.32.2.231 11.32.2.231 0 0 0 0 active MATCHED
WP066 WP - wp-0-5689bb5969-4hwsx 11.32.2.241 11.32.2.241 0 0 0 0 active MATCHED
WP067 WP - wp-0-5689bb5969-n54qr 11.32.2.229 11.32.2.229 0 0 0 0 active MATCHED
WP068 WP - wp-0-5689bb5969-5sxms 11.32.2.234 11.32.2.234 0 0 0 0 active MATCHED
WP069 WP - wp-0-5689bb5969-4p6cp 11.32.2.235 11.32.2.235 0 0 0 0 active MATCHED
WP070 WP - wp-0-5689bb5969-6vssf 11.32.2.233 11.32.2.233 0 1 1 1 active MATCHED
WP071 WP - wp-0-5689bb5969-577w9 11.32.2.252 11.32.2.252 0 0 0 0 active MATCHED
WP072 WP - wp-0-5689bb5969-7pqgz 11.32.2.222 11.32.2.222 0 0 0 0 active MATCHED
WP073 WP - wp-0-5689bb5969-4fwbf 11.32.2.232 11.32.2.232 0 1 0 0 active MATCHED
WP074 WP - wp-0-5689bb5969-825z6 11.32.2.203 11.32.2.203 0 1 1 1 active MATCHED
======================================================================
RECONCILIATION SUMMARY
======================================================================
AE server processes : 94
AE pods in cluster : 90
Pods to patch : 36
Pods already correct : 54
Pods not patchable : 4
Label changes (# of pods to patch):
b01 : 36
Anomalies:
CP017 ORPHANED_AE: Inactive AE process has no matching Kubernetes pod; hostname='jcp-ws-0-b5c587866-qfnjs'
CP018 ORPHANED_AE: Inactive AE process has no matching Kubernetes pod; hostname='jcp-rest-0-5bd4f8db8f-nk4r6'
CP019 ORPHANED_AE: Inactive AE process has no matching Kubernetes pod; hostname='jcp-rest-0-5bd4f8db8f-ff98l'
CP020 ORPHANED_AE: Inactive AE process has no matching Kubernetes pod; hostname='jcp-rest-0-8466d7cd74-cx6ln'
36 pod(s) will be modified.
4 result(s) will not be modified.
Apply these changes? [y/N]: n
No pods were patched.
The new script is much faster and more reliable than the previous solution, uc4_functions_k8s.sh, that relied almost entirely on parsing AE log files. This new script calls a small Java program to fetch AE server process information via the UC4 Java API (uc4.jar) — specifically, using the ServerList class.
The new script also incorporates the technique I previously devised to identify hung AE server processes.
I plan to develop this tool further to accomplish twp primary aims:
- Keep pod labels up-to-date.
- Identify discrepancies between the AE server and the Kubernetes cluster — in particular, non-responsive AE server processes.
I hope to have something to share soon.