Good day Experts,
While installing Dx OI 21.3.1 HF1 we receive the following error:
ng-acc-configserver-db-deployment is getting "CrashLoopBackOff" status.
describing the pod gives the following :
[root@globenode1 ~]# kubectl describe pod ng-acc-configserver-db-deployment-7d7d5f4fc6-x4q4j
Name: ng-acc-configserver-db-deployment-7d7d5f4fc6-x4q4j
Namespace: dxi
Priority: 0
PriorityClassName: <none>
Node: globenode1.xyz.zxy/192.168.1.40
Start Time: Fri, 27 Jan 2023 23:45:20 +0200
Labels: app=ng-acc-configserver-db
component=acc
dx-acc-cluster=001
dx-component=apm-acc
dx-product=acc
name=ng-acc-configserver-db
pod-template-hash=3838190972
Annotations: openshift.io/scc=dxi-custom
Status: Running
IP: 10.1.0.235
Controlled By: ReplicaSet/ng-acc-configserver-db-deployment-7d7d5f4fc6
Init Containers:
init-fs:
Container ID: docker://1b892da62ac4251421848d1a2b36ca0416f2b86013094b496b8da3f77e1ab83c
Image: docker-registry.default.svc:5000/dxi/busybox:1
Image ID: docker-pullable://docker-registry.default.svc:5000/dxi/busybox@sha256:dca71257cd2e72840a21f0323234bb2e33fea6d949fa0f21c5102146f583486b
Port: <none>
Host Port: <none>
Command:
/bin/sh
-c
set -ex; exec 2>&1;
mkdir -vp /data.all/cs/db /data.all/cs/db-logs ;
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 27 Jan 2023 23:45:26 +0200
Finished: Fri, 27 Jan 2023 23:45:26 +0200
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/data.all from pvol-postgresql-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from dxi-acc-token-qlvvm (ro)
Containers:
ng-acc-configserver-db-container:
Container ID: docker://31a6dd5e26c1934f9c87883fa8d5ba04d8e829cf89b23d0c63fc5f396c134031
Image: docker-registry.default.svc:5000/dxi/acc-postgresql:21.7.0.25
Image ID: docker-pullable://docker-registry.default.svc:5000/dxi/acc-postgresql@sha256:431b21919089c6ebf98c6260053c2e4c51d8d409972b5976b0c2794dacf1857e
Port: 5432/TCP
Host Port: 0/TCP
Args:
-c
pg_stat_statements.max=5000
-c
pg_stat_statements.track=all
-c
max_connections=2000
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 27 Jan 2023 23:52:54 +0200
Finished: Fri, 27 Jan 2023 23:53:56 +0200
Ready: False
Restart Count: 6
Limits:
cpu: 4
memory: 2Gi
Requests:
cpu: 2
memory: 1Gi
Liveness: exec [/bin/sh -i -c pg_isready -h 127.0.0.1 -U $POSTGRES_USER -d $POSTGRES_DB -p 5432] delay=30s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/bin/sh -i -c psql -h 127.0.0.1 -U $POSTGRES_USER -q -d $POSTGRES_DB -c 'SELECT 1'] delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
POSTGRES_USER: <set to the key 'username' in secret 'ng-acc-configserver-db-secret'> Optional: false
POSTGRES_PASSWORD: <set to the key 'password' in secret 'ng-acc-configserver-db-secret'> Optional: false
POSTGRES_LIQUIBASE_USER: <set to the key 'liquibase-username' in secret 'ng-acc-configserver-db-secret'> Optional: false
POSTGRES_LIQUIBASE_PASSWORD: <set to the key 'liquibase-password' in secret 'ng-acc-configserver-db-secret'> Optional: false
POSTGRES_DB: <set to the key 'database-name' in secret 'ng-acc-configserver-db-secret'> Optional: false
Mounts:
/var/lib/postgresql/data from pvol-postgresql-data (rw)
/var/lib/postgresql/logs from pvol-postgresql-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from dxi-acc-token-qlvvm (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
pvol-postgresql-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: dxi
ReadOnly: false
dxi-acc-token-qlvvm:
Type: Secret (a volume populated by a Secret)
SecretName: dxi-acc-token-qlvvm
Optional: false
QoS Class: Burstable
Node-Selectors: node-role.kubernetes.io/compute=true
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9m default-scheduler Successfully assigned dxi/ng-acc-configserver-db-deployment-7d7d5f4fc6-x4q4j to globenode1.xyz.zxy
Normal Pulled 9m kubelet, globenode1.xyz.zxy Container image "docker-registry.default.svc:5000/dxi/busybox:1" already present on machine
Normal Created 9m kubelet, globenode1.xyz.zxy Created container
Normal Started 9m kubelet, globenode1.xyz.zxy Started container
Normal Pulled 8m (x2 over 9m) kubelet, globenode1.xyz.zxy Container image "docker-registry.default.svc:5000/dxi/acc-postgresql:21.7.0.25" already present on machine
Normal Killing 8m kubelet, globenode1.xyz.zxy Killing container with id docker://ng-acc-configserver-db-container:Container failed liveness probe.. Container will be killed and recreated.
Normal Created 8m (x2 over 9m) kubelet, globenode1.xyz.zxy Created container
Normal Started 8m (x2 over 9m) kubelet, globenode1.xyz.zxy Started container
Warning Unhealthy 7m (x5 over 8m) kubelet, globenode1.xyz.zxy Liveness probe failed: 127.0.0.1:5432 - rejecting connections
sh: cannot set terminal process group (-1): Inappropriate ioctl for device
sh: no job control in this shell
Warning Unhealthy 4m (x25 over 9m) kubelet, globenode1.xyz.zxy Readiness probe failed: sh: cannot set terminal process group (-1): Inappropriate ioctl for device
sh: no job control in this shell
psql: error: FATAL: the database system is in recovery mode
Thanks for your prompt support and help !
------------------------------
Kind Regards,
Ahmed
------------------------------