Installing DX-APM - dxiportal error 404

[Osvan Widyowibowo]

Hi Everyone,

We have completed DX-APM installation and all the pods are in running state. But when opening http://apmservices-gateway.domain.com/dxiportal, the page returned http error code 404 Not Found.
I am not sure whether it is related, when verifying network services with kubectl get svc, the apm-em-xxx and apm-wv-xxx are missing.

[root@master ~]# kubectl get svc -ndxi
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
apmservices-cloudgw ClusterIP 10.101.19.207 <none> 5003/TCP 2d4h
apmservices-gateway ClusterIP 10.110.163.5 <none> 8004/TCP 2d4h
apmservices-kafka ClusterIP None <none> 9092/TCP 2d4h
axaservices-adminui ClusterIP 10.98.114.10 <none> 8080/TCP 2d4h
axaservices-amq ClusterIP 10.111.168.49 <none> 61616/TCP,8161/TCP 2d4h
axaservices-axa-indexer ClusterIP 10.111.153.232 <none> 8080/TCP 2d4h
......

Verifying kubectl get ingress --all-namespaces returned "No resources found".

Also the DX install log shows missing URL information after installation completed as below.

2021-06-23 05:39:17,343 INFO - [dxi-platform-init] [ OK ] Check log in x.x.x.x:/nfs/ca/dxi/dxi-platform-init-pod/dxi-platform-init-pod.log before login to DXI Manager
2021-06-23 05:39:17,344 TRACE - Fetching k8s ingress apmservices-gateway.
2021-06-23 05:39:17,370 INFO - Installation succeeded
2021-06-23 05:39:17,370 INFO - To create your first tenant, open this URL: /dxiportal. Log in using following credentials: "masteradmin" tenant and "masteradmin" user. Use the password that you entered as a masteradmin password during the installation

We deploy DX-APM behind corporate internet proxy.

Thanks,
Osvan W.

------------------------------
Solution Architect
Feiran Solutions Sdn Bhd
------------------------------

[Haruhiko Davis]

The output you provided shows no ingress endpoints were created.  Can you verify that with 'kubectl get ing --all-namespaces'?  If that's the case, something went wrong when the installer tried to apply the related yaml file.  You may be able to see some helpful errors by running 'kubectl get events --all-namespaces'.  You may also try to manually re-apply the manifest that creates them and see if you have any helpful errors. 

RE: INGRESS MANIFEST FILES
I'm afraid I don't have a working install to reference at the moment, but if I recall correctly, you can find all the manifest files in <installer>/files/meta.import/*.  Grep for "kind:.*ingress", this should give you a list of yaml files that create ingress entries in your k8s cluster.  You can test with any of these, but look for one named core-gateway.route.insecure.yml (or core-gateway.route.secure.yml if you chose the Secure Routes install option)... this is where the apmservices.<domain>/dxiportal ingress endpoint comes from.  Run kubectrl -n <dxi or your namespace> apply -f gateway.route.insecure.yml and take note of any errors.  You can also look inside this file to be sure that it has been properly updated for your installation (this update is performed by the installer). 

NEXT STEP:  REINSTALL
Once you address the issues with your ingress controller, I'd recommend you run a cleanup and re-install vs attempting to fix the current installation.

------------------------------
Solution Engineering
Broadcom
------------------------------

[Jörg Mertin]

If no ingress at all show up, then there will be no routing.

A "kubectl get ingress -A" should show you all existing ingress's - as Haruhiko already said, if there is no ingress, no routes are sent to the existing services. Also, an ingress will only be setup if the service exists! and the service will only be valid if the target endpoint (app usually) exists.

So - check the events for the entire cluster with:

kubectl get events -n dxi

That should give you an indication of what the ingress manager is waiting for.

------------------------------
Customer Solution Engineering
Broadcom Inc.
------------------------------

[Christopher Judd]

Hi Osvan,

The output you provided shows no ingress endpoints were created.  Can you verify that with 'kubectl get ing --all-namespaces'?  If that's the case, something went wrong when the installer tried to apply the related yaml file.  You may be able to see some helpful errors by running 'kubectl get events --all-namespaces'.  You may also try to manually re-apply the manifest that creates them and see if you have any helpful errors. 

RE: INGRESS MANIFEST FILES
I'm afraid I don't have a working install to reference at the moment, but if I recall correctly, you can find all the manifest files in <installer>/files/meta.import/*.  Grep for "kind:.*ingress", this should give you a list of yaml files that create ingress entries in your k8s cluster.  You can test with any of these, but look for one named core-gateway.route.insecure.yml (or core-gateway.route.secure.yml if you chose the Secure Routes install option)... this is where the apmservices.<domain>/dxiportal ingress endpoint comes from.  Run kubectrl -n <dxi or your namespace> apply -f gateway.route.insecure.yml and take note of any errors.  You can also look inside this file to be sure that it has been properly updated for your installation (this update is performed by the installer). 

NEXT STEP:  REINSTALL
Once you address the issues with your ingress controller, I'd recommend you run a cleanup and re-install vs attempting to fix the current installation.

[Osvan Widyowibowo]

Hi Everyone,

Thank you for your great workaround. Running the kubectl apply did show me something about the webhook validate service error as follows:

Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Error from server (InternalError): error when creating "core-gateway.route.insecure.yml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1beta1/ingresses?timeout=10s": Service Unavailable

Looking at the wireshark packet trace, the above http request returned http error Service Unavailable due to the corporate proxy's dns failed to resolve the name (X-Squid-Error: ERR_DNS_FAIL). 

I tried removing the ValidatingWebhookConfiguration and it fixed the problem. The ingress for the applied core-gateway.route can be created.

[root@master meta.import]# kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission
validatingwebhookconfiguration.admissionregistration.k8s.io "ingress-nginx-admission" deleted
[root@master meta.import]# kubectl -ndxi apply -f core-gateway.route.insecure.yml
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions/apmservices-gateway created
ingress.extensions/apmservices-cloudgw created

Then did the clean up and reinstall DX-APM as you suggested after deleting the Validating Webhook first. It may not be the good solution but the DX-APM seems to work well.

Thanks and regards,

Osvan Widyowibowo


------------------------------
Solution Architect
Feiran Solutions Sdn Bhd
------------------------------

[Jörg Mertin]

That about sounds right. If either side cannot resolve the FQDN target, it will fail anyway.

Good find :)