Layer7 API Management

Expand all | Collapse all

API Portal 4.5 - Kafka errors after upgrade

  • 1.  API Portal 4.5 - Kafka errors after upgrade

    Posted 03-04-2021 02:49 PM
    Hello guys, how are you?

    recently, in a lab, I upgraded a server to version 4.5. Looking at the logs with the journalctl -fu docker command I noticed that a message is logged several times:

    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC java.lang.IllegalStateException: Could not create admin
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC 	at org.springframework.kafka.core.KafkaAdmin.initialize(KafkaAdmin.java:161)
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC 	at com.ca.apim.druid.migration.config.TopicsHandler.lambda$new$0(TopicsHandler.java:33)
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC 	at java.lang.Thread.run(Thread.java:748)
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC Caused by: org.apache.kafka.common.KafkaException: Failed create new KafkaAdminClient
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC 	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:370)
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC 	at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:62)
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC 	at org.springframework.kafka.core.KafkaAdmin.initialize(KafkaAdmin.java:157)
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC 	... 2 common frames omitted
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC 	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:66)
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC 	at org.apache.kafka.clients.admin.KafkaAdminClient.<init>(KafkaAdminClient.java:409)
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC 	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:363)
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC 	... 4 common frames omitted
    portal_dispatcher.qd1mmnxm7e5asmxw1zt3qrh3o.fvy2ss9vf0xa28xba7o66x0u6 2021-03-04 19:09:40 UTC 127.0.0.1 - - [04/Mar/2021:19:09:40 +0000] "GET /nginx_status HTTP/1.1" 200 100 "-" "curl/7.61.1"
    portal_dispatcher.qd1mmnxm7e5asmxw1zt3qrh3o.fvy2ss9vf0xa28xba7o66x0u6 2021-03-04 19:09:40 UTC 2021/03/04 19:09:40 [info] 36#36: *119 client 127.0.0.1 closed keepalive connection
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:43 UTC [QueryServer.log] [2021-03-04 19:09:43,139] [INFO ] [o.a.k.c.a.AdminClientConfig] [Thread-12] [|] [|||] AdminClientConfig values: \n	bootstrap.servers = [kafka:9092]\n	client.id = \n	connections.max.idle.ms = 300000\n	metadata.max.age.ms = 300000\n	metric.reporters = []\n	metrics.num.samples = 2\n	metrics.recording.level = INFO\n	metrics.sample.window.ms = 30000\n	receive.buffer.bytes = 65536\n	reconnect.backoff.max.ms = 5000\n	reconnect.backoff.ms = 100\n	request.timeout.ms = 120000\n	retries = 5\n	retry.backoff.ms = 100\n	sasl.client.callback.handler.class = null\n	sasl.jaas.config = null\n	sasl.kerberos.kinit.cmd = /usr/bin/kinit\n	sasl.kerberos.min.time.before.relogin = 60000\n	sasl.kerberos.service.name = null\n	sasl.kerberos.ticket.renew.jitter = 0.05\n	sasl.kerberos.ticket.renew.window.factor = 0.8\n	sasl.login.callback.handler.class = null\n	sasl.login.class = null\n	sasl.login.refresh.buffer.seconds = 300\n	sasl.login.refresh.min.period.seconds = 60\n	sasl.login.refresh.window.factor = 0.8\n	sasl.login.refresh.window.jitter = 0.05\n	sasl.mechanism = GSSAPI\n	security.protocol = PLAINTEXT\n	send.buffer.bytes = 131072\n	ssl.cipher.suites = null\n	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]\n	ssl.endpoint.identification.algorithm = https\n	ssl.key.password = null\n	ssl.keymanager.algorithm = SunX509\n	ssl.keystore.location = null\n	ssl.keystore.password = null\n	ssl.keystore.type = JKS\n	ssl.protocol = TLS\n	ssl.provider = null\n	ssl.secure.random.implementation = null\n	ssl.trustmanager.algorithm = PKIX\n	ssl.truststore.location = null\n	ssl.truststore.password = null\n	ssl.truststore.type = JKS\n 
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:43 UTC [QueryServer.log] [2021-03-04 19:09:43,140] [WARN ] [o.a.k.c.ClientUtils] [Thread-12] [|] [|||] Removing server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka 
    portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:43 UTC [system-portal.log] [2021-03-04 19:09:43,141] [WARN ] [c.c.a.d.m.c.TopicsHandler] [Thread-12] [|] [|||] Failure while declaring topics on Kafka. Retrying in 5sec... 
    ​

    even reinstalling, deleting volumes and reconnecting the legacy database, the problem persists, which leads me to believe that this is a database problem. I also realized that I am no longer able to change the policy templates for APIs already created. Do you think these problems can be related?




    ------------------------------
    Roger - Sec4you

    ------------------------------


  • 2.  RE: API Portal 4.5 - Kafka errors after upgrade

    Broadcom Employee
    Posted 03-05-2021 06:45 PM
    Hi Rogerio.  I have not seen this before with the upgrade and deleting the volumes and installed cleanly should work but am wondering if you dropped external DB tables (assuming not using PostgreSQL)  as well if not concerned about saving any data.  As you likely know, Druid is all about the analytics so something went wrong and would have been good to get to the bottom of it before deleting the volumes.

    I don't think the two problems are related but one thing people often forget to do after an upgrade is to update the portal integration package on gateway cluster and restart it.

    Regards, Alex.

    ------------------------------
    Solution Architect - Security & Integration
    Symantec Enterprise Division, Broadcom Inc.
    Vancouver, B.C. Canada
    ------------------------------