Hi Rogerio. I have not seen this before with the upgrade and deleting the volumes and installed cleanly should work but am wondering if you dropped external DB tables (assuming not using PostgreSQL) as well if not concerned about saving any data. As you likely know, Druid is all about the analytics so something went wrong and would have been good to get to the bottom of it before deleting the volumes.
I don't think the two problems are related but one thing people often forget to do after an upgrade is to update the portal integration package on gateway cluster and restart it.
Regards, Alex.
------------------------------
Solution Architect - Security & Integration
Symantec Enterprise Division, Broadcom Inc.
Vancouver, B.C. Canada
------------------------------
Original Message:
Sent: 03-04-2021 02:49 PM
From: Rogerio Santos
Subject: API Portal 4.5 - Kafka errors after upgrade
Hello guys, how are you?
recently, in a lab, I upgraded a server to version 4.5. Looking at the logs with the journalctl -fu docker command I noticed that a message is logged several times:
portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC java.lang.IllegalStateException: Could not create adminportal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC at org.springframework.kafka.core.KafkaAdmin.initialize(KafkaAdmin.java:161)portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC at com.ca.apim.druid.migration.config.TopicsHandler.lambda$new$0(TopicsHandler.java:33)portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC at java.lang.Thread.run(Thread.java:748)portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC Caused by: org.apache.kafka.common.KafkaException: Failed create new KafkaAdminClientportal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:370)portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:62)portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC at org.springframework.kafka.core.KafkaAdmin.initialize(KafkaAdmin.java:157)portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC ... 2 common frames omittedportal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.serversportal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:66)portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC at org.apache.kafka.clients.admin.KafkaAdminClient.<init>(KafkaAdminClient.java:409)portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:363)portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:38 UTC ... 4 common frames omittedportal_dispatcher.qd1mmnxm7e5asmxw1zt3qrh3o.fvy2ss9vf0xa28xba7o66x0u6 2021-03-04 19:09:40 UTC 127.0.0.1 - - [04/Mar/2021:19:09:40 +0000] "GET /nginx_status HTTP/1.1" 200 100 "-" "curl/7.61.1"portal_dispatcher.qd1mmnxm7e5asmxw1zt3qrh3o.fvy2ss9vf0xa28xba7o66x0u6 2021-03-04 19:09:40 UTC 2021/03/04 19:09:40 [info] 36#36: *119 client 127.0.0.1 closed keepalive connectionportal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:43 UTC [QueryServer.log] [2021-03-04 19:09:43,139] [INFO ] [o.a.k.c.a.AdminClientConfig] [Thread-12] [|] [|||] AdminClientConfig values: \n bootstrap.servers = [kafka:9092]\n client.id = \n connections.max.idle.ms = 300000\n metadata.max.age.ms = 300000\n metric.reporters = []\n metrics.num.samples = 2\n metrics.recording.level = INFO\n metrics.sample.window.ms = 30000\n receive.buffer.bytes = 65536\n reconnect.backoff.max.ms = 5000\n reconnect.backoff.ms = 100\n request.timeout.ms = 120000\n retries = 5\n retry.backoff.ms = 100\n sasl.client.callback.handler.class = null\n sasl.jaas.config = null\n sasl.kerberos.kinit.cmd = /usr/bin/kinit\n sasl.kerberos.min.time.before.relogin = 60000\n sasl.kerberos.service.name = null\n sasl.kerberos.ticket.renew.jitter = 0.05\n sasl.kerberos.ticket.renew.window.factor = 0.8\n sasl.login.callback.handler.class = null\n sasl.login.class = null\n sasl.login.refresh.buffer.seconds = 300\n sasl.login.refresh.min.period.seconds = 60\n sasl.login.refresh.window.factor = 0.8\n sasl.login.refresh.window.jitter = 0.05\n sasl.mechanism = GSSAPI\n security.protocol = PLAINTEXT\n send.buffer.bytes = 131072\n ssl.cipher.suites = null\n ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]\n ssl.endpoint.identification.algorithm = https\n ssl.key.password = null\n ssl.keymanager.algorithm = SunX509\n ssl.keystore.location = null\n ssl.keystore.password = null\n ssl.keystore.type = JKS\n ssl.protocol = TLS\n ssl.provider = null\n ssl.secure.random.implementation = null\n ssl.trustmanager.algorithm = PKIX\n ssl.truststore.location = null\n ssl.truststore.password = null\n ssl.truststore.type = JKS\n portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:43 UTC [QueryServer.log] [2021-03-04 19:09:43,140] [WARN ] [o.a.k.c.ClientUtils] [Thread-12] [|] [|||] Removing server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka portal_analytics-server.qd1mmnxm7e5asmxw1zt3qrh3o.p2rxvzi33elpfbojqk30oagdz 2021-03-04 19:09:43 UTC [system-portal.log] [2021-03-04 19:09:43,141] [WARN ] [c.c.a.d.m.c.TopicsHandler] [Thread-12] [|] [|||] Failure while declaring topics on Kafka. Retrying in 5sec...
even reinstalling, deleting volumes and reconnecting the legacy database, the problem persists, which leads me to believe that this is a database problem. I also realized that I am no longer able to change the policy templates for APIs already created. Do you think these problems can be related?
------------------------------
Roger - Sec4you
------------------------------