Layer7 API Management

 View Only
  • 1.  Connection to Azure Event Hub via Kafka Producer

    Posted Dec 07, 2023 08:15 AM

    It seems I can't find the good documentation. I have a Azure Event Hub and with the IDE I can produce events. From the gateway I don't know how to configure the Kafka Client. In below picture I have tried to show what the configuration. Is there any documentation on this, or something I can try?

    The error in the logging is:
    Connection to node -1 (*****.servicebus.windows.net/<ip>:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.



    ------------------------------
    Peter van de Riet
    The Netherlands
    ------------------------------


  • 2.  RE: Connection to Azure Event Hub via Kafka Producer

    Broadcom Employee
    Posted Dec 07, 2023 10:31 AM

    Hello Peter,

    Here is our documentation for configuring Kafka clients on the Layer7 API Gateway. However, it doesn't include examples for Azure Event Hub.

    In fact, while Azure Event Hub supports Kafka protocol connections, it's not exactly Kafka and I expect your mileage might vary.

    However, based on the error you're getting and the screenshot you've provided, I suspect the problem might be the Username you specified. It looks like you're trying to use a context variable value, but with the wrong syntax. $ConnectionString should be ${ConnectionString}.



    ------------------------------
    Ben Urbanski
    Product Manager, API Gateway
    Layer7 API Management
    ------------------------------



  • 3.  RE: Connection to Azure Event Hub via Kafka Producer

    Broadcom Employee
    Posted Dec 08, 2023 01:07 AM

    Kafka client configurations are global and static in nature. As of now, configuring them using context variables is unsupported. Please revise the configuration with a specific user name.




  • 4.  RE: Connection to Azure Event Hub via Kafka Producer

    Posted Dec 08, 2023 07:58 AM

    Thanks for the answers, the screenshot was one of many tries. From my local machine in intellij it works with the $connection, that was a final try. The best I can think of is with username "SAS-Layer7" and password the value from primary key (ending in "SwE=") but that doesn't work either. Not with PLAIN nor SCRAM-SHA-256.
    This is the connection string (redacted)
    Endpoint=sb://******.servicebus.windows.net/;SharedAccessKeyName=SAS-Layer7;SharedAccessKey=******SwE=

    I asked the network team to check if maybe the traffic is blocked in some way.



    ------------------------------
    Peter van de Riet
    The Netherlands
    ------------------------------



  • 5.  RE: Connection to Azure Event Hub via Kafka Producer

    Broadcom Employee
    Posted Dec 08, 2023 09:17 AM

    That looks more correct when using Kafka's PLAIN mechanism for SAS per this documentation, https://learn.microsoft.com/en-us/azure/event-hubs/azure-event-hubs-kafka-overview#shared-access-signature-sas.

    You'll definitely want to verify network connectivity between your gateway and the endpoint.

    Also, is Azure providing any useful log information for the failure?

    In any case, you can open a support case, and we'll try to help you troubleshoot this further. However, we probably will not have access to Azure Event Hub ourselves.



    ------------------------------
    Ben Urbanski
    Product Manager, API Gateway
    Layer7 API Management
    ------------------------------



  • 6.  RE: Connection to Azure Event Hub via Kafka Producer

    Posted Dec 14, 2023 04:30 AM
    Edited by Peter van de Riet Dec 18, 2023 02:02 AM
      |   view attached

    Update with working configuration:

    First error was a missing certificate in the trust-store. No logging was pointing in that direction.

    After fixing the certificate and a lot of trial and error we have a working solution. Maybe not all is needed, but this is working:

    Changes needed:
    • enable TLS (with random private key, but that is a mandatory choice)
    • 2 properties in the producer-tab
      • producer.security.protocol=SASL_SSL
      • producer.sasl.mechanism=PLAIN
    • Username is $ConnectionString (that is the literal value, not a variable with missing {braces} )
    • Password is the Connection-String from Azure (starting Endpoint=sb:...) with appended: ;EntityPath=Layer7-topic (this will be automatic appended when creating the SAS configuration for one topic)
    • The topic must exist before sending messages



    ------------------------------
    Peter van de Riet
    The Netherlands
    ------------------------------



  • 7.  RE: Connection to Azure Event Hub via Kafka Producer

    Broadcom Employee
    Posted Dec 21, 2023 11:42 AM

    Thanks for the update, Peter. I'm glad to see that you got it working. I'll create a defect record for better error logging for the missing certificate.



    ------------------------------
    Ben Urbanski
    Product Manager, API Gateway
    Layer7 API Management
    ------------------------------