Idea Details

Test Step to interact with Apache kafka

Last activity 06-03-2019 08:24 PM
monika_mehta's profile image
07-18-2016 04:51 PM

Create a test Step that can help us Publish or Subscribe Kafka Queue/Topic using DevTest.


Comments

12-19-2017 01:12 PM

I have Kafka solution for DevTest and implemented in our project. Let me know if you need any help.

07-06-2017 12:31 PM

Thanks Kevin. Is there any workarounds??

07-06-2017 12:14 PM

Sorry, no Apache Kafka solution has been built into DevTest yet.

07-06-2017 10:57 AM

Hi, I'm looking for connecting & reading Kafka topics. Is there any solution available for the same?

03-01-2017 02:54 PM

Never thought about throughput but I will get back to you on this. 

Thanks.

03-01-2017 12:53 PM

I see.  We call that "payload-based" correlation, and we support it with existing messaging protocols (JMS, IBM MQ, RabbitMQ).  However, none of the existing messaging framework is really built to handle the kind of throughput that Kafka is designed for.  Just out of curiosity, do you have some idea of the throughput goal for your Kafka-based RPC application, in transactions per second?

02-28-2017 10:50 PM

hi Kevin, Sorry for the confusion.I meant an attribute in response like "item number" which has unique value.

02-28-2017 06:02 PM

What do you mean by 'attribute value'?  There is an 'attributes' field deep in the Kafka wire protocol but it's mostly internal use and can't contain an arbitrary ID. 

 

Are you taking about a 'key'?  Kafka messages have a 'value' plus an optional 'key'; are you using the key for an ID?

02-28-2017 05:48 PM

I agree with you kevin, but it is project requirement. There is no message or correlation ID as such. We testing it manually by using the attribute value which is unique. 

02-28-2017 12:29 PM

I'm a little baffled as to why you are doing this.  Everything I know and have read about Kafka suggests it's not appropriate for RPC-style request/response interactions.  What are you using for correlation, if anything?  Does your Kafka-based RPC application support more than one client at a time?

 

DevTest currently does not have built-in support for sending or receiving Kafka traffic.  If we were to build such support then the test step(s) probably would not look like the messaging Send Receive steps, and there wouldn't be a VSE component as I'm still unclear on how that would actually work.

02-28-2017 12:03 PM

I am looking it for functional testing and if possible virtualization as well. Requirement is just to send a json request message to the kafka producer queue and consume the response message from the consumer queue.

I have tried using command prompt step but since consumer keeps on running and need manual intervention to kill the process. I wasn't able to proceed. 

 

Any other way to implement the same ?

08-23-2016 12:16 PM

We have an idea logged already for MQTT.  I don't know about UDP, would you like to log an idea for that one as well?  From what I understand, Zigbee and Z-Wave are physical layer/data link layer protocols, which would be like trying to directly test/simulate Ethernet or Bluetooth.

 

We have to take each of these protocols individually.  Some are appropriate for DevTest and some probably aren't.

08-23-2016 11:45 AM

Like MQTT, UDP, Zigbee, Z-Wave, etc, I see Kafka support as integral to a complete testing solution for IoT. DevTest Workstation is already the most complete API testing solution for protocols used by corporates - the addition of IoT protocol support would put it in the forefront as a complete API testing solution for devices as well.

08-23-2016 10:58 AM

I'm interested to hear about your possible use cases for testing with Apache Kafka.  Functional testing?  Load testing?  What kind of validation?  What kinds of payloads?  Do you use keys in addition to payloads?

 

Kafka is not a "normal" messaging platform, like JMS, IBM MQ, RabbitMQ, or even MQTT.  It's more like a remote logging/event platform, with a focus on pure throughput rather than P2P, RPC, or anything that typically requires meta-data on an individual message level.  It's lightweight to the point where it cannot support most of the scenarios we typically see with other messaging platforms.

07-28-2016 02:58 PM

Hi Kevin,

 

We are using Apache Kafka 0.8.2.1.

 

Thanks.

07-27-2016 01:06 PM

If you're interested in this idea then can you also mention which version of Apache Kafka you are using?  There are significant API differences between v0.8.x/earlier and v0.9.0.