Create a test Step that can help us Publish or Subscribe Kafka Queue/Topic using DevTest.
I have Kafka solution for DevTest and implemented in our project. Let me know if you need any help.
Thanks Kevin. Is there any workarounds??
Sorry, no Apache Kafka solution has been built into DevTest yet.
Hi, I'm looking for connecting & reading Kafka topics. Is there any solution available for the same?
Never thought about throughput but I will get back to you on this.
I see. We call that "payload-based" correlation, and we support it with existing messaging protocols (JMS, IBM MQ, RabbitMQ). However, none of the existing messaging framework is really built to handle the kind of throughput that Kafka is designed for. Just out of curiosity, do you have some idea of the throughput goal for your Kafka-based RPC application, in transactions per second?
hi Kevin, Sorry for the confusion.I meant an attribute in response like "item number" which has unique value.
What do you mean by 'attribute value'? There is an 'attributes' field deep in the Kafka wire protocol but it's mostly internal use and can't contain an arbitrary ID.
Are you taking about a 'key'? Kafka messages have a 'value' plus an optional 'key'; are you using the key for an ID?
I agree with you kevin, but it is project requirement. There is no message or correlation ID as such. We testing it manually by using the attribute value which is unique.
I'm a little baffled as to why you are doing this. Everything I know and have read about Kafka suggests it's not appropriate for RPC-style request/response interactions. What are you using for correlation, if anything? Does your Kafka-based RPC application support more than one client at a time?
DevTest currently does not have built-in support for sending or receiving Kafka traffic. If we were to build such support then the test step(s) probably would not look like the messaging Send Receive steps, and there wouldn't be a VSE component as I'm still unclear on how that would actually work.
I am looking it for functional testing and if possible virtualization as well. Requirement is just to send a json request message to the kafka producer queue and consume the response message from the consumer queue.
I have tried using command prompt step but since consumer keeps on running and need manual intervention to kill the process. I wasn't able to proceed.
Any other way to implement the same ?
We have an idea logged already for MQTT. I don't know about UDP, would you like to log an idea for that one as well? From what I understand, Zigbee and Z-Wave are physical layer/data link layer protocols, which would be like trying to directly test/simulate Ethernet or Bluetooth.
We have to take each of these protocols individually. Some are appropriate for DevTest and some probably aren't.
Like MQTT, UDP, Zigbee, Z-Wave, etc, I see Kafka support as integral to a complete testing solution for IoT. DevTest Workstation is already the most complete API testing solution for protocols used by corporates - the addition of IoT protocol support would put it in the forefront as a complete API testing solution for devices as well.
I'm interested to hear about your possible use cases for testing with Apache Kafka. Functional testing? Load testing? What kind of validation? What kinds of payloads? Do you use keys in addition to payloads?
Kafka is not a "normal" messaging platform, like JMS, IBM MQ, RabbitMQ, or even MQTT. It's more like a remote logging/event platform, with a focus on pure throughput rather than P2P, RPC, or anything that typically requires meta-data on an individual message level. It's lightweight to the point where it cannot support most of the scenarios we typically see with other messaging platforms.
We are using Apache Kafka 0.8.2.1.
If you're interested in this idea then can you also mention which version of Apache Kafka you are using? There are significant API differences between v0.8.x/earlier and v0.9.0.