When British science fiction writer, Arthur C. Clarke called out his third law of prediction - "Any sufficiently advanced technology is indistinguishable from magic.", he likely had just experienced our team present the invention called Opaque Data Processing. (Well, maybe not since he passed away in 2008, but let's go with this for now.)
Opaque Data Processing (ODP) seems like magic because it's doing a ton of work in the background, unexposed to the user. Next when you start pulling out statements like "byte-level alignment", "genome sequencing algorithm" and "it just knows" even the brightest engineers are confused and therefore skeptical of it's accuracy. Since we released ODP in CA Service Virtualization 8.0, there has been a ton of interest, but it's rare when I do see someone actually try it out.
So, let's break down the magic a bit and then maybe you'll be confident that I'm not making this stuff up!
1. Record a bunch of transactions: We need something to match against so record a bunch of transactions from your existing system that you can't figure out because they're "OPAQUE"... i.e. binary data, not human readable, really difficult to figure out... etc. The more data you record, the higher the accuracy of ODP. Even better, if you know that there are transactions that do similar things (i.e. get user, delete user, etc.) record at least 2 of each so that ODP can find the similarities.
2. Find Your Match: In order to find an appropriate response to a request that comes into a virtual service, matching is key. We match the request coming in (incoming request) with an existing request that you recorded when creating the virtual service to find a proper response. Simple. Right? Well then, how do we match when we don’t know what we’re reading?
3. It's All About Those Bytes: ODP uses an algorithm to match the bytes in an incoming request against the bytes in an existing request to find the most appropriate match. If it finds something similar, it will highlight the differences and fill them in with blanks (it adjusts the matching alignment). Even better, Entropy weighting steps in and determines if and where there is a lack of order in the message. Where disorder is detected in the message, in is weighted lower in importance for the matching algorithm. In sum - we like data that looks similar in multiple messages (like an "operation") because if we find this "lack of entropy" we can find a quicker and more accurate match. ODP spots the difference and then says, "that's not as important as the stuff that is more similar".
4. Insert More Magic: The algorithm detects items that are similar in the request and the response and makes them variables (we like to call those 'magic strings'). For example, in the simple request and response below, the name “HENRY” is repeated in the request and in the response that is matched in the library. So, ODP takes the name “HENRY” makes it a variable, selects a similar string in the incoming request (“JAMES”) and makes that variable.
Request: | GET User/ JAMES |
Matching Request in Library: | GET User/ HENRY |
Matching Response in Library: | {"User":{"HENRY":{"ID":"12345"}}} |
Response: | {"User":{"JAMES":{"ID":"12345"}}} |
Essentially, it says:
If “HENRY” comes in, “HENRY” goes out.
If “JAMES” comes in, “JAMES” goes out.
If “NAME” comes in, “NAME” goes out.
5. Did you just respond to the unknown? Yes, that’s what I did. An unknown request with an unknown protocol, came into my ODP recorded virtual service, and I responded to that request at a high accuracy (measured between 99.6% and 100% accuracy - not too shabby).
Here's a quick diagram to show you how the processing and matching occurs using ODP:
Now that I’ve described the magic of matching, entropy weighting, byte-level alignment and the real magic of magic strings, I’ll share with you these two videos that will help both describe what ODP does and then give you a demo of the recording and matching.
Next Steps? I want to hear from you.
Have you tested out ODP? Do you have an unknown protocol you'd like us to test out? What are your questions? Do you see this as helpful for any projects you're working on today?
Let me know in the comments below.
Reference Blogs:
ODP makes moot the question, “Do you support that protocol?”
The 12 days of 8.0 - Day 4