Service Virtualization

Expand all | Collapse all

Scriptable Data Protocol Regex

  • 1.  Scriptable Data Protocol Regex

    Posted Nov 29, 2018 02:06 PM
      |   view attached

    Hi everybody i´m working with SV for System MQ and my requests and responses is just a character string with all the information and no delimiters. I'm using the "Scriptable Data Protocol" for DPH but i don´t know if is posible to regular expresions here and how to doit. Please if any one have made this before and have any for explain i´ll be gratefull.

     

    regards,

     

    Julio C. Ramos

    Attachment(s)

    zip
    mq_textmsg.xml.zip   2 KB 1 version


  • 2.  Re: Scriptable Data Protocol Regex

    Posted Nov 29, 2018 02:20 PM

    Hi Julio,

     

    Can you provide an example of your request payload with that single character line string and also share some details on how the regular expression match should work. For example, provide a sample regex for a transaction. And will there be a different regex pattern for each transaction? 



  • 3.  Re: Scriptable Data Protocol Regex

    Posted Nov 29, 2018 02:49 PM

    Hi William and thanks for your interest in my question.


    Sure, here the example and the code that i'm working right now but using substring function but this is not enough so i need to work with RegEx:

     

    This is the line:

    MSGCS104008JulioRamos20181116V27745215J356

     

    The Regex for first three values:

     

    ^\D+(?=\d) match "MSGCS"

    10([0-9]{4}) match "104008"

    and (?<=10([0-9]{4}))\D+ match JulioRamos

     

    My actual code in the DPH is:

     

    %beanshell%

    import com.itko.util.ParameterList;
    import com.itko.util.Parameter;
    import java.text;

    String theBody = lisa_vse_request.getBodyText();
    String bodyValue = theBody.substring(0, 6);

    ParameterList args = lisa_vse_request.getArguments();
    args.addParameters("key1=&key2=val2");
    String theVal1 = args.getParameterValue("val1");
    args.setParameterValue("key1", theVal1.substring(5, 11));
    args.setParameterValue("key2", theVal1.substring(0, 5));

     

    Regards,

     

    Julio C. Ramos



  • 4.  Re: Scriptable Data Protocol Regex

    Broadcom Employee
    Posted Nov 29, 2018 02:40 PM

    It's likely that your messages are constructed and decoded by copybooks. You should ask internally for the request and response copybooks associated with these messages, as they will describe how many characters are for each of your fields, what loops exist in the messages, what format the data fields can take, etc. DevTest has support for copybooks, including the generation of translation tables.

     

    If there really are no copybooks, the character string will have some other external description mechanism, which might be listed in spreadsheets or documents. If this is the case, a scriptable DPH can be used to convert these messages. I've created scriptable DPHs in the past for these use cases, but it's more user-friendly to investigate the copybook route first.



  • 5.  Re: Scriptable Data Protocol Regex

    Posted Nov 29, 2018 02:57 PM

    Hi Rick, i have requested for some file description or copybook but there is nothing about it, so i´m trying to use the DPH for scripting.

     

    Regards,

     

    Julio C. Ramos



  • 6.  Re: Scriptable Data Protocol Regex

    Broadcom Employee
    Posted Nov 29, 2018 03:23 PM

    Hi Julio --

     

    I suggest you don't use regex. My suggestion is that each of the elements is likely to be a fixed length. There will be elements that are part of a header, and elements that are part of the body. I do it like this for the request scriptable DPH:

    ///////////////////

    import com.itko.util.Parameter;
    String message = lisa_vse_request.getBodyText();
    var args = lisa_vse_request.getArguments();

    //the first 51 characters are the header. Everything else is the body

    var header = message.substring(0,51);
    var body = message.substring(51);

    testExec.setStateValue("flRequestHeader",header);
    testExec.setStateValue("flRequestBody",body);

     

    private getElementValues(String section, String[] elementNames, int[] elementLength) {
       int startPos = 0;
       int endPos = startPos;
       // testExec.setStateValue("SectionLength",section.length());
       for(int i=0;i<elementNames.length;i++) {
          endPos = startPos + elementLength[i];

          // testExec.setStateValue("Counter",i);
          // testExec.setStateValue("StartPos",startPos);
          // testExec.setStateValue("ElementLength",elementLength[i]);
          // testExec.setStateValue("EndPos",endPos);

          String element = section.substring(startPos, endPos);
          testExec.setStateValue("fl_"+elementNames[i],element);
          startPos = endPos;

          args.addParameter(new Parameter(elementNames[i],element));
       }
       return 0;
    }

     

    //
    //Header
    //

    // This array contains the list of element names, and we will use those names as arguments 
    String[] elementNames = {
    "APP_DATA_LEN",
    "EXT_HDR_LEN",
    "TRAN_CODE",
    "SOURCE",
    "DEST",
    "ERROR"
    };

    // This array contains the length of each of the element values
    int[] elementLength = {8,4,9,10,10,10};
    getElementValues(header, elementNames, elementLength);

    // This is what we will call the transaction
    lisa_vse_request.setOperation(testExec.getStateValue("fl_TRAN_CODE"));

     

    //

    //Body

    //

    // This array contains the list of element names, and we will use those names as arguments 

    String[] elementNames = {

    //All names deleted as they might contain client data
    };

    // This array contains the length of each of the element values
    int[] elementLength = {

    //All lengths deleted as they might contain client data
    };
    getElementValues(body, elementNames, elementLength);

    ///////////////////

    There needs to be both a recording response scriptable DPH and a replay response DPH.

     

    Recording:

    ///////////////////

    import com.itko.util.Parameter;
    var message = new String(lisa_vse_response.getBodyBytes());

     

    var header = message.substring(0,51);
    var body = message.substring(51);

    testExec.setStateValue("flRequestHeader",header);
    testExec.setStateValue("flRequestBody",body);

     

    private getElementValues(String tagname, String section, String[] elementNames, int[] elementLength) {
       int startPos = 0;
       int endPos = startPos;
       String myBody = "<"+tagname+">";
       // testExec.setStateValue("SectionLength",section.length());
       for(int i=0;i<elementNames.length;i++) {
          endPos = startPos + elementLength[i];

          // testExec.setStateValue("Counter",i);
          // testExec.setStateValue("StartPos",startPos);
          // testExec.setStateValue("ElementLength",elementLength[i]);
          // testExec.setStateValue("EndPos",endPos);

          String element = section.substring(startPos, endPos);
          testExec.setStateValue("fl_"+elementNames[i],element);
          startPos = endPos;

          myBody = myBody+"<"+elementNames[i]+">"+element+"</"+elementNames[i]+">";
       }
       myBody = myBody+"</"+tagname+">";
       return myBody;
    }

     

    //
    //Header
    //
    String[] elementNames = {
    "APP_DATA_LEN",
    "EXT_HDR_LEN",
    "TRAN_CODE",
    "SOURCE",
    "DEST",
    "ERROR"
    };
    int[] elementLength = {8,4,9,10,10,10};
    String xmlMessage = getElementValues("header", header, elementNames, elementLength);

     

    //

    //Body

    //

    String[] elementNames = {

    //Removed names
    };
    int[] elementLength = {

    //Removed lengths
    };

     

    xmlMessage = xmlMessage + getElementValues("body", body, elementNames, elementLength);
    lisa_vse_response.setBodyText(xmlMessage);
    return xmlMessage;

    ///////////////////

     

    Replay:

    ///////////////////

    String message = testExec.getStateString("lisa.vse.response","DEFAULT");

    int finalpos = message.lastIndexOf("<");
    int start = 0;
    int startpos = 0;
    int endpos = 0;
    String myString = "";
    do {
       startpos = message.indexOf(">",start);
       endpos = message.indexOf("<",startpos);
       myString = myString + message.substring(startpos+1,endpos);
       start = endpos;
       testExec.setStateValue("string",myString);
    } while(endpos < finalpos);

    lisa_vse_response.setBodyText(myString);
    return myString;

    ///////////////////

     

    I haven't syntax-checked these DPHs since I removed all PII data, so you will need to validate and modify the routines as necessary.

     

    Also, copy/paste sometimes runs double-quotes, so you want to check those.

     

    Whitespace at the start of lines is also ruined by copy/paste, but I think I've manually indented the important bits.

     

    Rick



  • 7.  Re: Scriptable Data Protocol Regex

    Broadcom Employee
    Posted Nov 30, 2018 06:06 AM

    Julio,

     

    I would find out what is the origin of the payload, the transport protocol is system MQ, I think it is most likely that these are COBOL copybooks. Even if they can't find the copybooks, it should be possible to assess the type of the payload.

     

    If it is so, I would reverse engineer the copybooks as you find out the structure of the payloads. There weren't much rockets around when COBOL was invented, hence the copybook format is not rocket science. If there are big chunks in the payload you don't care about you can ignore them by using FILLER directive.

     

    Those copybooks descriptions must be somewhere, at some point they will come out and you will be in much better shape to expand your solution as new transactions need to be added.

     

    Just my 2 cents,

    Danny



  • 8.  Re: Scriptable Data Protocol Regex

    Posted Dec 03, 2018 10:26 AM

    Hi Danny,

     

    The messages request come from an integrated process with this format, i have like answer that there is not a copybook associated.

     

    The format appear to be fixed length for each value but there is some lengths we are not sure and the regex seems to resolve this ones.

     

    Regards,

     

    Julio C. Ramos



  • 9.  Re: Scriptable Data Protocol Regex

    Posted Dec 03, 2018 10:09 AM

    Hi Rick, thanks for your comment.


    i have a doubt, Why do you suggest don't use regex, there is any particular to use just script code?

    I think that i need is more simple, is just split the request string into values that i need to handle even with magic strings like values for responses.

    I was checking the code in your comment but there is some details i don´t yet understend for the solution i need.

     

    Regards,

     

    Julio C. Ramos



  • 10.  Re: Scriptable Data Protocol Regex

    Broadcom Employee
    Posted Dec 03, 2018 10:52 AM

    Hi Julio --

     

    Your messages look to be comprised of fixed-length fields (which is what copybooks split), and my routines specifically do that splitting in code, as simply as possible, to make arguments and operationName from the request. What they are actually doing is making a little function (loop) to do the same thing as all of your manual substrings.

     

    A regex is able to match a specific number of characters, but it can do much more, and has its own syntax, so it has a much greater CPU overhead than a simple split. My routines use split, and you tell it (in elementNames) what to call each argument and (in elementLength) how many characters are in each argument.

     

    Every instance I've seen of copybook-style messages has both a header and a body, and I've seen some with a footer, so my routines follow that pattern to be understandable, reusable and simpler than trying to understand regex syntax & logic.

     

    You have an extra wrinkle in that your dates are standard mainframe date format, which is 8 digits. We don't support that date format for time-shifting (magic dates) because 8 digits could mean anything, so you want to add a function to convert your date arguments into better date formats (for example, from 20181116 to 2018/11/16) if you want to ensure your messages don't go stale or need manual updating.

     

    For the message snippet you posted, and using your suggested argument names, the bits to change for my routine to generate a proper request message for DevTest to store would be:

       elementNames = {"key2", "key1", "name", "date", "key3"};

       elementLengths = {5, 6, 10, 8, 13};

       getElementValues(theBody, elementNames, elementLength);

       lisa_vse_request.setOperation(testExec.getStateValue("fl_key2"));

     

    Rick



  • 11.  Re: Scriptable Data Protocol Regex

    Posted Dec 03, 2018 10:34 AM

    Hi, i have attached the raw traffic to help understand the handle that i need for requests.

     

    Regards, 

     

    Julio C. Ramos