Service Virtualization

Expand all | Collapse all

HTTP recording with Zlib compression

Jump to Best Answer
  • 1.  HTTP recording with Zlib compression

    Posted 08-30-2017 04:17 AM

    Hi,

    We are recording HTTP/soap traffic with payload is compressed with standard zlib.

    In the header, content-encoding is set to: gzip, deflate

     

    The recorder does not deflate the payload and keep it as binary, so no parsing of Soap arguments !

     Any idea on setting/configuration to instruct DevTest to defalte the payload ? 

     

    Thanks.

    Philippe.



  • 2.  Re: HTTP recording with Zlib compression
    Best Answer

    Posted 09-01-2017 05:03 AM

    Hi,

     

    I checked the request which "Content-Encooding" is "gzip, deflate" by using the DevTest10.1.0.

    I did not check the recorder level, but the playing back level I found these:
    When the "Content-Encooding" is gzip, then the virtual service can extract gzip and can get the argumrnts from the extracted SOAP request.
    However, when the "Content-Encooding" is deflate, the request data is not modified from the virtual service and treated as the binary data.

    To extract the deflated request, you need to define the Scriptable Data Protocol filter(for "Process Request") in the "Virtual HTTPS Listner" step and write script to handle the deflated request before the "Web Service(SOAP)" Data Protocol filter.
    Here is the sample script which I tested in my environment(DevTest 10.1.0).

    ---------------------- Script Start ----------------------------------
    %beanshell%
    import java.io.BufferedReader;
    import java.io.ByteArrayOutputStream;
    import java.io.IOException;
    import java.io.InputStream;
    import java.io.InputStreamReader;
    import java.util.zip.GZIPOutputStream;
    import java.util.zip.Deflater;
    import java.util.zip.Inflater;

    if (lisa_vse_request.isBinary()) {
        String out;
        Inflater decompresser = new Inflater();
        byte[] b = lisa_vse_request.getBodyBytes();
        InputStream bis = new ByteArrayInputStream(b);
        byte[] inpBuf = new byte[1024];
        byte[] outBuf = new byte[1024];
        int rd;
        rd = 1;
        do{
            rd = bis.read(inpBuf);
            if (rd > 0) {
                decompresser.setInput(inpBuf, 0, rd);
            }
            while (!decompresser.finished()) {
                int siz = decompresser.inflate(outBuf);
                if (siz > 0) {
                    out = new String(outBuf, 0, siz, "UTF-8");
                } else {
                     break;
                }
             }
        } while (rd > 0);
        bis.close();
        decompresser.end();

        lisa_vse_request.setBinary(false);
        lisa_vse_request.setBodyText(out);
    }
    ---------------------- Script End   ----------------------------------
    If the "Content-Encooding" is gzip, the value of lisa_vse_request.isBinary() is "false", so this script is for only deflated contents.
    I tested this script only by using the DevTest 10.1.0, but I hope this script will help you.

    Additionally, even if I used the DevTest 10.1.0, but the deflated contents still have problems. So I suggest to create an idea to this community as a feature request.

     

    Cheers,



  • 3.  Re: HTTP recording with Zlib compression

    Posted 09-15-2017 10:40 AM

    Hi,

     

    Tested your script and getting exception when executing this line of the script:   int siz = decompresser.inflate(outBuf);

    Target exception: java.util.zip.DataFormatException: incorrect header check
    in inline evaluation of: `` import java.io.BufferedReader; import java.io.ByteArrayOutputStream; import . . . '' at line number 25
    at bsh.BshScriptEngine.evalSource(BshScriptEngine.java:97)
    at bsh.BshScriptEngine.eval(BshScriptEngine.java:61)

    .....

    Wondering why using "inflater" when we try to deflate the request with Zlib ?

     

    Cheers



  • 4.  Re: HTTP recording with Zlib compression

    Posted 09-18-2017 09:19 PM

    Hi,

     

    The sample script which I wrote does following steps:
    1. Get the information that the request body is binary or not and if "not binary" then do nothing
    2. If the reqeust body is binary(deflated), then inflate the body information and get the text data.
    3. After getting the inflated(text) data then set the request body test and set the body text as "not binary"
    Could you please attach the full of error message after this comment?
    And if it is possible, could you please attach the VSM and VSI file that you tried to change?

     

    Cheers,



  • 5.  Re: HTTP recording with Zlib compression

    Posted 09-19-2017 05:13 AM

    Hi,

    I think I could reproduce your problem.
    I got the same type of error to send the gzipped content but the "Content-encoding" for the request is only "deflate" in my DevTest 10.1 environment.
    I did not get any problem when I sent the request with "Content-encoding" is "gzip, deflate". Maybe this will be a workaround.
    I will investigate more in DevTest 10.1 base, because this version is the newst version currently.
    If it is possible, I suggest to use Devtest 10.1 for your testing.


    Cheers,
    Yusuke



  • 6.  Re: HTTP recording with Zlib compression

    Posted 09-19-2017 07:51 AM
      |   view attached

    Hi

    We don't have 10.1 installed, but will do it in a sandbox.

    I tried the same with "Content-encoding" as "gzip, deflate"  but got the error:

    Caused by: java.util.zip.ZipException: Not in GZIP format
    at java.util.zip.GZIPInputStream.readHeader(Unknown Source)
    at java.util.zip.GZIPInputStream.<init>(Unknown Source)
    at java.util.zip.GZIPInputStream.<init>(Unknown Source)
    ... 17 more

    I run my tests from a recorded raw traffic file (only requests are compressed), so you can give a try.

     

    Cheers

    Philippe.

    Attachment(s)

    zip
    STG_FS_1.xml.zip   13K 1 version


  • 7.  Re: HTTP recording with Zlib compression

    Posted 09-21-2017 04:23 AM

    Hi,

    Thank you for attaching the raw traffic file.
    Based on the raw traffic file, the request information includes the "deflated" and "base64 encoded" information by using the URL:
    https://developers.onelogin.com/saml/online-tools/code-decode/base64-decode-inflate
    In this URL, I can extract the SOAP information.

     

    The original sample which I pasted in this question, I only wrote to "inflate" the data.
    Then you need to add the feature to do "base64 decode" the data in this sample.
    But it is very hard to determine what kind of "base64 decode" feature can be used for your request data.
    I checked several online "base64 decoder"s and found this URL can do the "base64 decode" your request information:
    https://convertstring.com/en/EncodeDecode/Base64Decode
    but other URLs like:
    https://developers.onelogin.com/saml/online-tools/code-decode/base64
    cannot do the "base64 decode" by using your request information.

     

    And I tried to do "base64 decode" in the script level twice with your request information but the results were:
    1. use org.apache.commons.codec.binary.Base64 but I could not get decoded data(same as before decoding)
    2. use java.util.Base64 but I got error message at the decoding feature and could not continue
    But I think if you use the "base64 decoder" which is the same kind of "base64 encorder" in your side, then you can extract the SOAP information like the URL:
    https://developers.onelogin.com/saml/online-tools/code-decode/base64-decode-inflate

    Cheers,
    Yusuke



  • 8.  Re: HTTP recording with Zlib compression

    Posted 09-12-2017 10:59 AM

    Hi Tsuji

     

    Thanks for the clarification on DevTest behavior with "deflate".

    I will test your script on 9.1 and let know.

     

    Cheers