Layer 7 API Management

Expand all | Collapse all

Why does the use of Message Streaming assertion still utilize disk/memory space?

  • 1.  Why does the use of Message Streaming assertion still utilize disk/memory space?

    Posted 07-26-2017 05:02 PM
      |   view attached

    We are trying to allow large response payloads for a REST service. After looking at Configure Message Streaming Assertion and Streaming large files through API Gateway the service was configured to allow unlimited response size with streaming enabled (no buffering). In testing, we discovered, large responses still buffered and written to disk at <SSG>/node/default/var/attachments/. In simple tests, the file-system quickly filled and caused "java.io.IOException: No space left on device" to be thrown, the connection to be terminated, and the ssg to send 500s. My assumption was that streaming would send the data to the client as it receives it without buffering on heap or disk. Is that assumption incorrect or is the policy/gateway configured improperly?

    Attachment(s)

    zip
    streamingPolicy.xml.zip   640B 1 version


  • 2.  Re: Why does the use of Message Streaming assertion still utilize disk/memory space?

    Posted 08-02-2017 03:33 PM

    The Configure Message Streaming is used prior to the Routing assertion to affect inbound requests and by default if no assertion or audit acts against the payload (not the header) then it will pass back being streamed. We have seen global policies such as message-completed affect the streaming capability.

     

    Sincerely,

     

    Stephen Hughes

    Director, CA Support



  • 3.  Re: Why does the use of Message Streaming assertion still utilize disk/memory space?

    Posted 08-02-2017 04:18 PM

    In this case we are wanting to stream the response, consequently, we have placed the Configure Message Streaming assertion after the route with a target of Response. We have one global message complete related to logging we can turn off and see if that changes anything.



  • 4.  Re: Why does the use of Message Streaming assertion still utilize disk/memory space?

    Posted 08-02-2017 04:36 PM

    If you find that removing the global policy does not correct the problem, please share the version you are running.

     

    Sincerely,

     

    Stephen Hughes

    Director, CA Support



  • 5.  Re: Why does the use of Message Streaming assertion still utilize disk/memory space?

    Posted 08-07-2017 10:28 AM

    So there are a couple of things here. It seems any message-completed global policy that references the response message will cause this behavior. Even if we add a conditional block to skip the logic in the global policy for the given service it still fails to stream properly. Is there anything we can do here to have both?

     

    Additionally, any amount of Log Sink with a category of "Traffic Log" will cause the same non-streaming behavior. 



  • 6.  Re: Why does the use of Message Streaming assertion still utilize disk/memory space?

    Posted 08-07-2017 07:30 PM

    Yes, robbiej ,

    Similar thing we can see for the gzip response -- any ref to response(eg. ${response.mainpart}), any operation on response(eg. "Response must match xpath"),  will make gateway to store the response, uncompress it, parse it...

    So, the streaming will not work, or the gzip response cannot go through without uncompress.

     

    The "Audit message in policy" assertion with "Always" save response option can cause the same problem.

    message-completed global policy itself won't cause the problem, only if there is any assertion ref to response.

    "Traffic Log" is the same, if the traffic log doesn't save response, it won't cause problem. If you want to log the response for most of other policies, there should be a work around -- if Cluster wide property trafficlogger.selective=true, traffic is logged only when the ${trafficlogger.select} context variable is defined in a policy and its value is "true". ie. you can disable traffic log for those policies using streaming.

     

    Regards,

    Mark



  • 7.  Re: Why does the use of Message Streaming assertion still utilize disk/memory space?

    Posted 08-13-2017 12:38 PM

    Rob,

     

    Did Mark's additional information assist in your configuration?

     

    Sincerely,

     

    Stephen Hughes

    Director, CA Support