There are error counts shown in the devtest portal, after which the service goes offline and this happens very intermittently.
I had a look at the vse.log, portal.log, <VSE_NAME>.log but no luck. I could not even get single trace of information of what caused the particular virtual service to go down.
If environment error: The step in Http/s Listen is set to 'Abort the test'
Please advise on how to find the root cause of this issue and the fix for it
Exception in Detail:
Does your Listen step have a filter?
This error indicates that there is a step with a filter.
The filter is the 'JSON Path Filter' and it requires some parameters to be filled in.
Executing the test in ITR will allow you to see the values that are used for the filter.
Thank you Marcy.
The problem is, since the issue is very intermittent, we are unable to see what type of request, is causing the above failure to test it in local ITR
I have even added a filter in Http/s Listen step to log out the incoming request to a file, but even that request is not logged.
You mention in the question title, error: "no date no incoming request found". Where are you seeing this error? Are you seeing this in the transaction view of the portal?
I am thinking that the JSON Path Filter error is a red herring, it is there because there is no valid request object.
It is probably the "If environment error" that makes the virtual service increment the error count and eventually go down (after 4 errors)
You can try setting the "if environment error" from "Abort the test" to "End the test". Most likely the count will stop going up and the virtual service will continue running.
I suggest that as a first step, if it has an effect then later we can take it further if you really want to find out the nature of this erroneous incoming request.
Thank you Danny, as suggested I will change "if environment error" from "Abort the test" to "End the test" and give it a try.
I also still eager to find out the root cause of it and fix it permanently.
To add on: We are using JSON DPH on DevTest10.3. Since the issue is intermittent we are not sure to confirm if upgrading to JSON 2.0 DPH will solve the issue
I guess what we would have expected to see either somewhere in your vse.log or in the inspection view or in ... is a java exception message, for example something as follows:
"exception is java.lang.IllegalArgumentException: Incoming request is not HTTP."
For example the above is often thrown when a IT Management vulnerability scan tool hits the virtual service. It does a call on the port but it isn't an HTTP call.
Because when you have some exception like the above then the HTTPS Listen Step is not able to create a lisa.vse.request object. And if you have no lisa.vse.request object then filters or DPHs will also throw errors because they need an existing lisa.vse.request object to work upon. So, I believe that the 'JSON Path Filter' error that you posted above is a secondary error and not part of the root cause.
So, no, I do not expect that JSON 2.0 DPH will resolve this error because it also needs a lisa.vse.request object.
If you already want to start digging for the root cause then I suggest that you change the "if environment error" assertion not to "end the test" but let it go to some additional step(s) that you can add to the .vsm. Then after that additional step you can end the test.
You can "query" the environment (testExec) and various properties in that step and try to write something to file which could point you to the root-cause.
Try looking at lisa.vse.request property, and lisa.vse.http.current.transaction.body as suggested by Prema. (But again, I expect those properties to be null)
Thank you for the response.
Please find my responses inline
1. What type of services are experiencing the stoppage? - HTTP
2. Have you had an opportunity to look at the vse.log and the VS_<service>.log during the window of time that the services stop working. I would be curious if there are any exceptions or unusual events occurring in the log. - I had a look at the below logs
1. vse.log - No Exceptions in the logs, I could only see the status of the service is shown as down
2. vse_matches.log - Following is the message printed continuously
2019-01-13 07:11:34,135Z (18:11)[PortServer:0.0.0.0/0.0.0.0:9443] INFO - Response on the Wire HTTP/1.1 400 Bad requestConnection: closeContent-Type: text/html; charset=US-ASCII
<html><body><title>Bad Request</title><p>The following line is not a valid HTTP request:<p><pre>
3. VS_<service>.log - No request or events are logged in this log file during this period
I have observed the similar behaviors as mentioned above during all 4 week period log dumps( only during Sundays' after 6pm).
There are similar copies of the same service which is deployed in the same servers are running fine. Only difference will be the type of application consuming the copies of the same virtual services will be different.
I have linked the reference error query that is related to this. Please find the ticket for more details:
Inspection view shows, no incoming request when you have the VSE log level set to WARN also. You can try changing the log level to INFO as below in logging.properties and then you can see the what request is causing in the inspection view.
log4j.logger.VSE=WARN, VSEAPP ---> log4j.logger.VSE=INFO, VSEAPP
You can also see the lisa.vse.request property which should provide information on request hitting VS and lisa.vse.http.current.transaction.body gives the payload information.
Thank you Prema.
The logging levels are already set to the above recommended values. The problem here is we are able to see the requests for all other hits on the same virtual services from the portal inspection view except for the transaction hit that has caused the above error on this virtual service. Please advise.
HiDid you ever find the solution to this issue? I have a very similar issue and have not found the root cause.
For me, I have narrowed it down to the fact that it will error out on lisa.vse.request
I've managed to figure out a solution for the error.
It looks like DevTest doesn't like when you use the JSON filter inside the Listener step and it tends to error out when you do. I created a Do-Nothing step right after the listener and applied the same exact filters as it originally was in the Listener step. The error stopped coming out and my service has been up for the past 2 days without any errors popping up and bringing it down.