Security violations behavior has changed in GUI client-server

[Douglas Seaver]

Recently (in the past few months) we have noticed a behavioral change when a security violation occurs.
what used to happen:

1. user logs on to mainframe via client. when the user tries to access a part of the application and no RACF access has been defined, they received a message box with TIRM621E: Error creating semaphore, can't access server. in the CICS logs we would see the message
   TISRVMSL TIML TASK=00000113 05/05/2023 07:31:34 GIVESOCKET NOT TAKEN FOR SERVER xxxx
   

Since early May 2023, when a user incurs a violation like that above, we no longer see the TIRM621E message at the client.  Instead the client window "spins" without ending and we see instead "<guiclient>.EXE is not responding", with options to RESTORE, CLOSE, or WAIT (picture attached).  On the server side we no longer see the GIVESOCKET NOT TAKEN message.

Our environment is

1. Client 
   1. Windows 10
   2. VS2015
2. Server
   1. z/OS 2.5
   2. CICS 5.6
   3. DB2 v12
   4. TISRVMSL
   5. TIRSLEXT LGVTIMS is set to 5 seconds.

     Wondering if anyone else has seen a similar change in behavior or may have an idea as to the cause.

------------------------------
Doug Seaver
Systems Development Services Specialist
Gen Tool Support
WisDOT
Madison, WI, USA
------------------------------

[Lynn Williams]

Hi Doug,

I notice you have z/OS 2.5. 

Is that a recent upgrade and if so did it occur just before the time in early May you saw the change in behaviour?

If yes and you are not up to date on Gen maintenance there could be a relevant Listener PTF LU08295 ("TISRVMSL LOOPS/DOES NOT PROCESS SOCKETS WHEN HANDLING ERRORS") that may help if you don't already have it installed.
That fixed a problem that from our logged customer cases only seems to get exposed with TCP/IP changes under z/OS 2.5. Basically, sockets related to error conditions like security violations were not getting released and eventually, they would get exhausted.
However, if it is that problem there should be other messages in the CICS MSGUSR log per the PTF description i.e.
START TRANSACTION FAILED ...
LISTENER TAKESOCKET BACK FAILED ...
ACCEPT CALL FAILED ...

Also if the TISRVMSL has been up for a while then after enough errors occur to use up the available sockets that is when the problem would start to occur but it should then impact all client connections and not just those that cause an error, so it does not really match your symptoms.

However, due to the hang/loop nature of your symptoms if you don't have the PTF applied it might still be worth doing so.

 

Hope it helps

Regards

Lynn

------------------------------
Lynn Williams
Senior Principal Support Engineer
Broadcom Software
Australia
------------------------------

[Douglas Seaver]

Thanks, Lynn. We have the listener PTF in our next round of production fixes, but those may be a while before they are deployed in production.
I will move it in separately in a test environment and do some testing. Thanks for the tip!
Regards,

Doug
Doug Seaver
DBM-BITS-ADS-DMV Core Systems Unit
Wisconsin Department of Transportation
[phone]

[Teresa Bredenkamp]

Hi Doug,

Thought you may like to know that this PTF affects TISRVMSL only, as it handles sockets management when sending the response - or not send it in this case. It does not impact your servers so the PTF should be easier to deploy.   

Regards,

------------------------------
Teresa Bredenkamp
CA Gen Development
Broadcom
------------------------------

[Douglas Seaver]

Good news – I have this PTF for TISRVMSL installed in our DEV TEST CICS region and it behaves as expected with appropriate messages on both client and server sides.

I will work on getting this deployed through the rest of our test environments, and plan for Production implementation on Sunday 9/10/2023.

Question - Do the following messages indicate the same type of response (install PTF) ? These errors occurred before z/OS 2.5 and they’ve only happened once, but I’m curious.



FYI. A ton of these just hit P202 at 9:40am (2,267 of them). Haven’t seen this one before or since. It appeared that all users were timing out trying to do anything in DMV Suite (our Gen client-server app) It just fixed itself 25 minutes later.



TISRVMSL TIML TASK=00000174 03/20/2023 09:40:31 DID NOT RECEIVE FULL CFB HEADER; SOCKET CLOSED

IP ADDR = 000.000.000.000

TISRVMSL TIML TASK=00000174 03/20/2023 09:40:31 DID NOT RECEIVE FULL CFB HEADER; SOCKET CLOSED

IP ADDR = 000.000.000.000

[Teresa Bredenkamp]

Hi Doug, 

It is hard to say without researching what is being processed when the messages are produced. 

The problem addressed by the PTF  was an incorrect check of the active sockets table that caused a follow on check of sockets with error conditions to be skipped so no message was returned to the client and the relevant socket closed. Eventually all the available sockets are active and no new sockets are accepted and TISRVMSL reports message  ACCEPT CALL FAILED                             
SOCKET ERROR, RETCODE=-0000001, ERRNO=00001002. 

The messages you report indicate that the client request triggered a Connection and a Socket Accept was completed but the subsequent Receive of data was not successful or was partially successful, so TISRVMSL cannot process the request.  It is possible that once there were no sockets available the backlog filled up, the network became congested and not all the data was available to be received.

But this is speculation on my part since I have nothing to confirm it.   

I remember you raising this but I do not think we got enough doc to investigate. If after applying PTF LU08295 you still see these messages please raise a new case and we will investigate.

I hope this explains what I think and why I cannot conclusively say this is the same problem.

Regards, 

------------------------------
Teresa Bredenkamp
CA Gen Development
Broadcom
------------------------------

[Lynn Williams]

Hi Doug,
Further to Teresa's reply, we do also have a knowledge article for this message: Gen TISRVMSL "DID NOT RECEIVE FULL CFB HEADER; SOCKET CLOSED" which indicates slow network data transmission could be a reason.
Also if the problem fixed itself after 25 minutes and no action was taken on the TISRVMSL i.e. it was not restarted to clear any socket saturation problem, then it would sort of suggest a network-related issue.

Regards

Lynn

------------------------------
Lynn Williams
Senior Principal Support Engineer
Broadcom Software
Australia
------------------------------