HI, I have seem too many enqueues recently on TMS-AUD and TMS-TMC, causing some medium impact to the systems, since sometimes it doesn't release the resource for some reason.
I would like to know what causes a job to lock the TMC/AUDIT as exclusive. I understand that every Tape Processing will update AUDIT / TMC, but if THIS is the reason for the LOCKS I should have enqueues every minute with the number of Tape jobs running at same time. Can you please clarify why a job requires exclusive access to TMC/AUDIT?
S=SYSTEM TMSQNAME TMS-AUD SYSNAME JOBNAME ASID TCBADDR EXC/SHR STATUS XX00 ABC12345 1234 007F83C0 EXCLUSIVE OWN XX00 XYZ12345 0123 007F83C0 EXCLUSIVE WAIT
Do you have a TMSCOPY running at the same time?
Do you use a blocket AUDIT?
Depending on the answers, I will let you know if you need to open a case or not.
There was no TMSCOPY running.
I am not sure what would be a blocked AUDIT so assume I do not use blocked Audit.
Honestly the issue is fixed now, and what a most want here is to understand WHY a job requires exclusive access to Audit / TMC ? It is not pointed in JCL DISP=(OLD), it is not even referenced on JCL. Basically that's all I need for now. Understand what causes EXCLUSIVE access to such vital file to Tape Processing
This is the way we protect the TMC and AUDIT updates from one tape to the others.
This is normally not measurable and very short in time except if you have a problem on tape devices or any other system tasks.
You have in the documentation some explanations about QNAME but I recognize not so much detailed.
Let me discuss with the labs to provide you a more detailed information.
Thanks for asking.
Thank you Pierre
As you said, the TMC is updated everytime there is a tape event (OPEN/CLOSE/EOV as well as a CATALOG/UN-CATALOG event). And, everytime the TMC is update we ALSO will update the AUDIT file. The order of the enq/deq is to first ENQ/LOCK the TMC, update the TMC, ENQ/LOCK the AUDIT, update the AUDIT, DEQ the AUDIT and finally DEQ the TMC. Now, some tape events (creating a secondary file for example) may cause 4 different updates to the TMC and AUDIT. So, creating many (thousands) of secondary-files in a short period of time will cause a LOT of I/O's to the TMC. And if each secondary-file is also cataloged; that will simply increase the number of I/O's and cause delays in the CATALOG address space.
One way to avoid the delay in the CATALOG address space is to run the SMFQ subtask of the CTS address space.
How many systems are sharing the TMC and AUDIT file, and are you converting our RESERVE to a global enqueue using a GRS-STAR or MIM configuration? (GRS-RING should not be used, much to slow.)
Thank you very much for your reply
We usually have about 4 LPARS sharing AUDIT/TMC, in this specific system I had issues this morning it was 4 LPARS.
We use GRS in this system.
We had a situation like this:
ISG343I 00.31.09 GRS STATUS 595 918 DEVICE:6101 VOLUME:XTTM05 RESERVED BY SYSTEM BT00 S=SYSTEM TMSQNAME TMS-AUD SYSNAME JOBNAME ASID TCBADDR EXC/SHR STATUS BT00 ZRPCPLD8 02AE 007F83C0 EXCLUSIVE OWN BT00 CLRQSTD8 02B3 007F83C0 EXCLUSIVE WAIT
S=SYSTEMS SYSIGGV2 SYS1.TIRKST.CATALOG SYSNAME JOBNAME ASID TCBADDR EXC/SHR GT00 CATALOG 0037 0078C148 EXCLUSIVEBT00 CATALOG /ZRPCPLD8 0037/02AE 0078C148 SHARE AT00 CATALOG 0037 0078C148 SHARE
Yes, that is showing the interaction between the MVS/Catalog and the TMC/AUDIT updates. You see, the logic I shared before (about the TMC and AUDIT being locked) gets longer and more complicated with the MVS/Catalogs. The reason is that the user-catalog (SYS1.TIRKST.CATALOG) is reserved while the MVS/Catalog is updating it (as expected); BUT it holds the reserve while it then writes an SMF record. Now, the CA 1 intercept then intercepts the SMF record being created, and updates the TMC. So now, the user-catalog is reserved UNTIL both the TMC and AUDIT have also been updated. This does cause a longer delay for the updating of the MVS/Catalog.
The easiest solution to this is to start the SMFQ subtask of the CTS address space. The SMFQ subtask only affect MVS/Catalog/un-catalog operations on the system it is running on, so if you have 4 very active systems you would need to run the SMFQ subtask on all 4 systems. If only 2 systems have a lot catalog/un-catalog operations; then you could just run the SMFQ subtask on those 2 systems and let the other 2 systems run without it. The SMFQ subtask if active will eliminate the delay in the MVS/Catalog processing, so while the same number of updates are done to the TMC - they won't be causing any delay in MVS/Catalog processing.
Of course, another option would be to enable Record-Level-Sharing for the MVS user-catalogs. But that is completely different solution and one that does not involve CA 1 directly.
Thank you Russ.
Just to make sure about how much time it takes in a perfect scenario.
The lock at AUDIT/TMC happens at the OPEN ?
The Unlock at Audit/TMC happens at the CLOSE ?
When exactly each step Lock/Unlock happens ?
BTW. Will check how to start SMFQ as suggested
Oh no. The AUDIT/TMC is locked ONLY while the update to the record is taking place. So, it is LOCKED, updated, un-locked at OPEN. Then it is LOCKED, updated, un-locked at CLOSE. Then, it is LOCKED, updated, un-locked at CATALOG time.
What the SMFQ does is "move" the "LOCK, update,un-lock" from the CATALOG address space (done while the user-catalog itself is still locked) to the CTS address space and is done independently from the lock on the user-catalog.
Again. Thank you for your contribution RussellWitt . I already started the implementation of SMFQ and it really seems to have fixed the enqueues we had.
I am just out of curiosity now. Why does the update on TMC/AUDIT requires a lock on user-catalog ? The other question is if it would be reasonable to have the updates being done without the lock on user-catalog by default, without SMFQ jumping into it. What component would have to be changed for that ? Thanks again
It is not that the update on the TMC/AUDIT that requires a lock on the user-catalog; but rather that the user-catalog happens to be locked (by design) while the SMF-record is being created. And when CA 1 intercepts the SMF-record (after it is created, but before it is posted back to actual SMF-creator) it then updates the TMC/AUDIT. The reason that the MVS/CATALOG logic holds the enqueue while the SMF-record is being created is because the SMF records are used for recovery purposes. And MVS/Catalog wants to make sure that the SMF-record has been cut before releasing the enqueue and going on to the next update.
I believe that IBM is coming out with an enhancement to SMF processing so that something like the IEFU83 exit is available AFTER the record has been cut (so the SMF-creator thinks it is all finished) but in the order of their creation and very shortly after it being created. It is very similar to the SMFQ however, and would basically do the same thing.
Sr. Consulting Software Engineer