Batch Cobol consuming all of storage pool 128

[Gary Cherlet]

Hello All:
I have a client who modified a CV Mode batch Cobol program (he says small modification) but now when this program executes it is consuming all of the
36 Megabytes defined to storage pool 128 which has Shared, Shared Kept, User and User Kept storage and when that fills up it takes all of storage pool 0 and then the CV just hangs up and freezes then the CV has to be cancelled. Has anybody seen or heard of anything like this?
William M. Allen, Jr.

[Gary Cherlet]

my first question would be: if this changed was backed out - would all go back to normal? or has something outside this program changed?
if it is the program, than is there a loop that continues to acquire storage? or a storage area accidentally defined to big?
is this in production? if so - did this program exhibit the same 'qualities' in testing that it does now in production? or was the change so minor that no one needed to test it <how many times have i heard that?>
before the change - how much pool 128 was this program using?
unless someone has a inspiration, I would want to know the answer to these questions in order to determine the path to resolution
Chris Hoelscher

[Gary Cherlet]

If the change is backed out everything returns to normal. This did cause a similar problem in testing with the SOS but the test CV had 1/3 of the resources that production had so I made the storage pools the same in test and production and the job then ran fine but nothing else was running on test and I (My fault) didn't look at the storage pool after the developer told me that it ran successfully. In production storage pool 128 rarely if ever went above 65% utilization so I don't know exactly how much this specific program was using. This is a 24 X 7 operation and they have hundreds of online users and many batch jobs executing all the time.
William M. Allen, Jr.

[Gary Cherlet]

does the storage get taken in one apparent bite - or does in gradually continue to consume the storage? Chris

[Gary Cherlet]

Gradually continue to consume the storage.

[Gary Cherlet]

Bill,
That would make me believe it is the called programs or lack of COMMITS.
Regards, John

[Gary Cherlet]

Do you have perfmon available? how about running a test with usertrace save turned on and then cancelling the task after a couple of seconds. that might give you an insight into what's happening.
Bob Wiklund

[Gary Cherlet]

I vote for record locks. Richard

[Gary Cherlet]

Record Locks would come out of a different Storage Pool I think ! User or User Kept and Shared or Shared Kept are in their Pool 128 - Record Locks would be DB or SYS - likely Pool 255 ? Gary

[Gary Cherlet]

There are no indicators that this is a record lock problem, LOCKMON Status is fine and DCMT DIS STAT SYS does not show a large amount of locks I have not looked at this transaction while it was running yet, it fills the storage pool in a matter of minutes and there are no signs that the CV is offloading journals so this does not appear to be volume related at this time. William M. Allen, Jr.

[Gary Cherlet]

Bill,
With that being the case, I would look at how the other programs are being called.
Best of Luck, John

[Gary Cherlet]

it's starting to smell like a loop to me.
Bob Wiklund

[Gary Cherlet]

I think record locks would first fill pool 255 and then overflow into pool 0. How about a loop of some kind?
Bob Wiklund

[Gary Cherlet]

Questions:
1) Update or Retrieval?
2) Any COMMIT processing (either case - this is important)?
3) Are there any database procedures on any of the records being processed? If there are - are they Cobol or Assembler?
Cheers - Gary

[Gary Cherlet]

1. Update.
2. Not sure if commits, but this is not a mass update program I have been told that there are less than 3,000 records.
3. No data base procedures.
William Allen, Jr

[Gary Cherlet]

Bill - if there are 3,000 records - is that the number being retrieved - or Updated? What is the Usage Mode?
Maybe only 3,000 updates but if there are 3,000,000 records being retrieved to do this - then there could be an issue if there are record locks being maintained. Look at the retrieval DML statements - are there any KEEP or KEEP EXCLUSIVE options?
Also - if I were you I would find out exactly what the "minor change" was - for my money somebody put a KEEP option onto a retrieval statement. That is why I asked about COMMITs - COMMIT ALL (with currency save / restore before and after) would be needed to release the locks in this case.
HTH - cheers - Gary

[Gary Cherlet]

Bill, Get out your 12" wooden school ruler and do what the Nuns used to do...smack the programmer across the nuckles untile he/she fesses up to what they did...... :wink: Bob Wiklund

[Gary Cherlet]

I really need to get to the source code and have a look at the modification.
I can't answer these questions at this time but this program has been in production for a very long time and never caused an issue until this latest modification so to sum it up I need to get more information from the developer for the list.
William M. Allen, Jr.

[Gary Cherlet]

Am I missing something or can we find out what the change was? ie. do a compare Now that COBOL allows data definitions in working storage of many megabytes, I'm not surprised 36M got sucked up. We used to be restricted to 132K for fixed records and 35K for variable length. Times have changed.
Margaret

[Gary Cherlet]

Margaret - regarding your comment about Cobol allowing larger Records today than in years gone by - please note that Batch Cobol Working Storage and Linkage Sections always represent storage from the batch region (for all practical purposes) - and would never use storage from the IDMS-CV/DC region. The following quote from a post by Chuck Hardee is well worth noting:
[color=#C00202]......... but the only 2 ways I can readily think of that a BATCH program can affect resources INSIDE CV is either record locks or database procedures allocating storage and not freeing it. " Chuck[color]
Cheers - Gary

[Gary Cherlet]

Worth checking, but if I recall correctly, overflow lock storage comes from pools 255 & 0 and not 128.
Gary mentioned database procedures and I have seen them cause this. It was good old IDMSCOMP/IDMSDCOM - the schema didn't specify to call them on FINISH/ROLLBACK and the batch process created a lot of run units (yes, not very efficient). As the cleanup didn't occur, each run unit grabbed a little bit of extra storage and over many many run-units, this added up. The schema was changed to make the call on FINISH/ROLLBACK and the problem went away. Of course the program creating less run-units would have helped too.
A long shot, but worth a look. Steve Franzon

[Gary Cherlet]

Compressed records.
Pgm is now issuing multiple BIND-FINISH sequences, but nobody has specified a CALL IDMSDCOM ON FINISH for the AREA so it never releases the work area acquired to decompress records. The work area only gets released at task end, or if specifically released on a FINISH call into DCOM.
Don Casey

[Gary Cherlet]

Don - is that your best bet - or the fact of the matter? Cheers - Gary

[Gary Cherlet]

Educated guess. Most people fail to include an AREA call to DCOM in the schema, since in most cases the clean-up happens automatically at end-of-task and nobody notices.
This situation may be due to a subroutine being called repetitively that issues a BIND-FINISH sequence while the main RU remains open or some other such construct.
Of course, I could be wrong. I forget what class of GETMAIN DCOM uses, but I THINK it's USER stg.
Don Casey

[color=#01020B]Note from monitor: [color]
[color=#061BFD] BROWSE IDMSR.SMP.DISTSRC(IDMSCOMP)

#GETSTG LEN=WSDSLEN,ADDR=PRUSER,TYPE=(USER,LONG),INIT=0, X
COND=DEAD,PLIST=*,RGSV=(R3-R7)
LTR R15,R15 BAIL OUT IF NO STG.
BNZ ERR56
LR R11,R1 R11-->IDMSCOMP WORKING STORAGE
[color]

[Gary Cherlet]

If that is the case, can we spare the programmer's knuckles?
Cindy Kline

[Gary Cherlet]

No, because if it is this, it means that the programmer has written a batch program that is creating many run-units. This is not very efficient. However it means we can also lay blame on the DBA. Steve Franzon

[Gary Cherlet]

I would tend to agree with Don on this one.
However, as Don says, educated guess. We could be both wrong and it could be something else, but the only 2 ways I can readily think of that a BATCH program can affect resources INSIDE CV is either [color=#0318FA]record locks[color] or [color=#0318FA]database procedures[color] allocating storage and not freeing it.
Chuck

[Gary Cherlet]

Hello All:
Many thanks to Don Casey and to everyone who responded tonight, it turns out that the record that the programmer was updating is a compressed record that used COMP/DCOM.
When I said that it didn't use data base procedures I was only thinking about user written procedures and not COMP/DCOM.
After I added the FINISH and ROLLBACK calls to the area in the Schema that contained the compressed record they are now running the job with hundreds of thousands of input records and so far storage pool 128 has only grown by 12K.
Yes this program is creating additional run units but not a 1 for 1 situation, every so often I see the additional run unit pop up in OPER.
This Schema was created a very long time ago and this record was created on 04/02/1990 this was before ARCH started supporting this Client.
In this case the programmer was not the cause of this issue, but, since I am the DBA I will accept the responsibility for the missing area calls in the Schema but it was around way before I arrived.
William M. Allen, Jr.

[Gary Cherlet]

As we say in the Marine Corps:
[color=#080CFF]"SAEPE EXPERTUS, SEMPER FIDELIS, FRATRES AETERNI"[color]
"Often tested, Always Faithful, Brothers Forever"
Update on the issue, I have searched the Schema and found 13 additional areas that have compressed records with calls to COMP/DCOM, is there a rule of thumb when to add these area calls? This one only showed up after the programmer added new code to modify and store the affected record last night, or should I just go add the calls to each of the areas just to be safe? William M. Allen, Jr.

[Gary Cherlet]

We always add them to the area.
Richard

[Gary Cherlet]

Ditto for our site.
Kay

[Gary Cherlet]

I should have said we always add the calls to areas containing compressed records.
Dick

[Gary Cherlet]

The big question in my mind remains, how did Bill manage to engage half a dozen of the best IDMS minds on a Tuesday night after working hours?
(Note: preceding line from earlier post from IDMS-L subscriber who has requested not to be cross-posted - but included in the following reply))
[color=#060AFF]And besides them, I chipped in too[color]
Chris Hoelscher

[Gary Cherlet]

Yes
IDMSCOMP BEFORE STORE
IDMSCOMP BEFORE MODIFY
IDMSDCOM AFTER GET
As far as how you got so many replies on a late Tuesday night, well only a very small majority were AWAY from our computers last night.
Thank You - Tim

[Gary Cherlet]

Tim - these are all RECORD calls - AREA calls which were the problem in this Thread would be along the lines of:
mod AREA containing-RECORDcalls-to-IDMSCOMP/DCOM
call IDMSDCOM after FINISH
HTH - cheers - Gary