Large VSAM ELIB Reorg: REPRO VSAM > BDAM

[Karen Turner]

Long ago and far away JvK added this paragraph to the Elib maintenance KD:

Note:  When dealing with large VSAM ELIBs that are in multiple physical extents, BC1PNCPY can take a long time to run. Although IDCAMS repro use is discouraged as the permanent resolution to your fragmented file it can be used as an intermediate step to speed up the BC1PNCPY process, by reducing the number of physical extents. So following the advice above:  Allocate 2 new ELIBs: 1 VSAM and 1 BDAM, ensuring both are large enough to fit the existing file into a single extent. Use BC1PNLIB to INIT the VSAM file.  Use IDCAMS to REPRO the fragmented ELIB to the un-initialized BDAM file, then use BC1PNCPY to copy the BDAM ELIB file to the newly initialized VSAM ELIB.

Has anyone done this successfully? 

I have a 2000 cyl production delta in 2 vsam extents and 5 elib extents.  I do not have a 20 hour window to BC1PNCPY it from old VSAM to new VSAM.

Testing with a smaller file I get an error on the VSAM to BDAM repro:

 REPRO INFILE(VSAM) OUTFILE(BDAM)                                        
IDC3300I  ERROR OPENING TTAP.END.TPROD.OTHDELTA                          
IDC3351I ** VSAM OPEN RETURN CODE IS 118                                 
IDC3302I  ACTION ERROR ON TTAP.END.TPROD.TSTDELTA.BDAM                   
IDC3309I ** RECORD X'0000000100' NOT WRITTEN.  LENGTH INVALID            
IDC3302I  ACTION ERROR ON TTAP.END.TPROD.TSTDELTA.BDAM                   
IDC3309I ** RECORD X'0000000200' NOT WRITTEN.  LENGTH INVALID            
IDC3302I  ACTION ERROR ON TTAP.END.TPROD.TSTDELTA.BDAM                   
IDC3309I ** RECORD X'0000000300' NOT WRITTEN.  LENGTH INVALID            
IDC3302I  ACTION ERROR ON TTAP.END.TPROD.TSTDELTA.BDAM                   
IDC3309I ** RECORD X'0000000400' NOT WRITTEN.  LENGTH INVALID            
IDC31467I MAXIMUM ERROR LIMIT REACHED.                                   
IDC0005I NUMBER OF RECORDS PROCESSED WAS 0                               
IDC3003I FUNCTION TERMINATED. CONDITION CODE IS 12                       

BDAM JCL:

//BDAMBR14 EXEC PGM=IEFBR14                                         
//BDAM DD DSN=TTAP.END.TPROD.TSTDELTA.BDAM,DISP=(NEW,CATLG,DELETE), 
//       SPACE=(CYL,(10,10)),MGMTCLAS=MCPERMMB,                     
//       DCB=(RECFM=FBS,LRECL=4096,BLKSIZE=4096,DSORG=DA)           

REPRO JCL:

//REPRO EXEC PGM=IDCAMS                                 
//SYSPRINT DD SYSOUT=*                                  
//SYSOUT   DD SYSOUT=*                                  
//VSAM DD DSN=TTAP.END.TPROD.OTHDELTA,DISP=OLD          
//BDAM DD DSN=TTAP.END.TPROD.TSTDELTA.BDAM,DISP=OLD     
//SYSIN DD *                                            
 REPRO INFILE(VSAM) OUTFILE(BDAM)                       

Any insight is greatly appreciated!

Thanks!

kt

------------------------------
Karen Turner
------------------------------

[Dave Harding]

Hi Karen,

No, I do not have any insight into that KB; I avoid ELIBs whenever
possible.

My immediate reaction is to copy it to a PDSE instead. PDSE is now the
recommended format for Delta libs. Do a dry run copying from ELIB to a
'throw-away' PDSE to see how long it'll take.

*Dave Harding *
Client Services Consultant • Mainframe Software Division
Broadcom Software
Mobile [phone] | [e-mail]
Upcoming vacation:
September 2 - September 10
October 4 - October 8



On Mon, Sep 18, 2023 at 12:06 PM Karen Turner via Broadcom <
[e-mail]> wrote:

> Long ago and far away JvK added this paragraph to the Elib maintenance KD:
> Note: When dealing with large VSAM ELIBs that are in multiple... -posted
> to the "Endevor" community
> [image: Broadcom]

[Karen Turner]

Excellent suggestion Dave!  Didn't think of the obvious ... test run took 45 minutes:

MEMBERS INPUT          344205  
MEMBERS ADDED          344205  
MEMBERS REPLACED            0  
MEMBERS FLUSHED             0  
MEMBERS EXCLUDED            0  
MEMBER RECORDS OUT   11885654  

Now I just have to see when the next 2 hour maintenance window is.

Thanks Again!

K.

------------------------------
Karen
[JobTitle]
[CompanyName]
[State]
------------------------------

[Eoin OCleirigh]

I HAVE used the JvK trick and it saved hours!
But you didn't provide a copy of your ELIB inquire output, that should always be the starting point, in any of these discussions
Yes, it looks like you have a valid BDAM defn for a PageSize 4096, but IS that the pageSize, or the VSAM record length/size.
I'd like to see Both the INQuire report and a LISTCAT ENTRY(your.current.elib.cluster) ALL before I comment further.

But I'd add to JvKs Scenarios to start with a rename of the existing library to .OLD and create the copy in .NEW
so that while you're reorg is running there is no chance of anybody else using an out of date image.
With modern zXX machines I'd be surprised if your actual copy took 20 hours anyway - but do check that LSERV and/or RLS aren't getting in your way.

------------------------------
Eoin O'Cleirigh
Lead Systems Engineer @ ANZ [phone]
------------------------------

[JIM AGGERSBURY]

Hi Karen,

We use the BDAM method as our ELIBs are massive. Your record length on the BDAM file was too big I think, it should be 4088. Also we create 2 BDAM files, we REPRO the VSAM file to the 1st BDAM file and then use BC1PNCPY to copy it to the 2nd BDAM file - this does the reorg. Then we repro the 2nd BDAM file to a new VSAM file.

Here is a sample of the JCL that we use, although I am pretty sure it was copied from Broadcom in the first place :) :-

//*------------------------------------------------------------------- 
//*  STEP JS000-- OWN THE ELIB FOR REORG - ACTUALLY A WASTE OF TIME    
//*  AS THE LAST REFERENCE IS ON THE COPY TO BDAM, AVAILABLE THEREAFTER
//*  RENAME THE ELIB TO .OLD BEFORE STARTING. CHECK THE NEW ONE        
//*  THEN RENAME THE .NEW TO THE ORIGINAL                              
//*------------------------------------------------------------------- 
//*                                                                    
//*                                                                    
//*S000     EXEC PGM=IEFBR14                                           
//*DB      DD  DISP=OLD,DSN=SYS7.ENDEVOR.PROD.CCM0.DELTA               
//*                                                                    
//*                                                                    
//*------------------------------------------------------------------- 
//*   STEP JS005 -- TIDY UP AFTER RERUN                                
//*------------------------------------------------------------------- 
//JS005    EXEC PGM=IEFBR14                                            
//DD1      DD  DISP=(MOD,DELETE,DELETE),SPACE=(CYL,(1,1)),             
//             DSN=SYS7.ENDEVOR.PROD.CCM0.DELTA.BDAM1                  
//DD2      DD  DISP=(MOD,DELETE,DELETE),SPACE=(CYL,(1,1)),             
//             DSN=SYS7.ENDEVOR.PROD.CCM0.DELTA.BDAM2                  
//*                                                                    
//*                                                                    
//*------------------------------------------------------------------- 
//*   STEP JS010 -- ALLOCATE A NEW VSAM ELIB                           
//*------------------------------------------------------------------- 
//JS010    EXEC PGM=IDCAMS,COND=(0,LT)                                 
//SYSPRINT DD  SYSOUT=*                                                
//SYSIN    DD  *                                                       
 DEFINE CLUSTER (NAME('SYS7.ENDEVOR.PROD.CCM0.DELTA.NEW') -            
        RECORDSIZE(4088 4088) -                                        
        CONTROLINTERVALSIZE(4096) -                                    
        SHAREOPTIONS(3,3) -                                            
        DATACLASS(DCXVSAM) -                                           
        CYLINDERS(3000,100) -                                          
        NONINDEXED           -                                         
        )      -                                                       
 DATA   (NAME('SYS7.ENDEVOR.PROD.CCM0.DELTA.NEW.DATA'))                
/*                                                                     
//*                                                                    
//*------------------------------------------------------------------- 
//*   STEP JS030 -- ALLOCATE 2 BDAM ELIBS (FOR OLD AND NEW)            
//*------------------------------------------------------------------- 
//JS030     EXEC PGM=IEFBR14,COND=(0,LT)                               
//DD1      DD  DISP=(NEW,CATLG),                                       
//             UNIT=SYSDA,SPACE=(CYL,(3000,),,CONTIG),                 
//             DCB=(DSORG=DA,BLKSIZE=4088,LRECL=4088,RECFM=FBS),       
//             DSN=SYS7.ENDEVOR.PROD.CCM0.DELTA.BDAM1                  
//DD2      DD  DISP=(NEW,CATLG),                                       
//             UNIT=SYSDA,SPACE=(CYL,(3000,),,CONTIG),                 
//             DCB=(DSORG=DA,BLKSIZE=4088,LRECL=4088,RECFM=FBS),       
//             DSN=SYS7.ENDEVOR.PROD.CCM0.DELTA.BDAM2                 
//*                                                                   
//*-------------------------------------------------------------------
//*   STEP JS030 -- INITIALIZE NEW BDAM ELIB USING BC1PNLIB UTILITY   
//*-------------------------------------------------------------------
//JS020    EXEC PGM=BC1PNLIB,COND=(0,LT)                              
//ELIB1    DD  DISP=OLD,DSN=SYS7.ENDEVOR.PROD.CCM0.DELTA.BDAM2        
//NDVTAG   DD  DUMMY                                                  
//SYSIN    DD  *                                                      
INIT       DDNAME = ELIB1                                             
           PAGE SIZE = 4088                                           
           ALLOCATE = (539999,18000)                                  
           RESERVE PAGES = 500                                        
           DIRECTORY PAGES = 16000                                    
.                                                                     
//BSTERR   DD  SYSOUT=*                                               
//SYMDUMP  DD  DUMMY                                                  
//SYSPRINT DD  SYSOUT=*                                               
//*                                                                   
//*-------------------------------------------------------------------
//*   STEP JS040 -- REPRO TO BDAM ELIB                                
//* - ELIB ELEMENTS AND FRAGMENTS ARE COPY TO BDAM ELIB.              
//*-------------------------------------------------------------------
//JS040    EXEC PGM=IDCAMS,COND=(0,LT)                                
//INFILE   DD  DISP=SHR,DSN=SYS7.ENDEVOR.PROD.CCM0.DELTA.OLD          
//OUTFILE  DD  DISP=SHR,DSN=SYS7.ENDEVOR.PROD.CCM0.DELTA.BDAM1        
//SYSPRINT DD  SYSOUT=*                                               
//SYSIN    DD  *                                                      
 REPRO  IFILE(INFILE) OFILE(OUTFILE)                                  
//*                                                                   
//*                                                                   
//*-------------------------------------------------------------------
//*   STEP JS050 -- COPY TO THE NEW BDAM ELIB                         
//* - ONLY ELIB ELEMENTS ARE COPY TO THE NEW BDAM ELIB.               
//* - ELIB COPY BETWEEN BDAM ELIB IS FASTER THAN THAT WITH VSAM ELIB. 
//*-------------------------------------------------------------------
//JS050    EXEC PGM=NDVRC1,PARM='BC1PNCPY',COND=(0,LT)                
//NDVTAG   DD  DUMMY                                                  
//IN       DD  DISP=SHR,DSN=SYS7.ENDEVOR.PROD.CCM0.DELTA.BDAM1        
//OUT      DD  DISP=SHR,DSN=SYS7.ENDEVOR.PROD.CCM0.DELTA.BDAM2        
//SYSPRINT DD  SYSOUT=*                                               
//BSTERR   DD  SYSOUT=*                                               
//SYSUDUMP DD  SYSOUT=*                                               
//ABNLIGNR DD  DUMMY                                                  
//SYSIN    DD  *                                                      
 COPY   INPUT  DDNAME IN                                              
        OUTPUT DDNAME OUT                                             
       .                                                              
/*                                                                    
//*-------------------------------------------------------------------
//*   STEP JS060 -- REPRO BDAM TO VSAM                                
//* - ELIB ELEMENTS AND FRAGMENTS ARE COPY TO BDAM ELIB.              
//*-------------------------------------------------------------------
//JS060    EXEC PGM=IDCAMS,COND=(0,LT)                                
//INFILE   DD  DISP=SHR,DSN=SYS7.ENDEVOR.PROD.CCM0.DELTA.BDAM2        
//OUTFILE  DD  DISP=SHR,DSN=SYS7.ENDEVOR.PROD.CCM0.DELTA.NEW          
//SYSPRINT DD  SYSOUT=*                                               
//SYSIN    DD  *                      
 REPRO  IFILE(INFILE) OFILE(OUTFILE)  
//*  

Regards                     

------------------------------
Jim
------------------------------

[Karen Turner]

Thanks Jim and Peter for pointing out my lrecl discrepancy, I’ll give that a try and add it to my copy of the KD. (which, with the exception of Jan’s BDAM/VSAM paragraph, Roberta and I wrote back in 19/20??) 😊
KT

[Ollivier-Francois Dorso]

Hi Karen,

I used to test this and it should work.
Can you confirm you initialized the target BDAM with the same attributes than the source ELIB ?

Regards,
Ollivier

[Karen Turner]

Ollivier & Eoin,

Thanks for the extra thoughts.

These deltas were created before my time, but they all use the default VSAM page size of 4088, and SHR 3,3

I've successfully re-orged using Dave's PDSe suggestion (45 mins vsam to pdse, 2 hours pdse to vsam, (and DISP=OLD)), but here are my original stats, we don't use Lserv or RLS.


ATTRIBUTES

KEYLEN-----------------0 AVGLRECL------------4088 BUFSPACE------------8192 CISIZE--------------4096

RKP--------------------0 MAXLRECL------------4088 EXCPEXIT----------(NULL) CI/CA----------------180

SHROPTNS(3,3) RECOVERY UNIQUE NOERASE NONINDEXED NOWRITECHK UNORDERED NOREUSE

NONSPANNED



ALLOCATION

SPACE-TYPE------CYLINDER HI-A-RBA------1548288000

IDCAMS SYSTEM SERVICES TIME: 14:57:27 09/18/23 PAGE 2

SPACE-PRI-----------2000 HI-U-RBA------1548288000

SPACE-SEC-------------20

VOLUME

VOLSER------------PEN030 PHYREC-SIZE---------4096 HI-A-RBA------1548288000 EXTENT-NUMBER----------2

DEVTYPE------X'3010200F' PHYRECS/TRK-----------12 HI-U-RBA------1548288000 EXTENT-TYPE--------X'40'

VOLFLAG------------PRIME TRACKS/CA-------------15

EXTENTS:

LOW-CCHH-----X'3E5A0000' LOW-RBA----------------0 TRACKS-------------30300

HIGH-CCHH----X'463D000E' HIGH-RBA------1489305599

LOW-CCHH-----X'2FEF0000' LOW-RBA-------1489305600 TRACKS--------------1200

HIGH-CCHH----X'303E000E' HIGH-RBA------1548287999



ENDEVOR-LIB INQUIRY: DDNAME=ELIB1

DDNAME: ELIB1

DSNAME: VGIO.END.GPROD.OTHDELTA

LIBRARY TYPE: VLB

PAGE SIZE: 4088

LAST PAGE: 378000

FILE PAGE SIZE: 4088

INITIALIZED: 11/01/2022 18:39:51.93

(RE)ALLOC STAMP: 341426

LAST REORG/EXPAND: 11/09/2022 15:18:05.75

FILE SIZE (PAGES) 377999

FIRST ALLOCATION: 359999

EXPAND ALLOCATION: 3600

RESERVE LIMIT: 10

BIT MAP PAGES: 12 STARTING AT 360984

DIRECTORY TARGET PAGES: 15000 STARTING AT 360996

ALLOC MAP BYTES: 47254

PAGES USED: 369961

PAGES FREE: 8038

LAST UPDATE STAMP: 367278

LAST UPDATE: 09/15/2023 10:37:14.04

LATEST MEMBER: FOVM0ZKA

AVG PAGES/MEMBER: 1

# DATA PAGES: 354948

BIT MAP PAGES: 12 STARTING AT 360984

DIRECTORY TARGET PAGES: 15000 STARTING AT 360996

ALLOC MAP BYTES: 47254

PAGES USED: 369961

PAGES FREE: 8038

LAST UPDATE STAMP: 367278

LAST UPDATE: 09/15/2023 10:37:14.04

LATEST MEMBER: FOVM0ZKA

AVG PAGES/MEMBER: 1

# DATA PAGES: 354948

# REPLACES: 2857

# ADDS: 354116

# DELETES: 9911

# MEMBERS: 344205

# DIRECTORY PAGES: 15000

# DIR OVERFLOWS: 0

LONGEST DIR OVFL: 0

# LIBRARY EXTENDS: 5

# DIRECTORY REORG: 1

Based on this statement in Jan's note

"Use IDCAMS to REPRO the fragmented ELIB to the un-initialized BDAM file"


I did not BC1PINIT the BDAM and used the sample JCL from the book to allocate.



Let me know if I'm missing something? I'd like to compare times btwn BDAM/VSAM copy and PDSE/VSAM copy.

Thanks!
KT



------------------------------------------------------------------------------
CONFIDENTIALITY NOTICE: If you have received this email in error,
please immediately notify the sender by e-mail at the address shown.
This email transmission may contain confidential information. This
information is intended only for the use of the individual(s) or entity to
whom it is intended even if addressed incorrectly. Please delete it from
your files if you are not the intended recipient. Thank you for your
compliance. Copyright (c) 2023 Cigna
==============================================================================

[Peter Pravda]

Hi Karen

I hope you are doing well. You might still remember me - JvK's apprentice :)

Couple of points to the discussion:

1) The problem with the Repro was due to incorrect LRECL/BLKSIZE used on your BDAM ELIB. It should be 4088 instead of 4096. The rule of thumb is:
    - For VSAM ELIBs: ELIB page size = CISIZE - 8
    - For BDAM ELIBs: ELIB page size = LRECL = BLKSIZE
Your VSAM CISIZE is 4096 and therefore the page size must be 4088.

2) I agree with Dave and also recommend using a PDSE. But I think you misunderstood his advice. You used a PDSE as an intermediate file instead of a BDAM and instead of Repro, you just used another BC1PNCPY step. Unfortunately this cannot bring any performance benefit. Directly copying VSAM ELIB to VSAM ELIB (with BC1PNCPY) would be faster. Dave's advice was about stopping using a VSAM ELIB in your production implementation and use PDSE instead. A permanent PDSE delta file. If you want to consider this option, to help you better decide, I should mention that I know of two limitations of a PDSE:
    - PDSE is limited to 1 volume. There is no other size limitation, only the size of the volume. So as long as you allocate a PDSE on a sufficiently large volume with large enough free space, this limitation would not pose a real-life problem.
    - PDSE can have no more than 522,239 members. Interestingly, PDS's have no such limitation, but nobody sane would use a PDS with that many members. In your previous post you mentioned you have 344,205 members. That is 66% of the limit, so unless you expect the number of your delta members to grow substantially, you should be fine. Just wanted to mention it because I remember some customers having 700k members in their ELIBs.

3) Internal sneak peek: We are working on a performance improvement for BC1PNCPY. Internal synthetic tests show that if the target is a VSAM ELIB, copying is 40x faster. That brings 20 hours task down to just half an hour (elapsed time). With this performance improvement, using the trick with an intermediate BDAM ELIB will no longer be necessary because direct copying will be faster. Disclaimer: BC1PNCPY performance improvement is an internal work item with low priority and undefined ETA. If there is any customer willing to perform real-life performance testing with the current (unpublished) version, feel free to use "Reply Privately" to this message and I'll get in touch with you with more details.

Best regards,
Peter Pravda
Software Engineer
Endevor Team
Broadcom Software

[Eoin OCleirigh]

Hi Karen,
Nice to hear you're all sorted.  With the details supplied, in retrospect, we can see that the BDAM file should have had LRECL=BLKSIZE=4088 - ie. it should match the ELIB pageSize.   The 4096 was the VSAM CISize which is 8 bytes bigger than the PageSize (and max/average record length).

Since it sounds like Broadcom are about to have a 40x speed improvement (how is this a low Priority?  this problem may be a thing of the past and you can easily perform simple reorgs within standard maintenance windows.
I'd echo the advice that in many cases PDS/E can now safely REPLACE Elibs - but perhaps the bigger underlying problem is having single, monolithic, ELIBs with hundreds of thousands of members, unfortunately separating the eggs into multiple baskets is not that easy, but maybe TYPE by TYPE is achievable, using the ANALYSE DIRECTORY to identify members by type (from their footprints) and copying those into a new dataset that includes type in it's name.... (and adjusting the TYPE definitions to point to the new library (maybe adding &C1TY to the DSName). rinse and repeat till every type has been moved off to it's own library, then archive/delete the old (everything) library.

Not something to try to do in the rush of an emergency, out of space/too fragmented situation, but calmly practiced - maybe with the .OLD and .BDAM1/2 libraries... Then, maybe performance benchmarked with the updated copy utility.

Cheers,

Eoin 

------------------------------
Eoin O'Cleirigh
Lead Systems Engineer @ ANZ [phone]
------------------------------

[Karen Turner]

Thanks Eoin…
They (we) have 2 deltas per stage, one for COBOL and one for all the other types. There has been a lot of JCL and NDM changes to add USER= to every jobcard, and for NDM PGM= within the PROCESS ESP to IWS and back to ESP again, I surmise that’s what pushed OTHDELTA over the edge. 😊

K.

[Ollivier-Francois Dorso]

Hello Karen,

Ok, I will review the content and the way apply article. https://knowledge.broadcom.com/external/article?articleId=9558
I wrote it with the assistance of Jan and it has been validated as is, but maybe something changed, so article should be amended. That's I will do if necessary

Regards,
Ollivier