need help /advice about IDMS DB archive to flat file

[Jan Binar]

Hello,
I need help with request form customer for archive IDMS databases.

Parameters:
Archive all IDMS database  (243 areas )
Archive it for 10 years

IDMS subsystem is now running (but it will stop - IDMS will not be available for whole 10 Y )
No planes transfer data to another database platform

goal:  - extract the data into a flat format suitable for long-term archival and reference purposes
          - long-term plan is to decommission the mainframe, so offline data access must be ensured
          - restoring the data in flat file form outside the mainframe (e.g. for browsing or reporting), rather than bringing IDMS back online.

Backup/restore function will be not fit because IDMS can be already shutdown, so restore can not be used that time. (and I should save/export logical and physical definition for those databases. ). I was think that most use full can be used EXPORT NETWORK and let generate all the output data files (data file, copybook file, and control file). But again it is for export /load using IDMS subsystem (need segment defined in DMCL in IDMS, ...). May be it is possible to read data by some tool like File Master, if we have data file and copy book.

I want ask if anyone face same problem and has any working scenario for long-time archive without IDMS subsystem. Also do you know about any tool which can read those files on z/OS or outside mainframe?

[Kevin Heard]

Jan:  I have served as the IDMS application SME for TWO iterations of Extract-Transform-Load integral to my employers exiting IDMS.... most recently in 2024.

I'm not sure how much I can say here; because - using an analogy now - the "Mainframe Forum" group prohibits providing any assistance to users intending to get off the mainframe.... they essentially "kick you out of the clubhouse".... and I certainly don't want to "run afoul" of a similar restriction here.  All of that admitted, it seems equally-if-not-more-so improper to not offer some perspective here.   Yes, it is absolutely "doable"; as the process involves using COBOL to obviously extract the data portions of the records; but crucially important is the replication of the "IDMS SETS/INDICES" construct (the actual DBKEY pointer values - likely converted to integers as 'keys' & 'sequential numbers') into the resulting COBOL files.  I have recent personal experience with this exercise in 2024 & received recognition/bonus for my involvement  (total 30 years IDMS), and would be glad to provide guidance and/or actual assistance in your effort.... SUBJECT TO THE BELOW CAVEAT:

** NOTE TO COMMUNITY MODERATOR:  Please feel free to reject this post if it is proven to violate any policies/rules akin to the above reference **
** I have no desire to "shoot myself" with this generous offer of user assistance **  

Kevin Heard

[e-mail]

[Jan Binar]

Thanks Kevin,

Yes, I also do not want to violate any Forum rules. (The recommendation to open this thread came from Broadcom – I opened case on same topic .) I fully understand that assistance for leaving IDMS/mainframe environments is not supported. However, I am only looking for a backup solution. The customer has already migrated the application away from the mainframe. Now, they likely need to satisfy certain external or internal regulations to keep the data accessible. This would not be on a daily basis, but rather as a proof of concept – to demonstrate that the data is stored and can be read, even if very slowly and only for a small, specific portion.

From the Broadcom case and from this discussion, I understand that there is no official scenario for this situation. It always requires careful planning and significant effort on the application side and DBA team.

I was curious to know if my thoughts are heading in the right direction. Would using a standard tool such as EXPORT NETWORK be sufficient, or would it be necessary to create a custom program to copy the data, for example, into a VSAM file?

[Kevin Heard]

I, too, have not previously used the tool. Both of my experiences utilized manually written COBOLs.  Is it possible that users in either Asia or Europe have done so???  Like you, I can envision essentially having to recreate-ore-reverse engineer the actual extraction code - presumably in COBOL - that would correctly replicate the flow of data thru the "reporting" modules (yuk).  I have a number of off-the-top concerns that I want to share... based on both of my previous efforts:  1) the Broadcom 19.0 doc specifically states that the tool is not usable on VSAM files (that would have been a real problem in my cases).  2)  I would be very concerned about SETS with no PRIOR pointers.  3) VIA Record Occurrences that - depending on which set is being traversed - contain completely disparate data [and how the resulting read/proof module correctly handles that condition]  4) Instances of REDEFINES on given records; 5) how otherwise "corrupted" data is addressed [for older databases, having apparently 'partial', 'changed' or all the other terms that speak to, "this DATA doesn't look right" is handled; 6) the Broadcom doc also specifically states that any LOGICALLY DELETED records must be ERASED prior to utilizing this tool; 7) How the current non-IDMS application will address "flagging" the presence of "additional archived data is available" ... and equally important, how the presumed batch job will be submitted and the mechanism(s) by which the resulting data will be passed back to the requestor for use.  If you're wanting a specific "let's do this" answer, I would suggest setting up a call with the current???? IDMS DBA (or the DBA for whichever DBMS currently houses their data) along with their Application SMEs, and let's walk thru one or two AREAs and discuss some of these points.   For its scope, I'd me more than happy to "contribute" a day-or-two to the effort.  I would also advise your client to expect to have some data not be converted/extracted/usable.... 10-12% wouldn't surprise me, particularly for "very mature" databases :) [e.g. data stored on IDMS record using OFFSET values "MOVE 'ABCD'  to 0415-EMP(28:4)"

I'm admittedly 'running on at the mouth'; but as we all appreciate, the 'net must be cast widely' :)   I would also expect to find a measurable number of IDMS records that were not INITIALIZED prior to storage, with the resulting data issues associated therewith.

I have chatted enough for now; but sincerely:  Feel free to set up a meeting for a few hours so that we can start moving forward with this .  It's not going to cost any of us anything to talk for bit.

[John Abell]

Send me a note offline as we have a lot of experience with similar requirements after a decision has been made to move off of IDMS.  [e-mail].

[Jan Binar]

Thank you for your responses. It is clear that there is no simple or “magic” solution available. The archive should be handled in the same manner as the customer performed the database transformation. I was in the study phase, so I have shared this information with the customer. Thank you once again.

[Margaret Sliming]

‘M trying to understand what your end goal is.  You want to archive all of your IDMS data to flat files.  Does this data have no use other than an occasional query?  Does all of the data need to be accessible?  If you don’t need these databases, why not back them up as is that way if you don’t need these need to access them down the road you can use a Mainframe-as-a-Service organization to access IDMS. You cannot access database data into flat files and maintain relationships. I was a SME on an IDMS to C#/Oracle conversion.  There is so much of IDMS that doesn’t convert and work-around are very complex  You may want to re-think your approach.

[Tommy Petersen]

Extract the data with the DB Keys, I believe you would need to get rid of logically deleted records first.  You do not need to extract prior keys for sets because the next key of the prior has the dbkey of the current.  Keeping the dbkeys would also resolve records that are not members of sets.  In order to read this mess, you would need to load it into a relational database or use Hadoop to read it.

We have a product from International Software Products that can replicate from IDMS to relational.  Comes with tools that convert the IDMS schema to DDL, data extraction from IDMS, and load into the relational database.  We use the tool to keep Oracle and PostgreSQL databases in sync with our IDMS production databases.

[Jan Binar]

Thanks for all comments. I also don't have all information, because how I already written, customer with their resources (Development team), was able migrate IDMS data outside IDMS and mainframe already and I wasn't involved to this project. Mainframe environment will be shutdown competently soon.  And now I can only guess. That customer not want pay any mainframe licence and mainframe people. Now they only checking possibilities if I ( service provider ) can backup IDMS mainframe databases outside mainframe by some magic cheaper way. ( because migration cost lots of time and people utilisation ).  Answer is no. Not simple way exist for copy data to flat file.