COBOL question

[Anon Anon]

I need to clean up a large string of data so that it does not contain any non-displayable characters. The goal is to make it XML-compatible. It's more than just alpha and numeric - I need to allow all displayable characters, and change the rest to spaces.

This is in an on-line program, so I am concerned about performance. I will need to compare each byte to the list of 95 allowable characters. What would be the most efficient way to do this: table lookup, large 88-level field, other?

We had discussed converting the data to base 64 while in flight, which would get it through the XML generator and parser on both ends. But that doesn't really clean it up.

All suggestions welcome!

Kay Rozeboom
Information Technology Enterprise
Iowa Department of Administrative Services
Telephone: [phone] Fax: [phone]
Email: [e-mail]

[Legacy User]

You could write a little assembler routine like this. This one translate everything except “a” thru “z”, “A” thru “Z” and “0” thru “9” to spaces. To add more “exempt” characters, add more ORGs to the table. One caveat, no warranty, expressed or implied, with this code. Use at your own risk as I have not tested it.

TRCHARS CSECT
STM 14,12,12(13) save caller’s registers
LM 2,3,0(1) load length and data address
LARL 4,TRTABLE point to translate table
CHI 2,256 256 chars or less?
JH USELOOP
BCTR 2,0 less one for effective length
LASTBIT DS 0H
EXRL 2,TRINST translate short style
LM 14,12,12(13) reload caller’s registers
SR 15,15 return code 0
BR 14 return to caller
*
TRINST TR 0(0,3),0(4) translate according to needs
*
USELOOP DS 0H
AHI 2,-256 subtract 256 from remaining length
JM DOLAST if less than 256, we went < 0
TR 0(256,3),0(4) translate next 256 bytes
LA 3,256(,3) advance to next 256 bytes
J USELOOP go do it again
*
DOLAST DS 0H
AHI 2,255 add back to get byte count
JM ALLDONE count was zero, we’re done
J LASTBIT go do final translate
*
TRTABLE DS 0H
DC 256C' '
ORG TRTABLE+C'a'
DC X'818283848586878889'
ORG TRTABLE+C'j'
DC X'919293949596979899'
ORG TRTABLE+C's'
DC X'A2A3A4A5A6A7A8A9'
ORG TRTABLE+C'A'
DC X'C1C2C3C4C5C6C7C8C9'
ORG TRTABLE+C'J'
DC X'D1D2D3D4D5D6D7D8D9'
ORG TRTABLE+C'S'
DC X'E2E3E4E5E6E7E8E9'
ORG TRTABLE+C'0'
DC X'F0F1F2F3F4F5F6F7F8F9'

Charles (Chuck) Hardee
Senior Systems Engineer
Database Administration
Information Technology Services
Thermo Fisher Scientific
300 Industry Drive
Pittsburgh, PA 15275
Direct: 724-517-2633
FAX: [phone]
[e-mail]

From: CA IDMS (IUA EIUA) Global User Community [mailto:[e-mail]]
Sent: Thursday, April 18, 2013 1:23 PM
To: [e-mail]
Subject: [DBA Issues - Open Forum] COBOL question

I need to clean up a large string of data so that it does not contain any non-displayable characters. The goal is to make it XML-compatible. It's more than just alpha and numeric - I need to allow all displayable characters, and change the rest to spaces.

This is in an on-line program, so I am concerned about performance. I will need to compare each byte to the list of 95 allowable characters. What would be the most efficient way to do this: table lookup, large 88-level field, other?

We had discussed converting the data to base 64 while in flight, which would get it through the XML generator and parser on both ends. But that doesn't really clean it up.

All suggestions welcome!

Kay Rozeboom
Information Technology Enterprise
Iowa Department of Administrative Services
Telephone: [phone] Fax: [phone]
Email: [e-mail]
Posted by:Kay.Rozeboom
--
CA Communities Message Boards
100829683
[e-mail]
https://communities.ca.com

[Chris Hoelscher]

Is this data pulled from a database (or other “permanent” source), or entered by an operator, or some other source?

Chris hoelscher
Technology Architect | Database Infrastructure Services
Technology Solution Services
[Description: Description: cid:image001.png@01CD13D7.1A57CAF0]
123 East Main Street |Louisville, KY 40202
[e-mail]
Humana.com
[phone] – office
[phone] - blackberry
Keeping CAS and Metavance safe for all HUMANAty

From: CA IDMS (IUA EIUA) Global User Community [mailto:[e-mail]]
Sent: Thursday, April 18, 2013 1:23 PM
To: [e-mail]
Subject: [DBA Issues - Open Forum] COBOL question

I need to clean up a large string of data so that it does not contain any non-displayable characters. The goal is to make it XML-compatible. It's more than just alpha and numeric - I need to allow all displayable characters, and change the rest to spaces.

This is in an on-line program, so I am concerned about performance. I will need to compare each byte to the list of 95 allowable characters. What would be the most efficient way to do this: table lookup, large 88-level field, other?

We had discussed converting the data to base 64 while in flight, which would get it through the XML generator and parser on both ends. But that doesn't really clean it up.

All suggestions welcome!

Kay Rozeboom
Information Technology Enterprise
Iowa Department of Administrative Services
Telephone: [phone] Fax: [phone]
Email: [e-mail]<mailto:[e-mail]>
Posted by:Kay.Rozeboom
--
CA Communities Message Boards
100829683
[e-mail]<mailto:[e-mail]>
https://communities.ca.com

The information transmitted is intended only for the person or entity to which it is addressed
and may contain CONFIDENTIAL material. If you receive this material/information in error,
please contact the sender and delete or destroy the material/information.

[Anon Anon]

It’s bad data in a database. Yes, it needs to be cleaned up. But that could take a while. In the mean time, the current middleware software, which is used only for this one application, is costing us over $1,000 per week. (Yes, that was “per week”.)

Kay Rozeboom
Information Technology Enterprise
Iowa Department of Administrative Services
Telephone: [phone] Fax: [phone]
Email: [e-mail]

From: CA IDMS (IUA EIUA) Global User Community [mailto:[e-mail]]
Sent: Thursday, April 18, 2013 12:56 PM
To: [e-mail]
Subject: RE: [DBA Issues - Open Forum] COBOL question

Is this data pulled from a database (or other “permanent” source), or entered by an operator, or some other source?

Chris hoelscher
Technology Architect | Database Infrastructure Services
Technology Solution Services
[Description: Description: cid:image001.png@01CD13D7.1A57CAF0]
123 East Main Street |Louisville, KY 40202
[e-mail]<mailto:[e-mail]>
Humana.com
[phone] – office
[phone] - blackberry
Keeping CAS and Metavance safe for all HUMANAty

From: CA IDMS (IUA EIUA) Global User Community [mailto:[e-mail]]
Sent: Thursday, April 18, 2013 1:23 PM
To: [e-mail]<mailto:[e-mail]>
Subject: [DBA Issues - Open Forum] COBOL question

I need to clean up a large string of data so that it does not contain any non-displayable characters. The goal is to make it XML-compatible. It's more than just alpha and numeric - I need to allow all displayable characters, and change the rest to spaces.

This is in an on-line program, so I am concerned about performance. I will need to compare each byte to the list of 95 allowable characters. What would be the most efficient way to do this: table lookup, large 88-level field, other?

We had discussed converting the data to base 64 while in flight, which would get it through the XML generator and parser on both ends. But that doesn't really clean it up.

All suggestions welcome!

Kay Rozeboom
Information Technology Enterprise
Iowa Department of Administrative Services
Telephone: [phone] Fax: [phone]
Email: [e-mail]<mailto:[e-mail]<mailto:[e-mail]%3cmailto:[e-mail]>>
Posted by:Kay.Rozeboom
--
CA Communities Message Boards
100829683
[e-mail]<mailto:[e-mail]<mailto:[e-mail]%3cmailto:[e-mail]>>
https://communities.ca.com

The information transmitted is intended only for the person or entity to which it is addressed
and may contain CONFIDENTIAL material. If you receive this material/information in error,
please contact the sender and delete or destroy the material/information.
Posted by:ChrisHoelscher
--
CA Communities Message Boards
100830404
[e-mail]<mailto:[e-mail]>
https://communities.ca.com

[Gary Cherlet]

This can be done in "native" Cobol with an INSPECT verb - an example follows. The only problem in coding is to code all the non-Display characters - there's a fair few of them. The advantage is that you have a great deal of control of what each non-Display character is translated to - it is nice and "visible" in the code - in the sense that you don't need to go to an assembler program and look at the translate table to work out what the "from:to" will be. Exposing the rules in this way makes it easier too if you want different replacements in some instances, in the same or different programs - you don't need to have special assembler routines for each case. The example below shows how to code the HEX values for the non-display characters - it also shows how you can do "string" conversions for strings of more than one character!

HTH - take care - cheers - GaryC

[font=Courier New][color=#017d96] inspect [color=#050000]W-TCP-BUFFER (1:W-READ-LENGTH-IN) [color]
converting[color=#050000] '[]'[color] to[color=#050000] '{}'[color]
replacing all [color=#050000]'[' [color] by [color=#050000] '{' [color]
[color=#050000] ']'[color] by [color=#050000] '}' [color]
[color=#050000] x'B0' [color]by[color=#050000] x'5F'[color]  
[font] [color]

[Anon Anon]

Gary, do you know if you can code multiple “replacing” statements on a single INSPECT commands?

Kay Rozeboom
Information Technology Enterprise
Iowa Department of Administrative Services
Telephone: [phone] Fax: [phone]
Email: [e-mail]

From: CA IDMS (IUA EIUA) Global User Community [mailto:[e-mail]]
Sent: Thursday, April 18, 2013 6:25 PM
To: [e-mail]
Subject: [DBA Issues - Open Forum] RE: COBOL question

This can be done in "native" Cobol with an INSPECT verb - an example follows. The only problem in coding is to code all the non-Display characters - there's a fair few of them. The advantage is that you have a great deal of control of what each non-Display character is translated to - it is nice and "visible" in the code - in the sense that you don't need to go to an assembler program and look at the translate table to work out what the "from:to" will be. Exposing the rules in this way makes it easier too if you want different replacements in some instances, in the same or different programs - you don't need to have special assembler routines for each case. The example below shows how to code the HEX values for the non-display characters - it also shows how you can do "string" conversions for strings of more than one character!

HTH - take care - cheers - GaryC

inspect W-TCP-BUFFER (1:W-READ-LENGTH-IN)
converting '[]' to '{}'
replacing all '[' by '{'

']' by '}'

x'B0' by x'5F'


Posted by:Gary_Cherlet
--
CA Communities Message Boards
100833117
[e-mail]<mailto:[e-mail]>
http://communities.ca.com

[Legacy User]

Kay,
I’ll send you the page from the IBM language reference for Enterprise COBOL version 4 Release 2 with the syntax tree, but the short answer is yes.
Chuck

Charles (Chuck) Hardee
Senior Systems Engineer
Database Administration
Information Technology Services
Thermo Fisher Scientific
300 Industry Drive
Pittsburgh, PA 15275
Direct: 724-517-2633
FAX: [phone]
[e-mail]

From: CA IDMS (IUA EIUA) Global User Community [mailto:[e-mail]]
Sent: Friday, April 19, 2013 8:26 AM
To: [e-mail]
Subject: RE: [DBA Issues - Open Forum] RE: COBOL question

Gary, do you know if you can code multiple “replacing” statements on a single INSPECT commands?

Kay Rozeboom
Information Technology Enterprise
Iowa Department of Administrative Services
Telephone: [phone] Fax: [phone]
Email: [e-mail]

From: CA IDMS (IUA EIUA) Global User Community [mailto:[e-mail]]
Sent: Thursday, April 18, 2013 6:25 PM
To: [e-mail]
Subject: [DBA Issues - Open Forum] RE: COBOL question

This can be done in "native" Cobol with an INSPECT verb - an example follows. The only problem in coding is to code all the non-Display characters - there's a fair few of them. The advantage is that you have a great deal of control of what each non-Display character is translated to - it is nice and "visible" in the code - in the sense that you don't need to go to an assembler program and look at the translate table to work out what the "from:to" will be. Exposing the rules in this way makes it easier too if you want different replacements in some instances, in the same or different programs - you don't need to have special assembler routines for each case. The example below shows how to code the HEX values for the non-display characters - it also shows how you can do "string" conversions for strings of more than one character!

HTH - take care - cheers - GaryC

inspect W-TCP-BUFFER (1:W-READ-LENGTH-IN)
converting '[]' to '{}'
replacing all '[' by '{'

']' by '}'

x'B0' by x'5F'


Posted by:Gary_Cherlet
--
CA Communities Message Boards
100833117
[e-mail]<mailto:[e-mail]>
http://communities.ca.com
Posted by:Kay.Rozeboom
--
CA Communities Message Boards
100837341
[e-mail]
https://communities.ca.com

[Gary Cherlet]

Kay - if what you are wanting to know is if you can specify lots and lots of things to "replace x by z" - then the answer is yes.

Just code one "replacing" and multiple "x by z" clauses for that statement - the formatting in these posts didn't really show that as clearly as I wanted to - I'll try it this way to see if it is clearer:

[font=Courier New]inspect W-TCP-BUFFER (1:W-READ-LENGTH-IN)
converting '[]' to '{}'
replacing all [color=#1200b6]'[' by '{' [color] [color=#b60006]']' by '}' [color] [color=#1200b6] x'B0' by x'5F' [color] [color=#b60000]x'00 by x'40'[color] etc[font]

HTH - cheers - GaryC

These links should prove helpful:

IBM Cobol examples

IBM Cobol syntax tree

[Anon Anon]

Cool! I knew about INSPECT, but about not it allowing multiple clauses. I thought it would have to read through the field umpteen times. Glad to be proved wrong.

Kay Rozeboom
Information Technology Enterprise
Iowa Department of Administrative Services
Telephone: [phone] Fax: [phone]
Email: [e-mail]

From: CA IDMS (IUA EIUA) Global User Community [mailto:[e-mail]]
Sent: Friday, April 19, 2013 8:20 AM
To: [e-mail]
Subject: RE: [DBA Issues - Open Forum] RE: COBOL question

Kay - if what you are wanting to know is if you can specify lots and lots of things to "replace x by z" - then the answer is yes.

Just code one "replacing" and multiple "x by z" clauses for that statement - the formatting in these posts didn't really show that as clearly as I wanted to - I'll try it this way to see if it is clearer:

inspect W-TCP-BUFFER (1:W-READ-LENGTH-IN)
converting '[]' to '{}'
replacing all '[' by '{' ']' by '}' x'B0' by x'5F' x'00 by x'40' etc

HTH - cheers - GaryC

These links should prove helpful:

IBM Cobol examples<http://publib.boulder.ibm.com/infocenter/iadthelp/v7r0/index.jsp?topic=/com.ibm.etools.iseries.langref.doc/c0925395439.htm>

IBM Cobol syntax tree<http://pic.dhe.ibm.com/infocenter/pdthelp/v1r1/index.jsp?topic=%2Fcom.ibm.entcobol.doc_3.4%2Frlpsinsp.htm>
Posted by:Gary_Cherlet
--
CA Communities Message Boards
100838040
[e-mail]<mailto:[e-mail]>
http://communities.ca.com

[Legacy User]

That’s interesting Gary. The IBM manual shows CONVERTING only in Format 4 of the INSPECT verb and that format does NOT include the REPLACING sub-clause.

Only Formats 2 and 3 show the REPLACING sub-clause. Format 2 is “INSPECT … REPLACING ….” and Format 3 is “INSPECT … TALLYING … REPLACING …”.


Charles (Chuck) Hardee
Senior Systems Engineer
Database Administration
Information Technology Services
Thermo Fisher Scientific
300 Industry Drive
Pittsburgh, PA 15275
Direct: 724-517-2633
FAX: [phone]
[e-mail]

From: CA IDMS (IUA EIUA) Global User Community [mailto:[e-mail]]
Sent: Friday, April 19, 2013 9:20 AM
To: [e-mail]
Subject: RE: [DBA Issues - Open Forum] RE: COBOL question

Kay - if what you are wanting to know is if you can specify lots and lots of things to "replace x by z" - then the answer is yes.

Just code one "replacing" and multiple "x by z" clauses for that statement - the formatting in these posts didn't really show that as clearly as I wanted to - I'll try it this way to see if it is clearer:

inspect W-TCP-BUFFER (1:W-READ-LENGTH-IN)
converting '[]' to '{}'
replacing all '[' by '{' ']' by '}' x'B0' by x'5F' x'00 by x'40' etc

HTH - cheers - GaryC

These links should prove helpful:

IBM Cobol examples<http://publib.boulder.ibm.com/infocenter/iadthelp/v7r0/index.jsp?topic=/com.ibm.etools.iseries.langref.doc/c0925395439.htm>

IBM Cobol syntax tree<http://pic.dhe.ibm.com/infocenter/pdthelp/v1r1/index.jsp?topic=%2Fcom.ibm.entcobol.doc_3.4%2Frlpsinsp.htm>
Posted by:Gary_Cherlet
--
CA Communities Message Boards
100838040
[e-mail]
http://communities.ca.com

[Gary Cherlet]

The original example with the converting and replacing clauses in a single statement is straight out of a working DC-Cobol program. I "enhanced" the last example just to make the point ....
... don't argue with success ... even if it doesn't quite agree with what you think is 100% correct .... is something that has helped me in my career :wink:

Cheers - GaryC

[rob.klan.1]

Given main program is Cobol and INSPECT is an expensive instruction.

I wonder if Cobol main calling a compositely linked assembler TRT sub-routine would be more efficient.

I do not recall or know if IDMS has same LE overhead as CICS does.




From: CA IDMS (IUA EIUA) Global User Community [mailto:[e-mail]]
Sent: Friday, April 19, 2013 10:01 AM
To: [e-mail]
Subject: RE: [DBA Issues - Open Forum] RE: COBOL question

The original example with the converting and replacing clauses in a single statement is straight out of a working DC-Cobol program. I "enhanced" the last example just to make the point ....
... don't argue with success ... even if it doesn't quite agree with what you think is 100% correct .... is something that has helped me in my career [Image removed by sender. emoticon]

Cheers - GaryC
Posted by:Gary_Cherlet
--
CA Communities Message Boards
100839493
[e-mail]
http://communities.ca.com

[Chris Hoelscher]

What about a database procedure to clean up the record before it is stored ?

Chris hoelscher
Technology Architect | Database Infrastructure Services
Technology Solution Services
[Description: Description: cid:image001.png@01CD13D7.1A57CAF0]
123 East Main Street |Louisville, KY 40202
[e-mail]
Humana.com
[phone] – office
[phone] - blackberry
Keeping CAS and Metavance safe for all HUMANAty

From: CA IDMS (IUA EIUA) Global User Community [mailto:[e-mail]]
Sent: Friday, April 19, 2013 12:52 PM
To: [e-mail]
Subject: RE: [DBA Issues - Open Forum] RE: COBOL question

Given main program is Cobol and INSPECT is an expensive instruction.

I wonder if Cobol main calling a compositely linked assembler TRT sub-routine would be more efficient.

I do not recall or know if IDMS has same LE overhead as CICS does.




From: CA IDMS (IUA EIUA) Global User Community [mailto:[e-mail]]<mailto:[mailto:[e-mail]]>
Sent: Friday, April 19, 2013 10:01 AM
To: [e-mail]<mailto:[e-mail]>
Subject: RE: [DBA Issues - Open Forum] RE: COBOL question

The original example with the converting and replacing clauses in a single statement is straight out of a working DC-Cobol program. I "enhanced" the last example just to make the point ....
... don't argue with success ... even if it doesn't quite agree with what you think is 100% correct .... is something that has helped me in my career [Image removed by sender. emoticon]

Cheers - GaryC
Posted by:Gary_Cherlet
--
CA Communities Message Boards
100839493
[e-mail]<mailto:[e-mail]>
http://communities.ca.com
Posted by:rob.klan.1
--
CA Communities Message Boards
100843353
[e-mail]<mailto:[e-mail]>
https://communities.ca.com

The information transmitted is intended only for the person or entity to which it is addressed
and may contain CONFIDENTIAL material. If you receive this material/information in error,
please contact the sender and delete or destroy the material/information.

[Anon Anon]

Thanks everybody, for all the suggestions.

Kay Rozeboom
Information Technology Enterprise
Iowa Department of Administrative Services
Telephone: [phone] Fax: [phone]
Email: [e-mail]

[Anon Anon]

Kay,

When you have a chance, please let us know which solution you used.

Thanks
Linda

[Anon Anon]

I ended up using an assembler program I got from someone in a different department at the State of Iowa, since that one was already being used at our site. It was similar to the one provided to the list by Chuck Hardee.

If no one had given me a complete assembler program, I would have used the multi-substitution INSPECT command described by Gary Cherlet.

Kay Rozeboom
Information Technology Enterprise
Iowa Department of Administrative Services
Telephone: [phone] Fax: [phone]
Email: [e-mail]

From: CA IDMS (IUA EIUA) Global User Community [mailto:[e-mail]]
Sent: Monday, April 22, 2013 11:18 AM
To: [e-mail]
Subject: RE: [DBA Issues - Open Forum] RE: COBOL question

Kay,

When you have a chance, please let us know which solution you used.

Thanks
Linda
Posted by:lindajcasey
--
CA Communities Message Boards
100866342
[e-mail]<mailto:[e-mail]>
https://communities.ca.com