Consistent Masking across applications

[Swapnil_Rasane]

Hi,

 

I have a scenario where one of my application is on IMS and other on SQL Server. Both applications have customer demographic details and I have to privatize these details.

 

To ensure "Mark" is "Peter" for customer ID 12345 (for example) in both applications after masking, that is, to have consistent masking what are the best practices in FDM?

 

I know we can use hashlov function but what should be the parameters to this function?

 

For masking IMS, I am using FileMaster to generate copybook and CA TDm to generated transformation maps and for SQL Server I am using FDM directly.

 

Also we have million records in a table to be masked using hash function, how the masking speed can be improved as my previous run took 30 hrs for masking firstname for 0.3 million records that too using 30 threads.

[gilta03]

Hi Swapnil, we are currently researching your questions and will get back to you shortly.

[gilta03]

Hi Swapnil,

 

Here is some more information about understanding masking functions and parameters in FDM, and some FDM best practices:

Understand Masking Functions and Parameters - CA Test Data Manager - 3.5 - CA Technologies Documentation 

Fast Data Masker Best Practices - CA Test Data Manager - 3.5 - CA Technologies Documentation

 

If FDM connects to the same DB2 table on the mainframe to get its seed data, AND if option SEEDTABLEINDEXCOLUMN=RL_RN is set, they should both mask the same.

 

Required mainframe options are :

TRIMMEDHASHLOV=Y

HASHTYPE=JAVA

CASEINSENSITIVEHASHLOV=Y

 

Please let us know if this helps or if you have any follow up questions.

[Swapnil_Rasane]

Thanks Taylor.

 

I tried running the masking job in 10 threads however got the error. I have attached option and masking file here.

 

Also, I could not find SEEDTABLEINDEXCOLUMN in FDM Is it available on mainframe only?

[Swapnil_Rasane]

Masking File:

 

Table,Column,Function,Parm1,Parm2,Parm3,Parm4,KeepNulls,DateFormat,Cross Reference,Override SQL,Unique Columns,Xpath Element,Substr start,Substr length,Notes,Preformat,update,Use Masked Values,restart column

esit_gcm_name,,Where,ciskey like'1%',,,,N,,,,,,,,,,,,

esit_gcm_name,first_name,HASHLOV,firstnames.txt,,ciskey,,Y,,,,ciskey,,,,,,,,

esit_gcm_name,,Where,ciskey like'2%',,,,N,,,,,,,,,,,,

esit_gcm_name,first_name,HASHLOV,firstnames.txt,,ciskey,,Y,,,,ciskey,,,,,,,,

esit_gcm_name,,Where,ciskey like'3%',,,,N,,,,,,,,,,,,

esit_gcm_name,first_name,HASHLOV,firstnames.txt,,ciskey,,Y,,,,ciskey,,,,,,,,

esit_gcm_name,,Where,ciskey like'4%',,,,N,,,,,,,,,,,,

esit_gcm_name,first_name,HASHLOV,firstnames.txt,,ciskey,,Y,,,,ciskey,,,,,,,,

esit_gcm_name,,Where,ciskey like'5%',,,,N,,,,,,,,,,,,

esit_gcm_name,first_name,HASHLOV,firstnames.txt,,ciskey,,Y,,,,ciskey,,,,,,,,

esit_gcm_name,,Where,ciskey like'6%',,,,N,,,,,,,,,,,,

esit_gcm_name,first_name,HASHLOV,firstnames.txt,,ciskey,,Y,,,,ciskey,,,,,,,,

esit_gcm_name,,Where,ciskey like'7%',,,,N,,,,,,,,,,,,

esit_gcm_name,first_name,HASHLOV,firstnames.txt,,ciskey,,Y,,,,ciskey,,,,,,,,

esit_gcm_name,,Where,ciskey like'8%',,,,N,,,,,,,,,,,,

esit_gcm_name,first_name,HASHLOV,firstnames.txt,,ciskey,,Y,,,,ciskey,,,,,,,,

esit_gcm_name,,Where,ciskey like'9%',,,,N,,,,,,,,,,,,

esit_gcm_name,first_name,HASHLOV,firstnames.txt,,ciskey,,Y,,,,ciskey,,,,,,,,

esit_gcm_name,,Where,ciskey like'0%',,,,N,,,,,,,,,,,,

esit_gcm_name,first_name,HASHLOV,firstnames.txt,,ciskey,,Y,,,,ciskey,,,,,,,,

[Swapnil_Rasane]

Option File:

AUDIT=SAMPLE10

AUDITDIR=G:\Westpac TDM Team\Output Files\Rahul\Workspace\ISSMARIN\logs

AUDITFILE=myaudit.csv

AUDITPASSWORD=Rahul123

AUDITZIP=winzip

BACKUPDIR=G:\Westpac TDM Team\Output Files\Rahul\Workspace\ISSMARIN\\backup

DIAGLEVEL=1

LOGDIR=G:\Westpac TDM Team\Output Files\Rahul\Workspace\ISSMARIN\logs

PARALLEL=10

[Swapnil_Rasane]

Error Log:

############################################################
#   Default Logging Configuration File
#
# You can use a different file by specifying a filename
# with the java.util.logging.config.file system property. 
# For example java -Djava.util.logging.config.file=myfile
############################################################

############################################################
#   Global properties
############################################################

# "handlers" specifies a comma separated list of log Handler
# classes.  These handlers will be installed during VM startup.
# Note that these classes must be on the system classpath.
# By default we only configure a ConsoleHandler, which will only
# show messages at the INFO and above levels.
handlers= java.util.logging.ConsoleHandler

# To also add the FileHandler, use the following line instead.
#handlers= java.util.logging.FileHandler, java.util.logging.ConsoleHandler

# Default global logging level.
# This specifies which kinds of events are logged across
# all loggers.  For any given facility this global level
# can be overriden by a facility specific level
# Note that the ConsoleHandler also has a separate level
# setting to limit messages printed to the console.
.level= SEVERE

############################################################
# Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################

# default file output is in user's home directory.
java.util.logging.FileHandler.pattern = %h/java%u.log
java.util.logging.FileHandler.limit = 500000
java.util.logging.FileHandler.count = 1
java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter

# Limit the message that are printed on the console to INFO and above.

java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter

# Example to customize the SimpleFormatter output format
# to print one-line log message like this:
#     <level>: <log message> [<date/time>]
#
# java.util.logging.SimpleFormatter.format=%4$s: %5$s [%1$tc]%n

############################################################
# Facility specific properties.
# Provides extra control for each logger.
############################################################

# For example, set the com.xyz.foo logger to only log SEVERE
# messages:
#com.wibu.xpm.Runtime.level = FINEST
com.wibu.xpm.Runtime.level = SEVERE

[gilta03]

Hi Swapnil, If you are still facing masking issues, please open a support case so that we can investigate this issue further. I want to make sure we are not posting too much of your system and database information.

If you are no longer facing this issue, we would appreciate it if you marked your question as answered and posted the way you resolved the issue.

[Swapnil_Rasane]

Hi Taylor,

 

I am not able to find SEEDTABLEINDEXCOLUMN in FDM option tab. FDM Version: 4.2.200.50

[Dwijadas]

Hi, 

I have similar requirement. I dont use DB2, instead IAM file for Seed data in mainframe. I can export the seed file into SQL table to be used by FDM. 

Do you have any guide in this area that I can refer to ? or any suggestion ?

[Billy Keefer]

Hello Dwijadas.  Could you please be a bit more specific about what you are trying to do.

[Dwijadas]

Hi Billy, 

Thanks for your reply. I will try to describe here more.  We have common customer information in mainframe(VSAM) and Linux/Oracle and I want to use HASHLOV for name masking. I have to ensure that output of masking in both platform is same. 

In mainframe the SEED list is a VSAM file, I want to be able to use that for masking using FDM in case of Oracle. I was suggested that to achieve this I have to ensure both platform (Mainframe & Linux) connect to a common DB2 seed list. 

This is not possible as Mainframe, I cant connect to external DB, even if its technically possible.

To me it really doesn't make sense as masking logic/algorithm same in both platform, why cant sync HASHLOV across platform. 

You may check :Case 32969844 , since HASHLOV is not possible, I used FormatEncrypt in both platform as solution. 

Should you need more info, just contact me using my contact details in the CASE. 

Thank you.
Dwija

[Billy Keefer]

Hello Dwijadas.  Apologies for the late reply...totally missed your question.  With hashlov the size of the seed table plays a part in the algorithm. If the seed table rows for the data category chosen within your vsam file is exactly the same as the seed table rows for the same data category   for your oracle db seed table and you specify to use the java hash method  (on the mainframe), then it should be possible.

[Jaspal Singh]

Hi Dwija, You have a seedlist for masking data sources using FDM. Migrate that seedlist to mainframe DB2 (not external DB2). Since both the seedlist will be exactly same (having same index/value) you will achieve consistent masking ​for any Mainframe / Non Mainframe data source. 

Please refer to the below link for more detail:

https://techdocs.broadcom.com/us/en/ca-enterprise-software/devops/test-data-management/4-9/installing/mainframe-installation-and-upgrade/install-mainframe-components-v5-4.html

Also as Billy mentioned, you can also use Vsam for hosting your seedlist on Mainframe. Both solutions will produce consistent masking as long as seedlists are same.