FIle Masking and Automation Set up

[Vinoth Gopi]

Hi All,

Is it possible can we achieve file masking through CA TDM Portal.

I need to establish a connection between a source and target.

The source file needs to be picked for masking and placed in a target location.

It Need to be schedule on an hourly basis.

Thanks

[Bradley Gray]

Hi Vinoth,

Currently, you can use FDM to mask flat files, but we don't support this in Portal yet. I believe this is a feature that product Management and Engineering are working on. It should be available in a future release of Portal.

For more information, see Mask Data Stored in Flat Files

Regards,
Brad

[Vinoth Gopi]

Hi Brad,

Thanks for the information. 

That's a great news where it's being supported from portal since we work for automation set up as well, so when it comes to the portal it would be easy for us to set up the automation.

Also, I would like to know do we have any file watcher so that we can have an end-to-end automation set up.

Thanks 

Vinoth

[Bradley Gray]

Hi Vinoth,

The only way I can think of would be to create a Javelin flow that could check the file for updates. I'm not a Javelin expert, so I don't know all that is needed to go about setting this up. 

Regards,
Brad

[Rafael Cintra]

Hey Vinoth,

You should be able to use FDM and Javelin together to be able to accomplish what you are looking for. There is at least two ways to go about this:

1. Use Javelin to monitor a file and trigger actions if a file is update or some other condition is met
2. If the condition on step 1 is meet do X Javen actions within your workflow which will include using FDM to perform masking based on a pre-define masking configuration
3. Copy your masked file to any given destination
4. Repeat steps 1-3 as needed every X minutes

The trick here is how to trigger this process to run every X minutes and check your condition to validate if your file has been updated.

Currently there is no way you can through TDM Portal auto scheduled something to be run every X minutes, therefore you would have to have some additional scheduling tool to do that for you. It can be something simple as your Windows Server Task scheduling or a cron job running on some Linux server.

That schedule task would trigger your TDM Portal Generator which would have a dummy table or csv file associated with a Workflow action which would be able to execute the logic defined on steps 1-3.

If that makes sense to you I ca give a you a sample Javelin workflow showing how to integrate with FDM to mask files using a predefined set of FDM configuration files. If you need more customized masking rules you could also use Javelin to build your masking rules configuration on the fly before masking.

There is also a option where you can have the steps on a infinite running job on the TDM Portal which would also depend on a similar architecture the difference being you have one job which would always be running on your TDM Portal which would have internal Javelin logic to check the same condition on step 1 every X minutes and repeat this flow X many items or for X many minutes. The first execution would have to potentially be done manually or by some external process.

The downside to this option is that over time this process could start consuming a lot of memory but would need to validated based on your own environment on a test run. A sample workflow can be build to demonstrate that option if you think that would be a better option.

Let me know which option makes more sense to you. 

Thanks,

[Vinoth Gopi]

Hi Rafel,

Thanks for the update.

I created a file watcher using Power shell and now trying to invoke the CA TDM data generation job using the API services.

I am trying to use the Swagger API do we have any document related to this.

I am using powershell to trigger the CA TDM generation job once the new file is placed in the target folder.

Thanks

Vinoth 

[Rafael Cintra]

Hello Vinoth,

I'm assuming you just need help on how to use our TDM APIs to submit a Test Data Generator execution. If this is the case you can find what you need from the example below which is using CURL.

To be able to execute this on your side you will have to update the following:

name, description, projectId, scheduledTime and all the generator related data.

{
"name":"Publish_travel_Add People",
"description":"Publish to travel using Add People", "projectId":2200, "type":"PUBLISHJOB", "origin":"generation", "scheduledTime":"2018-05-23T09:11:00Z",
"email":"[e-mail]",
"jobs":[
],
"parameters":{
"variableDefaults":[],
"generatorId":2211,
"jobType":"PUBLISH",
"title":"Publish to travel using Add People", "publishTo":"TGT", "target":"dbo", "actionOnDuplicate":"exit", "actionOnGenDuplicate":"exit", "dataTargetProfile":"travel",
"dataSourceProfile":"travel", "repeatCount":1, "tables":[ { "tableNo":36, "tableName":"PEOPLE",
"status":1
},
{ "tableNo":42,
"tableName":"CREDIT_CARDS",
"status":1
}
],
"email":"[e-mail]"
}
}

CURL:
curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJBZG1pbmlzdHJhdG9yIiwiYXVkIjoiQUxMIiwiUFdEX0hBU0hfQ0xBSU0iOiI4MzkzMTQyODMiLCJpc3MiOiJDQSBUZWNobm9sb2dpZXMiLCJVU0VSX0lEIjoiMSIsImV4cCI6MTUyNzE2NTkxNSwiaWF0IjoxNTI3MDc5NTE1LCJBQ0NFU1NfUEVSTUlTU0lPTlMiOiJ7XCJBTExfUFJPSkVDVFNcIjpbMTAwXX0ifQ.KUcG5-mMcKiNaoMtZbTkhhTzJAs1wzcwd2HWPNyiSvM' -d '{ \
"name":"Publish_travel_Add People", \
"description":"Publish to travel using Add People", \
"projectId":2200, \
"type":"PUBLISHJOB", \
"origin":"generation", \
"scheduledTime":"2018-05-23T09:11:00Z", \
"email":"x%40ca.com", \
"jobs":[ \
], \
"parameters":{ \
"variableDefaults":[], \
"generatorId":2211, \
"jobType":"PUBLISH", \
"title":"Publish to travel using Add People", \
"publishTo":"TGT", \
"target":"dbo", \
"actionOnDuplicate":"exit", \
"actionOnGenDuplicate":"exit", \
"dataTargetProfile":"travel", \
"dataSourceProfile":"travel", \
"repeatCount":10, \
"tables":[ \
{ \
"tableNo":36, \
"tableName":"PEOPLE", \
"status":1 \
}, \
{ \
"tableNo":42, \
"tableName":"CREDIT_CARDS", \
"status":1 \
} \
], \
"email":"x%40ca.com" \
} \
}' 'https://localhost:8443/TDMJobService/api/ca/v1/jobs'

Request URL: https://localhost:8443/TDMJobService/api/ca/v1/jobs

Headers:
{ "Accept": "application/json",
"Authorization": "Bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJBZG1pbmlzdHJhdG9yIiwiYXVkIjoiQUxMIiwiUFdEX0hBU0hfQ0xBSU0iOiI4MzkzMTQyODMiLCJpc3MiOiJDQSBUZWNobm9sb2dpZXMiLCJVU0VSX0lEIjoiMSIsImV4cCI6MTUyNzE2NTkxNSwiaWF0IjoxNTI3MDc5NTE1LCJBQ0NFU1NfUEVSTUlTU0lPTlMiOiJ7XCJBTExfUFJPSkVDVFNcIjpbMTAwXX0ifQ.KUcG5-mMcKiNaoMtZbTkhhTzJAs1wzcwd2HWPNyiSvM"
}

It's usually easier if you execute this generator through the UI and review what you have under the network tab using your browser developer tools to get what is being sent to the TDMJobService API one time and copy and paste to your PowerShell script.

Are you not planning to use Javelin in this case and going only with Powershell?

Thanks,

[Vinoth Gopi]

Hi Rafel,

Hope you are doing good.

I tried with copying the Power shell script from View source for particular Job ID. But I am not able successfully invoke the generation job.

Do we have any documents related to this which will help us to trigger the data generation job using power shell.

Thanks

Vinoth Gopi

[Rafael Cintra]

Hey @Vinoth Gopi you do not anything specific to Powershell under our documentation but I would be able to build an example for you. Would you mind sharing what you have done so far in Powershell so I can use as a baseline?

You can remove any private IPs, usernames and such. I just need the logic.

Thanks,