Data Generation velocity for publishing to SQL Server (using Docker)

[Amanpreet Kaur Aulakh]

Hi Team

To generate 1 million rows of data (for 7 columns using docker containers) and publish to:

• database( SQL server) took around 6 hours.
• CSV/ SQL file took around 1 hour

Is there any way to speed up the publish (generated data) to SQL server?

Thanks
Amanpreet Aulakh

[Abderrahmane Zahrir]

If you are inserting new data and do not expect issues with duplicates then you can enable batch mode (update application.properties) so that generated data is not pushed one row at a time to the database.

You should also make sure that network bandwidth is high enough between your portal and database


Regards, Dahman

[Amanpreet Kaur Aulakh]

Since, I am performing data generation locally on docker containers:

Do I have to enable batch mode (application.properties) as follows:

• tdmweb.publish.batchCommit=true

Or

Do I have to edit docker-compose.yml file as follows:

• environment:

                  - 'APPLICATION_PROP="tdmweb.publish.batchCommit=true"'

[Abderrahmane Zahrir]

would update docker compose and do not forget to set the batch size 

tdmweb.publish.iterationsBeforeCommit=20000

Regards, Dahman

[Amanpreet Kaur Aulakh]

Thanks for the reply Dahman, will update docker-compose file.

Regards
Amanpreet Aulakh