Test Data Manager

Expand all | Collapse all

Data Generation velocity for publishing to SQL Server (using Docker)

Jump to Best Answer
  • 1.  Data Generation velocity for publishing to SQL Server (using Docker)

    Posted 10-17-2019 01:47 PM
    Hi Team

    To generate 1 million rows of data (for 7 columns using docker containers) and publish to:
    • database( SQL server) took around 6 hours.
    • CSV/ SQL file took around 1 hour
    Is there any way to speed up the publish (generated data) to SQL server?

    Thanks
    Amanpreet Aulakh


  • 2.  RE: Data Generation velocity for publishing to SQL Server (using Docker)

    Posted 10-18-2019 05:12 AM
    If you are inserting new data and do not expect issues with duplicates then you can enable batch mode (update application.properties) so that generated data is not pushed one row at a time to the database.

    You should also make sure that network bandwidth is high enough between your portal and database


    Regards, Dahman


  • 3.  RE: Data Generation velocity for publishing to SQL Server (using Docker)

    Posted 10-18-2019 11:58 AM
    Since, I am performing data generation locally on docker containers:

    Do I have to enable batch mode (application.properties) as follows:
    • tdmweb.publish.batchCommit=true

    Or

    Do I have to edit docker-compose.yml file as follows:
    • environment:
                      - 'APPLICATION_PROP="tdmweb.publish.batchCommit=true"'


  • 4.  RE: Data Generation velocity for publishing to SQL Server (using Docker)
    Best Answer

    Posted 10-18-2019 12:09 PM
    would update docker compose and do not forget to set the batch size 

    tdmweb.publish.iterationsBeforeCommit=20000

    Regards, Dahman





  • 5.  RE: Data Generation velocity for publishing to SQL Server (using Docker)

    Posted 10-18-2019 12:41 PM
    Edited by Amanpreet Kaur Aulakh 10-18-2019 12:41 PM
    Thanks for the reply Dahman, will update docker-compose file.

    Regards
    Amanpreet Aulakh