Test Data Manager

Expand all | Collapse all

Data Generation velocity for publishing to SQL Server (using Docker)

Jump to Best Answer
  • 1.  Data Generation velocity for publishing to SQL Server (using Docker)

    Posted 29 days ago
    Hi Team

    To generate 1 million rows of data (for 7 columns using docker containers) and publish to:
    • database( SQL server) took around 6 hours.
    • CSV/ SQL file took around 1 hour
    Is there any way to speed up the publish (generated data) to SQL server?

    Thanks
    Amanpreet Aulakh


  • 2.  RE: Data Generation velocity for publishing to SQL Server (using Docker)

    Posted 28 days ago
    If you are inserting new data and do not expect issues with duplicates then you can enable batch mode (update application.properties) so that generated data is not pushed one row at a time to the database.

    You should also make sure that network bandwidth is high enough between your portal and database


    Regards, Dahman


  • 3.  RE: Data Generation velocity for publishing to SQL Server (using Docker)

    Posted 28 days ago
    Since, I am performing data generation locally on docker containers:

    Do I have to enable batch mode (application.properties) as follows:
    • tdmweb.publish.batchCommit=true

    Or

    Do I have to edit docker-compose.yml file as follows:
    • environment:
                      - 'APPLICATION_PROP="tdmweb.publish.batchCommit=true"'


  • 4.  RE: Data Generation velocity for publishing to SQL Server (using Docker)
    Best Answer

    Posted 28 days ago
    would update docker compose and do not forget to set the batch size 

    tdmweb.publish.iterationsBeforeCommit=20000

    Regards, Dahman





  • 5.  RE: Data Generation velocity for publishing to SQL Server (using Docker)

    Posted 28 days ago
    Edited by Amanpreet Kaur Aulakh 28 days ago
    Thanks for the reply Dahman, will update docker-compose file.

    Regards
    Amanpreet Aulakh