Test Data Manager

 View Only
  • 1.  java.lang.OutOfMemoryError: Java heap space

    Posted Nov 06, 2019 04:35 AM
    ​Hi Team,

    I am trying to apply  the masking on 21 million records and using "Fixed Value " option. When I am executing the Masking job, its showing below error message. Could you please help me on this.

    >2019-11-06 01:16:44.514 Cursor Fetchsize:1000
    >2019-11-06 01:16:44.514 Autocommit:false
    >java.lang.OutOfMemoryError: Java heap space
    >Dumping heap to java_pid6480.hprof ...
    >Unable to create java_pid6480.hprof: Permission denied
    >2019-11-06 01:17:03.899 java.lang.OutOfMemoryError: Java heap space
    > at com.microsoft.sqlserver.jdbc.TDSPacket.<init>(IOBuffer.java:6215)
    > at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6405)
    > at com.microsoft.sqlserver.jdbc.TDSCommand.detach(IOBuffer.java:7249)
    > at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2925)
    > at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:248)
    > at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:223)
    > at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:2074)
    > at com.grid_tools.products.datamasker.SQLUpdate.batchPreparedMasking(SQLUpdate.java:4771)
    > at com.grid_tools.products.datamasker.SQLUpdate.run(SQLUpdate.java:5316)
    > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    > at java.lang.Thread.run(Thread.java:748)
    >
    >2019-11-06 01:17:04.071 
    >2019-11-06 01:17:04.243 masking complete at 2019.11.06 01:17:03.899 PST
    >2019-11-06 01:17:04.243
    >Allocated memory      :910,336k
    >2019-11-06 01:17:04.243 Approx max memory used:1,132,543k (124% of allocated)
    >2019-11-06 01:17:04.260 exit value = 0

    ------------------------------
    Nitin Dixit
    ------------------------------


  • 2.  RE: java.lang.OutOfMemoryError: Java heap space

    Posted Nov 06, 2019 07:55 AM
    Hi Nitin,

    I have a similar experience with Fast Data Masker's fixed value masking in a large table. In the end I used a SQL Server element in Javelin to do a direct update. What took almost an entire day to mask was now done in a couple of minutes. 

    Best regards,
    Klaas-Jan


  • 3.  RE: java.lang.OutOfMemoryError: Java heap space

    Broadcom Employee
    Posted Nov 06, 2019 08:30 AM
    You need to increase the value of the parameter, -Xmx, in your FDM bat file.


  • 4.  RE: java.lang.OutOfMemoryError: Java heap space

    Posted Nov 06, 2019 09:00 AM
    Increasing both memory parameters was the first thing I did but ít did not improve the performance significantly.


  • 5.  RE: java.lang.OutOfMemoryError: Java heap space

    Posted Nov 07, 2019 03:43 AM
    ​Hi Billy,

    Are you talking about  "bigfdm64" batch file? Could you please help me with the maximum value which we can update here. Thanks,

    @echo off
    if "%1"=="" goto error
    if "%2"=="" goto error
    if "%5"=="high" goto highlabel
    if "%5"=="HIGH" goto highlabel
    java -Xms100M -Xmx%1M -jar Fastdatamasker.jar %2 %3 %4
    goto end
    :highlabel
    start /high java -Xms100M -Xmx%1M -jar gtsdm.jar %2 %3 %4
    goto end
    :error
    echo.
    echo ** ERROR SPECIFYING PARAMETERS
    echo.
    echo ** TO RUN:  BIGFDM memory_size_in_meg_for_example_1000 connection.txt map.csv [options.txt] [high]
    :end

    ------------------------------
    Nitin Dixit
    ------------------------------



  • 6.  RE: java.lang.OutOfMemoryError: Java heap space
    Best Answer

    Posted Nov 07, 2019 07:41 AM
    FastDataMasker creates a .bat file (along with the options- and definitionfile) in which the memory usage is configured. The bat file starts with the following string:
    java -Dfile.encoding=UTF-8 -Djava.util.logging.config.file="D:/Program Files/Grid-Tools/FastDataMasker/logging.properties" -Xms100M -Xmx1000M -cp


  • 7.  RE: java.lang.OutOfMemoryError: Java heap space

    Posted Nov 08, 2019 02:02 AM
    ​Thanks Klaas,

    I have increased the  value of the parameter, -Xmx, in FDM bat file. If it will not work, will try the above option suggested by you "I used a SQL Server element in Javelin to do a direct update". Thanks for the quick response.


    ------------------------------
    Nitin Dixit
    ------------------------------