java.lang.OutOfMemoryError: Java heap space

[NITIN DIXIT]

​Hi Team,

I am trying to apply  the masking on 21 million records and using "Fixed Value " option. When I am executing the Masking job, its showing below error message. Could you please help me on this.

>2019-11-06 01:16:44.514 Cursor Fetchsize:1000
>2019-11-06 01:16:44.514 Autocommit:false
>java.lang.OutOfMemoryError: Java heap space
>Dumping heap to java_pid6480.hprof ...
>Unable to create java_pid6480.hprof: Permission denied
>2019-11-06 01:17:03.899 java.lang.OutOfMemoryError: Java heap space
> at com.microsoft.sqlserver.jdbc.TDSPacket.<init>(IOBuffer.java:6215)
> at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6405)
> at com.microsoft.sqlserver.jdbc.TDSCommand.detach(IOBuffer.java:7249)
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2925)
> at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:248)
> at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:223)
> at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:2074)
> at com.grid_tools.products.datamasker.SQLUpdate.batchPreparedMasking(SQLUpdate.java:4771)
> at com.grid_tools.products.datamasker.SQLUpdate.run(SQLUpdate.java:5316)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>2019-11-06 01:17:04.071 
>2019-11-06 01:17:04.243 masking complete at 2019.11.06 01:17:03.899 PST
>2019-11-06 01:17:04.243
>Allocated memory      :910,336k
>2019-11-06 01:17:04.243 Approx max memory used:1,132,543k (124% of allocated)
>2019-11-06 01:17:04.260 exit value = 0

------------------------------
Nitin Dixit
------------------------------

[Klaas-Jan Lute]

Hi Nitin,

I have a similar experience with Fast Data Masker's fixed value masking in a large table. In the end I used a SQL Server element in Javelin to do a direct update. What took almost an entire day to mask was now done in a couple of minutes. 

Best regards,
Klaas-Jan

[Billy Keefer]

You need to increase the value of the parameter, -Xmx, in your FDM bat file.

[Klaas-Jan Lute]

Increasing both memory parameters was the first thing I did but ít did not improve the performance significantly.

[NITIN DIXIT]

​Hi Billy,

Are you talking about  "bigfdm64" batch file? Could you please help me with the maximum value which we can update here. Thanks,

@echo off
if "%1"=="" goto error
if "%2"=="" goto error
if "%5"=="high" goto highlabel
if "%5"=="HIGH" goto highlabel
java -Xms100M -Xmx%1M -jar Fastdatamasker.jar %2 %3 %4
goto end
:highlabel
start /high java -Xms100M -Xmx%1M -jar gtsdm.jar %2 %3 %4
goto end
:error
echo.
echo ** ERROR SPECIFYING PARAMETERS
echo.
echo ** TO RUN:  BIGFDM memory_size_in_meg_for_example_1000 connection.txt map.csv [options.txt] [high]
:end

------------------------------
Nitin Dixit
------------------------------

[Klaas-Jan Lute]

FastDataMasker creates a .bat file (along with the options- and definitionfile) in which the memory usage is configured. The bat file starts with the following string:
java -Dfile.encoding=UTF-8 -Djava.util.logging.config.file="D:/Program Files/Grid-Tools/FastDataMasker/logging.properties" -Xms100M -Xmx1000M -cp

[NITIN DIXIT]

​Thanks Klaas,

I have increased the  value of the parameter, -Xmx, in FDM bat file. If it will not work, will try the above option suggested by you "I used a SQL Server element in Javelin to do a direct update". Thanks for the quick response.


------------------------------
Nitin Dixit
------------------------------

[Rakesh Jogi]

This error appears to occur when the masking job runs out of Java heap space while processing a very large volume of data. During execution, the JVM needs to keep many objects in memory, especially while performing batch updates. Thank you for providing the data. Your heap allocation may not be sufficient to handle this job or workload. The logs show that approximately 1.1 GB of heap is utilized for the job. This number exceeds your allocated memory. This actually results in a java.lang.OutOfMemoryError: Java heap space.

From the attached stack trace, it is clear that the failure occurs during executeBatch() in the SQL Server JDBC driver. JDBC driver seems not to have write access. The message  "Permission denied" is actually indicating that the JVM does not have write access to the current directory. To resolve this issue, it is recommended to increase the JVM heap size.  We can fix the permission issue by specifying a valid path, such as "-XX:HeapDumpPath". By doing so, the heap dumps can be generated even if the problem persists.  

To learn more interesting details about the OutOfMemoryError: Java heap space error, You can check out this blog How to Solve OutOfMemoryError: Java heap space. This will give you more insights about this Java Heap Space error.

-------------------------------------------