Spring

 View Only

 How do I use the Spring Cloud Stream App Starters - Jdbc source app to query 400 million records from avGreenplum table.

Adams Tan's profile image
Adams Tan posted Jun 12, 2018 05:18 PM

I'm using the jdbc(source) Spring Cloud Stream App Starters 

to query all data from a table in Greenplum Database. This jdbc source is connected to a processor(which performs some transformation) which is then connected to a sink.

 

The table contains 400 million records, and I encountered the out-of-memory 

error when I do a select all in the jdbc source app.

 

What are the attibutes of the jdbc app which i can set to 

do a batch query?

Daniel Mikusa's profile image
Daniel Mikusa

The configuration options for the JDBC starter are here.

 

https://docs.spring.io/spring-cloud-stream-app-starters/docs/Celsius.SR2/reference/htmlsingle/#spring-cloud-stream-modules-jdbc-source

 

I think you want:

jdbc.max-rows-per-poll Max numbers of rows to process for each poll. (Integer, default: 0)

 

Adams Tan's profile image
Adams Tan

What are the attributes of the jdbc app which I can use to limit the number of results per poll, so as to prevent the out-of-memory.