Redis

 View Only

 redis.clients.jedis.exceptions.JedisMovedDataException: MOVED

Franck Bideau's profile image
Franck Bideau posted Feb 04, 2019 09:08 PM

My SpringBoot app is connecting to a Redis sharded cluster in Azure for better scalability.

Using Jedis to set up the connection and the RedisClusterConfiguration class, my app is unable to perform

Redis any commands: redis.clients.jedis.exceptions.JedisMovedDataException: MOVED

If Jedis is not proper framework for a sharded Redis cluster, what is the alternative framework and where could I find a reference implementation? 

Daniel Mikusa's profile image
Daniel Mikusa

With Spring Data & Boot, I don't think you need to do anything but set your boostrap cluster node IP/ports. Have you taken a look at this example?

 

https://github.com/spring-projects/spring-data-examples/tree/master/redis/cluster

Franck Bideau's profile image
Franck Bideau

My config class:

 

@Bean

public RedisConnectionFactory connectionFactory() {

return new JedisConnectionFactory(getClusterConfiguration());

}

 

@Bean

public RedisClusterConfiguration getClusterConfiguration() {

Map<String, Object> source = Maps.newHashMap();

source.put("spring.redis.cluster.nodes", "xxxx.redis.cache.windows.net:6380");

source.put("spring.redis.cluster.timeout", "30");

source.put("spring.redis.cluster.password", "xxxxxx");

return new RedisClusterConfiguration(new MapPropertySource("RedisClusterConfiguration", source));

}

 

Franck Bideau's profile image
Franck Bideau

yeah, but Azure does not give me access to the underlying nodes . Azure Redis has a load balancer in front of the sharded cluster.

 

This does not work either:

@Bean

public RedisConnectionFactory connectionFactory() {

JedisConnectionFactory conn = new JedisConnectionFactory();

conn.setHostName("ds1-redic-poc.redis.cache.windows.net");

conn.setPassword("GfZ417sFRDlMBR7yqsO0611oaEN5L6SiANjQujoz7WE=");

conn.setPort(6380);

conn.setUseSsl(true);

conn.setUsePool(true);

conn.setTimeout(socketTO);

 

JedisPoolConfig config = new JedisPoolConfig();

config.setMaxIdle(maxIdle);

config.setMaxTotal(maxTotal);

config.setMinIdle(minIdle);

config.setMaxWaitMillis(maxWaitMillis);

config.setBlockWhenExhausted(blockWhenExhausted);

 

conn.setPoolConfig(config);

 

return conn;

}

 

Does Springboot support lettuce?

 

 

Daniel Mikusa's profile image
Daniel Mikusa

>Does Springboot support lettuce?

 

Yes. It looks like it should be the default, according to this. At least in 2.0 + 2.1.

 

https://docs.spring.io/spring-boot/docs/2.1.2.RELEASE/reference/htmlsingle/#boot-features-redis

 

If it's not being picked up check and see if you have Jedis being pulled in somewhere and maybe try excluding the dependency. Also, make sure Lettuce is being pulled in as a dependency and not excluded. If it's on the class path it should use Lettuce.

Daniel Mikusa's profile image
Daniel Mikusa

What's your Spring Boot version? I can put something together.

Franck Bideau's profile image
Franck Bideau

Daniel - Managing the library dependencies is not as smooth as that. A full reference iplementation/sample project would be very helpful.

 

 

org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.redis.connection.RedisConnectionFactory]: Factory method 'connectionFactory' threw exception; nested exception is java.lang.NoClassDefFoundError: com/lambdaworks/redis/RedisException

 

--- gradle build ----

compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-redis'

compile group: 'com.lambdaworks', name: 'lettuce' , version: '2.3.3'

compile group: 'io.lettuce', name: 'lettuce-core', version: '5.0.0.RELEASE'

compile 'redis.clients:jedis'

 

--- config class --

@Bean

public PdfLocationRepositoryCustom pdfLocationRepositoryImpl(){

return new PdfLocationRedisRepositoryImpl(redisTemplate());

}

 

@Bean

public RedisConnectionFactory connectionFactory() {

return new LettuceConnectionFactory(getRedisClusterConfiguration());

//return new JedisConnectionFactory(getRedisClusterConfiguration());

}

 

 

@Bean

public RedisClusterConfiguration getRedisClusterConfiguration() {

Map<String, Object> source = Maps.newHashMap();

source.put("spring.redis.cluster.nodes", "ds1-redic-poc.redis.cache.windows.net:6380");

source.put("spring.redis.cluster.timeout", "30");

//source.put("spring.redis.cluster.max", ?);

source.put("spring.redis.cluster.password", "xxxxxx");

//source.put("spring.redis.cluster.max-redirects", ?);

return new RedisClusterConfiguration(new MapPropertySource("RedisClusterConfiguration", source));

 

}

 

@Bean

public RedisTemplate<String, Object> redisTemplate() {

RedisTemplate<String, Object> template = new RedisTemplate<>();

template.setConnectionFactory(connectionFactory());

template.setKeySerializer(new StringRedisSerializer());

template.setHashValueSerializer(new GenericToStringSerializer<>( Object.class ));

return template;

}

 

 

 

 

 

Franck Bideau's profile image
Franck Bideau

I am getting actually this exception with the above build

Factory method 'connectionFactory' threw exception; nested exception is java.lang.NoClassDefFoundError: com/lambdaworks/redis/AbstractRedisClient

Franck Bideau's profile image
Franck Bideau

With all these libs checked in gradle:

compile group: 'org.springframework.boot', name: 'spring-boot-starter-data-redis'

//compile group: 'org.apache.commons', name: 'commons-pool2'

compile group: 'com.lambdaworks', name: 'lettuce' , version: '2.3.3'

compile group: 'biz.paluch.redis', name: 'lettuce', version: '4.0.Final'

compile group: 'io.lettuce', name: 'lettuce-core', version: '5.0.0.RELEASE'

compile 'org.springframework.data:spring-data-redis'

compile 'redis.clients:jedis'

 

Franck Bideau's profile image
Franck Bideau

it looks like this class org.springframework.data.redis.connection.lettuce is missing

import com.lambdaworks.redis.resource.ClientResources;

 

 

Franck Bideau's profile image
Franck Bideau

1.5.9.RELEASE

 

Daniel Mikusa's profile image
Daniel Mikusa

This worked for me. It's using the latest 1.5.x. It is also not connecting to a cluster as I don't have one to test against. It should get you past the dependency conflicts though, and you can configure the ConnectionFactory from there.

 

Hope that helps!

Daniel Mikusa's profile image
Daniel Mikusa

Hmm, it seems like there could be multiple things that cause this. I did find this Github issue which seems to indicate that Lettuce 4.2+ supports Azure Redis Clusters. If you're on 4.5 like in my demo app that should be good enough.

 

https://github.com/lettuce-io/lettuce-core/issues/246

 

You might need to hunt for more clues as to what's failing. Maybe try turning up the log level for `org.springframework.data` and/or Lettuce (not sure the package name off hand). Similarly, you might try `tcpdump` or Wireshark to capture the traffic and dig in more, although that's challenging if TLS is involved.

 

Aside from that Pivotal doesn't support Lettuce itself, so if there is a core issue with it, I would suggest opening a Github issue. The author seems quite active.

 

Hope that helps!

Franck Bideau's profile image
Franck Bideau

Thanks looking at it. Noticed I was using an older version

compile group: 'biz.paluch.redis', name: 'lettuce', version: '4.0.0.Final'

 

Franck Bideau's profile image
Franck Bideau

Finally getting my app with Lettuce running in PCF / Azure.

Facing now a new issue and investigating:

Caused by: org.springframework.data.redis.RedisSystemException: Redis exception; nested exception is com.lambdaworks.redis.RedisException: Cannot retrieve initial cluster partitions from initial URIs [RedisURI [host='ds1-redic-poc.redis.cache.windows.net', port=6380]]

Franck Bideau's profile image
Franck Bideau

Ok - I got it to connect now. Needed to set SSL to true. Getting a strange memory exception now:

com.lambdaworks.redis.RedisException: io.netty.handler.codec.EncoderException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16776960 byte(s) of direct memory (used: 66816, max: 10485760)

Daniel Mikusa's profile image
Daniel Mikusa

OK, looks like it's using Direct Memory. The Java buildpack defaults the max direct memory to 10M. It looks like you will need more than that. You can bump that up and it should help. See the two sections here.

 

https://github.com/cloudfoundry/java-buildpack/blob/master/docs/jre-open_jdk_jre.md#java-options

Daniel Mikusa's profile image
Daniel Mikusa

From what I read earlier, it didn't seem like there was a standard in terms of how servers are configured. It could be something MS is doing in their implementation that's just not compatible with the clients. Java is a big space though so you'd think they could provide one Java Redis client that works. Jedis & Lettuce are two popular clients, so if neither works, it sounds like it's time to check with Microsoft Support & see what they have to say.

 

Post back if you're able to work out a solution or have more questions. Happy to help where we can from the Pivotal side.

Franck Bideau's profile image
Franck Bideau

Thanks Dan - Makes sense. I will give it a shot this morning.

Franck Bideau's profile image
Franck Bideau

Dan

 

Bad news. I am back to square one . Now that I have Lettuce connecting to an Azure Redis cache with 2 shards, I am getting the same exception I was getting with the other framework, Jedis:

org.springframework.data.redis.RedisSystemException: Error in execution; nested exception is com.lambdaworks.redis.RedisCommandExecutionException: MOVED 13292 52.167.224.109:15003

If I connect my app to an Azure Redis cache with 1 shard, I am able to run different CRUD operations against it. (I also tried to use the non SSL port and that did not help. )

It is very disappointing as I followed Microsoft's recommended solution to use Lettuce.

A tutorial like this one (https://docs.microsoft.com/en-us/java/azure/spring-framework/configure-spring-boot-initializer-java-app-with-redis-cache?view=azure-java-stable) is helpful for a simple initial deployment but is not enough to demonstrate how Azure cache can support an Elastic Cache solution when 

Azure does not provide a mechanism to access the underlying Redis nodes that seemed to be needed by these frameworks. (https://github.com/lettuce-io/lettuce-core/blob/5.1.3.RELEASE/src/test/java/io/lettuce/examples/ConnectToMasterSlaveUsingElastiCacheCluster.java)

 

THoughts?

 

Unless Microsoft is able to demonstrate with a reference implementation that we can use the Elastic Cache, we are going to have to look at the alternative approach with Redis Lab on our side.

 

Any others thoughts on your side?

 

Franck Bideau's profile image
Franck Bideau

I have been with MSFT support on this issue since day one :-)

I am waiting for them as well to answer these questions

Franck Bideau's profile image
Franck Bideau

Dan

 

Do you know if there is a way to configure the LettuceConnection with both a cluster and a pool object?

 

LettuceConnectionFactory clientConfig = new LettuceConnectionFactory(lettucePool); // DefaultLettucePool

LettuceConnectionFactory factory = new LettuceConnectionFactory(clusterConfig); //RedisClusterConfiguration

 

 

Oscar Pastas's profile image
Oscar Pastas

Dear All,

I appreciate if you could provide if you could get Jedis or Lettuce working with Azure Redis cache service in Pivotal Cloud Foundry to create pooling configuration. Please advice if Jedis/Lettuce is supported in PCF ?

 

I am using following configuration:

 

 private JedisPoolConfig buildPoolConfig() {

    final JedisPoolConfig poolConfig = new JedisPoolConfig();

    poolConfig.setMaxTotal(20);

    poolConfig.setMaxIdle(20);

    poolConfig.setMinIdle(5);

    poolConfig.setTestOnBorrow(true);

    poolConfig.setTestOnReturn(true);

    poolConfig.setTestWhileIdle(true);

    poolConfig.setMinEvictableIdleTimeMillis(Duration.ofSeconds(20).toMillis());

    poolConfig.setTimeBetweenEvictionRunsMillis(Duration.ofSeconds(20).toMillis());

    poolConfig.setNumTestsPerEvictionRun(-1);

    poolConfig.setBlockWhenExhausted(true);

    return poolConfig;

 

Also using in the application.yml:

 

redis:

  jedis:

   pool:

    max-active: 3

    max-idle: 3

    max-wait: 1

    min-idle: 1

  timeout: 3000

 

Thanks in advance.

 

Best Regards,

 

OP

Daniel Mikusa's profile image
Daniel Mikusa

@Oscar P - There is nothing specific to PCF in how you configure a Jedis/Lettuce connection pool. It is the same process on PCF as on your laptop or on a server/VM. The only difference is that you may choose to use different values for development on your local machine vs production on PCF.

 

https://docs.spring.io/spring-boot/docs/2.2.x/reference/html/spring-boot-features.html#boot-features-redis

 

See also all the properties you can set to configure things (Find in Page "redis" on the link below):

 

https://docs.spring.io/spring-boot/docs/2.2.x/reference/html/appendix-application-properties.html#data-properties

 

In addition, you can choose to get Redis configuration from a bound service. That is typically only for things like host, port and creds. Your pool configuration would not typically come from the bound service.

 

As an FYI, if you want to pull bound service info, you can use https://github.com/pivotal-cf/java-cfenv. That's the recommended approach.

 

Hope that helps!

 

PS. Please start a new thread if you have more questions.