Handling Spring boot gemfire + stale data + missing diskstore

[Vaidhyanathan Pranatharthiharan]

We have 4 physical servers which has 1 spring boot gemfire locator and server on each of the server . We loaded data into the cache and stopped one server and tried to start the stopped one . In the logs we are seeing its failing with an error for the partitioned regions which has redundancy level conifgured as 2

My persistent id:

<<server id that is waiting>>

 

<<ooffline members with potentially new data>>>

<<the other server>>

 

<Redundany logger >One of the regions (region name) has potentially stale data Buckets[1...20] are waiting for other memebers to recover the stale data

 

Use the "gfsh show missing-disk-stores" command to see all disk stores that are being waited on by other members.

 

How do we handle this error ? Since we are using spring boot gemfire and not gemfire installable is ther e a way to revoke the diskstore ?

 

Secondly , one more issue that i observed is if 2 servers have server group 'A' and other 2 servers have server group 'B' and id all the servers are in a cluster , if servers under server group B is down its not synching the region data under group 'B' to group 'A' . My understanding is if its a cluster it has to do that . Server group 'A' has spring boot gemfire server jar with rregions A,B and server group 'B' has spring boot gemfire server jar with regions c,d . If server group B is down then a copy of regions c,d should be under a,b if redundancy level is configured ?

[Juan Ramos]

Hello Vaidhyanathan,

 

Thanks for contacting the Pivotal Support Community!.

That said, in order to avoid the issue in the first place, you should orderly start and shutdown the members with persistent data; please have a look at Start Up and Shut Down with Disk Stores [1] for further details. To answer your first question, you can still use the revoke missing-disk-store [2] command, even when using a spring-boot application, you just need to make you manually delete the folders afterwards as thee member won't be able to re-join the distributed system with a revoked disk store, you can get more information about this in Handling Missing Disk Stores [3].

Regarding your second question, I don't fully understand the issue... I'd suggest to create a new community post and add more details so we can help (a new post basically to avoid polluting this one with unrelated issues/questions).

Hope this helps.

Best regards.

 

[1]: https://gemfire.docs.pivotal.io/98/geode/managing/disk_storage/starting_system_with_disk_stores.html

[2]: https://gemfire.docs.pivotal.io/98/geode/tools_modules/gfsh/command-pages/revoke.html

[3]: https://gemfire.docs.pivotal.io/98/geode/managing/disk_storage/handling_missing_disk_stores.html

[Vaidhyanathan Pranatharthiharan]

Thank you .In our case we are seeing that the data synch is not happening asychronously while we restart the servers​. We are starting one server agter other in sequence.

​

Most of the time the first server is stuck with stale data error and the region creation is blocked .

​

Also, how do we identify which region has most recent data?

 

[Juan Ramos]

Hello Vaidhyanathan,

 

I'm not sure I'm following you here... if you shutdown and start your members following the steps described in Start Up and Shut Down with Disk Stores [1] then you shouldn't hit this issue at all. During the startup the members will synchronise between each other and decide which has the most up to date data, preventing you from having to execute any manual steps.

 

>> Also, how do we identify which region has most recent data?

I can't think of an easy way of doing this when the members are offline, other than manually checking the last modified time of the persistent files on disk.

 

 

Best regards.

 

[1]: https://gemfire.docs.pivotal.io/98/geode/managing/disk_storage/starting_system_with_disk_stores.html