VMware GemFire

 View Only

 Deserialization exception on long running OQL

Unmesh Joshi's profile image
Unmesh Joshi posted Mar 05, 2019 12:49 PM

We are using some gemfire queries over all the data loaded on gemfire (its around 80 Gb). We started getting following excpetion recently. Whats the relevance of deserialization of EntryEventImpl?

 

g.apache.geode.cache.execute.FunctionException: org.apache.geode.cache.query.QueryInvocationTargetException

 

    at org.apache.geode.management.internal.beans.QueryDataFunction$LocalQueryFunction.execute(QueryDataFunction.java:545)

 

    at org.apache.geode.internal.cache.PartitionedRegionDataStore.executeOnDataStore(PartitionedRegionDataStore.java:2911)

 

    at org.apache.geode.internal.cache.partitioned.PartitionedRegionFunctionStreamingMessage.operateOnPartitionedRegion(PartitionedRegionFunctionStreamingMessage.java:98)

 

    at org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:332)

 

    at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374)

 

    at org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:440)

 

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

 

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

 

    at org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:665)

 

    at org.apache.geode.distributed.internal.DistributionManager$9$1.run(DistributionManager.java:1111)

 

    at java.lang.Thread.run(Thread.java:748)

 

Caused by: org.apache.geode.cache.query.QueryInvocationTargetException

 

    at org.apache.geode.cache.query.internal.AttributeDescriptor.readReflection(AttributeDescriptor.java:119)

 

    at org.apache.geode.cache.query.internal.AttributeDescriptor.read(AttributeDescriptor.java:82)

 

    at org.apache.geode.cache.query.internal.PathUtils.evaluateAttribute(PathUtils.java:71)

 

    at org.apache.geode.cache.query.internal.CompiledPath.evaluate(CompiledPath.java:139)

 

    at org.apache.geode.cache.query.internal.CompiledIteratorDef.evaluateCollection(CompiledIteratorDef.java:174)

 

    at org.apache.geode.cache.query.internal.RuntimeIterator.evaluateCollection(RuntimeIterator.java:111)

 

    at org.apache.geode.cache.query.internal.CompiledSelect.doNestedIterations(CompiledSelect.java:809)

 

    at org.apache.geode.cache.query.internal.CompiledSelect.doNestedIterations(CompiledSelect.java:841)

 

    at org.apache.geode.cache.query.internal.CompiledSelect.doIterationEvaluate(CompiledSelect.java:699)

 

    at org.apache.geode.cache.query.internal.CompiledSelect.evaluate(CompiledSelect.java:543)

 

    at org.apache.geode.cache.query.internal.CompiledGroupBySelect.evaluate(CompiledGroupBySelect.java:162)

 

    at org.apache.geode.cache.query.internal.CompiledGroupBySelect.evaluate(CompiledGroupBySelect.java:47)

 

    at org.apache.geode.cache.query.internal.DefaultQuery.executeUsingContext(DefaultQuery.java:558)

 

    at org.apache.geode.internal.cache.PRQueryProcessor.executeQueryOnBuckets(PRQueryProcessor.java:382)

 

    at org.apache.geode.internal.cache.PRQueryProcessor.executeSequentially(PRQueryProcessor.java:347)

 

    at org.apache.geode.internal.cache.PRQueryProcessor.executeQuery(PRQueryProcessor.java:138)

 

    at org.apache.geode.internal.cache.PartitionedRegionQueryEvaluator.executeQueryOnLocalNode(PartitionedRegionQueryEvaluator.java:909)

 

    at org.apache.geode.internal.cache.PartitionedRegionQueryEvaluator.executeQueryOnRemoteAndLocalNodes(PartitionedRegionQueryEvaluator.java:327)

 

    at org.apache.geode.internal.cache.PartitionedRegionQueryEvaluator.queryBuckets(PartitionedRegionQueryEvaluator.java:445)

 

    at org.apache.geode.internal.cache.PartitionedRegion.doExecuteQuery(PartitionedRegion.java:1880)

 

    at org.apache.geode.internal.cache.PartitionedRegion.executeQuery(PartitionedRegion.java:1809)

 

    at org.apache.geode.cache.query.internal.DefaultQuery.execute(DefaultQuery.java:361)

 

    at org.apache.geode.cache.query.internal.DefaultQuery.execute(DefaultQuery.java:319)

 

    at org.apache.geode.management.internal.beans.QueryDataFunction$LocalQueryFunction.execute(QueryDataFunction.java:540)

 

    ... 10 more

 

Caused by: java.lang.reflect.InvocationTargetException

 

    at sun.reflect.GeneratedMethodAccessor429.invoke(Unknown Source)

 

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

 

    at java.lang.reflect.Method.invoke(Method.java:498)

 

    at org.apache.geode.cache.query.internal.AttributeDescriptor.readReflection(AttributeDescriptor.java:100)

 

    ... 33 more

 

Caused by: org.apache.geode.InternalGemFireError: unexpected typeCode: 44

 

    at org.apache.geode.internal.InternalDataSerializer.decodePrimitiveClass(InternalDataSerializer.java:1880)

 

    at org.apache.geode.DataSerializer.readClass(DataSerializer.java:264)

 

    at org.apache.geode.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2389)

 

    at org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2813)

 

    at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2961)

 

    at org.apache.geode.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:99)

 

    at org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1911)

 

    at org.apache.geode.internal.cache.EntryEventImpl.deserialize(EntryEventImpl.java:1904)

 

    at org.apache.geode.internal.cache.VMCachedDeserializable.getDeserializedValue(VMCachedDeserializable.java:134)

 

    at org.apache.geode.internal.cache.EntrySnapshot.getRawValue(EntrySnapshot.java:113)

 

    at org.apache.geode.internal.cache.EntrySnapshot.getRawValue(EntrySnapshot.java:101)

 

    at org.apache.geode.internal.cache.EntrySnapshot.getValue(EntrySnapshot.java:130)

 

    ... 37 more

Juan Cassella's profile image
Juan Cassella

Hello Unmesh,

 

Thanks for contacting the Pivotal Support Community!!.

My name is Juan José Ramos Cassella, and I am a GemFire Technical Support Engineer with Pivotal.

That said, this Exception is thrown by GemFire whenever it can't deserialise an actual entry from the region within the cache.

From this particular stack trace it's clear that GemFire is trying to determine the primitive data type (boolean, char, short, byte, float, int, etc) during deserialisation but it can't... it's hard to say why you got to this point without looking at logs and statistics?, would you mind opening a Support Ticket through the Pivotal Support Portal so we can assist you?.

As a workaround, you can try to shutdown the entire cluster (including clients as they might store some stale metadata) and start it up again.

Best regards.