VMmark4 benchmark fails in NoSQLBench Workload setup

5. RE: VMmark4 benchmark fails in NoSQLBench Workload setup

Recommend

Sivakumar Alagumalai

Posted Apr 06, 2026 05:47 PM
Edited by Sivakumar Alagumalai Apr 06, 2026 05:48 PM

on Delete Tiles, we followed the pprocedure described in user guide. what we followed was

1) delete the PVCs using the pod_pvc_cleanup.sh

2) followed by delete tiles using "vmmark4service --mode delete_all_vmmark4"

this is exactly we followed. let us know if this is right procedure.

2) on the file upload, yes will work with my team here and update you in case if we can enable access to you to download files from our sharepoint

-------------------------------------------

Original Message

Original Message:
Sent: Apr 06, 2026 05:17 PM
From: David Morse
Subject: VMmark4 benchmark fails in NoSQLBench Workload setup

1) It's very strange that changing to CPUs with more cores would cause problems even at 1 tile when you were able to run 1-2 tiles with the lower-core count CPUs.

2) Okay, 3 NVMe disks may become a bottleneck at higher tile counts, but I'm glad to hear you have been have to run 2. What were the scores for each?

3) When you deleted the tiles, did you delete them through the vSphere Client, or did you follow the "Delete the VMmark 4 Tiles" section in the User's Guide, namely running the "vmmark4service --mode delete_all_vmmark4" command on the prime client?

4) and 5) Does your network/company provide another file sharing mechanism that I could use to download them?

Thanks,
David

Original Message:
Sent: Apr 06, 2026 02:59 PM
From: Sivakumar Alagumalai
Subject: VMmark4 benchmark fails in NoSQLBench Workload setup

Hi David,

Thanks for the response to our issues. appreciate your time and effort on this.

1) Initially we could run 1 Tiles and 2 Tiles and got the results. when we decided to move to 4 Tiles, we wanted to change the CPU because we felt the for 2 tiles itself the cpu utilization was about more than 100%...so we changed CPU to higher core cpus and then provisioned 4 tiles. After this, so far, we could not even run for 1 Tile. for every try the same issue has been experienced

2) on the storage, we setup an iSCSI storage of 25TB capacity with 3 NVMe SSD disks

3) after this, I have even deleted the entire Tiles, re provisioned the 4 Tiles, still the same issue.

4) to your "box" location, we will not be able to upload any files because it is restricted in our network

5) is there any other way, I can share the provision files?

Original Message:
Sent: Apr 06, 2026 01:34 PM
From: David Morse
Subject: VMmark4 benchmark fails in NoSQLBench Workload setup

Hi Sivakumar,

We have not seen this particular error before. Some questions:

From the screenshot, it appears the restore failed on at least the first two tiles (0 and 1). You say you're trying to run 4 tiles. Have you successfully run 1, 2, and/or 3 tiles prior to this?
NoSQLBench is disk I/O intensive. Can you send me the details of your storage subsystem (for example: # of disks, vSAN vs. SAN, RAID level, etc.)?
You said the log file upload failed. On your primeclient, please go to the provision directory (cd /root/VMmark4/provision) and zip up all of the provisioning folders (zip -r provision.zip Provision*). I will send you a separate private reply with an alternative way to upload the zip file.

Thanks,
David

Original Message:
Sent: Apr 05, 2026 01:18 AM
From: Sivakumar Alagumalai
Subject: VMmark4 benchmark fails in NoSQLBench Workload setup

we are trying to run VMmark4.0.3 benchmark @ 4 tiles. when initiated the benchmark, it fails in workload setup with the following error. we tried re-provisioning the VMmark4 tiles, still the issue persists. attached the log file for further analysis. Please let us know if any one of you experienced this issue and resolved from your end, how this issue can be resolved? Appreciate your input on this. Thanks.

Since my log file upload is failing, this is the log file content:

Fri Apr 3 03:05:39 PM PDT 2026 : Starting NoSQLBench restore v. 04152025, writing to /NoSQLBenchA0_restore.txt

Fri Apr 3 03:05:39 PM PDT 2026 : cassandra is running on NoSQLBenchA.

Fri Apr 3 03:05:40 PM PDT 2026 : Node NoSQLBenchA reports all nodes up and normal.

Fri Apr 3 03:05:40 PM PDT 2026 : cassandra is running on NoSQLBenchB.

Fri Apr 3 03:05:41 PM PDT 2026 : Node NoSQLBenchB reports all nodes up and normal.

Fri Apr 3 03:05:42 PM PDT 2026 : cassandra is running on NoSQLBenchC.

Fri Apr 3 03:05:43 PM PDT 2026 : Node NoSQLBenchC reports all nodes up and normal.

Fri Apr 3 03:05:44 PM PDT 2026 : Restoring snapshots on all nodes.

Fri Apr 3 03:19:41 PM PDT 2026 : Restoring snapshots was unsuccessful:

/var/lib/cassandra/data/baselines/keyvalue-b7ecac202f7b11f198d4838bbb6a0e70

Datacenter: datacenter1

=======================

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

-- Address Load Tokens Owns (effective) Host ID Rack

UN 198.18.4.19 248.25 KiB 16 31.6% 797cdb7e-3dd7-40e7-b31b-4562a6586a36 rack1

UN 198.18.4.20 249.75 KiB 16 35.7% a63ff010-aa67-464e-b152-38103420f026 rack1

UN 198.18.4.18 240.66 KiB 16 32.7% 7f37d39e-0e82-41d8-bb97-94a056653605 rack1

Doing a quick import - skipping sstable verification and row cache invalidation

error: Directory /var/lib/cassandra/data/baselines/keyvalue-b7ecac202f7b11f198d4838bbb6a0e70/snapshots/nosqlsnapA does not exist

-- StackTrace --

java.lang.RuntimeException: Directory /var/lib/cassandra/data/baselines/keyvalue-b7ecac202f7b11f198d4838bbb6a0e70/snapshots/nosqlsnapA does not exist

at org.apache.cassandra.db.SSTableImporter.getSSTableListers(SSTableImporter.java:238)

at org.apache.cassandra.db.SSTableImporter.importNewSSTables(SSTableImporter.java:75)

at org.apache.cassandra.db.ColumnFamilyStore.importNewSSTables(ColumnFamilyStore.java:741)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)

at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)

at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)

at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)

at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)

at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)

at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)

at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)

at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)

at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)

at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)

at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)

at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)

at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)

at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)

at sun.rmi.transport.Transport$1.run(Transport.java:200)

at sun.rmi.transport.Transport$1.run(Transport.java:197)

at java.security.AccessController.doPrivileged(Native Method)

at sun.rmi.transport.Transport.serviceCall(Transport.java:196)

at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)

at java.security.AccessController.doPrivileged(Native Method)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:750)

Doing a quick import - skipping sstable verification and row cache invalidation

error: Directory /var/lib/cassandra/data/baselines/keyvalue-b7ecac202f7b11f198d4838bbb6a0e70/snapshots/nosqlsnapB does not exist

-- StackTrace --

java.lang.RuntimeException: Directory /var/lib/cassandra/data/baselines/keyvalue-b7ecac202f7b11f198d4838bbb6a0e70/snapshots/nosqlsnapB does not exist