There are a couple of things you can do:
1. You can use the keytool to delete the certificates (similar to importing them). First you will want to use keytool -list to list the aliases in the keystore, then delete them as in the following example (which shows deleting the 'wasp' alias certificate):
keytool -delete -alias wasp -keystore wasp_keystore.jks.
2. You can stop the wasp probe, delete the existing keystore, and restart wasp. Then, use the Probe Utility (CTRL+P in infrastructure manager) to run the callback called "ssl_reinitalize_keystore" vs. the wasp probe. This will generate a new keystore (you must provide a new password.)