Deployment fails after cluster node restart

[Dirk Woywode]

Identity Manager 14.3
JBoss EAP 6.4.0
RHEL 7.3
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0
2 node cluster is configured as Unicast tcp ping with replication.

I have the following problem with my configuration. Any ideas on what missconfiguration should i look for or how should i go about debugging the cause are very much welcome.

Steps to reproduce the problem:
1. Start JBoss on node1; OK
2. Start JBoss on node2; OK
3. Shutdown JBoss on node2; OK
4. Start JBoss on node2;
Deployment fails with missing dependencies. deployments/iam_im.ear.failed contains:
21:02:11,183 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-7) JBAS010401: Bound JCA ConnectionFactory [java:/iam/im/rar/Workflow]21:02:11,193 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014612: Operation ("deploy") failed - address: ([("deployment" => "iam_im.ear")]) - failure description: {"JBAS014771: Services with missing/unavailable dependencies" => [    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.ServerCommandsEJB.Validator is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.ServerCommandsEJB]",    "jboss.deployment.subunit.\"iam_im.ear\".\"iam_im_identityminder_ejb.jar\".INSTALL is missing [jboss.deployment.subunit.\"iam_im.ear\".\"management_console.war\".deploymentCompleteService]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.WorkItemPvt_EJB.env.queue.wpUtilQueue is missing [jboss.naming.context.java.queue.wpUtilQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.WorkItemUpdatePvt_EJB.env.jms.wpConnectionFactory is missing [jboss.naming.context.java.jms.wpConnectionFactory]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.ServerAutomatedActivityMDBean.env.jms.wpConnectionFactory is missing [jboss.naming.context.java.jms.wpConnectionFactory]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.ResourcePvt_EJB.env.queue.wpEventQueue is missing [jboss.naming.context.java.queue.wpEventQueue]",    "jboss.deployment.subunit.\"iam_im.ear\".\"iam_im_wpServer.jar\".component.EventMDBean.CREATE is missing [jboss.ra.hornetq-ra]",    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.SubscriberMessageEJB.ValidatorFactory is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.SubscriberMessageEJB]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.MonitorPvt_EJB.env.jms.wpConnectionFactory is missing [jboss.naming.context.java.jms.wpConnectionFactory]",    "jboss.deployment.subunit.\"iam_im.ear\".\"manage_redirect.war\".INSTALL is missing [jboss.deployment.subunit.\"iam_im.ear\".\"redirect.war\".deploymentCompleteService]",    "jboss.deployment.subunit.\"iam_im.ear\".\"management_console.war\".INSTALL is missing [jboss.deployment.subunit.\"iam_im.ear\".\"user_console.war\".deploymentCompleteService]",    "jboss.deployment.subunit.\"iam_im.ear\".\"iam_im_wpServer.jar\".component.UtilityMDBean.CREATE is missing [jboss.ra.hornetq-ra]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.ServerAutomatedActivityMDBean.env.queue.wpUtilQueue is missing [jboss.naming.context.java.queue.wpUtilQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.RuntimeStatusDetailEJB.ValidatorFactory is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.RuntimeStatusDetailEJB]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.ServerAutomatedActivityMDBean.env.queue.wpEventQueue is missing [jboss.naming.context.java.queue.wpEventQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.SubscriberMessageEJB.ORB is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.SubscriberMessageEJB]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.UtilityMDBean.env.queue.wpServAutoActQueue is missing [jboss.naming.context.java.queue.wpServAutoActQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.AlertPvt_EJB.env.jms.wpConnectionFactory is missing [jboss.naming.context.java.jms.wpConnectionFactory]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.WorkItemUpdatePvt_EJB.env.queue.wpServAutoActQueue is missing [jboss.naming.context.java.queue.wpServAutoActQueue]",    "jboss.deployment.subunit.\"iam_im.ear\".\"redirect.war\".INSTALL is missing [jboss.deployment.subunit.\"iam_im.ear\".\"iam_im_identityminder_ejb.jar\".deploymentCompleteService]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.JobUpdatePvt_EJB.env.queue.wpUtilQueue is missing [jboss.naming.context.java.queue.wpUtilQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.ForceActionUpdatePvt_EJB.env.queue.wpServAutoActQueue is missing [jboss.naming.context.java.queue.wpServAutoActQueue]",    "jboss.deployment.unit.\"iam_im.ear\".deploymentCompleteService is missing [jboss.deployment.subunit.\"iam_im.ear\".\"ca-nim-sm.war\".deploymentCompleteService, jboss.deployment.subunit.\"iam_im.ear\".\"redirect.war\".deploymentCompleteService, jboss.deployment.subunit.\"iam_im.ear\".\"manage_redirect.war\".deploymentCompleteService, jboss.deployment.subunit.\"iam_im.ear\".\"user_console.war\".deploymentCompleteService, jboss.deployment.subunit.\"iam_im.ear\".\"iam_im_identityminder_ejb.jar\".deploymentCompleteService, jboss.deployment.subunit.\"iam_im.ear\".\"management_console.war\".deploymentCompleteService]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.JobUpdatePvt_EJB.env.queue.wpEventQueue is missing [jboss.naming.context.java.queue.wpEventQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.UtilityMDBean.env.jms.wpConnectionFactory is missing [jboss.naming.context.java.jms.wpConnectionFactory]",    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.WorkFlowCallBackEJB.Validator is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.WorkFlowCallBackEJB]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.UniqueIDUpdatePvt_EJB.env.queue.wpUtilQueue is missing [jboss.naming.context.java.queue.wpUtilQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.ScriptExecAsyncPvt_EJB.env.jms.wpConnectionFactory is missing [jboss.naming.context.java.jms.wpConnectionFactory]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.ForceActionUpdatePvt_EJB.env.queue.wpEventQueue is missing [jboss.naming.context.java.queue.wpEventQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.UniqueIDUpdatePvt_EJB.env.queue.wpServAutoActQueue is missing [jboss.naming.context.java.queue.wpServAutoActQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.WorkFlowCallBackEJB.HandleDelegate is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.WorkFlowCallBackEJB]",    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.ServerCommandsEJB.ValidatorFactory is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.ServerCommandsEJB]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.AlertPvt_EJB.env.queue.wpEventQueue is missing [jboss.naming.context.java.queue.wpEventQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.WorkItemUpdatePvt_EJB.env.queue.wpEventQueue is missing [jboss.naming.context.java.queue.wpEventQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.ServerCommandsEJB.ORB is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.ServerCommandsEJB]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.WorkItemPvt_EJB.env.jms.wpConnectionFactory is missing [jboss.naming.context.java.jms.wpConnectionFactory]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.ScriptExecutePvt_EJB.env.queue.wpEventQueue is missing [jboss.naming.context.java.queue.wpEventQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.EventPvt_EJB.env.jms.wpConnectionFactory is missing [jboss.naming.context.java.jms.wpConnectionFactory]",    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.RuntimeStatusDetailEJB.ORB is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.RuntimeStatusDetailEJB]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.GroupTypePvt_EJB.env.jms.wpConnectionFactory is missing [jboss.naming.context.java.jms.wpConnectionFactory]",    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.SubscriberMessageEJB.Validator is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.SubscriberMessageEJB]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.ResourcePvt_EJB.env.jms.wpConnectionFactory is missing [jboss.naming.context.java.jms.wpConnectionFactory]",    "jboss.deployment.subunit.\"iam_im.ear\".\"user_console.war\".INSTALL is missing [jboss.deployment.subunit.\"iam_im.ear\".\"ca-nim-sm.war\".deploymentCompleteService]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.ForceActionUpdatePvt_EJB.env.jms.wpConnectionFactory is missing [jboss.naming.context.java.jms.wpConnectionFactory]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.EventMDBean.env.jms.wpConnectionFactory is missing [jboss.naming.context.java.jms.wpConnectionFactory]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.ForceActionUpdatePvt_EJB.env.queue.wpUtilQueue is missing [jboss.naming.context.java.queue.wpUtilQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.WorkFlowCallBackEJB.ORB is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.WorkFlowCallBackEJB]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.ServerAutomatedActivityMDBean.env.queue.wpServAutoActQueue is missing [jboss.naming.context.java.queue.wpServAutoActQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.RuntimeStatusDetailEJB.HandleDelegate is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.RuntimeStatusDetailEJB]",    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.SubscriberMessageEJB.HandleDelegate is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.SubscriberMessageEJB]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.EventPvt_EJB.env.queue.wpEventQueue is missing [jboss.naming.context.java.queue.wpEventQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.UniqueIDUpdatePvt_EJB.env.jms.wpConnectionFactory is missing [jboss.naming.context.java.jms.wpConnectionFactory]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.ScriptExecutePvt_EJB.env.jms.wpConnectionFactory is missing [jboss.naming.context.java.jms.wpConnectionFactory]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.WorkItemUpdatePvt_EJB.env.queue.wpUtilQueue is missing [jboss.naming.context.java.queue.wpUtilQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.RuntimeStatusDetailEJB.Validator is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.RuntimeStatusDetailEJB]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.JobUpdatePvt_EJB.env.jms.wpConnectionFactory is missing [jboss.naming.context.java.jms.wpConnectionFactory]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.ScriptExecAsyncPvt_EJB.env.queue.wpEventQueue is missing [jboss.naming.context.java.queue.wpEventQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.ServerCommandsEJB.HandleDelegate is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.ServerCommandsEJB]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.GroupTypePvt_EJB.env.queue.wpEventQueue is missing [jboss.naming.context.java.queue.wpEventQueue]",    "jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.WorkFlowCallBackEJB.ValidatorFactory is missing [jboss.naming.context.java.comp.iam_im.iam_im_identityminder_ejb.WorkFlowCallBackEJB]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.MonitorPvt_EJB.env.queue.wpEventQueue is missing [jboss.naming.context.java.queue.wpEventQueue]",    "jboss.deployment.subunit.\"iam_im.ear\".\"iam_im_wpServer.jar\".component.ServerAutomatedActivityMDBean.CREATE is missing [jboss.ra.hornetq-ra]",    "jboss.naming.context.java.comp.iam_im.iam_im_wpServer.JobUpdatePvt_EJB.env.queue.wpServAutoActQueue is missing [jboss.naming.context.java.queue.wpServAutoActQueue]"]}

If node1 is down i can restart node2 without a problem. Also if node2 is down i can restart node1 without a problem.
The same happens if Start Node2, Start Node1, Stop Node1, Restart Node1 

Best regards,
Dirk

[Bill Patton]

There is a support ticket opened and the opinion is that this is a core jboss issue.
Maybe try to delete the /tmp/ /data/ directories and try again.

Perhaps there are others in the communities who have experience in doing this and we invite them to comment here also.

Another option may be to reach out to our partner HCL Technologies to see in what way they can assist further. The Enterprise Studio team of HCL can be reached at [e-mail]. https://www.hcltech.com/enterprise-studio

[Dirk Woywode]

The problem was solved by applying the latest JBoss EAP patch (6.4.22).