We face this issue on and off with our IdentityMinder application which is running on a cluster of two servers. We frequently run into issues with tasks getting stuck in a 'In Progress' state.
I have had a look at the tech articles http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec498092.aspx & http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec615197.aspx. While they speak about how to provide access for tasks to be resubmitted individually, I need to know a way to resubmit a mass number of tasks. To illustrate with an example, on some days we have thousands of tasks getting stuck it is not possible for us to resubmit 3000-5000 tasks individually.
I need to figure out a way to -
a) Resubmit tasks in a 'mass' manner
b) Find the root cause for these tasks getting stuck and provide a long term fix
On running the following query on our DB server - select * from tasksession12_5 WHERE (state NOT IN (128, 512)) I get almost 450K rows. I don't necessarily need to resubmit all these tasks, I need find a way to resubmit tasks stuck for a selected date.
Also we have the following row counts for the various tables in the Task DB. I understand that these are really high in number, but is there a specific rule based on which I need to calculate what is the max limit on the number rows for the below tables before I purge these -
tasksession12_5 - 722715 rows
objects12_5 - 5519587 rows
lock12_5 - 5351804 rows
event12_5 - 4076086 rows
Note : This is from one of the DB servers, the other DB server has more or less a similar number.
Could you please share your version of IdentityMinder along with the version of your application server ?
IdentityMinder V12.6 SP2 and Jboss 5.1
Please try the below steps which should solve your issue.
Hi, we are already using SQL and not the hypersonic db.
Unfortunately, there's no simple way to resubmit all of the stuck tasks. In other cases where this has occurred, it often is counter productive to resubmit these tasks, as people have submitted new changes assuming that the original task is "dead" You can submit an idea for an enhancement request to have this considered for inclusion in a future version.
As for root cause, there are a couple of things to look at. First, what sort of tasks were running when this bottleneck occurred?
Was an explore and correlate just run? This is the most common cause of a large number of in progress tasks. When a E/C is run, every global user that is modified or created needs to callback to IM to get any updates back to the corporate user store. In this case a new IM task is created for every user that was affected by the E/C. Check the etanotify.log for any errors.
If you have not modified the JMS queue in JBoss to use a RDBMS, you should look at the tech doc that ashokpearl posted, as the JMS queue corruption in the default Hypersonic queues can cause problems with task data consistency, as all IM tasks are placed in the JMS queue for processing.
If this doesn't cover your situation, I would suggest that you open a support ticket and we can work further on the issue and get to resolution.
I was hoping there was maybe some way to 're-push' these tasks from the SQL level if nothing in IdentityMinder exists.
It would be difficult to pinpoint where exactly the problem started, but in all probability it would because we have integrated our ERP HR application with IdentityMinder with TEWS and the create/modify tasks received from the HR app are what usually get stuck.
I am not sure if a E & C runs everytime the these tasks are received or if we have scheduled trigger for E & C task. But I did have a look at the etanotify.log and I did not find any errors, it is always filled with lines like this - 20150126:161425:TID=0021bc:I: NOT SENDING NOTIFICATIONS: Transmissions have been suspended by configuration.
We use SQL for JMS queue, the size of the db is not so large (2.6K rows) so I am not sure if that is the cause of the issue.
I did open a ticket with CA support, but their finds tied down the issue to the large size of the tables in the Task Persistence database (details of which I have posted in my starter post). We have a six monthly purge of the TP db so I believe we should be good there.