Application triggering in APPLWAIT but only that generation active

[Neil_Simmonds]

We had to reboot the servers this morning after a system hang and we now seem to have an issue with 6 applications that have the "wait for previous generation" option set. They are triggering in an APPLWAIT state and have to be unwaited. As a workaround I have amended them to turn off "wait on previous generation" and trun on "do not trigger if active" but this is less than ideal due to the fact that it could potentially stop running altogether if there is an issue. 

 

Has anyone ever sen this before or have any idea how to resolve it?

[powdo03]

Hi, 

I am assuming this is a dSeries scheduler. 

 

From the CLI issue the command to see all the active applications. Do you find one that does not show up on your screen? 

LISTAPPLICATION   APPLICATION('applicationnamehere')

[Neil_Simmonds]

No active applications are listed with this command. 

[Olusegun Oduntan]

There are situations where system hang issue can cause data inconsistency problem in the database so the server still considered some previous generations of the applications as active. Also, it's possible that some completed applications records were not updated and removed causing the applications to get stuck in APPLWAIT. Sometimes performing a changerole can resolve the issue if you have HA configured and/or run the purgecompletedjobs command on the applications.

[SHARON SHIMANEK]

hi,

Just following this and I have a question  - Neil said they changed the application properties from WAIT for prev generation to do not trigger if active.  In my mind if the system still thinks there is an active application causing it to go into APPWAIT, then changing it to DO NOT TRIGGER IF ACTIVE should not have worked either. I am I wrong?

 

Sharon

[Olusegun Oduntan]

Sharon, you're not wrong it should not have worked. Such applications should be monitored closely when they are scheduled to be triggered.

[Neil_Simmonds]

purgecompletedjobs has run over the weekend (we run it daily) and it's still not working. We haven't done a changerole yet but are looking at doing that to test something else probably tomorrow so we'll see how that goes. As for Sharon's comment, I never thought of that. Thanks for pointing it out Sharon and you're right, it shouldn't really work but it does for these 6 problem applications. We are on 11.3.1 so perhaps it's fixed in newer versions so it wouldn't work.  

[SHARON SHIMANEK]

we run this sql before doing a cold restart of DE to get a list of active applications.  If you run this do you see anything out of place with the applications in question

 

Select Name, Type, Thestate, Status, Starttime, Endtime From WA.Esp_Rtwob
  Where Thestate Not In('COMPLETE') 

[Neil_Simmonds]

Hi Shimaneks, I ran a variant as all the applications I have the issue with are CYC*

Select Name, Type, Thestate, Status, Starttime, Endtime From Esp_Rtwob
Where Thestate Not In('COMPLETE') and NAME like '%CYC%'

 

This returned some entries where Thestate was BYPASSED for several applications (of which 3 were problem ones) but nothing for 4 of the problem applications 

[Neil_Simmonds]

Since posting this I have involved CA Support and on there suggestion, I deleted one of the problem applications and it's trigger event. I then ran PURGECOMPLETEDJOBS for that application and tried to Subscribe with filter which returned no applications. I then opened the XML file of the application that I had saved and recreated the event then uploaded the application. 

All the above had zero effect on the issue. The application still came on queue in APPLWAIT at the next trigger. 

[Anon Anon]

Hi Neil,

 

Unfortunately, it is looking like a Cold Start is the only thing that will clear up this situation. Trying to perform manual updates on the DB to get the applications to stop getting stuck APPLWAIT is risky and can lead to further corruption.

 

-Paolo

[Neil_Simmonds]

Well we still have the issue, but thanks to over 200,000 rows on ESP_MANAGER_OUTQ last night and the CA_Workload_Automation_17599 service refusing to stay up (Java 911 errors) we are now on 11.3.3. I currently have 8 CYC applications that I have had to remove "wait for previous generation" from to prevent them triggering in APPLWAIT and I also now have 11 BATCH applications (run once a day) that I've had to remove it from as well. 

[Neil_Simmonds]

Following the upgrade we have been able to identify the problem entries on ESP_APPLICATION and ESP_GENERIC_JOB and are planning a DB update next week for the 31 applications affected.  

[Neil_Simmonds]

Bit of a late update but following up on this due to us now having caught up with movehistorydata I have now found by running the following, 

 

Select Name, Type, Thestate, Status, Starttime, Endtime From Esp_Rtwob
Where Thestate = 'READY'
and STARTTIME like '%20180522%'

 

that all the applications that have an issue return an entry in that sql query and the column "thestate" contains the word READY. Is this something we could update via an SQL update or are we still looking at a cold start to clear it?