How to cancel that which can not be cancelled?

[Carsten Schmitz]

Greetings.

Our users somehow keep creating objects (jobs, jobf mostly) which are in "waiting for predecessor", but though they have a parent RunID in monitoring, that parent RunID doesn't exist.

Any attempt to cancel these objects results in:

Cancel/Quit of task 'JOBS.MAKING_HALF_BAKED_SOFTWARE_KILLS_POLAR_BEARS' (RunID '012345678') not possible because it is not in a cancelable status.

One can manually put these into a cancelled state ("Modify status manually") but here's the catch-22: You can not multi-select more than one and do this. There simply is no menu option to "Modify status manually" when selecting more than one.

Anyone had this before, or have any good idea how to cancel the nearly 2000 objects that have been spawned over the last days?

Because right now, the only supported way I see is to modify the status of about 2000 items one by one, and unless I am given access to an at least medium sized army of small trained monkeys that assist, this is not feasible. Plus, even if I would spend my work day cancelling nearly 2000 objects like the world's most sad and grindy action RPG, our users keep spawning them faster than I can possibly get rid of them.

Thanks.

p.s. this is with AWI 12.3, so it's the latest and greatest.

[Peter Grundler]

Hi Carsten,

what if you open the parent monitor and cacel the parent task instead?

kind regards,
Peter

[Carsten Schmitz]

Hi Peter,

good idea in principle, but all of these have no parent monitor, because they have no active parent. There is no menu option for a parent monitor.

If I go hunting for the parent by doing a "search for use", then look at the statistics of the (thankfully) only job plan it finds, I can not even find the runID given for the parent in there, because those parent runs have deactivated and rotated out of the database.

In fact, I suspect that this kind of stalemate happens precisely when our users cancel and deactivate the parents without the children (how ever they manage).

Interesting side note: When I open the monitor of these stuck "waiting for predecessor" children, it opens the JCL tab. These objects are in some weird limbo.

Kind regards,
Carsten

[Will Boyd]

Whenever we see this issue we run some sql statement provided by the vendor to remove the objects from the activities window. You may want to reach out to Broadcom. There are many different sql commands to remove objects but its best not to run without the vendor support.

[Carsten Schmitz]

​Cheers. I had indeed contacted Broadcom in the mean time and they will send me an "officially approved" delete statement.

[Reed Byers]

At Oregon State University, we've only just started migrating from Application Manager to AWA 12.3, and will be going live in a year.

The prospect of a horde of zombie jobs filling our Process Monitor and demanding our brains is... quite daunting.

And if anyone's customers are capable of causing this to happen, *OURS* can.  :)

If you could share any resolution to this problem you find on this thread, it would be greatly appreciated.

Thanks!


------------------------------
Reed Byers
Programmer/Analyst
Oregon State University
------------------------------

[Carsten Schmitz]

> ​If you could share any resolution to this problem you find on this thread, it would be greatly appreciated.

I've received an sql statement from support to deal with this, this is usally the way those things get cleaned up. Havent run it yet, but these are usually straight forward and work. How do i know? Well, I we had similar zombie objects in the past, too, e.g. with V10. So yeah, not sure if this is good news or not, but on the plus side, it's not an entirely new issue with v12 ;)


> And if anyone's customers are capable of causing this to happen, *OURS* can.  :)

Your people and our people should join forces. We also had users that dig out the most obscure script funtions that Automic probably implemented for a poster child customer and then hoped nobody would ever use them again. Then they built huge includes arround those functions and moved on to greener pastu^h^h^w I mean to other challenges.

Together, I bet our users could write a great suite of tests that Automic could then run before each release :)

Best regards,
Carsten

[Carsten Schmitz]

I'm truly sorry. I spoke to soon.

This time they have sent me 35 SQL statements, one of which has an absolutely pointless sub select.​ I do believe the support rep was probably supposed to send only a subset of these, but not sure, maybe they also go for a "nuke it from orbit" approach. Either way, all of these sql statements are not qualified enough, so if I were to execute them, I'd run the risk of deleting object instances and modify statistics and reports of object instances that are not affected by this bug, and thus cause major issues for (and probably with :) the user base.

I tend to forget things, but I now remember that with V10 I'd also escalate such a ticket and in the end provide my own, fully qualified SQL query and only had support give final approval to it (which, icing on the cake, my query had a copy/paste error when I submitted it in the ticket system and was approved despite being faulty in the ticket).

Current state of affairs: went back to Automic to ask them to reconsider their submission.

[Josef Scharl]

Hi,

I created Knowledge Articles for the topic "cleanup of activities" many years ago. It looks like this is what you are looking for. The last update of the content was in October 2018, when I was still with UC4/Automic/CA…

I don't know, why this knowledge is not used. Anyhow, the good news is that these articles are still online:  

 

MSSQL / Oracle / DB2

I've compared v12.2 and version 12.3 db schema, there are no changes in the area of E* tables, so the same will work also for 12.3.

KR, Josef

------------------------------
Application Manager
Raiffeisen Informatik GmbH
------------------------------

[Carsten Schmitz]

Hi Josef,

first of all, many thanks! I think this article is very helpful BUT I can only find in the article how to build the sub selects. I can NOT find the actual SQL statements operating on the E* tables that the sub select is then used in. Case of "Betriebsblindheit" on my part (it's possible), or was there maybe an attachment or a link that did not get migrated? I am using the Oracle article btw.

Also, I do think the support rep probably even used your article, at least to an extend. The 35 statements he gave do operate on E* (and AH and RH), and the final one on EH. It's just that I presently believe he ignored all the qualifier options that we'd need in this case such as EH_STATUS and EH_STARTTIME, and gave us a block of DELETE statements that exclusively use EH_NAME and EH_CLIENT. Which, of course, would not only wipe the broken instances, but any legit instances running as of today as well. But then, what's a potential extended downtime, some data loss and a restore of a database from backup between friends ...? :)

Cheers,
Carsten

[Josef Scharl]

You are so right! A support rep needs incredibly product and general IT know how. Otherwise it can be dangerous for customers.

Hmm, don't know what they migrated or what was truncated. I've just uploaded my source documents – free to use at own risk.

Attachment(s)

ProcessMonitoringCleanupDB2DB.docx   21 KB 1 version

ProcessMonitoringCleanupMSSQLDB.docx   21 KB 1 version

ProcessMonitoringCleanupOracleDB.docx   21 KB 1 version

[Carsten Schmitz]

@Josef Scharl ​Brilliant, many thanks! You probably saved me a LOT of time with this.

With these, I can confirm that the support rep did in fact send your statements but he didn't consider the (vital) part about building the correct sub select at all. He just handed out the statements as they were (with client id and object name).

I will now build a matching sub select myself, and then send the finished SQL to Automic for token approval :)
​

[Carsten Schmitz]

​Oh, I see it now:

As the cleanup of Tasks is a serious action, it should be done guided by CA Support engineers only! Therefore, the statements themselves are not posted in this article.

I do not wish to post the statements suggested in the support ticket here because that would counteract your safekeeping. But can you confirm there were about 35 statements overall, 33 for E* and one for AH, one for RH?

Thanks!

[Will Boyd]

35 total statements plus the commit with two only 2 update statements for an Oracle DB.