Stability of Automic Platform

[Michael TMO8Y2tl]

Dear Community,

are there some mechanism in order to prevent damage on the automic engine?
e.g. prevent infinitive loops, limit the amount of allowed activities, query which identifies objects which are causing more then X RT,AH records etc.

If so, how can this be done?

[Carsten Schmitz]

​Hi.

Good question.

Disclaimer: The following is my opinion.

There are certain built-in protections, e.g. against certain infinite loops. It will throttle down the execution of an object if it detects an infinite loop and print a message to the message window to that effect. This is, however, not all-encompassing, there are various ways to circumvent this and still generate infinite loops. Some of these have been declared as to not being malfunctions, hence they will not be mitigated.

So while there are some protections, a malicious actor can still find various ways to fill up db tables or overload the engine.

The only full solution is to limit access to the engine to non-malicious actors with sufficient training. In our organisation, we have created a best practice guidebook which (also) aims to educate users how to avoid certain infinite loop scenarios and some other pitfalls that create excessive load on the system and/or database.

Hth,

[Frank Muffke]

Hi

in our company we handle it similar to Carstens attempt - (almost) no limit for AE, but as much restrictions for the users as suitable & possible & easy to handle.

No access for users without UC4 basics training, No writeable access for users without Full UC4 training.

The system itself is with redundancy (2 Servers, Linux) and mirrored DB (oracle RAC). This saved or lives several times so far :-)

cheers, Wolfgang

------------------------------
Support Info:
if you are using one of the latest version of UC4 / AWA / One Automation please get in contact with Support to open a ticket.
Otherwise update/upgrade your system and check if the problem still exists.
------------------------------

[Pete Wirfs]

"limit the amount of allowed activities"

There are multiple tricks available.  You can throttle concurrent activities via QUEUE object restrictions.  You can throttle an individual task by limiting how many of them can run at the same time.  I've also used SYNC objects to throttle a family of tasks.  There may be more ways(?)

Regarding finding objects that cause more RT records, I use this SQLServer query;

select oh_name as job_name
     --, ah_timestamp1 as activation_time
     , dateadd(hour, datediff(hour, getutcdate(), getdate()), ah_TimeStamp2) as Start_time
     --, ah_runtime as runtime
     , CONVERT(varchar, DATEADD(ms, ah_runtime * 1000, 0), 114) as runtime
     , count(*) as report_size
from uc4.dbo.rt
   , uc4.dbo.ah
   , uc4.dbo.oh
where ah_timestamp1 > cast('20191201 00:00:00:000' as DATETIME) -- how far back in time
and   ah_oh_idnr = oh_idnr
and   rt_ah_idnr = ah_idnr
and   not oh_name in ('APPUTILP', 'AE_PROD#WP001') -- Objects to exclude from the selection
group by oh_name, ah_timestamp1, ah_timestamp2, ah_runtime
having count(*) > 500 -- MAX_REPORT_SIZE is set in UC_HOSTCHAR_DEFAULT
order by 2;

------------------------------
Pete
------------------------------

[Michael TMO8Y2tl]

Unfortunately, this query dont work for me because we're using Oracle (SQL Developer)..

Perfect would be something like this:

Client | Job_Name | Count of RT records

sorted by the job with most RT records. Filtered by specific date range.

[Antoine Sauteron]

​Hi,

You can try something along these lines :

SELECT AH_Client, AH_NAME, COUNT (RT_AH_IDNR)
FROM uc4.AH, uc4.RT
WHERE AH_Idnr= RT_AH_Idnr
GROUP BY AH_NAME, AH_CLIENT
ORDER BY 3 DESC;

However it's pretty slow and can surely be improved. As RT is one of the biggest tables maybe it'd make sense checking / rebuilding the indexes before running this SQL.

Regarding the time frame you can either use the timestamps from AH or RT. Those from AH are more informative because they will tell you when the job was activated, started, when it has ended and when it was deactivated. On the other hand RT only has one timestamp, assumably the time when the report line was written to the DB.

Best regards,
Antoine

[Michael TMO8Y2tl]

Thanks, this is exactly what I'm looking for! I just found a job with more than 500.000 pages. Is there some general option in order to limit the allowed reportlines?

[Carsten Schmitz]

Hi,

See MAX_REPORT_SIZE, it limits how many pages end up in the DB vs. in the agent file system.

https://docs.automic.com/documentation/webhelp/english/AA/12.3/DOCU/12.3/Automic%20Automation%20Guides/help.htm#AWA/Variables/UC_HOSTCHAR_DEFAULT.htm?Highlight=MAX_REPORT_SIZE

Various job types have different options, e.g. for SAP jobs you can limit the amount of report being kept right in the job object.

Hth,

[Michael TMO8Y2tl]

MAX_REPORT_SIZE is already set to 20. However, there is a rest job with several post processing (POST) pages (+4.000.000). Can this be limited on a global level instead of an object level?

For this particular job Write agent log  to job report was enabled.

[Michael TMO8Y2tl]

is there an option on how to restrict POST / LOG - Reports?

I'm also searching for a query which identifies the runid of jobs with infinitive loops.

[Michael TMO8Y2tl]

Sometimes the MELD-Windows shows the following error-message:

U00010037 Object: 'XYZ', client: 'XX': processing interrupted, possible loop! Script processing will continue in '128' seconds.

Can someone provide an sql query in order to identify the runid's of those jobs? I've written this query. Maybe there is an more elegant way of getting this information. (Currently scenario that loop has been fixed in the meanwhile is not covered)

SELECT ah_idnr       RunID, 
       ah_name       JobName, 
       ah_client     UC4_Client, 
       ah_timestamp1 JobStart 
FROM   uc4.ah 
WHERE  ah_name IN (SELECT DISTINCT jobname 
                   FROM   (SELECT Substr(meld_msginsert, 1, 
                                  Instr(meld_msginsert, '|') 
                                  - 1) AS 
                                  Jobname, 
                                  meld_client 
                           FROM   uc4.meld 
                           WHERE  meld_msgnr = '10037') 
                   WHERE  meld_client != '0') 
       AND ( ah_timestamp4 IS NULL 
             AND ah_timestamp3 IS NULL );