Automic Workload Automation

Expand all | Collapse all

What is your opinion on Event Driven Data Automation / Event Engine?

  • 1.  What is your opinion on Event Driven Data Automation / Event Engine?

    Posted 02-07-2018 02:27 AM
    Processes must become more and more intelligent and the term Intelligent Automation is omnipresent. What use cases can you think of for which you wish you had an intelligent system  in your company? What comes to your mind when you hear the terms Intelligent Automation or Event Engine?

    In version 12.1 we included an intelligent mechanism to process incoming events. Watch the video below for an example in which we combine ServiceNow with our Event Engine and Automation Engine.

    qvt3kld8gilp.pnghttps://us.v-cdn.net/5019921/uploads/editor/76/qvt3kld8gilp.png" width="372">
    https://www.youtube.com/watch?v=LfSxgj63yBs&feature=youtu.be

    What are the use cases you can think of where you wish for a system like this in your company?
     
    If you want to discuss or read about the technical aspects, this thread could be interesting for you: https://community.automic.com/discussion/comment/34044


  • 2.  What is your opinion on Event Driven Data Automation / Event Engine?

    Posted 02-07-2018 05:30 AM
    Hi.

    Since you asked ...

    What comes to mind, when I hear the terms "Event Engine" and "Intelligent System" is that one of these terms is vastly more useful than the other.

    As for the event driven approach, that has merrit, but is more a logical evolution of a job scheduling system, isn't it? As of today, jobs already start based on events - not only UC4's file events, but a scheduler triggering is nothing but an event, too. From a technical standpoint, UC4 has always been polling for events and reacted to those (mostly by starting jobs).

    Of course, here's the thing: UC4 is actually very flexible already. As of this day, I can theoretically create and handle many events myself. Want to have a job start when an SMS comes in? Poll the industry modem's spool with a shell script, write out a file when the SMS comes, react to that file with a file event in UC4. Want to react to a Nagios event? Poll the URL with wget from a scheduler and start a job if the coffee maker reports a "critical" status for the coffee bean level.

    So many of those things are already possible. What Automic appears to be doing these days under the moniker "Event Engine" appears twofold: optimizing the event handling code (optimization is a very welcome thing, if done properly), and adding more interfaces for things then labeled "events". Automic can certainly add convinience (i.e. free customers from having to build these solutions themselves) and make it more obvious what can be done, and the process can be optimized by rooting event handling deeper in the Engine. These can be valuable things if done right (it's the if done right part that somewhat worries me). But let's also be clear, none of this is reinventing the wheel. And if I may say so, it appears it's at least partially in a search for use cases: Maybe because the really needed use cases have already been built as described by clients or consultants, or maybe because for some customers, the way they use AE is much more trivial than the forefront of Automic's strategists would sometimes like to envision.

    Yes, it's kinda cool to see UC4 responsing to Alexa (once. if it works). And no, we're not doing it. We're instead struggling these days to get much more basic things to work reliably: Our experience with AE V12.1 has been so abysmal, with many basics that used to work in old versions not reliably working anymore. Basics such as simple file transfers (PRB00216843). Basics such as opening old versions of jobs (PRB00218930). And more.
    And that's with having even skipped the "Zero Release", as is oftentimes suggested wisdom among more than a few Automic clients.

    We want to upgrade our production systems to AE 12.1 in March. Not half an hour ago, I had to answer to a representative of our users currently testing the version 12.1, who appealed to me that it's impossible to go "productive" with the current state of affairs (a realization that is forming with the admin staff for quite some time as well).

    But maybe it's us? Maybe we're using it wrong? Maybe we installed it wrong? Hm. Nope. It appears, without diving into sources, we are by far not alone with this overall assessment. So, before adding more "Event Driven" interfaces, before any more new developments that users and clients will latch on to, like Zero Downtime Upgrades or "Intelligent Automation": please consider vastly improving the stability, reliability, usability and, last not least, QA'ing of what you already have.

    I'm not sure if, for example, this:

    "Version 13, now rock solid, partly rewritten from scratch, over 500 bugs fixed"

    would market better than this

    "Version 13: Now 'Intelligent' ".

    But it's certainly what this client would want.

    Kind regards,
    Carsten Schmitz



    (edit, p.s. one aspect I missed that should be mentioned:

    if the Event campaign leads to the polling engine to have a granularity of under a minute (as is the case for schedulers), that would be truly awesome. But this would take a major rework of the core which I'm at least sceptical about it happening near-term).


  • 3.  Re: What is your opinion on Event Driven Data Automation / Event Engine?

    Posted 05-01-2018 04:57 AM

    I also fully support what Carsten Schmitz says:

    "...So, before adding more "Event Driven" interfaces, before any more new developments that users and clients will latch on to, like Zero Downtime Upgrades or "Intelligent Automation": please consider vastly improving the stability, reliability, usability and, last not least, QA'ing of what you already have..."

     

    ..I couldn't agree more!

     

    Regards

    Keld Mollnitz.



  • 4.  Re: What is your opinion on Event Driven Data Automation / Event Engine?

    Posted 09-17-2018 02:13 PM

    Carsten Schmitz wrote:

     

    Hi.

    Since you asked ...

    (edit, p.s. one aspect I missed that should be mentioned:

    if the Event campaign leads to the polling engine to have a granularity of under a minute (as is the case for schedulers), that would be truly awesome. But this would take a major rework of the core which I'm at least sceptical about it happening near-term).

    The current event engine uses a backend of kafka and flink to queue and track event requests. Current event templates are driven through this backend.  The speed of this response is near instant, most events I have set up point at the Analytics Rest end point.  The response to these events are contained in the AE UI, run X script or job.  The Kafka and Flink backend control the bandwidth of event requests to the AE.

     

    There are some community offerings for SQL and logfile monitoring which run a java process.  I did not write these but they require parameters to the Analytics Rest Endpoint, so I assume they are making rest calls with the data they retrieve.

     

    The design consideration above was chosen so that AE users could architect with the same objects and scripting language that they use for existing use cases.



  • 5.  What is your opinion on Event Driven Data Automation / Event Engine?

    Posted 02-07-2018 11:40 AM
    nothing to add from my side & fully support Carsten's Post.


  • 6.  What is your opinion on Event Driven Data Automation / Event Engine?

    Posted 02-08-2018 04:54 AM
    Hello,

    I know this is slightly off thread but we are in the process if implementing/using Automic on site for the first time.  What version would people recommend we start at? Sounds like v12.1 may give us some unexpected results?

    Cheers,

    Dan.


  • 7.  What is your opinion on Event Driven Data Automation / Event Engine?

    Posted 02-08-2018 05:12 AM
    daniel gates said:
    Hello,

    I know this is slightly off thread but we are in the process if implementing/using Automic on site for the first time.  What version would people recommend we start at? Sounds like v12.1 may give us some unexpected results?

    Cheers,

    Dan.
    You may not like the answer (I certainly wouldn't) but ... I'm afraid 10 or earlier is definetly out of the question due to no reasonable support anymore (and soon to be wholy unsupported).

    11.2, well you could go with that, but if you start today, I personally would not want to start with the Java client in the knowledge that I'd have to switch to the web interface (and set up servers for it) very soon anyway. It depends on the number of users, too: A small job building team may be okay with version 11.2 for a year or two and then switch to 12. If you have 50 users like we have - I'd not want to train them on one client and retrain them in a year from now or so. And as a whole, 11.2 will run out of support not too long into the future, too.

    So that leaves 12.0 or 12.1. I don't have that much experience with 12.0, but I'd still wager (alhtough I heard different opinions) that 12.0 and 12.1 are not that different, and that 12.1 should have bugfixes and additions over 12.0. Thus, I'd bet that we'd probably see the same or similar issues we see with 12.1 with 12.0 as well.

    So the final answer I'll give, from my point of view and with a slight feeling of uneasiness, is: Go for 12.1 (or 12.0) directly, because the other choices are not useful. But make sure you test it on a test system first with realistic usage patterns, as is advised with any Automic update (or by extension, installation, for that matter). If you find that 12.1 gives you show-stopper issues, I don't have any good advise what to do next. Except inviting you to report them and hope for them to be fixed in due time.

    Just my $0.02,

    Carsten


  • 8.  What is your opinion on Event Driven Data Automation / Event Engine?

    Posted 02-08-2018 05:36 AM
    Cheers Carsten,

    Thanks again for your knowledge and input .. I will report this to the powers that be :)

    Dan


  • 9.  What is your opinion on Event Driven Data Automation / Event Engine?

    Posted 02-08-2018 05:59 AM
    Hi Sandra,

    I may get back to the questions you asked and trie to imagine a good example for a practical use case. But actually, I can't (find a good example). If I understod correctly there are 2 dependencies for this to work.


    • Analytics must be installed
    • The calling software needs to send a REST call to the engine

    For the Analytics part, we decided not to install it yet, as we don't want another database to maintain and don't want to hire a Postgres Administrator just for graphical statistics (that may be different in other companies and may change when the Event Engine becomes usefull for us).

    I also tried to imagine which software, that we use, could make a REST call in case of an event. But the only one that comes to my mind is SAP or software that we build ourself. We use the SAP Agent to handle SAP events and most of our own software doesn't have a connection to the AE.So in our case, I have to say: There are to many technical requirements in Event Engine to be usefull for us.

    Hope this may help.

    Regards, Matthias