Automic Workload Automation

Expand all | Collapse all

Getting Started with EDDA (Event and Data Driven Automation)

  • 1.  Getting Started with EDDA (Event and Data Driven Automation)

    Posted 11-18-2017 11:19 AM
      |   view attached

    What EDDA is good at:

        True Event Driven Automation.

        => Allowing outside systems (programs, applications, scripts, SAAS solutions, etc.) to send events to Automic and trigger Automic Objects intelligently as a response. All of that on a large scale: up to 15k events per second.

    How EDDA Works:

       Edda works by providing a simple & secure Rest interface that allows sending Events to AE (from anywhere) and automatically filtering / processing the  content of those events via rules (new object type in AE). The rules map Events to Automic Executable Objects and their variables / Promptsets.

    What you need to know before you start:

        Edda is very scalablebecause built on top ofApache KafkaandApache Flink(=Streaming Platform and Rules Engine). This post is not about those technologies so that is as far as we will go here on this topic :).

        Edda is integrated into Automic’s Analytics component, therefore the Rest endpoint is the same and the URLs / requests are very similar. The port for the Rest Endpoint is also the same for Analytics and EDDA.

        Sending Events to Edda willrequire an API key(to keep it secure and within a bounded scope (ex: a particular AE Client)). Such API Key can be generated by a Rest call to Edda.

        Generating API Keys requires you toknow your own System API Key(see below on how to retrieve it).

       Edda introduces 2 new objects to AE: 

            - a newVARA Objectof type: EVENT_DEFINITION

            - a newEVENT Objectof type: SIMPLE_RULE

            => both objects can be found by typing “IA” in the search bar when creating objects (if you must know, IA = Intelligent Automation :))

            EVENT_DEFINITION:

               => basically describes attributes expected in a given Rest Event

                => ex: “count” as a “number”

                => ex: “success” as a “boolean"

            SIMPLE_RULE:

                => describes a set a Rules for filtering events & their content (ex: if attribute “count” is > 2 & if attribute “success” is “True”)

                => also describes an Action (an AE object to execute when a Rest event matches the Rules) (ex: Map “Count” to Variable &COUNT# and execute JOBP Object accordingly)

     

        Lastly but importantly: get familiar with this post: https://community.automic.com/discussion/10794/getting-started-with-the-rest-api-for-automic-v12-1 and download the zip file attached to it (all the examples showed below with are contained within this file and uses Postman).


    Part 1: Sending Events to EDDA

       Since Edda exposes a Rest Endpoint, sending events to Edda is done through (POST) Rest Requests, and since Edda is secure, you need to first generate one or several API keys for the Rest Requests to work. In order to generate your API keys, you first need to retrieve yoursystem API key.

     

        A- Getting your System API Key:

     

            From the server where AWI (Automic Web Interface) is installed, navigate to the AWI folder structure and locate your “plugin.properties” under the “web-plugin-analytics” folder. You should be able to find it under (in most cases):

     

                C:\Automic\External.Resources\apache-tomcat-7.0.81\webapps\awi\config\webui-plugin-analytics\plugin.properties

     

            Open the file and locate “backend.api_key”, write down your api key somewhere. The line will look something like this:

     

                backend.api_key=513a5133-0909-44cb-a33b-b226c5bf4717

     

         B- Generating your EDDA API Keys:

     

            Use a POST Request to generate your first EDDA API Key.

     

            => the request’s header should contain an “Authorization” key with yourSystem API Key (see above) as a value:

                wd78zya5lqqn.png

     

            => the request’s Body should contain a JSON structure specifying a “scope”, adescriptionand aclient number(to limit the scope of the Key to a given AE client):

                v6cxqgl8ev9t.png 

            

            Execute the request: you should get a response similar to below: Write the “key” somewhere: that’syour first EDDA API Keyyou will use to send Rest Events!

                mjk7xhj83cjo.png

     

    C- Sending / Writing an Event:

     

        A Rest Event is very simple. It uses the EDDA API Key and a “type" to specify the scope of the event, and sends values through the Body of a POST Request.

     

        The EDDA API Key needs to be specified in the Header of the request (“Authorization”):

     

    9w4rq4nh2dbi.png        

        

    The Body of the request is a JSON structure and needs to contain:

     

        - “type”: this needs to be the name of the EVENT_DEFINITION (VARA) Object you will create in AE (see EVENT_DEFINITION above for description and below for creation)

        - “timestamp”: in the same format as below, it does not need to be right now, it can be in the past (you can use the one below)

        -a set of values: THIS is the actual content of the Event. It can contain any numberof actual “key/value” pairs but is restricted to3 types:

            -Strings(ex: “message” below)

            -Numbers(ex: “count” below)

            -Booleans(true or false, ex: “success” below)

     

            => the values you are passing will depend on what you are trying to achieve.. they could contain anything, for example: a ticket number in ServiceNOW, a description of a ticket,  the status of a process running somewhere else, an ID number that needs to be processed, etc.

     

        Ex: 

    kmm8roahqexf.png     

     

        At this point, youwill not be able to send an event to AE: we first need to create the Objects in AE that are necessary to the processing of this event. So let’s put this aside for now.

     

    Part 2: Processing Events with EDDA & AE:

     

       The purpose of EDDA is to Trigger AE Objects based on the content of an event. EDDA Maps the attributes from your Rest Body to the Variables needed for an AE Object to run.

     

       Ex: I want to pass the value of my attributes “count”, “message” and “success” to a SCRI Object and run it.

     

       EDDA does this by reading the attributes of an event, mapping them to PROMPTSET Variables of an AE Object and then submitting the AE Object accordingly.

     (All Automic Objects below can be found in the zip file attached to this post).

       A-  Let’s create the SCRI object and its PROMPTSET first then. Here is an example:

     

            My SCRI Object:    

    lmsheebs2b3v.png

            

     

            And its promptest:

    uywi8ea0glje.png        

            (Number represents variable &MYCOUNT#, Message represents &MYMESSAGE# and Success is &MYSUCCESS#)

            

     

        B- Let’s now create the mapping between our Rest Event and the SCRI execution:

     

            Create an EVENT_DEFINITION VARA Object.

     

            In it, specify the structure that you are expecting to extract (the list of attributes from the body and their type, essentially):

     0vgdkfzxfco7.png

            

     

            => this (I suspect) is only used internally to list all the attributes that need to be extracted by EDDA from the Event (Ex: the event could contain any number of other useless attributes, only listing the ones we need here offers better performance when parsing the body of the Request)

     

            Create a SIMPLE_RULE EVENT Object.

               In it, specify the VARA Object you created in the previous step.

                

                In the “Conditions” tab, specify what constitutes a “Valid” event. This allows you to filter events depending on their content.

                => you need to use this particular syntax for attributes “event.<name of attribute>”. Ex:event.count, event.message, event.success

     

                In the “Action” tab, specify which AE Object you want to trigger if an Event passes the “Conditions” successfully

                => you should Map your attributes to Promptest Variables for the object you want to trigger.

     

                Here is a complete example of this:

     rpq6pislvrp9.png


           Execute your SIMPLE_RULE Object

            => you should see it run in the Process Monitoring and it should have an “Event Count” of 0 (in the Details):

     2j200t97p40c.png


     

        Go back to Postman and send your Event as previously defined (play with the values in it to see what happens):

     

    w0u39xqjxjkc.png         

     

        The “Event Count” should increment every time you trigger an event:

     o9iqrrzc3ef3.png

                 

     

        The “RULE” will also trigger “children jobs” every time:

     md9ui4tqpdtf.png

            

     

            And each one of these should trigger your SCRI object with the values taken from the Event’s attributes:

     c14ac5hulg4q.png

             

     

     

    Attachment(s)

    zip
    EDDA-Example.zip   3K 1 version


  • 2.  Getting Started with EDDA (Event and Data Driven Automation)

    Posted 11-19-2017 08:49 AM
    Hi Brendan

    Very cool!

    What happens to REST-calls that are sent to the REST interface if no active event object is processing them? Is there any queue / replay mechanism? I guess the REST interface will be up and running (returning HTTP 200) so the recipient system does not know that event processing is currently disabled.

    Regards
    Joel


  • 3.  Getting Started with EDDA (Event and Data Driven Automation)

    Posted 12-11-2017 09:07 AM

    I just created some Powershell code to create an application key // send an EDDA event. It's quiet straight-forward and can be easily re-arranged to be used in a generic use-case.

     

    https://gist.github.com/JoelWiesmann/e6246f5e5b87e41dc53cc087da5b3ca5

    BTW smibe06 gist embedding support would be cool in the forum.



  • 4.  Getting Started with EDDA (Event and Data Driven Automation)

    Posted 12-12-2017 03:54 AM
    Some things I observed / open questions:

    The combination of:
    - If an event is sent using a type that is not known on the system it gets approved with HTTP 200 anyway. I would expect something like HTTP 500 / 501.
    - The caller gets no feedback at all whether the event was processed or not. So the caller has no chance to cache / resend events but does fire & forget. This is rarely an option in the use cases I have in my mind.
    - I could not find any log files beside the access log that indicated access to the event service. Especially for
    ...will result in hard work to debug issues with people stating that they sent events but have not been processed. I'm missing some audit trail here.

    Also:
    - It seems I can not limit a sender to an event type. So once an application has a token it can send any type of event. IMHO REST keys should have a "group" attribute and events should be able to access this information to restrict executions if needed.
    - Exporting the event with references won't include the definition VARA. This is not recorded in the restrictions for export with references. So guess this is a bug.
    - Deleting application keys that do not exist (or have already been deleted) does not return an error.

    Tested on 12.1.0GA01.

    Regards
    Joel


  • 5.  Getting Started with EDDA (Event and Data Driven Automation)

    Posted 12-18-2017 10:00 AM
    Hey joel_wiesmann_automic, thank you for the feedback!! i have shared it with Product Management.


  • 6.  Getting Started with EDDA (Event and Data Driven Automation)

    Posted 12-21-2017 04:38 AM
    Hi Joel, 

    Thanks for the feedback questions, i try to answer them here:

    What happens to REST-calls that are sent to the REST interface if no active event object is processing them? Is there any queue / replay mechanism? I guess the REST interface will be up and running (returning HTTP 200) so the recipient system does not know that event processing is currently disabled.

    Events send to the REST endpoint are written to the Streaming Platform (Kafka).  We accept events as long as they are following the format and we can write them to the Kafka queue. Currently all events are accepted regardless if you have an event definition for it or not. As long as we can receive and queue the events we return a 200, only if this is not the case or the request is malformed we return the appropriate status.  

    If an event is sent using a type that is not known on the system it gets approved with HTTP 200 anyway. I would expect something like HTTP 500 / 501.
    This is by design at the moment, you could add the type and rules later. But i get the point. I think this should be attacked by allowing to fine tune the EVENT scope for keys to define only specific event types (like {scoped key: BA..., scope: EVNT, types: OSMTERIC;SYSTEM_ALERT, client: 100})   
    The caller gets no feedback at all whether the event was processed or not. So the caller has no chance to cache / resend events but does fire & forget. This is rarely an option in the use cases I have in my mind.
    That is the basic idea of an event system, the sender only gets informed that the system received the event and 0 or multiple rules will then make use of the event. 
    If you need to know that you have started something in the ae and the run_id, ... it might be better to use the new inbound rest api but then you lose the benefits the event engine offers.   

    I could not find any log files beside the access log that indicated access to the event service. Especially for
    ...will result in hard work to debug issues with people stating that they sent events but have not been processed. I'm missing some audit trail here.

    Definitely true and we are already working on that. I would like to share our current findings and possible solutions with you beginning of next year and will therefore reach out to you soon.

    It seems I can not limit a sender to an event type. So once an application has a token it can send any type of event. IMHO REST keys should have a "group" attribute and events should be able to access this information to restrict executions if needed.
    At the moment it is only possible to restrict the scope for an API KEY  to AE clients. Being able to further restrict those keys is a very good idea. 
    2  Options: 
    • Define on the keys which events can be send or
    • Define on the Event which keys are allowed to send 
    Which would you prefer?



    Exporting the event with references won't include the definition VARA. This is not recorded in the restrictions for export with references. So guess this is a bug.
    Deleting application keys that do not exist (or have already been deleted) does not return an error.

    Bugs, thank you for reporting, i created tickets for those things



  • 7.  Re: Getting Started with EDDA (Event and Data Driven Automation)

    Posted 07-17-2018 04:03 AM

    1. Trying my hand on EDDA example given and SIMPLE.RULE.1 ends with Status FAULT_OTHER - Start Impossible. Other error. It has Last message : U00033504 Rule engine unavailable.

     

    UC4 Service Manager shows RULE-ENGINE in running Status (I tried a restart of services as well, still the same). Any guidance?

     

    2. At present there is a POC going on for ASO and a requirement will be "ServiceNow ticket has to trigger a run book (ASO workflow)". I am confused between using REST API or EDDA. Logically, REST API to Execute JOBP will suffice this requirement - correct? then, what value EDDA will add here to this requirement?



  • 8.  Re: Getting Started with EDDA (Event and Data Driven Automation)

    Posted 07-17-2018 10:11 AM

    Regarding point 2 - you are correct, the REST API is able to meet this requirement. The Event Engine (EDDA) adds value when 'rules' need to be applied to determine whether or not an action should be taken by the Automation Engine. In essence the Event Engine acts as an intelligent filter which can reduce the volume of events that the Automation Engine has to analyze and respond to, thus removing an unnecessary load on the Automation Engine. This is particularly valuable when there is a high volume of incoming events which may or may not all require an action to be taken by the Automation Engine. For example a single occurrence of an event may not warrant a workflow to be executed, but a flood of similar events might justify the execution of that workflow.



  • 9.  Re: Getting Started with EDDA (Event and Data Driven Automation)

    Posted 01-30-2019 04:38 AM

    I am stuck at Part 2 - step B,

     

    B- Let’s now create the mapping between our Rest Event and the SCRI execution:

     

            Create an EVENT_DEFINITION VARA Object.

     

    Because the EVENT_DEFINITION VARA object doesn't appear same as shown in original thread (part 2 - step B) and hence I am unable to create a mapping. (PFA sanpshot of VARA object in my setup).

     

    Any guidance?

     

    EVENT_DEFINITION VARA