Automic Workload Automation

 View Only
Expand all | Collapse all

Dashboard XML

  • 1.  Dashboard XML

    Posted Oct 27, 2020 04:59 AM
    Edited by Michael A. Lowry Oct 27, 2020 12:10 PM
    We recently decided to begin allowing our users to deploy dashboards. We have enabled the dashboards feature in our development environment, and we are starting to plan for development of the capability to deploy DASH objects from DEV to TEST and PROD. As a part of this work, I have for the first time taken a look at the XML of some exported DASH objects. What I found was somewhat discouraging.

    <?xml version="1.0" encoding="UTF-8"?> <uc-export clientvers="12.3.2+build.1581365228664"> <DASH name="TEST1.DASH"> <HEADER state="1"> <Title>Test dashboard</Title> <ArchiveKey1 /> <ArchiveKey2 /> <OH_SubType /> </HEADER> <DASH state="1"> <DOC><![CDATA[ Base64-encoded XML ]]></DOC> </DASH> <DOCU_Docu state="1" type="text"> <DOC /> </DOCU_Docu> <DOCU_Runbook- state="1" type="xml"> <DOC /> </DOCU_Runbook-> </DASH> </uc-export>

    Almost all of the content is opaque, embedded in a Base64-encoded string inside a CDATA block. When decoded, this reveals dashboard-relevant XML content, including one entry for each widget. Again, all of the content is encoded. Widget content is embedded escaped XML. Configuration content is embedded Base64-encoded JSON.

    <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
    <dashboard cols="3" rows="2" static="false">
    <widgets>
    <entry>
    <widget id="com.uc4.ae.processmonitoring"> Escaped XML </widget>
    <geometry height="1" width="3" x="0" y="0"/>
    </entry>
    </widgets>
    <configurations>
    <item id="analytics.dashboard.datasource"> Base64-encoded JSON </item>
    </configurations>
    </dashboard>

    The unescaped XML of the widgent looks like this:

    <?xml version="1.0" encoding="UTF-8"?> <configuration xmlns:ns2="http://www.automic.com/ECC/core/usersettings" xmlns:ns3="http://www.automic.com/ECC/core/usersettings/common" xmlns:ns4="http://www.automic.com/ECC/core/usersettings/textsearch" xmlns:ns5="http://www.automic.com/ECC/ae/usersettings" xmlns:ns6="http://www.automic.com/ECC/core/usersettings/optiontree" xmlns:ns7="http://www.automic.com/ECC/core/usersettings/optiongroup" xmlns:ns8="http://www.automic.com/ECC/core/usersettings/specifictaskpropertiesfilter" xmlns:ns9="http://www.automic.com/ECC/core/usersettings/optionsearch"> <refreshInterval>AUTO_MIN_1</refreshInterval> <title /> <description /> <treeMode>true</treeMode> <filter includeDeactivatedTasks="false" useSearchStringAsTabDescription="false"> <ns5:taskNameSearchConfiguration inverse="false"> <ns4:searchString /> </ns5:taskNameSearchConfiguration> <ns5:taskTypeConfiguration /> <ns5:statusConfiguration> <ns6:selectedKeys xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">1800-1899</ns6:selectedKeys> <ns6:selectedKeys xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">1560-1564</ns6:selectedKeys> <ns6:selectedKeys xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">1600-1799</ns6:selectedKeys> <ns6:selectedKeys xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">1300-1599</ns6:selectedKeys> </ns5:statusConfiguration> <ns5:hostNameSearchConfiguration inverse="false"> <ns4:searchString /> </ns5:hostNameSearchConfiguration> <ns5:hostTypeConfiguration /> <ns5:userNameSearchConfiguration currentUser="false" inverse="false"> <searchString /> </ns5:userNameSearchConfiguration> <ns5:ak1NameSearchConfiguration inverse="false"> <ns4:searchString /> </ns5:ak1NameSearchConfiguration> <ns5:ak2NameSearchConfiguration inverse="false"> <ns4:searchString /> </ns5:ak2NameSearchConfiguration> <ns5:customAttributeConfigurations /> <ns5:queueConfiguration /> <ns5:messageNumberSearchConfiguration> <value>0</value> </ns5:messageNumberSearchConfiguration> <ns5:messageTextSearchConfiguration inverse="false"> <ns4:searchString /> </ns5:messageTextSearchConfiguration> <ns5:clientConfiguration /> <ns5:timeFrameConfiguration> <filterOption>LAST</filterOption> <filterUnits>DAYS</filterUnits> <units>1</units> <from>2020-10-26T19:08:20.103Z</from> <to>2020-10-26T19:08:20.103Z</to> <filterType>NO_TIME_FRAME</filterType> </ns5:timeFrameConfiguration> <ns5:runIdConfiguration> <value>0</value> </ns5:runIdConfiguration> <ns5:topRunIdConfiguration> <value>0</value> </ns5:topRunIdConfiguration> <ns5:parentRunIdConfiguration> <value>0</value> </ns5:parentRunIdConfiguration> <ns5:zduVersionConfiguration /> <ns5:syncUsageConfiguration> <selectedItemId /> </ns5:syncUsageConfiguration> <ns5:taskAliasConfiguration inverse="false"> <ns4:searchString /> </ns5:taskAliasConfiguration> <ns5:specificTaskPropertiesConfiguration> <ns8:name /> <ns8:value /> </ns5:specificTaskPropertiesConfiguration> <ns5:additionalTaskFilterConfiguration commented="false" modified="false" /> </filter> </configuration>

    The decoded JSON content of the configuration key looks like this:

    {
    "configuration" : {
    "primary_entity" : "JOB",
    "time_frame" : {
    "between" : {
    "start" : "2018-02-01",
    "end" : "2018-02-08"
    }
    },
    "query_tree_root" : {
    "leaves" : [ ],
    "branches" : [ ],
    "query_tree_root_operator" : "ALL"
    }
    }
    }

    This design leaves much to be desired. We want to examine the XML of objects to ensure validity before allowing users to deploy the objects to production. When important content is encoded, or even doubly encoded, this obscures information from the validation tool.

    Is there a good reason the XML schema for dashboards is designed this way?


  • 2.  RE: Dashboard XML

    Posted Oct 29, 2020 04:10 AM
    Edited by Michael A. Lowry Nov 02, 2020 07:05 AM
    I figured out why the XML schema for DASH objects is so opaque: it's just a direct dump from the database. When the dashboards feature was added, this was apparently done using an off-the-shelf toolkit that had its own XML-based schema. Dashboards can be quite complicated, and Automic apparently decided that it would be prohibitively costly or involved to create a new table (or more likely, multiple tables) in the DB dedicated to DASH objects. Instead, Automic opted to just store the foreign dashboard-specific XML inside DOCU sections/tabs. 

    When a DASH object is loaded, the AE/AWI combines chunks extracted from the DB, decodes them from Base64 to XML, and hands the XML off to the classes that handle dashboards.

    You can extract the XML of dashboards using SQL like this:
    select OH_Name, OH_Idnr, ODOC_Lnr, blob_to_clob(decode_base64(blob_to_clob(ODOC_Content))) from OH left join ODOC on OH_IDNr = ODOC_OH_IDNr where OH_OType = 'DASH' and OH_DeleteFlag = 0​


    You can find the blob_to_clob and decode_base64 functions in the discussion AWI Configuration Settings. where you will learn that AWI configuration settings are stored in a similar way.

    This way of storing data using foreign XML schemas is a compromise. It obscures functional data behind layers of encoding and complicates the load/store process. But for data whose use is limited to small parts of the application, it probably makes sense. It eliminates the need to do significant re-engineering or to implement translation from one schema to another. Like AWI configuration settings, dashboards are needed only by the AWI, so these data do not need to be stored in a way that is easily accessible to the rest of the Automation Engine application.

    This finding does throw water on the idea of getting a more sane XML schema for these objects. We will now do the extra engineering work of adding a decode class for dashboards to our AE XML validation tool.



  • 3.  RE: Dashboard XML

    Posted Oct 29, 2020 10:54 AM
    Edited by Michael A. Lowry Oct 29, 2020 10:54 AM
    Given the opaqueness of the dashboard XML schema, I wonder if it might be better to approach this problem in a different way. The reason we want to examine dashboard content is to see if any widgets are configured with a refresh interval smaller than 5 minutes. We do not want lots of dashboards active that are constantly polling the Automation Engine at short intervals, and putting an unduly high load on the system.

    Is there a way to see what dashboards are currently active? If we had a way to do this, we could perhaps identify offending dashboards that way instead.

    (My guess is that active dashboard instances exist only at the level of the AWI, and do not have any entries in the AE DB.)



  • 4.  RE: Dashboard XML

    Posted Nov 02, 2020 07:29 AM
    Edited by Michael A. Lowry Nov 02, 2020 07:30 AM
    I wrote a script to parse the XML of an exported dashboard object, and to list the refresh intervals of any defined widgets.
    #!/bin/bash DashboardXmlFile=$1 Dashboard="$(basename ${DashboardXmlFile})" Dashboard="${Dashboard%.*}" printf "%-45s " ${Dashboard} DashboardXml=$(cat "${DashboardXmlFile}" ) ExportError=$(printf "${DashboardXml}" | xmllint --xpath 'string(/uc-export/error)' - ) if [[ "${ExportError}" != "" ]]; then ExportErrorNum=$(printf "${DashboardXml}" | xmllint --xpath 'string(/uc-export/error/text())' - ) printf "export error U%08d\n" "$ExportErrorNum" else DashDocB64Text=$(printf "${DashboardXml}" | xmllint --nocdata --xpath '/uc-export/DASH/DASH/DOC/text()[1]' - ) if [[ "${DashDocB64Text}" == "" ]]; then printf "no content\n" else (( counter=0 )) DecodedDashXml="$(printf "${DashDocB64Text}" | base64 --decode)" echo "$DecodedDashXml" | grep -q 'widget' if [[ ${PIPESTATUS[1]} -ne 0 ]]; then printf "no widgets defined\n" else printf "at least one widget defined" while read Widget; do (( counter++ )) WidgetId=$(printf "$Widget" | xmllint --xpath 'string(/widget/@id)' - ) WidgetXml=$(printf "$Widget" | xmllint --xpath '/widget/text()' - ) WidgetRefreshInterval=$(printf "$WidgetXml" | recode xml..ascii | xmllint --xpath '/configuration/refreshInterval/text()' - 2>/dev/null ) if [[ "$WidgetRefreshInterval" == "" ]]; then WidgetRefreshInterval="NA"; fi if [[ ${counter} -eq 1 ]]; then echo; printf " # %-42s %s\n" "Widget Id" "Refresh Interval"; fi printf " %-2d %-42s %s\n" $counter $WidgetId $WidgetRefreshInterval done <<< "$(printf "$DecodedDashXml" | xmllint --xpath '/dashboard/widgets/entry/widget' - | sed 's:\/widget>:\/widget>\n:g')" if [[ $counter -eq 0 ]]; then printf "no widgets defined\n"; fi fi fi fi


    I then wrote a UNIX job to run the above script once for each DASH object in the system.

    :SET &DashboardList# = "UC4.DV.LIST.VARA_SEC_SQLI" :SET &TempDir# = "/tmp" echo "Looking up refresh intervals of dashboards defined in &$SYSTEM#." :SET &DashListHnd# = PREP_PROCESS_VAR(&DashboardList#) :PROCESS &DashListHnd# : SET &Dashboard# = GET_PROCESS_LINE(&DashListHnd#,2) : SET &OutputFile# = "&TempDir#/&Dashboard#.xml" : SET &ExportRC# = EXPORT(&Dashboard#,&OutputFile#) : IF &ExportRC# <> 0 : PRINT "ERROR &ExportRC# from EXPORT function." : ELSE : PRINT "EXPORT return code: &ExportRC#" /opt/scripts/parse_dash.sh &OutputFile# rm &OutputFile# : ENDIF :ENDPROCESS :CLOSE_PROCESS &DashListHnd#


    The SQL that lists the DASH objects is very simple.

    select OH_Name from OH where OH_OType = 'DASH' and OH_DeleteFlag = 0


    Below is an example of what the output of the job looks like.

    Looking up refresh intervals of dashboards defined in UC4_DEV. DASH.NEW.1 no widgets defined DASH.NEW.2 export error U04002032 DASH.NEW.TEST at least one widget defined # Widget Id Refresh Interval 1 ecc.common.text NA 2 com.uc4.ae.processmonitoring AUTO_MIN_1 STEFAN.DASHBOARD at least one widget defined # Widget Id Refresh Interval 1 com.uc4.ae.processmonitoring AUTO_MIN_5 DEFAULT.DASH at least one widget defined # Widget Id Refresh Interval 1 com.uc4.ae.processmonitoring MANUAL 2 com.uc4.ae.socialchangelog AUTO_MIN_1

    I might later add some steps to the post-process to check if the refresh interval of any widget is set to less than 5 minutes. It could then send an email to the owner of the dashboard to advise him to increase the interval. (The user who last modified the object could be added as a second column to the SQL query.)

    Enjoy.




  • 5.  RE: Dashboard XML

    Posted Nov 02, 2020 07:33 AM
    The awful Higher Logic software corrupted all of the code blocks that I had painstakingly inserted into the above message. It's so discouraging to get my hopes up that this community might be usable, only to be shot down again and again by reality. Broadcom seems dead set on losing customers.

    @Jason McClellan: fix it if you want. I won't even try.



  • 6.  RE: Dashboard XML

    Posted Nov 02, 2020 11:10 AM
    I can't seem to use code blocks as one would expect either.  I've been using paragraph=preformatted as a workaround;
    Now I can format stuff nicely;

    if condition1
    then do the following
    else
    don't do it
    endif
    But learning how to use paragraph=preformatted takes some time and effort.  Definitely frustrating.

    ------------------------------
    Pete Wirfs
    SAIF Corporation
    Salem Oregon USA
    ------------------------------



  • 7.  RE: Dashboard XML

    Community Manager
    Posted Nov 02, 2020 10:42 PM
    @Michael A. Lowry  are you using the insert code button in the editor to create a wrapper around the code?
    Insert Code into Popup


    ------------------------------
    Thank you
    Jason
    Broadcom Community Platform Admin, IT
    ------------------------------



  • 8.  RE: Dashboard XML

    Posted Nov 03, 2020 01:30 AM
    Yes.


  • 9.  RE: Dashboard XML

    Community Manager
    Posted Nov 03, 2020 09:41 PM
    @Michael A. Lowry &  ​@Pete Wirfs​   - I've opened a ticket with the vendor to check the configuration file.  Back in the Lithium days when we customized the tinymce editor you could choose what code to support which color-coded it and formatted it correctly.   I'll post back to the thread to see what our options are with this editor.   Jason

    ------------------------------
    Thank you
    Jason
    Broadcom Community Platform Admin, IT
    ------------------------------



  • 10.  RE: Dashboard XML

    Posted Nov 04, 2020 10:24 AM
    While you're at it, please add syntax highlighting for: