DX Unified Infrastructure Management

 View Only
Expand all | Collapse all

Anyone use grafana?

  • 1.  Anyone use grafana?

    Posted Aug 23, 2021 04:18 PM
    Hello all!
    I don't know about you but in our case, the robot cabi dashboard is really slow and painful to load. The "metric" view loads faster but the filter on target acts weird.

    So we've decide to use grafana instead. Is there any of you that use grafana? I attached dashboard that we use maybe we could share ideas.
    Asking for the last 15 minutes or the last 30 days of data take the same amount of time: less than 10 sec
    The box at the top contains the list the HUB and the robot from this HUB so it's not like if we need to manually populate the list of robots.
    You can also select from the list the probe to show the probe config inside a table.

    What do you think?

    Thanks


  • 2.  RE: Anyone use grafana?

    Posted Aug 24, 2021 01:44 PM
    I have a plan to do it on ,may be will reach you on this . Is it is plan to do do only for HUB and primary UIM servers or all robots ?


  • 3.  RE: Anyone use grafana?

    Posted Aug 24, 2021 04:24 PM
    Edited by Guillaume Briere Aug 24, 2021 04:26 PM
    Hello,
    I have a different dashboard only for HUB that contain other type  of data (like robot online / robot offline / robot in maintenance from this hub). The screenshot is the one I've made for robots data.

    At the top there's a dropbox with the HUB name and the robot list. The data of the panel will load the information from the selected robot.

    Both of these boxes are populate from a query to the api /hub to get the list of the hubs and to /robot with a param hub:$hubList to retreive the robot list of this HUB.



  • 4.  RE: Anyone use grafana?

    Posted Aug 25, 2021 01:11 AM
    very nice - I love it! 

    i've tried to utilise grafana a couple of times via the info on these two pages but had no luck:
    https://medium.com/@hirzani/grafana-simple-json-datasource-for-ca-uim-integration-a5399cab6bb1
    https://medium.com/@hirzani/ca-uim-grafana-plugin-b350e7fd1b36

    Any chance you're open to sharing how to do it?

    Cheers



  • 5.  RE: Anyone use grafana?

    Posted Aug 25, 2021 09:38 AM

    Hello,

    Yes I can share.
    JSON API plugin as a datasource.
    You create a data source that will query the uimapi

    Then add panel to your dashboard, we will use CPU data as an example. The QOS data is retrieve through the <operator console>/uimapi/metrics. You can use the UIMAPI swagger to play with the parameters and find the right one for you.

    I use this to retrieve the latest data.

    For the last 30 days, last hour, etc I use these params

    With that settings, you will be able to use grafana's timerange selector at the top to control the number of days you want to retrieve.

    Another important thing is the target, if you want to reuse the same dashboard for all you robots, they all need to be configure the same way. So, in cdm if you have check the following box

    All your robots need to have this box check otherwise the "target" that will be return is the robotname. It does not really matter if the box is check or not as long as you are consistent accross your cdm QOS configuration. If you want to retrieve "cpu wait, cpu idle, etc" you need to adjust your params target.

    Then, you need to parse the data into "fields" and that's the part with a lot of try and errors (at least in my case it was).

    Here's a hint that could save you a lot of troubleshooting time. The JSON API plugin could not retrieve data that are on a different path level when the data's return.
    Example:

    [
      {
        "origin": "BIS",
        "id": "M6E54552497E1A29825C670D7A96029C1",
        "type": "1.5:1",
        "self": "http://whatever/uimapiM6E54552497E1A29825C670D7A96029C1",
        "source": "robotname",
        "target": "Total",
        "probe": "cdm",
        "for_computer_system": {
          "id": "32855",
          "self": "http://whatever/uimapi/devices/32855",
          "name": "robotname",
          "ip": "1.1.1.1"
        },
        "for_device": {
          "id": "D7D7AB049C102BC4359B7125F586FC2D1",
          "self": "http://whatever/uimapi/devices/D7D7AB049C102BC4359B7125F586FC2D1"
        },
        "for_configuration_item": {
          "id": "CAD117CA66004ACBD1CA4368EAC6412E5",
          "self": "http://whatever/uimapi/TBD/CAD117CA66004ACBD1CA4368EAC6412E5",
          "name": "CPU-Total",
          "qosName": "QOS_CPU_USAGE",
          "description": "CPU Usage",
          "unit": "%"
        },
        "uimMetricDefinition": null,
        "minSampleValue": 28.4,
        "maxSampleValue": 28.4,
        "meanSampleValue": 28.4,
        "sample": [
          {
            "time": "2021-08-25T12:58:07.000Z",
            "timeSinceEpoch": 1629896287,
            "value": 28.4,
            "rate": 300
          }
        ]
      }
    ]

    As you can see the "sample" (the data we want to show) is on a different level. You need to use $.0[sample].[value] and $.0[sample].[time] inside "field" to retrieve the data. For a panel that you only want to show 1 "target" there's no problem. 
     
    If you want a historical view of all the mount point inside the same graph it will not be possible because you will need to show the target (first level), the value (2nd level under sample) and the time (2nd level under sample) and the plugin will throw you an error. So for the historical view of the mount point it could only be done one mount point at the time.

    But you can cheat with the latest data only. You can do something that looks like this:

    The reason you can cheat is because at the first level (so at the same level as the target name (aka the mount point)) you have 3 values.
    "minSampleValue": 28.4,
    "maxSampleValue": 28.4,
    "meanSampleValue": 28.4,

    Inside the params if you select "latest" as the period you receive only the last sample and the value for the minimum, maximum and the average will all be the same as the sample (since there's only one) so you can build a bar gauge panel with one of the value.

    If you're familiar with grafana you will know this but just in case, you will need to configure dashboard variable to retrieve the mount point if you don't want to hardcode the mount point name for a disk space historical view.

    In my case, I use 3 variables to build my dashboard
    $hubList
    $RobotName
    $MountPoint

    I do a variable to get the list of HUB, then from this HUB I query the robot list then I query the list of mount point of the robotname

    At the top of the dashboard, there's a box and you can select the mount point that will build your disk space usage over time ...

    if you use the $MountPoint variable inside your params


    Maybe this does not seem to make sense at first or seems difficult to integrate but starts playing with it and it will make more sense.

    Start by hardcoding the value inside the params and when you're comfortable go with the variable.

    Hope it helps




  • 6.  RE: Anyone use grafana?

    Posted Aug 27, 2021 03:56 AM
    Edited by Steve B Aug 27, 2021 09:01 AM
    That's some great info and got me most of the way there! Thank you so much :)

    Any chance that you could please post a couple of screenshots of your variable configurations? This is where I'm currently stuck...

    I worked it out! Never used grafana but I'm enjoying this :)




  • 7.  RE: Anyone use grafana?

    Posted Aug 27, 2021 09:05 AM
    Hello,
    Yes sure.
    hubList:
    Field: $.hub[*].address
    Path: /hubs
    Params: None

    Regex: /<your uim domain>\/(.*)\/.*\//
    I extracted the HUB Name but if you don't use a regex will you have the long format /Domain/HUBNAME/Hostname/hub
    Sort: Alphabetical (asc)

    RobotName:
    Field: 
    $.*[robot]
    Path: /robots
    Params: 
        key: hub
        Value: $hubList

    Regex: None
    Sort: Alphabetical (asc)

    MountPoint:
    Field:
    $.*[target]
    Path: /metrics
    Params:
    I could have use period: latest but maybe some mount point has been remove since the last 30 days and the user will still want to see the data until it was remove.

    Hope it helps


  • 8.  RE: Anyone use grafana?

    Posted Nov 26, 2021 05:39 PM
    Dear Guillaume Briere

    I'm trying to use this example but in the query Query inspector I have null values so therer's "No data" in that panel

    for_configuration_item:Object

    name:"C:\"
    qosName:"QOS_DISK_USAGE_PERC"
    description:"Disk Usage (%)"
    unit:"%"
    uimMetricDefinition:null
    minSampleValue:null
    maxSampleValue:null
    meanSampleValue:null
    sample:Array[0]

    Another question is the variable host $.[*].devId without automatic update on all panels option when selected

    Thanks


  • 9.  RE: Anyone use grafana?

    Posted Nov 29, 2021 11:17 AM

    Hello,

    If you go to https://<operator console>/uimapi/swagger-ui.html and you select "metrics" are you able to retrieve the data?

    You need to be able to retrieve the data from this page first. You need to find the right combination of ID and Target.
    You also need to be sure that the QOS is configured on the probe and if you overwrite the source of the QOS, that's the value you need to use as the ID.

    When you are able to retrieve the information from this page then you can  build your grafana query

    I don't understand your question "Another question is the variable host $.[*].devId without automatic update on all panels option when selected"





  • 10.  RE: Anyone use grafana?

    Posted Nov 29, 2021 11:39 AM
    Edited by Paulo Pires Nov 29, 2021 11:42 AM
      |   view attached
    Guillaume Briere, taking your example

    <metrics>
      <metric>
        <origin>XXX_hub</origin>
        <id>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</id>
        <type>1.1:3</type>
        <self>http://XXXXXXXXXX/uimapiXXXXXXXXXXXXXXXXXXXXXX</self>
        <source>XXXXXXXXX</source>
        <target>C:\</target>
        <probe>cdm</probe>
        <for_computer_system>
          <id>1</id>
          <self>http://XXXXXXXX/uimapi/devices/1</self>
          <name>XXXXXXX</name>
          <ip>XXXXXXXXX</ip>
        </for_computer_system>
        <for_device>
          <id>XXXXXXXXXXXXXXXXXXXXXXXXX</id>
          <self>http://XXXXXXXXXX/uimapi/devices/XXXXXXXXXXXXXXXXX</self>
        </for_device>
        <for_configuration_item>
          <id>XXXXXXXXXXXXXXXX</id>
          <self>http://XXXXXXX/uimapi/TBD/XXXXXXXXXXXXXXXXXXX</self>
          <name>C:\</name>
          <qosName>QOS_DISK_USAGE_PERC</qosName>
          <description>Disk Usage (%)</description>
          <unit>%</unit>
        </for_configuration_item>
        <uimMetricDefinition/>
        <minSampleValue/>
        <maxSampleValue/>
        <meanSampleValue/>
        <samples/>
      </metric>



  • 11.  RE: Anyone use grafana?

    Posted Nov 29, 2021 11:44 AM
    Edited by Paulo Pires Nov 29, 2021 11:46 AM
      |   view attached
    and for CDM probe


  • 12.  RE: Anyone use grafana?

    Posted Aug 31, 2021 09:32 AM
    Hey Steve! 

    Do you want to share your progress?

    If you use this as a params

    With a variable like this one as the source

    And another one that extract the target you can create a generic panel for a probe.

    I did this for net_connect and url_resonse


    You can create a generic panel for a specific probe.


  • 13.  RE: Anyone use grafana?

    Posted Aug 31, 2021 11:31 PM

    Hi Guillaume,

    Happy to share what I've come up with over the past couple of days!

    First step was to create a "host metrics" dashboard to get my head around grafana, variables and the UIM API. That was simple enough:


    After I was comfortable, I started working on a SQL metrics dashboard (still a work in progress):


    Thanks again for your help with it all! The info you provided had all the details required to get anyone started.

    Cheers




  • 14.  RE: Anyone use grafana?

    Posted Sep 01, 2021 09:14 AM
    Hello,
    Very clean and very nice. I'm proud that I'm not the only one that integrate UIM inside grafana successfully.


  • 15.  RE: Anyone use grafana?

    Posted Oct 12, 2021 11:16 AM
    Dear Steve B
    Thanks for sharing this lovely "host metrics" dashboard.
    Can you share how you did it? Or is there a way to share your dashboard?
    I'm trying to do as Guillaume Briere says, but always with "No data"


  • 16.  RE: Anyone use grafana?

    Posted Oct 12, 2021 05:38 PM
    Edited by Steve B Oct 13, 2021 07:53 AM
    Hi @Paulo Pires,

    The first thing to check is the Data Source. This is what mine looks like:


    The accept header has this: application/json

    Try downloading this file (rename from .txt to .json):  https://community.broadcom.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=05adbdaf-587f-471e-9bb7-fe2a1b8ac9f5




  • 17.  RE: Anyone use grafana?

    Posted Oct 13, 2021 03:51 AM
    Edited by Paulo Pires Oct 16, 2021 03:14 PM
    Hi Steve B

    Thanks for sharing. Check the image :(

    The accept header value is this: application/json ???




  • 18.  RE: Anyone use grafana?

    Posted Oct 13, 2021 04:12 AM

    Can you get to your uimapi url via a browser?

    I've got ours set to https://uimhost/uimapi but we have it listening on 443 and have a valid certificate. You might need to drop back to http:// if you dont have it setup properly




  • 19.  RE: Anyone use grafana?

    Posted Oct 13, 2021 04:25 AM
    Edited by Paulo Pires Oct 13, 2021 05:00 AM
    Hi @Steve B

    ​http://<OC_HOST>/uimapi

    Now I'm collecting some metrics on the dashboard, I had to set the datasource as default




  • 20.  RE: Anyone use grafana?

    Posted Oct 13, 2021 07:44 AM
    Hi Steve,
    Thank you for sharing inputs. While importing your JSON content into dashboard , I am getting below error.



    ------------------------------
    Regards,

    Eshwar
    ------------------------------



  • 21.  RE: Anyone use grafana?

    Posted Oct 13, 2021 07:51 AM
      |   view attached
    Try this file, just rename from .txt to .json

    Attachment(s)



  • 22.  RE: Anyone use grafana?

    Posted Oct 13, 2021 08:43 AM
    Thanks Steve,
    It's working  now. What was the issue previously?

    ------------------------------
    Regards,

    Eshwar
    ------------------------------



  • 23.  RE: Anyone use grafana?

    Posted Oct 13, 2021 02:52 PM
    Edited by Paulo Pires Oct 16, 2021 03:15 PM
    Dear Steve B and @Purneswara Rao Konda

    I had to set the datasource JSON API as default ;)

    I'm receiving some data but just "CPU Utilisation %"

    Update: @Steve Bagshaw, you are using metricas like "QOS_MEMORY_PHYSICAL_PERC" and "QOS_MEMORY_PHYSICAL" in these 2 panels, I only have "QOS_MEMORY_PERC_USAGE" ;) I need to check my CDM probe :)






  • 24.  RE: Anyone use grafana?

    Posted Oct 13, 2021 03:01 PM
    Dear @Purneswara Rao Konda

    Copy and paste the Json content and before save, just delete the last } and add it again ;)

    Thanks

    Paulo​​


  • 25.  RE: Anyone use grafana?

    Posted Oct 13, 2021 02:55 PM
    Edited by Paulo Pires Oct 16, 2021 03:13 PM
    Dear Steve B

    Can you share this specific dashboard?

    Thanks

    Paulo​


  • 26.  RE: Anyone use grafana?

    Posted Oct 13, 2021 04:19 PM
    Unfortunately not, there's a lot of stuff in them specific to our UIM build but you should have everything you need to start your grafana/uim journey!


  • 27.  RE: Anyone use grafana?

    Posted Oct 20, 2021 01:45 PM

    Hello,

    Just a hint for those who could have issue finding the right QOS and which parameters to configure inside the request to the api.

    I use a lot the admin console, it gives the QOS name of the data to use.

    I also use a lot the "metrics" page of a probe from the documentation, example: https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/ca-unified-infrastructure-management-probes/GA/alphabetical-probe-articles/cdm-cpu-disk-memory-performance-monitoring/cdm-metrics.html

    I also use the data_engine gui from the IM. It gives valuable information about the unit of the data

    This is helpful when you build the visualization of the data inside grafana since it automatically convert unit into a human readable format. Even if the original source is in mb, grafana will show the number in GB or TB instead of giving you an insane long number that you need to convert in your head.


    Then finally I use the uimapi I start only with my robot name and the QOS name without any other filter to see what's the data looks like.


    If I don't find any data, I try with the fqdn of the robot. If I found the data, that means that I need to modify the controller probe.


    Keyword is consistency, if you check the box it needs to be check everywhere if you want to use your dashboard for multiple robot. If it's not check, make sure it's not check everywhere. But it's easier to manage if it's check everywhere.

    Thank you




  • 28.  RE: Anyone use grafana?

    Posted Oct 21, 2021 09:35 AM

    Oh! And one last thing to help you troubleshoot the "no data" message.
    If you go to OC / inventory / Metrics and you don't find your metric somewhere here

    That's an indication that there's a problem, probably the QOS is not configured or you change the source of the QOS.

    But if you find your QOS here and you are able to see data, you will be able to show it inside grafana. You just need to do more testing inside the api swagger.

    Hope it helps




  • 29.  RE: Anyone use grafana?

    Posted Dec 22, 2021 04:53 AM
      |   view attached
    Hi,

    Am very New to UIM / Grafana . am trying to create a datasource and getting the below error . 
    Our UIM https://********/uimapi/  resolve  to https://*******/uimapi/docs/index.html# 

    Please help in this regard.


  • 30.  RE: Anyone use grafana?

    Posted Oct 16, 2021 01:13 PM
    Edited by Paulo Pires Oct 16, 2021 07:44 PM
    Hi Steve B

    I'm trying to access data from SQL servers, so I duplicate one of your dashboards, but so far "no data". I am using the example "QOS_SQLSERVER_ACTIVE_USERS".

    Can you explain how you arrived at these variables? Where are they documented?
    • "id_lookup" > "by_device_id"
    • "metric_type_lookup" > "by_metric_name"
    • "period" > "latest"

    Update > /metrics/map ;)
    Returns a mapping of metric ids to their corresponding computer system/configuration item/device ids.






  • 31.  RE: Anyone use grafana?

    Posted Oct 18, 2021 07:55 AM
    Hi @Steve B

    Can you share how you did it with that SQL Server backups dashboard? I'm having trouble accessing SQL Server metrics.

    Thanks

    Paulo