DX Infrastructure Manager

Expand all | Collapse all

Anyone use grafana?

  • 1.  Anyone use grafana?

    Posted 08-23-2021 04:18 PM
    Hello all!
    I don't know about you but in our case, the robot cabi dashboard is really slow and painful to load. The "metric" view loads faster but the filter on target acts weird.

    So we've decide to use grafana instead. Is there any of you that use grafana? I attached dashboard that we use maybe we could share ideas.
    Asking for the last 15 minutes or the last 30 days of data take the same amount of time: less than 10 sec
    The box at the top contains the list the HUB and the robot from this HUB so it's not like if we need to manually populate the list of robots.
    You can also select from the list the probe to show the probe config inside a table.

    What do you think?

    Thanks


  • 2.  RE: Anyone use grafana?

    Posted 08-24-2021 01:44 PM
    I have a plan to do it on ,may be will reach you on this . Is it is plan to do do only for HUB and primary UIM servers or all robots ?


  • 3.  RE: Anyone use grafana?

    Posted 08-31-2021 09:32 AM
    Hey Steve!

    Do you want to share your progress?

    If you use this as a params

    With a variable like this one as the source

    And another one that extract the target you can create a generic panel for a probe.

    I did this for net_connect and url_resonse


    You can create a generic panel for a specific probe.


  • 4.  RE: Anyone use grafana?

    Posted 08-25-2021 01:11 AM
    very nice - I love it!

    i've tried to utilise grafana a couple of times via the info on these two pages but had no luck:
    https://medium.com/@hirzani/grafana-simple-json-datasource-for-ca-uim-integration-a5399cab6bb1
    https://medium.com/@hirzani/ca-uim-grafana-plugin-b350e7fd1b36

    Any chance you're open to sharing how to do it?

    Cheers



  • 5.  RE: Anyone use grafana?

    Posted 08-24-2021 04:24 PM
    Edited by Guillaume Briere 08-24-2021 04:26 PM
    Hello,
    I have a different dashboard only for HUB that contain other type  of data (like robot online / robot offline / robot in maintenance from this hub). The screenshot is the one I've made for robots data.

    At the top there's a dropbox with the HUB name and the robot list. The data of the panel will load the information from the selected robot.

    Both of these boxes are populate from a query to the api /hub to get the list of the hubs and to /robot with a param hub:$hubList to retreive the robot list of this HUB.



  • 6.  RE: Anyone use grafana?

    Posted 08-25-2021 09:38 AM

    Hello,

    Yes I can share.
    JSON API plugin as a datasource.
    You create a data source that will query the uimapi

    Then add panel to your dashboard, we will use CPU data as an example. The QOS data is retrieve through the <operator console>/uimapi/metrics. You can use the UIMAPI swagger to play with the parameters and find the right one for you.

    I use this to retrieve the latest data.

    For the last 30 days, last hour, etc I use these params

    With that settings, you will be able to use grafana's timerange selector at the top to control the number of days you want to retrieve.

    Another important thing is the target, if you want to reuse the same dashboard for all you robots, they all need to be configure the same way. So, in cdm if you have check the following box

    All your robots need to have this box check otherwise the "target" that will be return is the robotname. It does not really matter if the box is check or not as long as you are consistent accross your cdm QOS configuration. If you want to retrieve "cpu wait, cpu idle, etc" you need to adjust your params target.

    Then, you need to parse the data into "fields" and that's the part with a lot of try and errors (at least in my case it was).

    Here's a hint that could save you a lot of troubleshooting time. The JSON API plugin could not retrieve data that are on a different path level when the data's return.
    Example:

    [
      {
        "origin": "BIS",
        "id": "M6E54552497E1A29825C670D7A96029C1",
        "type": "1.5:1",
        "self": "http://whatever/uimapiM6E54552497E1A29825C670D7A96029C1",
        "source": "robotname",
        "target": "Total",
        "probe": "cdm",
        "for_computer_system": {
          "id": "32855",
          "self": "http://whatever/uimapi/devices/32855",
          "name": "robotname",
          "ip": "1.1.1.1"
        },
        "for_device": {
          "id": "D7D7AB049C102BC4359B7125F586FC2D1",
          "self": "http://whatever/uimapi/devices/D7D7AB049C102BC4359B7125F586FC2D1"
        },
        "for_configuration_item": {
          "id": "CAD117CA66004ACBD1CA4368EAC6412E5",
          "self": "http://whatever/uimapi/TBD/CAD117CA66004ACBD1CA4368EAC6412E5",
          "name": "CPU-Total",
          "qosName": "QOS_CPU_USAGE",
          "description": "CPU Usage",
          "unit": "%"
        },
        "uimMetricDefinition": null,
        "minSampleValue": 28.4,
        "maxSampleValue": 28.4,
        "meanSampleValue": 28.4,
        "sample": [
          {
            "time": "2021-08-25T12:58:07.000Z",
            "timeSinceEpoch": 1629896287,
            "value": 28.4,
            "rate": 300
          }
        ]
      }
    ]

    As you can see the "sample" (the data we want to show) is on a different level. You need to use $.0[sample].[value] and $.0[sample].[time] inside "field" to retrieve the data. For a panel that you only want to show 1 "target" there's no problem.

    If you want a historical view of all the mount point inside the same graph it will not be possible because you will need to show the target (first level), the value (2nd level under sample) and the time (2nd level under sample) and the plugin will throw you an error. So for the historical view of the mount point it could only be done one mount point at the time.

    But you can cheat with the latest data only. You can do something that looks like this:

    The reason you can cheat is because at the first level (so at the same level as the target name (aka the mount point)) you have 3 values.
    "minSampleValue": 28.4,
    "maxSampleValue": 28.4,
    "meanSampleValue": 28.4,

    Inside the params if you select "latest" as the period you receive only the last sample and the value for the minimum, maximum and the average will all be the same as the sample (since there's only one) so you can build a bar gauge panel with one of the value.

    If you're familiar with grafana you will know this but just in case, you will need to configure dashboard variable to retrieve the mount point if you don't want to hardcode the mount point name for a disk space historical view.

    In my case, I use 3 variables to build my dashboard
    $hubList
    $RobotName
    $MountPoint

    I do a variable to get the list of HUB, then from this HUB I query the robot list then I query the list of mount point of the robotname

    At the top of the dashboard, there's a box and you can select the mount point that will build your disk space usage over time ...

    if you use the $MountPoint variable inside your params


    Maybe this does not seem to make sense at first or seems difficult to integrate but starts playing with it and it will make more sense.

    Start by hardcoding the value inside the params and when you're comfortable go with the variable.

    Hope it helps




  • 7.  RE: Anyone use grafana?

    Posted 08-27-2021 03:56 AM
    Edited by Steve B 08-27-2021 09:01 AM
    That's some great info and got me most of the way there! Thank you so much :)

    Any chance that you could please post a couple of screenshots of your variable configurations? This is where I'm currently stuck...

    I worked it out! Never used grafana but I'm enjoying this :)




  • 8.  RE: Anyone use grafana?

    Posted 08-27-2021 09:05 AM
    Hello,
    Yes sure.
    hubList:
    Field: $.hub[*].address
    Path: /hubs
    Params: None

    Regex: /<your uim domain>\/(.*)\/.*\//
    I extracted the HUB Name but if you don't use a regex will you have the long format /Domain/HUBNAME/Hostname/hub
    Sort: Alphabetical (asc)

    RobotName:
    Field: 
    $.*[robot]
    Path: /robots
    Params: 
        key: hub
        Value: $hubList

    Regex: None
    Sort: Alphabetical (asc)

    MountPoint:
    Field:
    $.*[target]
    Path: /metrics
    Params:
    I could have use period: latest but maybe some mount point has been remove since the last 30 days and the user will still want to see the data until it was remove.

    Hope it helps


  • 9.  RE: Anyone use grafana?

    Posted 08-31-2021 11:31 PM

    Hi Guillaume,

    Happy to share what I've come up with over the past couple of days!

    First step was to create a "host metrics" dashboard to get my head around grafana, variables and the UIM API. That was simple enough:


    After I was comfortable, I started working on a SQL metrics dashboard (still a work in progress):


    Thanks again for your help with it all! The info you provided had all the details required to get anyone started.

    Cheers




  • 10.  RE: Anyone use grafana?

    Posted 09-01-2021 09:14 AM
    Hello,
    Very clean and very nice. I'm proud that I'm not the only one that integrate UIM inside grafana successfully.


  • 11.  RE: Anyone use grafana?

    Posted 4 days ago
    Dear Steve B
    Thanks for sharing this lovely "host metrics" dashboard.
    Can you share how you did it? Or is there a way to share your dashboard?
    I'm trying to do as Guillaume Briere says, but always with "No data"


  • 12.  RE: Anyone use grafana?

    Posted 3 days ago
    Edited by Paulo Pires 13 hours ago
    Dear Steve B

    Can you share this specific dashboard?

    Thanks

    Paulo​


  • 13.  RE: Anyone use grafana?

    Posted 15 hours ago
    Edited by Paulo Pires 8 hours ago
    Hi Steve B

    I'm trying to access data from SQL servers, so I duplicate one of your dashboards, but so far "no data". I am using the example "QOS_SQLSERVER_ACTIVE_USERS".

    Can you explain how you arrived at these variables? Where are they documented?
    • "id_lookup" > "by_device_id"
    • "metric_type_lookup" > "by_metric_name"
    • "period" > "latest"

    Update > /metrics/map ;)
    Returns a mapping of metric ids to their corresponding computer system/configuration item/device ids.






  • 14.  RE: Anyone use grafana?

    Posted 4 days ago
    Edited by Steve B 3 days ago
    Hi @Paulo Pires,

    The first thing to check is the Data Source. This is what mine looks like:


    The accept header has this: application/json

    Try downloading this file (rename from .txt to .json):  https://community.broadcom.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=05adbdaf-587f-471e-9bb7-fe2a1b8ac9f5




  • 15.  RE: Anyone use grafana?

    Posted 4 days ago
    Edited by Paulo Pires 13 hours ago
    Hi Steve B

    Thanks for sharing. Check the image :(

    The accept header value is this: application/json ???




  • 16.  RE: Anyone use grafana?

    Posted 3 days ago
    Hi Steve,
    Thank you for sharing inputs. While importing your JSON content into dashboard , I am getting below error.



    ------------------------------
    Regards,

    Eshwar
    ------------------------------



  • 17.  RE: Anyone use grafana?

    Posted 4 days ago

    Can you get to your uimapi url via a browser?

    I've got ours set to https://uimhost/uimapi but we have it listening on 443 and have a valid certificate. You might need to drop back to http:// if you dont have it setup properly




  • 18.  RE: Anyone use grafana?

    Posted 4 days ago
    Edited by Paulo Pires 3 days ago
    Hi @Steve B

    ​http://<OC_HOST>/uimapi

    Now I'm collecting some metrics on the dashboard, I had to set the datasource as default




  • 19.  RE: Anyone use grafana?

    Posted 3 days ago
      |   view attached
    Try this file, just rename from .txt to .json

    Attachment(s)



  • 20.  RE: Anyone use grafana?

    Posted 3 days ago
    Dear @Eshwar K

    Copy and paste the Json content and before save, just delete the last } and add it again ;)

    Thanks

    Paulo​​


  • 21.  RE: Anyone use grafana?

    Posted 3 days ago
    Thanks Steve,
    It's working  now. What was the issue previously?

    ------------------------------
    Regards,

    Eshwar
    ------------------------------



  • 22.  RE: Anyone use grafana?

    Posted 3 days ago
    Edited by Paulo Pires 13 hours ago
    Dear Steve B and @Eshwar K

    I had to set the datasource JSON API as default ;)

    I'm receiving some data but just "CPU Utilisation %"

    Update: @Steve Bagshaw, you are using metricas like "QOS_MEMORY_PHYSICAL_PERC" and "QOS_MEMORY_PHYSICAL" in these 2 panels, I only have "QOS_MEMORY_PERC_USAGE" ;) I need to check my CDM probe :)






  • 23.  RE: Anyone use grafana?

    Posted 3 days ago
    Unfortunately not, there's a lot of stuff in them specific to our UIM build but you should have everything you need to start your grafana/uim journey!