vCenter

 View Only
Expand all | Collapse all

Getting data from a CSV into Hyperic?

  • 1.  Getting data from a CSV into Hyperic?

    Posted Aug 15, 2013 03:36 PM

    Hi all,

    I have a CSV file that is produced by a script. It contain the columns: timestamp, seconds. How can i get the data into Hyperic so that it displays a graph of this? Will something like this work?

    hq-plugin.xml

    ---------------------

    <?xml version="1.0"?>

    <!--  Define the type of plugin this is, and which classes implement each plugin type -->

    <plugin>

         <server name="Script Reader">

           <plugin type="measurement" class="org.hyperic.hq.product.MeasurementPlugin"/>

     

         <property name="service_name" value="Monitoring Latencye"/>

             <config>       

              <option name="script" description="Data Collector script" default="/opt/monitoringDaemon/script.bat"/> 

         </config>

         <filter name="template" value="exec:file=%script%"/>

         <metric name="Script Data" category="PERFORMANCE" indicator="true" template="${template}:w/s"/>

         <metric name="Availability" template="win32:Service=${service_name}:Availability" indicator="true"/>

               

         <service name="Ping Service">

         <metric name="Availability" indicator="true"/>

         <metric name="Response Time 1" units="ms" indicator="true"/>

         <metric name="Response Time 2" units="ms" indicator="true"/>

         <metric name="Response Time 3" units="ms" indicator="true"/>

         <metric name="Response Time 4" units="ms" indicator="true"/>

         </service>

          

         </server>

    </plugin>

    /opt/monitoringDaemon/script.bat

    ---------------------------------

    TYPE datafile.csv

    /opt/monitoringDaemon/times.csv

    ---------------------------------

    timestamp | 0.012



  • 2.  RE: Getting data from a CSV into Hyperic?

    Posted Aug 16, 2013 05:32 PM

    There is no method to be able to put CSV data into the HQ database as you describe.

    Since your plugin looks like you're interested in Ping response, any reason you're not just using the Ping Platform Service already in HQ?



  • 3.  RE: Getting data from a CSV into Hyperic?

    Posted Aug 19, 2013 09:33 AM

    The ping response is from another script which connects over IPsec to various servers and outputs the results in a csv. this cannot be changed.

    I have written a shell script which echos the values and added a script service. I get a response time but no values in Hyperic under "Result Value". It is giving Execution Time values but 0% Availability. My script:

    #!/bin/bash

    INPUT=finalTimes.csv

    OLDIFS=$IFS

    IFS=,

    [ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }

    while read timestamp r1 r2 r3 r4

    do

        echo "Timestamp : $timestamp"

        echo "Response 1 : $r1"

        echo "Response 2 : $r2"

        echo "Response 3 : $r3"

        echo "Response 4 : $r4"

    done < $INPUT

    IFS=$OLDIFS



  • 4.  RE: Getting data from a CSV into Hyperic?

    Posted Aug 19, 2013 11:42 AM

    Two things.

    HQ wants key-value pairs returned with assignment.

    Try removing the spaces, avoids premature end of variable testing.

    Try changing your script to:

        echo "Response 1=$r1"

        echo "Response 2=$r2"

        echo "Response 3=$r3"

        echo "Response 4=$r4"

    I'd make it more like this to get rid of the spaces.

    echo Response1=$r1

    etc.


    I also use camelCase and the same names of the metric in both the plugin and the script to avoid confusion:

    <metric name="ResponseTime1" units="ms" indicator="true"/>

    ... in bash:

    echo ResponseTime1=$rs1



  • 5.  RE: Getting data from a CSV into Hyperic?

    Posted Aug 19, 2013 12:51 PM

    Hi, I'm just running the script through a service atm. Do i actually need a plugin as well? if so how do i get the script to use the plugin?



  • 6.  RE: Getting data from a CSV into Hyperic?

    Posted Aug 19, 2013 01:01 PM

    You already have a plugin in your first entry.

    You just need to modify your script being called to return the metric key/value pairs correctly and make sure the string in the 'echo' matches the metric name.



  • 7.  RE: Getting data from a CSV into Hyperic?

    Posted Aug 19, 2013 01:11 PM

    Ok i have this plugin uploaded to Hyperic and set up a service to point to the script under my Hyperic server platform. However it is still getting 0% availability and only 1 "Result Values".

    hq-plugin.xml

    ---------------------

    <?xml version="1.0"?>

    <!--  Define the type of plugin this is, and which classes implement each plugin type -->

    <plugin>

         <server name="Script Reader">

           <plugin type="measurement" class="org.hyperic.hq.product.MeasurementPlugin"/>

         <property name="service_name" value="Monitoring Latency"/>

            <config>    

                <option name="script" description="Data Collector script" default="/local/home/hyperic/scripts/csv-reading-script.sh"/>

            </config>

         <filter name="template" value="exec:file=%script%"/>

            <service name="Ping Service">

            <metric name="ResponseTime1" units="ms" indicator="true"/>

            <metric name="ResponseTime2" units="ms" indicator="true"/>

            <metric name="ResponseTime3" units="ms" indicator="true"/>

            <metric name="ResponseTime4" units="ms" indicator="true"/>

         </service>

         </server>

    </plugin>

    csv-reading-script.sh

    ---------------------------------

    #!/bin/bash

    INPUT=finalTimes.csv

    OLDIFS=$IFS

    IFS=,

    [ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }

    while read rs1 rs2 rs3 rs4

    do

        echo "ResponseTime1=$rs1"

        echo "ResponseTime2=$rs2"

        echo "ResponseTime3=$rs3"

        echo "ResponseTime4=$rs4"

    done < $INPUT

    IFS=$OLDIFS



  • 8.  RE: Getting data from a CSV into Hyperic?

    Posted Aug 19, 2013 01:24 PM

    You have 0% availability because you took out this statement:

    <metric name="Availability" template="win32:Service=${service_name}:Availability" indicator="true"/>

    Turn on agent debug and you will see what is being returned to the management server.

    In agent.properties

    agent.log.Level=DEBUG



  • 9.  RE: Getting data from a CSV into Hyperic?

    Posted Aug 19, 2013 01:27 PM

    Ah OK, what is the Linux equivalent for that line of code as references Windows?

    In the agent logs:

    [pool-1-thread-15] [ExecutableProcess@239] [/local/home/hyperic/scripts/csv-reading-script.sh]: Timeout running [/local/home/hyperic/scripts/csv-reading-script.sh ]

    19-08-2013 14:31:12,517 BST ERROR [pool-1-thread-9] [VSphereCollector@163] org.hyperic.hq.product.PluginException: ServiceInstance(https://xxx/sdk, xxx\SVC_xxx): com.vmware.vim25.InvalidLogin

    Message was edited by: GodAtum



  • 10.  RE: Getting data from a CSV into Hyperic?

    Posted Aug 19, 2013 01:33 PM

    Depends on what is running in your system.

    See the PTQL docs below.

    https://support.hyperic.com/display/SIGAR/PTQL



  • 11.  RE: Getting data from a CSV into Hyperic?

    Posted Aug 19, 2013 01:46 PM

    I have tried:

    <metric name="Availability" template="linux:Service=${service_name}:Availability" indicator="true"/>

    but still 0% avail. Looking at the logs it says:

    [pool-1-thread-15] [ExecutableProcess@239] [/local/home/hyperic/scripts/csv-reading-script.sh]: Timeout running [/local/home/hyperic/scripts/csv-reading-script.sh ]

    19-08-2013 14:31:12,517 BST ERROR [pool-1-thread-9] [VSphereCollector@163] org.hyperic.hq.product.PluginException: ServiceInstance(https://xxx/sdk, xxx\SVC_xxx): com.vmware.vim25.InvalidLogin

    and

    [system-0] [ScheduleThread@602] [3:10787:AVAILABILITY] Metric='exec:exec=,file=/local/home/hyperic/scripts/csv-reading-script.sh,args=,timeout=120:Availability' -> 0 timestamp=1376922000625

    [system-0] [ScheduleThread@544] collecting data for meas=[derivedId=21066|dsnId=21066]|interval=60000]

    [system-0] [ScheduleThread@602] [3:10787:THROUGHPUT] Metric='exec:exec=,file=/local/home/hyperic/scripts/csv-reading-script.sh,args=,timeout=120:ExecTime' -> 120,058 timestamp=1376922000625

    [system-0] [ScheduleThread@544] collecting data for meas=[derivedId=21067|dsnId=21067]|interval=60000]

    [tomcat-0] [ScheduleThread@602] [3:10315:UTILIZATION] Metric='Catalina:j2eeType=Servlet,name=jsp.resource.application.inventory.ChangeOwner_jsp,WebModule=//localhost/,J2EEApplication=none,J2EEServer=none:requestCount__RATE__=1m:jmx.url=ptql%3AState.Name.re%3Djava|jsvc%2CState.Name.Pne%3Djsvc%2CArgs.*.sw%3D-Dcatalina.base%3D/local/home/hyperic/server-5.0.0-EE/hq-engine/hq-server,jmx.username=system,jmx.password=****************' -> 0 timestamp=1376922000625

    [system-0] [ScheduleThread@602] [3:10787:UTILIZATION] Metric='exec:exec=,file=/local/home/hyperic/scripts/csv-reading-script.sh,args=,timeout=120:ResultValue' -> 0.009 timestamp=1376922000626

    Metric task 'exec:exec=,file=/local/home/hyperic/scripts/csv-reading-script.sh,args=,timeout=120:ResultValue' complete, duration: 67

    Message was edited by: GodAtum



  • 12.  RE: Getting data from a CSV into Hyperic?

    Posted Aug 21, 2013 10:44 AM

    Hi,

    Thank you for your help so far. I have manged to get the script itself working but am stuck with no vlaues apperaing in hyperic. In the debug logs:

    21-08-2013 11:36:37,147 BST DEBUG [pool-1-thread-7] [Execute@348] Done exit=0 /local/home/hyperic/scripts/csv-reading-script.sh

    21-08-2013 11:36:37,147 BST DEBUG [pool-1-thread-7] [ExecutableProcess@247] [/local/home/hyperic/scripts/csv-reading-script.sh]: Timestamp=2013-08-16 16:17:58

    ResponseTime1=0.016

    ResponseTime2=0.916

    ResponseTime3=0.022

    ResponseTime4=0.019

    21-08-2013 11:36:37,148 BST DEBUG [pool-1-thread-7] [Collector@403] name=exec, thread=pool-1-thread-7, result=Wed Aug 21 11:36:37 BST 2013 [Info] (/local/home/hyperic/scripts/csv-reading-script.sh) Timestamp=2013-08-16 16:17:58

    ResponseTime1=0.016

    ResponseTime2=0.916

    ResponseTime3=0.022

    ResponseTime4=0.019

    values={Availability=1.0, ResponseTime=27.0, ResponseTime4=0.019, ResponseTime2=0.916, ResponseCode=0.0, ResponseTime3=0.022, ResponseTime1=0.016, Timestamp=2013-08-16 16:17:58}



  • 13.  RE: Getting data from a CSV into Hyperic?

    Posted Aug 22, 2013 08:53 AM

    How do I get the script to reference the plugin?



  • 14.  RE: Getting data from a CSV into Hyperic?

    Posted Aug 19, 2013 04:12 PM

    I have edited the script so it only reads the last line in the csv:

    #!/bin/bash

    INPUT=finalTimes.csv

    OLDIFS=$IFS

    [ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }

    lastline=`cat $INPUT | tail -1`

    timestamp=`echo $lastline |cut -d, -f1`

    rs1=`echo $lastline |cut -d, -f2`

    rs2=`echo $lastline |cut -d, -f3`

    rs3=`echo $lastline |cut -d, -f4`

    rs4=`echo $lastline |cut -d, -f4`

        echo "timestamp=$timestamp"

        echo "ResponseTime1=$rs1"

        echo "ResponseTime2=$rs2"

        echo "ResponseTime3=$rs3"

        echo "ResponseTime4=$rs4"

    IFS=$OLDIFS

    however it does not give back any results in Hyperic. The output when runnijg the script via command line is:

    Timestamp=2013-08-16 16:17:58
    ResponseTime1=0.016
    ResponseTime2=0.916
    ResponseTime3=0.022
    ResponseTime4=0.019



  • 15.  RE: Getting data from a CSV into Hyperic?

    Posted Aug 21, 2013 05:59 PM

    Please UNSUBCRIBE ME

    On Mon, Aug 19, 2013 at 6:11 PM, GodAtum <hq-users@hyperic.org> wrote:

    I have edited the script so it only reads the last line in the csv:

    #!/bin/bash

    INPUT=finalTimes.csv

    OLDIFS=$IFS

    && { echo "$INPUT file not found"; exit 99; }

    lastline=`cat $INPUT | tail -1`

    timestamp=`echo $lastline |cut -d, -f1`

    rs1=`echo $lastline |cut -d, -f2`

    rs2=`echo $lastline |cut -d, -f3`

    rs3=`echo $lastline |cut -d, -f4`

    rs4=`echo $lastline |cut -d, -f4`

        echo "timestamp=$timestamp"

        echo "ResponseTime1=$rs1"

        echo "ResponseTime2=$rs2"

        echo "ResponseTime3=$rs3"

        echo "ResponseTime4=$rs4"

    IFS=$OLDIFS

    however it does not give back any results in Hyperic.

    >