CA APM symbiotic relationship with Splunk?

[Billy Cole]

So, it has come to pass.  Our company's Splunk team has been joined with the CA APM/CEM/UIM and HP SiteScope team to form an Enterprise Monitoring team.

 

Last year, I posted the first shot into this topic and had no responses, maybe this time, someone has cut through the jungles, trail blazed a path that they will share.

 

Log Analytics - Splunk and Gartner APM MQ 2014 report 

 

My general view is that if CA APM does not author, direct, join the story of CA APM and Splunk then someone else will

 

 

What is CA's integration answer for integrating Splunk to APM?  

 

What are the factors would need to be reviewed and discovered in Splunk or APM during the integration effort?

 

Is there or can there be a symbiotic relationship between CA APM and Splunk or are they on a collision course?

 

<Please, no "what ever you want it to do" because I don't know and if left to our own devices, one or the other will suffer in the light of the other>

 

My history is very heavy in APM (5+ years), and very light in Splunk (< 1 year) terms and features.  

 

Other APM vendor stories with Splunk:

 

AppDynamics & Splunk – Better Together

 

Dynatrace Application Performance Management | Splunkbase 

 

Application Delivery | Application Performance Management | Splunk 

 

Splunk App for CA APM | Devpost 

[Hallett German]

Hi Billy:

Thanks for asking this question. I have forwarded your note to an internal APM group requesting a response. No guarantee  that you will get one. But more people are now aware of your query.

 

Thanks

Hal German

[Chris_Kline]

Hi Billy:

Thanks for reaching out and opening the topic.  This is the type of topic that's more easily had verbally.  Would you like to meet with our product team after the holidays for a conversation?

 

Thanks,

 

Chris

[Billy Cole]

Thank you Chris.

 

Yes, a meeting on this topic would be very helpful.  Please send a meeting request with the date and times.  On the time, I'm located in PA so eastern time zone.   I the early shift, so in by 7:00 am, lunch between 11-12 then out of the office after 3:30 pm.

 

Billy

[Hallett German]

Thanks Chris for replying! Much appreciated

Hal 

[Billy Cole]

For it has come to pass.  The CA APM/HP SiteScope performance testing and monitoring team and the Splunk team are becoming one team, Enterprise Monitoring.  The testing part has been spun off to a testing center of excellence. 

 

Now the really tough questions are getting asked such as, hey do you know about Splunk Glass Table? How does that differ from CA APM Team Center?  Hey should we invest in the Splunk Glass Table expansion modules?  Splunk stores all the data for a very long time but CA APM holds all the data for a week, then 4:1 for a month then 40:1 for two months.

 

I tried the typical, What is the KPI? What value do you place on that KPI? How useful is that KPI after a week, month or months? Where would you get value from strategy? It sort of worked.  Seemed like a Splunk sales person, a really good sales person showed the Splunk glasstable stuff with lots of graphs, flashy widgets, then the standard kill shot - type statement

 

"Traditionally, the end-to-end performance of systems and apps supporting digital strategies is difficult for the business team to monitor against SLAs and KPIs they have established without bolting this view together with a variety of products," according to Maureen Fleming, vice president of BPM and middleware research,"

 Splunk IT Service Intelligence - Glass Table Overview (Graphic: Business Wire) - TheStreet 

 

 

Has anyone had any interaction between CA APM and Splunk?  

 

How did that turn out?  

 

Does anyone have any suggestions on how to play nice since Splunk is not going away?

 

Splunk has a foothold with auditing access information and we have invested quite a bit into that effort.

 

Billy

[Hallett German]

Hi Billy:

 

>Has anyone had any interaction between CA APM and Splunk?  

  I see past notes on this topic. https://communities.ca.com/message/241900084?commentID=241900084#comment-241900084 But these deal more with Log Reading Splunk Data and a field pack to do so

Did you talk with Product Management as listed above? If not, this might be a good time now.

 

Thanks

Hal German

[Billy Cole]

Hey Hal,

 

I haven't heard back from Chris or anyone else from Product Management on this topic. 

 

As far as the interaction between CA APM and Splunk, this isn't so much how to get them to interact or carving APM logs up with APM instead of Splunk, but more of being able to sell both and what solutions can be done with one or the other, or both. 

 

Why develop a log parser for APM when you ingest the log into Splunk, you run a query and here ya go.  What, you need another field in a log that is already in Splunk, here ya go, here is the query for that.  We had three different log parsers, searching for specific text within a log and sending how many occurrences were found.  Yes, quite a bit of logic for log rolling and publishing a change value by storing the previous count in a file.   So since Splunk is already ingesting the logs, here are the alerts that does the same as all the APM log parsers.

 

So Splunk has quite a few expansions, applications, that have a out of the box query set for various enterprise security purposes.  This provides a foot-hold into the enterprise performance space, one that the sole citizen was APM.  Now, Splunk has all of the system, application and access logs, and look...with a simple expression, we can query the average response time over a day, week and quarter.  Pay no mind to the man behind the curtain.  Look, we can turn that simple expression into a pretty graph, you like graphs, everyone likes graphs.

 

 

Billy

[Hallett German]

Hi Billy:

  Sent a note to PM to contact you. (Which I hope happens.) Mike's note is thoughtful as I would expect

Thanks

Hal German

[spyderjacks]

Regards the "play nice" portion of your questions, every APM tool has a sweet spot.  For Splunk, it works on log info, and logging is part of ANY APM landscape.

 

Using CA-APM, via the EPAgent, for log scraping.  It's OK - if you don't have another tool.  Splunk does a great job here, for Logs.

 

So when you look at your portfolio, especially for those apps where instrumentation is not possible with the OOTB agent, AND they can generate logs -> Splunk is your answer - a great fit.

 

Regards the "Glass Table" - this may be an important use case in your shop.  Splunk has a full suite of analytics, purpose built by a PhD Data Scientist - freak'n fabulous.  But how many APM shops really do data analytics?  Not many.

 

There are some hard questions to ask when you start going for the "single version of the truth" that a "Glass Table" represents:

   - what are your first priorities, that you employ APM visibility for?

   - what visibility gaps remain - areas where you have problems, but no metrics of significance?

   - what business value is enhanced by bringing APM visibility (all your tools) to more people?

 

These don't really go away, with the arrival of a "dead ****" tool set.

 

You need to communicate to the business the value each of the tools bring, to your monitoring initiative, and what gaps remain unaddressed.  Ultimately, how the combined tool set helps you do a better job - and hopefully the politics of who is better, and who owns the single view, ya-da, ya-da... will recede in front of the main purpose for your organization, and what you need to do the best job going forward.

 

"Do not go softly in that dark night"... use the deployment planning, capacity and sizing techniques you have refined for APM and ask those questions of Splunk-centric universe.  When you get an alert from APM... when do you see those metrics in Splunk?  You don't need 'kill points' - you need to use your tools for what they are good at!  ;-)

[Billy Cole]

Thank you Michael.  I've tried to apply the concepts from your "APM best Practices" book, but the APM took a direction that APM has been hammered, sort of the square peg into a round hole.  

 

The prospective of the context and questions certainly helps try to bring this into focus and find a target. 

 

Currently the majority of the alert structures within our APM is based on the custom epagent plugin metrics (df/netstat/vmstat/lparstat/free/lsps/etc) with maybe the average response time being alerted on.  With this the APM isn't really used by the business folks, application development, or the product support folks but the system groups, websphere, Unix.  Which is, in my opinion caused the square peg in the round hole issue.

 

The next issue with our APM implementation is the company's aversion to using anything that is not contractually supported.  So the field packs that provide transaction linkage between different points, not allowed.  Customization to an agent to tie the transaction id to the next layer, nope.  So we typically have one maybe two jumps within a JVM to the databases or a socket before the trace stops.   Hopefully with our pending upgrade to 10.5 I can get the cross JVM with RMI calls to provide a couple more trace steps.

 

- What have been our first priorities, that we employ APM visibility for?

Over 90% of our current APM position (alerts/dashboards) is for the middle tier, websphere, MQ and a bit down to the databases.  Over the last few years, tried to provide more application and services (SOA/webservices) metrics and alerts but very little traction on that field.

 

- what visibility gaps remain? 

Back in 9.6 when TIM was only supported on RedHat, we shut down TIM/CEM since at that time we did not have a RedHat support contract but even when we did have it, we couldn't get enough business attention to configure, identify, and provide useful business metrics.    From the end user to the front door, with the browser agent or the mobile agent both of which we do not have deployed, would provide coverage to those points.    Next would be the distance between the application and the physical/virtual world of CPU/RAM/NIC/Disk.  We have been developing plugins to the epagent to provide coverage.  Through a long story we have CA UIM also but got blocked due to the robots having to be installed as root.  So with that, we keep expanding the epagents.

 

- what business value is enhanced by bringing APM visibility (all your tools) to more people?

Only in the last few months have I been able to really engage more than the mid and database tier groups.  So really working toward providing directed, focused dashboards and alerts to directed product owners.   The three or four meetings I have had with one of the business product owners was fairly successful.  The base dashboard has as the center-piece the mainframe CPU since when the CPU approaches high levels, the average response times of the primary service shoots up and end users start to complain.  Then from there, the services five KPI (average response, concurrent, responses, stall, error) with alerts on the average response time since that is typically the driver of the application's issues.  

 

Now the new kid on the block is coming onto the court, Splunk.  I know very little about Splunk in the grand scheme of things but really doesn't help when I get cornered with, well today was Splunk Glass table and having to try to understand if they just saw a bunch of pretty lights and really don't have a business need or case then talk them away from the edge decision of buying a disco ball or first learn to dance.

 

[Billy Cole]

While this link is a few years old, Apr 1, 2015, it is a good discussion on the Splunk vs APM 

   https://www.quora.com/Which-APM-tool-is-better-Splunk-or-AppDynamics

One interesting developments is Splunk having a Gigamon visibility

   Gigamon Visibility for Splunk Enterprise | Gigamon 

What has me concerned, is we have a gigamon for the CEM/TIM, which is currently idle and we could re-purpose the gigamon to feed Splunk.  Yes, this is a rather large amount of data to feed into Splunk, several terabytes per day.

Then, Gartner listed Splunk as an APM Innovator

Splunk Named An Application Performance Management (APM) Innovator 

Splunk leverages Dynatrace Application Performance Monitor for visibility inside of applications

   Dynatrace Application Performance Management | Splunkbase 

Every time I look at new developments with Splunk in the APM or enterprise monitoring space, seems like it is nibbling away at the APM of the gaps.

[Hallett German]

Thanks Billy for doing this!!!

[Gnanawathy Hareendran]

Thank you ..