DX Application Performance Management

Expand all | Collapse all

New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

Anon Anon04-30-2013 06:44 PM

  • 1.  New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 04-24-2013 05:36 PM

    Have you ever wanted to automatically create User Groups in CEM based on geographic location or CIDR? Do you currently group users by IP Subnet, but are tired of ambigous names like "UserGroup-63.117.33.0" and "UserGroupTim-10.1.178.0"? Have you been forced to manually put users into geographic groups to support regional SLAs?

    Now, with the recently uploaded GeoIPCemPlugin, you can use MaxMind's IP geolocation data to group users in a more meaningful way! When fully implemented, this plugin allows CEM to group detected users by customizable geographic data, such as city, country, or time-zone, among others. This HTTP Analyzer plugin for CEM works by parsing an HTTP request parameter that contains the client IP address, such as X-Forwarded-For. It then performs a IP geolocation query for this IP Address and outputs an additional request parameter that contains a customizable string of identifying geographic data for the user. Additionally, IP address ranges or CIDR Notation can be used to specify custom geographies or groups instead of geographic data or when no geolocation data is available for the user's IP address. CEM can then be configured to use this Plug-in request parameter to define the User Groups for Business Applications. For Enterprise applications in CEM, this is a more sustainable and meaningful mechanism than grouping by IP Subnet.

    Please note: IP Geolocation is not an exact science; not all IP Addresses have known or strict geographies, and if a request is coming from a proxy server rather than a user’s IP Address, the geolocation data will be for the proxy, not the user. Nevertheless, this data can be extremely useful in logically grouping users and allowing for more impactful SLA definitions without requiring manual user maintenance in CEM.

    As always, feedback is appreciated, either as part of this thread or as part of the Community Document comments.



  • 2.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Broadcom Employee
    Posted 04-25-2013 10:08 AM
    Very cool! I'll put it through some stuff on my test VMs.

    Thanks again for all your effort!


  • 3.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 04-25-2013 06:05 PM
    Hiko,
    Even if it's to point out a typo in the documentation, I look forward to hearing your thoughts on it! :grin:

    - Jack


  • 4.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 09-09-2013 03:52 PM
    Hello --

    We are trying to implement this plugin for a Enterprise Business Application in our Nonprod CEM environment but we are unable to get the plugin to work. We have followed the word doc and configured the env as described, however, we are not seeing user groups populate as we test hitting our app from an External IP address.

    We did notice however for a different APM env we have setup, we see the plugin working as expected for a CEM defined E-Commerence application.

    We are able to see the external ip when looking at the logs after the URL is hit.

    Do you recommend any troubleshooting steps or traces to take as to why the plugin is not creating the user groups for an Enterprise Application inside CEM?

    Thank you!


  • 5.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 09-09-2013 04:25 PM
    Have you confirmed that the extention is not having a problem? The first place to do this is from the Tim Plugin Status page, which is accesible here (where <TIM_IP> is the IP Address of your TIM appliance): http://<TIM_IP>/cgi-bin/wily/packages/cem/tim/index. You should see the name "GeoIP CEM Plugin" listed with an enabled value of Yes. If you do not, then the plugin was not able to start correctly. If so, the TIM Log should contain an error message as to why. Via the command line, the TIM Log is located here: /usr/local/wily/cem/tim/logs/timlog.txt, but can also be viewed on the view via the View Tim Log page (same URL as before).

    If it appears to be installed and running successfully, then my guess would be a configuration problem. Let me know what you find out.


  • 6.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 09-09-2013 04:34 PM
    We are seeing "GeoIP CEM Plugin" with a value of Yes when we navigated to the url you mentioned.

    WIll take a look again at the timlog.txt however initially we did not see anything. Will let you know if we see any issues in timlog.


  • 7.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 04-30-2013 04:10 PM
    Wow Jack! Thanks for sharing this with the APM Community. Has anyone checked it out yet?

    Mary


  • 8.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 04-30-2013 06:44 PM
    Slick! Nice work, cant wait to try it


  • 9.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 02-05-2014 12:50 PM

    Have there been any recent updates to this plugin?  I would really like to see things like extending the lookups beyond just maxmind, something like a map for internal IP's that would override the masking of "internal address".



  • 10.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 02-10-2014 05:43 PM

    We've not made any updates, as we've not received much feedback. Can you write up a brief summary of what you'd like to see and how you envision it functioning? 



  • 11.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 02-11-2014 01:54 PM

    It would work very much like it does today.  Since it looks like the address spaces are already being filtered out for internal addresses, there could be an option to declare a lookup against a CSV or other database.  The value returned would be the group name on those internal addresses.

    -Ramiro



  • 12.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 02-11-2014 02:41 PM

    When specifying the internal "geographies", would you want it to support IP Address ranges, specific IP Adresses, subnet masks, or a combination thereof? Additionally, would you want it to support IPv4, IPv6, or both? 



  • 13.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 02-11-2014 04:40 PM

    My usecase would be for it to support an incoming IP to match to an IP range in cidr.  Such as 10.100.0.0/22

    Only for IPv4 for now.

     

     

     



  • 14.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 02-11-2014 06:41 PM

    Awesome! Thanks, I'll give it a go!



  • 15.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 02-11-2014 06:34 PM

    I'm getting a "Documents and Media is temporarily unavailable" message when I try to update this Plugin, so I've attached the newest version here. It should now provide you with the ability you want.

    EDIT: Now also supports CIDR notation for specifying IP Address ranges.  

    EDIT #2: The Community Document has been re-added and the links in this thread have been updated. 



  • 16.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 02-18-2014 08:00 PM

    Seem to be having some issues with CIDR addresses.

    I have downloaded the maxmind DB because it seems that the plugin fails without one and have loaded the properties file with many ranges all of which have mutliple CIDR entries (comma separated) but they dont want to match.

    Example I have an IP of "10.62.80.60" that should match a range specified with "10.0.0.0/8" but the plugin returns "Unknown Region".

    I've tried removing all of the other IP ranges/names as well as made sure there is only one CIDR in the ranges for that location but with no success.


    Any help is appreciated.

     

    Thanks!



  • 17.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 02-19-2014 11:19 AM

    Thanks for the feedback! I'll look into this and hopefully post an update by end-of-day. 



  • 18.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 02-21-2014 12:37 PM

    Okay. Here's the latest version [EDIT: now back in Community Documents, thanks to Hiko] - it passed the expanded unit tests. If you have problems, set the log level to DEBUG, disable and re-enable the plugin (or restart the TIM), and send me the log file after it runs for a while. 



  • 19.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 02-21-2014 12:55 PM

    So far so good Jack, Thanks!  I just loaded the updated version, I'll let it run for a while and post back.  I already see many of my ranges matching with a few that arent, so I'll take a closer look later to see if its just my configuration.

    Thanks again for this, it is extremely helpful!

    -Ramiro



  • 20.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 02-25-2014 11:45 AM

    Ramiro, 

    Is the plug-in still working as you'd hoped? 



  • 21.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 02-26-2014 01:53 PM

    Yeah it's going pretty well so far.  I am noticing though that I have a lot of groups getting created with "null" wherever it can't find the field value.  Not sure if this is because we're using the free version of maxmind or if there was an error in the plugin.  

    I was thinking just now, if it is cause by the value not being found that rather than printing "null", it would be cleaner to only include what it can find.  Say the format is "City, Region, CountryName", if only Region, CountryName are found, that it print those and leave out City.  Or any combination thereof.

    What do you think?



  • 22.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 02-26-2014 06:51 PM

    Hmm. I'd like to see a debug log. I agree, you should not be seing "null" in the values, and I'd like to investigate why you are. Can you set the log level to DEBUG for the plugin and either restart the TIM or disable / enable the plug-in via CEM? Once a decent log file is produced, please attach it to this thread or send to me via private message. 



  • 23.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 03-04-2014 03:29 PM

    We have installed the Geo pluin following the word document thoroughly. I see plugin is enable and active in both TIM and CEM.

    We tried to test it by creating some slow transactions, we got defects but dont see the user location we were expecting.  what would have went wrong?

    FYI.. we are not filtering traffic to the plugin yet.

    Tue Mar  4 14:15:27 2014 28438 ! Warning: MTPComm: MTP socket "/var/run/nqcapd_sock" does not exist
    Tue Mar  4 14:15:27 2014 28438   AdapterManager: "eth1": thread starting
    Tue Mar  4 14:15:27 2014 28438   DeviceCtl: running: /usr/sbin/ethtool --offload eth1 gro off 2>&1
    Tue Mar  4 14:15:27 2014 28438   JavaHttpPlugin: analysis thread 0x2aaab403d550 attached to JVM
    Tue Mar  4 14:15:27 2014 28438   JavaHttpPlugin: analysis thread 0x2aaab403d550 initialized for JVM
    Tue Mar  4 14:15:27 2014 28438   AdapterManager: "eth1": network device opened
    Tue Mar  4 14:15:27 2014 28438   AdapterManager: "eth1": initializing to reject all packets
    Tue Mar  4 14:15:27 2014 28438   AdapterManager: "eth1": receive buffer size for is 125829120
    Tue Mar  4 14:15:27 2014 28438   AdapterManager: "eth1": packet filter set
    Tue Mar  4 14:25:26 2014 28438   Service: running: tim-ageout -data /etc/wily/cem/tim/data -minFreePercent 25 -reclaimPercent 10
    Tue Mar  4 14:25:26 2014 28438   Service: tim-ageout exited with status 0
    Tue Mar  4 14:26:13 2014 28438   WebServer: POST request for /tess/setpluginconfig from 127.0.0.1
    Tue Mar  4 14:26:13 2014 28438   WebServer: request forwarded from 156.81.248.191
    Tue Mar  4 14:26:13 2014 28438   WebServer: loading javapluginconfig
    Tue Mar  4 14:26:13 2014 28438   ConfigFile: writing configuration file /etc/wily/cem/tim/config/javapluginconfigs/1.xml
    Tue Mar  4 14:26:13 2014 28438   JavaPluginManager: received config file
    Tue Mar  4 14:26:13 2014 28438   JavaPluginConfig: id=1, enabled=0, version=1
    Tue Mar  4 14:26:13 2014 28438   JavaHttpPlugin: stopping plugin 1 ("GeoIP CEM Plugin")
    Tue Mar  4 14:26:16 2014 28438   WebServer: POST request for /tess/setpluginconfig from 127.0.0.1
    Tue Mar  4 14:26:16 2014 28438   WebServer: request forwarded from 156.81.248.191
    Tue Mar  4 14:26:16 2014 28438   WebServer: loading javapluginconfig
    Tue Mar  4 14:26:16 2014 28438   ConfigFile: writing configuration file /etc/wily/cem/tim/config/javapluginconfigs/1.xml
    Tue Mar  4 14:26:16 2014 28438   JavaPluginManager: received config file
    Tue Mar  4 14:26:16 2014 28438   JavaPluginConfig: id=1, enabled=1, version=1, name="GeoIP CEM Plugin"
    Tue Mar  4 14:26:16 2014 28438   JavaPluginConfig: URL filter is not defined
    Tue Mar  4 14:26:16 2014 28438   JavaPluginConfig: no server filter
    Tue Mar  4 14:26:16 2014 28438   JavaHttpPlugin: starting plugin 1 ("GeoIP CEM Plugin") with:
    Tue Mar  4 14:26:16 2014 28438   JavaHttpPlugin:   /etc/wily/cem/tim/config/javaplugins/1.jar
    Tue Mar  4 14:26:16 2014 28438   JavaHttpPlugin: plugin "GeoIP CEM Plugin" started

     

     



  • 24.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 03-04-2014 03:43 PM

    And here is what is configured in plugin property

    geoip.city.file=/usr/local/wily/cem/tim/data/GeoLiteCity.dat
    geoip.log.level=DEBUG
    geoip.cem.incoming.parameter.name=X-Forwarded-For
    geoip.cem.outgoing.parameter.name=X-GeoIPLocation
    geoip.internal.networks=A,B,C,D
    geoip.internal.networks.A.name=A
    geoip.internal.networks.A.ranges=11.0.0.0-255.0.0.0
    geoip.internal.networks.B.name=B
    geoip.internal.networks.B.ranges=157.146.0.0-255.255.0.0
    geoip.internal.networks.C.name=C
    geoip.internal.networks.C.ranges=156.82.0.0-255.255.0.0
    geoip.internal.networks.D.name=D
    geoip.internal.networks.D.ranges=169.225.0.0-255.255.0.0
    geoip.cem.parameter.format={0}, {1}
    geoip.cem.parameter.value.0=countryName
    geoip.cem.parameter.value.1=timeZone



  • 25.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 03-05-2014 11:37 AM

    It sounds like everything is working as expected. The plug-in has started up normally without errors and you've configured some IP ranges that will override the GeoIP lookups. The reason you are not seeing any results is that you have not directed any traffic to be handled by the GeoIPCemPlugin: 
     

    Tue Mar  4 14:26:16 2014 28438   JavaPluginConfig: URL filter is not defined
    Tue Mar  4 14:26:16 2014 28438   JavaPluginConfig: no server filter
    

     

    This means that you've likely not set the IP Address range and URL filter for the plugin. I recommend using 0.0.0.0 - 255.255.255.255 for the IP Address Range and /* for the URL match pattern. 



  • 26.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 03-05-2014 05:59 PM

    We change the url pattern to /* as you have suggested and we see the user groups created as “international” and “United States, America/New_York”.

     

    We have the following in our properties file

    geoip.city.file=/usr/local/wily/cem/tim/data/GeoLiteCity.dat

    geoip.log.level=DEBUG

    geoip.cem.incoming.parameter.name=client_ip

    geoip.cem.outgoing.parameter.name=X-GeoIPLocation

    geoip.internal.networks=International,test1,test2,test3

    geoip.internal.networks.International.name=International

    geoip.internal.networks.International.ranges=11.0.0.0-255.0.0.0

    geoip.internal.networks.test1.name=test1

    geoip.internal.networks.test1.ranges=157.146.0.0-255.255.0.0

    geoip.internal.networks.test2.name=test2

    geoip.internal.networks.test2.ranges=156.82.0.0-255.255.0.0

    geoip.internal.networks.test3.name=test3

    geoip.internal.networks.test3.ranges=169.225.0.0-255.255.0.0

    geoip.cem.parameter.format={0}, {1}

    geoip.cem.parameter.value.0=countryName

    geoip.cem.parameter.value.1=timeZone

     

     

    We are expecting to see 156.82.244.165 in test2 user group, but it shows currently under international user group. So, we are wondering why it is not mapping properly.

    Please help



  • 27.  RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 03-06-2014 11:46 AM

    It is mapping properly, it's just that your IP Address ranges are huge and overlapping.  You seem to be using a subnet mask as the last part of your IP range, which might be part of the confusion. I recommend reading up on IP range blocks to get a better understanding of how you can specify these. 

    Also, the IP range overrides are first-match; since the International range is specified first in the geoip.internal.networks property, it's the first match for the 156.82.244.165 address, even though it could also be contained by the test2 range. Once you redefine your ranges so that they do not overlap, you should get your expected results. 



  • 28.  RE: [CA APM General Discussion] RE: New HTTP Analyzer Plug-in: CEM User Gro

    Posted 03-06-2014 12:04 PM
    Thanks much for the response
    We figured it out last evening that we were providing subnet mask rather than the range. We corrected yesterday and it shows properly now.
    Except that – for most of them it shows the correct location in the http trace for that defect (for example as -

    X-GeoIPLocation:

    International

    )

    However it does not put the user in that group. More details below:

    Now:

    User group : new users

    [cid:image001.png@01CF392B.BD413D20]

    What we were expecting:

    User group: International

    [cid:image001.png@01CF392B.BD413D20]



    From: CA APM/Wily Global User CommunityMessage Boards [mailto:CommunityAdmin@communities-mail.ca.com]
    Sent: Thursday, March 06, 2014 10:46 AM
    To: mb_message.2307983.110612813@myca-email.ca.com
    Subject: [CA APM General Discussion] RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location


    It is mapping properly, it's just that your IP Address ranges are huge and overlapping. You seem to be using a subnet mask as the last part of your IP range, which might be part of the confusion. I recommend reading up on IP range blocks<http://www.mediawiki.org/wiki/Help:Range_blocks> to get a better understanding of how you can specify these.

    Also, the IP range overrides are first-match; since the International range is specified first in the geoip.internal.networks property, it's the first match for the 156.82.244.165 address, even though it could also be contained by the test2 range. Once you redefine your ranges so that they do not overlap, you should get your expected results.
    Posted by:jakbutler
    --
    CA Communities Message Boards
    110615353
    mb_message.2307983.110612813@myca-email.ca.com<mailto:mb_message.2307983.110612813@myca-email.ca.com>
    https://communities.ca.com


  • 29.  RE: [CA APM General Discussion] RE: New HTTP Analyzer Plug-in: CEM User Gro

    Posted 03-06-2014 01:59 PM

    If you're seeing the output parameter, the plug-in is working correctly. I would double-check the Business Application definition and make sure that, for each business application (including Default), the X-GeoIPLocation Plug-in Parameter is defined for the User Group Identification settings. 



  • 30.  RE: [CA APM General Discussion] RE: [CA APM General Discussion] RE: New HTT

    Posted 03-06-2014 02:07 PM
    Thanks for the response!

    We found that the user group info is not accurate due to users previously in new users group. We delete all previous user groups and cleaned up users and now we see plugin is working perfectly. We intend to do more testing though. Thanks much for the plugin







    From: CA APM/Wily Global User CommunityMessage Boards [mailto:CommunityAdmin@communities-mail.ca.com]
    Sent: Thursday, March 06, 2014 12:59 PM
    To: mb_message.2307983.110621385@myca-email.ca.com
    Subject: [CA APM General Discussion] RE: [CA APM General Discussion] RE: New HTTP Analyzer Plug-in: CEM User Gro


    If you're seeing the output parameter, the plug-in is working correctly. I would double-check the Business Application definition and make sure that, for each business application (including Default), the X-GeoIPLocation Plug-in Parameter is defined for the User Group Identification settings.
    Posted by:jakbutler
    --
    CA Communities Message Boards
    110623925
    mb_message.2307983.110621385@myca-email.ca.com<mailto:mb_message.2307983.110621385@myca-email.ca.com>
    https://communities.ca.com


  • 31.  RE: [CA APM General Discussion] RE: [CA APM General Discussion] RE: New HTT

    Posted 03-06-2014 05:31 PM

    I'm glad to hear that! Let me know if you run into any difficulty or if you have enhancement requests. 



  • 32.  Re: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 06-25-2014 12:50 PM

    We wanted to disable geo plugin as for some reason its not giving the accurate location probably based on geolite data. I disabled it in CEM console and i see NOT enabled in TIM.

    But for some we are still seeing the usergroups from Geoplugin. Can you help provide the proper process to disable the plugin?

    Thanks for your help



  • 33.  Re: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 06-25-2014 03:19 PM

    Are you seeing new users get put into the geographic groups, or are you referring to the existing users still being assigned to the user group from the GeoIPCemPlugin? Even when disabled, the existing groups will not be removed or modified unless you manually re-group the users yourself (which can be done group by group via the CEM UI).

     

    It sounds like you've followed all the necessary steps to disable, and you can confirm this by logging on to the TIM appliance and looking at the GeoIPCemPlugin.log (/usr/local/wily/cem/tim/logs/GeoIPCemPlugin.log). Additionally, upon restart of the TIM, it will usually log which extensions have been turned on and should not list the GeoIPCem plugin (/usr/local/wily/cem/tim/logs/timlog.txt); the TIM log is also visible through the CEM UI.



  • 34.  Re: RE: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Posted 05-24-2016 04:49 PM

    I really appreciate this feature.

    Thanks for your effort.

    I have some questions regarding the installation since it seems that it's not working and I assume that I missing something.

     

    I'm looking for geolocalize my internet users but even when it seems that is working nor groups are discovered and the last line in the log is "Tue May 24 17:37:09 ART 2016 [INFO] Plugin succesfully Initialized". Even when the propertie file sets the log in DEBUG mode.  Also I can say that the plugin is correctly enabled in the TIM.

     

    I assume that I have X-Forwarded-For but using autodiscovery to check how the CEM captures the IP I could not find that field. I do find the field URL ClientIP.

    How do I know if I setting correctly the incoming and outgoing parameters.? I saw an example posted where they are using a different field : client_ip. Should be another one? How do I know if its working ?

     

    I send you the properties file just in case.

     

    Best regards.

     

     

    geoip.city.file=data/GeoLiteCity.dat

    geoip.log.level=DEBUG

    geoip.log.max.byte.size=2000000

    geoip.cem.incoming.parameter.name=X-Forwarded-For

    geoip.cem.outgoing.parameter.name=X-GeoIPLocation

    geoip.internal.networks=A,B,C

    geoip.internal.networks.A.name=Internal Network, 24-bit block

    geoip.internal.networks.A.ranges=10.0.0.0/8

    geoip.internal.networks.B.name=Internal Network, 20-bit block

    geoip.internal.networks.B.ranges=172.17.0.0/12

    geoip.internal.networks.C.name=Internal Network, 16-bit block

    geoip.internal.networks.C.ranges=192.168.0.0/16

    geoip.cem.parameter.format={0}, {4}

    geoip.cem.parameter.value.0=countryName

    geoip.cem.parameter.value.1=city



  • 35.  Re: New HTTP Analyzer Plug-in: CEM User Groups by Geographical Location

    Broadcom Employee
    Posted 05-24-2016 05:50 PM

    Hi Lenardo:

    Check your TIM Collector Logs for errors related to this feature

     

    Thanks

    Hal German