DX Application Performance Management

 View Only
Expand all | Collapse all

Cannot Enable TIMs - Urgent Please Help!

Jose Javier Sanchez Martin

Jose Javier Sanchez MartinApr 22, 2016 10:52 AM

Manish Parikh

Manish ParikhApr 22, 2016 08:46 AM

  • 1.  Cannot Enable TIMs - Urgent Please Help!

    Posted Apr 22, 2016 08:31 AM

    When I am trying to enable my TIMs, I get the following error

     

    An updated configuration could not be sent to one or more of the selected Monitors. These monitors were not enabled.

     

    When I click on "Communication Error 500"

    I get the following:

    Communication Error 500

    500 Error cannot load plugin "21" while it is enabled

     

    Also, see the following error log on TIM

     

    Fri Apr 22 08:33:01 2016 39732 hub: WebServer: wrote plugin to temporary file /apps/CA/APM/tim/config/cemjavaplugin.jar.tmp

    Fri Apr 22 08:33:01 2016 39732 **ERROR: hub: JavaPluginManager: cannot load plugin "21" while it is enabled

     

    I've already gone into "defects" folder and removed all the defects (after stopping the TIM and the TCS Collector). Restarted everything back up.

     

    Any/all help will be greatly appreciated.

     

    Thanks

    Manish



  • 2.  Re: Cannot Enable TIMs - Urgent Please Help!
    Best Answer

    Broadcom Employee
    Posted Apr 22, 2016 08:41 AM

    Hi Manish,

     

    There is a TEC DOC available that seems to match your issue:

     

    http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec1324169.aspx

     

    Hope it helps

     

    Regards,

    Jose



  • 3.  Re: Cannot Enable TIMs - Urgent Please Help!

    Posted Apr 22, 2016 08:45 AM

    Jose-Javier-Sanchez

     

    That TEC doc talks about plugin "53". My issue concerns plugin "21"



  • 4.  Re: Cannot Enable TIMs - Urgent Please Help!

    Posted Apr 22, 2016 09:48 AM

    Thanks Jose-Javier-Sanchez for helping out. Also, thanks to Hallett_German

     

    I followed the steps and was able to 'enable' the TIMs.



  • 5.  Re: Cannot Enable TIMs - Urgent Please Help!

    Broadcom Employee
    Posted Apr 22, 2016 10:52 AM

    Glad to hear it worked

     

    Thanks

    Jose



  • 6.  Re: Cannot Enable TIMs - Urgent Please Help!

    Posted Apr 09, 2018 10:04 AM

    Jose-Javier-Sanchez Hallett_German

    Looks like I am coming across the same issue again w/ v10.5.2 upgrade. I clicked on the link above to access the TEC DOC but the link seems NOT to be working. Can someone please look into this. I will also follow the other procedures to see if it resolves the issue



  • 7.  Re: Cannot Enable TIMs - Urgent Please Help!

    Broadcom Employee
    Posted Apr 09, 2018 10:10 AM

    Dear Manish:

          As you may have noticed, some knowledge docs were retired in reent months. Mostly these are due to low number of accesses

         Will recreate the above kd  as time allows

    Thanks 

    Hal German



  • 8.  Re: Cannot Enable TIMs - Urgent Please Help!

    Broadcom Employee
    Posted Apr 09, 2018 10:28 AM

    Manish, I sent you a copy of that KB in the support case. Looks like an issue loading the plugin in again...

    4/09/18 09:31:53.816 AM EDT [WARN] [Thread-89840] [Manager.com.timestock.tess.services.tim.TimIo] Cannot read jar file cem/tess/plugins/NTLM-SessionID.jar
    java.io.FileNotFoundException: cem/tess/plugins/NTLM-SessionID.jar (No such file or directory)

     

    Please try the same steps and let us know how it goes. 



  • 9.  Re: Cannot Enable TIMs - Urgent Please Help!

    Posted Apr 09, 2018 10:29 AM

    Hal,

    So in the meantime can someone tell me what that KD's resolution was?

     

    Thnaks

    Manish



  • 10.  Re: Cannot Enable TIMs - Urgent Please Help!

    Posted Apr 09, 2018 10:32 AM

    here is what the log says

     

    4/09/18 09:31:45.429 AM EDT [INFO] [Thread-89839] [Manager.com.timestock.tess.services.tim.TimIo] enabling monitor id=700000000000000646
    4/09/18 09:31:45.429 AM EDT [INFO] [Thread-89840] [Manager.com.timestock.tess.services.tim.TimIo] enabling monitor id=700000000000000647
    4/09/18 09:31:45.495 AM EDT [INFO] [Thread-89839] [Manager.com.timestock.tess.services.tim.TimIo] sent timconfig to monitor id=700000000000000646
    4/09/18 09:31:45.496 AM EDT [INFO] [Thread-89839] [Manager.com.timestock.tess.services.tim.TimIo] Setting TessIpAddr 10.60.168.38 for monitor 10.60.169.33
    4/09/18 09:31:45.506 AM EDT [INFO] [Thread-89839] [Manager.com.timestock.tess.services.tim.TimIo] sent TessIpAddr to monitor id=700000000000000646
    4/09/18 09:31:45.506 AM EDT [INFO] [Thread-89839] [Manager.com.timestock.tess.services.tim.TimIo] setting TessCollectorIpAddr for monitor 700000000000000646 to '10.60.168.40'
    4/09/18 09:31:45.520 AM EDT [INFO] [Thread-89839] [Manager.com.timestock.tess.services.tim.TimIo] sent TessCollectorIpAddr to monitor id=700000000000000646
    4/09/18 09:31:45.640 AM EDT [INFO] [Thread-89840] [Manager.com.timestock.tess.services.tim.TimIo] sent timconfig to monitor id=700000000000000647
    4/09/18 09:31:45.640 AM EDT [INFO] [Thread-89840] [Manager.com.timestock.tess.services.tim.TimIo] Setting TessIpAddr 10.60.168.38 for monitor 10.60.137.33
    4/09/18 09:31:45.754 AM EDT [INFO] [Thread-89840] [Manager.com.timestock.tess.services.tim.TimIo] sent TessIpAddr to monitor id=700000000000000647
    4/09/18 09:31:45.754 AM EDT [INFO] [Thread-89840] [Manager.com.timestock.tess.services.tim.TimIo] setting TessCollectorIpAddr for monitor 700000000000000647 to '10.60.168.40'
    4/09/18 09:31:45.877 AM EDT [INFO] [Thread-89840] [Manager.com.timestock.tess.services.tim.TimIo] sent TessCollectorIpAddr to monitor id=700000000000000647
    4/09/18 09:31:46.566 AM EDT [INFO] [Thread-89839] [Manager.com.timestock.tess.services.tim.TimIo] sent domainconfig to monitor id=700000000000000646
    4/09/18 09:31:46.566 AM EDT [INFO] [Thread-89839] [Manager.com.timestock.tess.services.tim.TimIo] updating license for monitor id=700000000000000646
    4/09/18 09:31:46.731 AM EDT [INFO] [Thread-89839] [Manager.com.timestock.tess.services.tim.TimIo] sent license to monitor id=700000000000000646
    4/09/18 09:31:46.731 AM EDT [INFO] [Thread-89839] [Manager.com.timestock.tess.services.tim.TimIo] updating introscope for monitor id=700000000000000646
    4/09/18 09:31:46.787 AM EDT [INFO] [Thread-89839] [Manager.com.timestock.tess.services.tim.TimIo] sent introscopeconfig to monitor id=700000000000000646
    4/09/18 09:31:46.787 AM EDT [INFO] [Thread-89839] [Manager.com.timestock.tess.services.tim.TimIo] updating siteminder for monitor id=700000000000000646
    4/09/18 09:31:46.807 AM EDT [INFO] [Thread-89839] [Manager.com.timestock.tess.services.tim.TimIo] sent siteminderconfig to monitor id=700000000000000646
    4/09/18 09:31:46.823 AM EDT [INFO] [Thread-89839] [Manager.com.timestock.tess.services.tim.TimIo] loading plugin id=41 for monitor id=700000000000000646
    4/09/18 09:31:46.823 AM EDT [WARN] [Thread-89839] [Manager.com.timestock.tess.services.tim.TimIo] Cannot read jar file cem/tess/plugins/NTLM-SessionID.jar
    java.io.FileNotFoundException: cem/tess/plugins/NTLM-SessionID.jar (No such file or directory)
    at java.io.FileInputStream.open0(Native Method)
    at java.io.FileInputStream.open(FileInputStream.java:195)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at java.io.FileInputStream.<init>(FileInputStream.java:93)
    at com.timestock.tess.services.tim.TimIo.loadPlugin(TimIo.java:1976)
    at com.timestock.tess.services.tim.TimIo.enable(TimIo.java:990)
    at com.timestock.tess.services.tim.TimIo$EnableMonitorThread.perform(TimIo.java:3696)
    at com.timestock.tess.services.tim.TimIoSubThread.run(TimIoSubThread.java:74)

     

    So I checked that file location and "NTLM-SessionID.jar" file doesn't exist. Was I supposed to copy it prior to the upgrade?



  • 11.  Re: Cannot Enable TIMs - Urgent Please Help!

    Broadcom Employee
    Posted Apr 09, 2018 10:44 AM

    Ideally, yes the jar should have been copied. If you have access to your older Tim files, it should be in /config/javaplugins. 



  • 12.  Re: Cannot Enable TIMs - Urgent Please Help!

    Posted Apr 09, 2018 11:09 AM

    I've copied that "NTLM-SessionID.jar" file from 10.1 MoM (EM) to 10.5.2 MoM.(EM). Am restarting MoM, TCS Coll, and both TIMs. Will keep you posted

     

    Thanks

    Manish



  • 13.  Re: Cannot Enable TIMs - Urgent Please Help!

    Posted Apr 09, 2018 11:39 AM

    Still no joy:

     

    4/09/18 11:33:25.905 AM EDT [INFO] [Thread-246] [Manager.com.timestock.tess.services.tim.TimIo] enabling monitor id=700000000000000646
    4/09/18 11:33:25.907 AM EDT [INFO] [Thread-247] [Manager.com.timestock.tess.services.tim.TimIo] enabling monitor id=700000000000000647
    4/09/18 11:33:25.926 AM EDT [ERROR] [Thread-247] [Manager.com.timestock.tess.services.tim.TimIoSubThread] enable monitor: exception processing monitor 700000000000000647
    java.lang.NullPointerException
    at org.springframework.jdbc.datasource.DataSourceUtils.doCloseConnection(DataSourceUtils.java:341)
    at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.closeConnection(LocalDataSourceConnectionProvider.java:97)
    at org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:445)
    at org.hibernate.jdbc.ConnectionManager.aggressiveRelease(ConnectionManager.java:400)
    at org.hibernate.jdbc.ConnectionManager.afterStatement(ConnectionManager.java:275)
    at org.hibernate.jdbc.AbstractBatcher.closePreparedStatement(AbstractBatcher.java:535)
    at org.hibernate.jdbc.AbstractBatcher.closeStatement(AbstractBatcher.java:265)
    at org.hibernate.jdbc.AbstractBatcher.closeQueryStatement(AbstractBatcher.java:281)
    at org.hibernate.jdbc.AbstractBatcher.closeQueryStatement(AbstractBatcher.java:209)
    at org.hibernate.loader.Loader.doQuery(Loader.java:714)
    at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:224)
    at org.hibernate.loader.Loader.loadEntity(Loader.java:1851)
    at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:48)
    at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:42)
    at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:2992)
    at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:395)
    at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:375)
    at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:139)
    at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:98)
    at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878)
    at org.hibernate.impl.SessionImpl.immediateLoad(SessionImpl.java:836)
    at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:66)
    at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:111)
    at org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.invoke(CGLIBLazyInitializer.java:140)
    at com.timestock.tess.data.objects.Entity$$EnhancerByCGLIB$$8120a580.getIpAddressType(<generated>)
    at com.timestock.tess.services.tim.TimIo.checkEntityIPAddress(TimIo.java:1233)
    at com.timestock.tess.services.tim.TimIo.enable(TimIo.java:974)
    at com.timestock.tess.services.tim.TimIo$EnableMonitorThread.perform(TimIo.java:3696)
    at com.timestock.tess.services.tim.TimIoSubThread.run(TimIoSubThread.java:74)
    4/09/18 11:33:25.974 AM EDT [INFO] [Thread-246] [Manager.com.timestock.tess.services.tim.TimIo] sent timconfig to monitor id=700000000000000646
    4/09/18 11:33:25.975 AM EDT [INFO] [Thread-246] [Manager.com.timestock.tess.services.tim.TimIo] Setting TessIpAddr 10.60.168.38 for monitor 10.60.169.33
    4/09/18 11:33:25.984 AM EDT [INFO] [Thread-246] [Manager.com.timestock.tess.services.tim.TimIo] sent TessIpAddr to monitor id=700000000000000646
    4/09/18 11:33:25.985 AM EDT [INFO] [Thread-246] [Manager.com.timestock.tess.services.tim.TimIo] setting TessCollectorIpAddr for monitor 700000000000000646 to '10.60.168.40'
    4/09/18 11:33:26.000 AM EDT [INFO] [Thread-246] [Manager.com.timestock.tess.services.tim.TimIo] sent TessCollectorIpAddr to monitor id=700000000000000646
    4/09/18 11:33:27.087 AM EDT [INFO] [Thread-246] [Manager.com.timestock.tess.services.tim.TimIo] sent domainconfig to monitor id=700000000000000646
    4/09/18 11:33:27.087 AM EDT [INFO] [Thread-246] [Manager.com.timestock.tess.services.tim.TimIo] updating license for monitor id=700000000000000646
    4/09/18 11:33:27.225 AM EDT [INFO] [Thread-246] [Manager.com.timestock.tess.services.tim.TimIo] sent license to monitor id=700000000000000646
    4/09/18 11:33:27.225 AM EDT [INFO] [Thread-246] [Manager.com.timestock.tess.services.tim.TimIo] updating introscope for monitor id=700000000000000646
    4/09/18 11:33:27.258 AM EDT [INFO] [Thread-246] [Manager.com.timestock.tess.services.tim.TimIo] sent introscopeconfig to monitor id=700000000000000646
    4/09/18 11:33:27.258 AM EDT [INFO] [Thread-246] [Manager.com.timestock.tess.services.tim.TimIo] updating siteminder for monitor id=700000000000000646
    4/09/18 11:33:27.329 AM EDT [INFO] [Thread-246] [Manager.com.timestock.tess.services.tim.TimIo] sent siteminderconfig to monitor id=700000000000000646
    4/09/18 11:33:27.353 AM EDT [INFO] [Thread-246] [Manager.com.timestock.tess.services.tim.TimIo] loading plugin id=41 for monitor id=700000000000000646
    4/09/18 11:33:27.459 AM EDT [ERROR] [Thread-246] [Manager.com.timestock.tess.services.tim.TimIo] cannot send plugin to monitor id=700000000000000646: Communication Error 500: 500 Error
    cannot load plugin "41" while it is enabled

    4/09/18 11:33:28.110 AM EDT [INFO] [btpool0-1] [Manager.com.timestock.tess.services.tim.TimIoSubThread] enable monitor: updated monitor 700000000000000646
    4/09/18 11:33:28.360 AM EDT [INFO] [btpool0-1] [Manager.com.timestock.tess.services.tim.TimIoSubThread] enable monitor: updated monitor 700000000000000647
    4/09/18 11:33:28.362 AM EDT [INFO] [btpool0-1] [Manager.com.timestock.tess.services.tim.TimIo] enabled 0 monitors
    4/09/18 11:33:28.363 AM EDT [WARN] [btpool0-1] [Manager.com.timestock.tess.services.tim.TimIo] can't enable 2 monitors



  • 14.  Re: Cannot Enable TIMs - Urgent Please Help!

    Broadcom Employee
    Posted Apr 09, 2018 11:46 AM

    Try deleting the plugin outright then from the CEM UI and see if we can enable the Tim again. We can reload the plugin once TIM is enabled...

     

    •Delete the plugins from the CEM UI -> Setup -> Plug-ins page. 
    •Restart the MOM, the EM running the TIM Collection Service, and the TIM. 
    •Make sure TIM is enabled.  
    •Reload plugin if needed. 



  • 15.  Re: Cannot Enable TIMs - Urgent Please Help!

    Posted Apr 09, 2018 03:05 PM

    Yeah, I was gonna do that next anyways. Looks like that worked. So, coping the .jar file doesn't work. You HAVE to delete the "CA CEM HTTP Analyzer" and restart MoM, TCS Coll, and TIMs.

     

    So, can we have a new knowledge or TEC doc build to provide this info?

     

    Also, it would be good if this also provided int eh docops site (if it is true for any/all APM upgrades for customers who have an HTTP Plug-in). This is a tidbit that folks would wanna know, I would assume!!!



  • 16.  Re: Cannot Enable TIMs - Urgent Please Help!

    Broadcom Employee
    Posted Apr 10, 2018 06:12 PM

    As I promised Manish, I created https://comm.support.ca.com/kb/seeing-plugin-lo/KB000077045   which will take some hours to become publish. 



  • 17.  Re: Cannot Enable TIMs - Urgent Please Help!

    Posted Apr 10, 2018 06:16 PM

    Hallett_German - you know I've never doubted you 

     

    Much appreciated.



  • 18.  Re: Cannot Enable TIMs - Urgent Please Help!

    Broadcom Employee
    Posted Apr 22, 2016 08:43 AM

    Hi Manish:

    Was Javi's response helpful or do you need additional assistance?

    Thanks

    Hal German



  • 19.  Re: Cannot Enable TIMs - Urgent Please Help!

    Posted Apr 22, 2016 08:46 AM

    Hallett_German

     

    see my response above.



  • 20.  Re: Cannot Enable TIMs - Urgent Please Help!

    Posted Apr 22, 2016 08:48 AM

    MOM Error shows:

     

    4/22/16 08:27:45.026 AM EDT [INFO] [Thread-212] [Manager.com.timestock.tess.services.tim.TimIo] enabling monitor id=700000000000000647

    4/22/16 08:27:45.160 AM EDT [INFO] [Thread-212] [Manager.com.timestock.tess.services.tim.TimIo] sent timconfig to monitor id=700000000000000647

    4/22/16 08:27:45.161 AM EDT [INFO] [Thread-212] [Manager.com.timestock.tess.services.tim.TimIo] setting TessCollectorIpAddr for monitor 700000000000000647 to '10.60.168.40'

    4/22/16 08:27:45.290 AM EDT [INFO] [Thread-212] [Manager.com.timestock.tess.services.tim.TimIo] sent TessCollectorIpAddr to monitor id=700000000000000647

    4/22/16 08:27:46.291 AM EDT [INFO] [Thread-212] [Manager.com.timestock.tess.services.tim.TimIo] sent domainconfig to monitor id=700000000000000647

    4/22/16 08:27:46.291 AM EDT [INFO] [Thread-212] [Manager.com.timestock.tess.services.tim.TimIo] updating license for monitor id=700000000000000647

    4/22/16 08:27:46.638 AM EDT [INFO] [Thread-212] [Manager.com.timestock.tess.services.tim.TimIo] sent license to monitor id=700000000000000647

    4/22/16 08:27:46.638 AM EDT [INFO] [Thread-212] [Manager.com.timestock.tess.services.tim.TimIo] updating introscope for monitor id=700000000000000647

    4/22/16 08:27:46.782 AM EDT [INFO] [Thread-212] [Manager.com.timestock.tess.services.tim.TimIo] sent introscopeconfig to monitor id=700000000000000647

    4/22/16 08:27:46.782 AM EDT [INFO] [Thread-212] [Manager.com.timestock.tess.services.tim.TimIo] updating siteminder for monitor id=700000000000000647

    4/22/16 08:27:46.917 AM EDT [INFO] [Thread-212] [Manager.com.timestock.tess.services.tim.TimIo] sent siteminderconfig to monitor id=700000000000000647

    4/22/16 08:27:46.922 AM EDT [INFO] [Thread-212] [Manager.com.timestock.tess.services.tim.TimIo] loading plugin id=21 for monitor id=700000000000000647

    4/22/16 08:27:47.484 AM EDT [ERROR] [Thread-212] [Manager.com.timestock.tess.services.tim.TimIo] cannot send plugin to monitor id=700000000000000647: Communication Error 500: 500 Error^M

    cannot load plugin "21" while it is enabled^M

    4/22/16 08:27:47.599 AM EDT [INFO] [btpool0-21] [Manager.com.timestock.tess.services.tim.TimIoSubThread] enable monitor: updated monitor 700000000000000647

    4/22/16 08:27:47.600 AM EDT [INFO] [btpool0-21] [Manager.com.timestock.tess.services.tim.TimIo] enabled 0 monitors

    4/22/16 08:27:47.600 AM EDT [WARN] [btpool0-21] [Manager.com.timestock.tess.services.tim.TimIo] can't enable 1 monitors

    4/22/16 08:31:43.881 AM EDT [INFO] [btpool0-29] [com.timestock.tess.web.controllers.LoginController] Logging out user "manish.a.parikh" from host "10.60.168.20"

    4/22/16 08:32:31.858 AM EDT [INFO] [btpool0-32] [Manager.com.timestock.tess.services.security.TsAuthenticationProcessingFilter] onPreAuthentication() was called: POST

    4/22/16 08:32:32.903 AM EDT [INFO] [btpool0-32] [Manager.com.timestock.tess.services.security.APMAuthenticationProvider] User "manish.a.parikh" logged in successfully from host "localhost"



  • 21.  Re: Cannot Enable TIMs - Urgent Please Help!

    Broadcom Employee
    Posted Apr 22, 2016 08:55 AM

    Dear Manish:

    Regardless of the plugin number in the error,the solution is appropriate for you:

     

    • Delete the plugins from the CEM UI -> Setup -> Plug-ins page.
    • Restart the MOM, the EM running the TIM Collection Service, and the TIM.
    • Reload the plugins using the CEM UI -> Setup -> Plug-ins page and "Synchronize All Monitors."


    Read more at http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec1324169.aspx

     

    Did you try that?

     

    Thanks

    Hal German



  • 22.  Re: Cannot Enable TIMs - Urgent Please Help!

    Posted Apr 22, 2016 08:59 AM

    So, are you referring to the CA HTTP Analyzer Plugin?



  • 23.  Re: Cannot Enable TIMs - Urgent Please Help!

    Broadcom Employee
    Posted Apr 22, 2016 09:36 AM

    Correct - HTTP Plugin



  • 24.  Re: Cannot Enable TIMs - Urgent Please Help!

    Posted Apr 22, 2016 09:05 AM

    I am now getting additional problems with stopping TIMs

     

    [root@txanunxlapmd005 defects]# service tim stop

    Stopping tim: /sbin/service: line 66: 43668 Killed                  env -i PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/${SERVICE}" ${OPTIONS}

    [root@txanunxlapmd005 defects]# service tim status

    tim dead but subsys locked