CEM stop working

[Gabriel Orlando]

Hello,
Yesterday after a restart of MOM and Collectors service (using EMCtrl.sh stop), the CEM stop collecting transactions. I check the em.log of the Collector with tim service collector role and i see this:

9/16/20 04:56:54.872 PM ART [ERROR] [Thread-26] [Manager.com.timestock.tess.services.initialization.RunInitializers] Exception in bean initializer; some features won't work
java.util.concurrent.ExecutionException: org.hibernate.LazyInitializationException: could not initialize proxy - no Session
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.timestock.tess.services.initialization.RunInitializers$RunInitializeMethods.run(RunInitializers.java:111)
Caused by: org.hibernate.LazyInitializationException: could not initialize proxy - no Session
at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:132)
at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:174)
at org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.invoke(CGLIBLazyInitializer.java:173)
at com.timestock.tess.data.objects.TranDefGroup$$EnhancerByCGLIB$$497d5d69.getName(<generated>)
at com.wily.apm.tess.isengard.BizDefnitionBeanHelper.getBusinessTransaction(BizDefnitionBeanHelper.java:170)
at com.wily.apm.tess.isengard.BizDefnitionBeanHelper.getBizDefinitionObjects(BizDefnitionBeanHelper.java:366)
at com.wily.apm.tess.isengard.BizDefinitionObjectCache.initialize(BizDefinitionObjectCache.java:113)
at com.timestock.tess.services.initialization.RunInitializers$IntializeWrapper.call(RunInitializers.java:87)
at com.timestock.tess.services.initialization.RunInitializers$IntializeWrapper.call(RunInitializers.java:1)
at com.timestock.tess.util.ThreadPool$ConcurrentCallableTask.call(ThreadPool.java:218)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
9/16/20 04:56:54.904 PM ART [WARN] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.service.CheckServices] EM earlier was not shutdown properly, TIM.Collection.Svc service is trying to recover...
9/16/20 04:56:54.908 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.service.TimCollectionService] TIM.Collection.Svc starting...
9/16/20 04:56:54.914 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.AutogenProcessor] AutogenProcessor starting...
9/16/20 04:56:54.935 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.AutogenProcessor] AutogenProcessor started
9/16/20 04:56:54.936 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.BizEventProcessor] IncidentProcessor starting...
9/16/20 04:56:55.011 PM ART [INFO] [Thread-28] [Manager.com.timestock.tess.services.processors.BizEventProcessor] IncidentProcessorThread starting
9/16/20 04:56:55.012 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.BizEventProcessor] IncidentProcessor started
9/16/20 04:56:55.012 PM ART [INFO] [Thread-28] [Manager.com.timestock.tess.services.processors.BizEventProcessor] Checking Pending Incidents
9/16/20 04:56:55.019 PM ART [INFO] [Thread-28] [Manager.com.timestock.tess.services.processors.BizEventProcessor] Finished Checking Pending Incidents
9/16/20 04:56:55.031 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.BtStatsProcessor] BtStatsProcessor started
9/16/20 04:56:55.031 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.service.CaseSensitivityProvider] CaseSensitivityService starting...
9/16/20 04:56:55.039 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.service.CaseSensitivityProvider] CaseSensitivityService started
9/16/20 04:56:55.041 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.DefectProcessor] DefectProcessor starting...
9/16/20 04:56:55.042 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.util.DbUtils] building biz event cache...
9/16/20 04:56:55.077 PM ART [ERROR] [ServicesCheck.Thread1] [org.hibernate.LazyInitializationException] could not initialize proxy - no Session
org.hibernate.LazyInitializationException: could not initialize proxy - no Session
at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:132)
at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:174)
at org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.invoke(CGLIBLazyInitializer.java:173)
at com.timestock.tess.data.objects.TranSetDef$$EnhancerByCGLIB$$773ea934.getName(<generated>)
at com.timestock.tess.data.objects.BizEvent.getTranSetName(BizEvent.java:89)
at com.timestock.tess.util.DbUtils.addBizEventToCache(DbUtils.java:8108)
at com.timestock.tess.util.DbUtils.initializeBizEvents(DbUtils.java:8090)
at com.timestock.tess.services.processors.DefectProcessor.startService(DefectProcessor.java:235)
at com.timestock.tess.services.service.ServiceBase.startService(ServiceBase.java:116)
at com.timestock.tess.services.service.CheckServices.startService(CheckServices.java:209)
at com.timestock.tess.services.service.CheckServices.checkEntityServices(CheckServices.java:151)
at com.timestock.tess.services.service.CheckServices.run(CheckServices.java:58)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
9/16/20 04:56:55.078 PM ART [ERROR] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.DefectProcessor] Unexpected exception starting DefectProcessor
org.hibernate.LazyInitializationException: could not initialize proxy - no Session
at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:132)
at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:174)
at org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.invoke(CGLIBLazyInitializer.java:173)
at com.timestock.tess.data.objects.TranSetDef$$EnhancerByCGLIB$$773ea934.getName(<generated>)
at com.timestock.tess.data.objects.BizEvent.getTranSetName(BizEvent.java:89)
at com.timestock.tess.util.DbUtils.addBizEventToCache(DbUtils.java:8108)
at com.timestock.tess.util.DbUtils.initializeBizEvents(DbUtils.java:8090)
at com.timestock.tess.services.processors.DefectProcessor.startService(DefectProcessor.java:235)
at com.timestock.tess.services.service.ServiceBase.startService(ServiceBase.java:116)
at com.timestock.tess.services.service.CheckServices.startService(CheckServices.java:209)
at com.timestock.tess.services.service.CheckServices.checkEntityServices(CheckServices.java:151)
at com.timestock.tess.services.service.CheckServices.run(CheckServices.java:58)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
9/16/20 04:56:55.078 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.DefectProcessor] DefectProcessor stopping...
9/16/20 04:56:55.085 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.DefectProcessor] DefectProcessor stopped
9/16/20 04:56:55.085 PM ART [ERROR] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.service.TimCollectionService] DefectProcessor can't be started
9/16/20 04:56:55.085 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.AutogenProcessor] AutogenProcessor stopping...
9/16/20 04:56:55.085 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.AutogenProcessor] AutogenProcessor stopped
9/16/20 04:56:55.086 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.BizEventProcessor] IncidentProcessor stopping...
9/16/20 04:56:55.086 PM ART [INFO] [Thread-28] [Manager.com.timestock.tess.services.processors.BizEventProcessor] IncidentProcessorThread terminated
9/16/20 04:56:55.086 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.BizEventProcessor] IncidentProcessor stopped
9/16/20 04:56:55.086 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.BtStatsProcessor] BtStatsProcessor stopped
9/16/20 04:56:55.086 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.service.CaseSensitivityProvider] CaseSensitivityService stopping...
9/16/20 04:56:55.086 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.service.CaseSensitivityProvider] CaseSensitivityService stopped...
9/16/20 04:56:55.087 PM ART [ERROR] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.service.CheckServices] Can't start service 'TIM.Collection.Svc' on entity 3
9/16/20 04:56:55.113 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.service.StatsAggregationService] Stats.Aggregation.Svc starting...
9/16/20 04:56:55.114 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.StatsAggregator] StatsAggregator starting...
9/16/20 04:56:55.535 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.StatsAggregator] First scheduled daily aggregation will run at 2020-09-17 00:02:00
9/16/20 04:56:55.544 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.processors.StatsAggregator] StatsAggregator started
9/16/20 04:56:55.574 PM ART [INFO] [ServicesCheck.Thread1] [Manager.com.timestock.tess.services.service.StatsAggregationService] Stats.Aggregation.Svc started
9/16/20 04:56:58.301 PM ART [INFO] [main] [org.apache.tiles.context.AbstractTilesApplicationContextFactory] Initializing Tiles2 application context. . .
9/16/20 04:56:58.304 PM ART [INFO] [main] [org.apache.tiles.context.AbstractTilesApplicationContextFactory] Finished initializing Tiles2 application context.

I check the tim server and is collecting transactions without any problem:

This is the last cem event collected:

Anyone who can help on this?

Regards

------------------------------
Cordinador Ingenieria y Soporte
Banco Patagonia
------------------------------

[Jörg Mertin]

Hi Gabriel,
first question - why did you restart the EM/MoM? and how did you restart it?
Then, did you change anything on the EM/MoM side? with this I mean - did you perform an update, apply a hotfix etc.?

Also, what exact version are you using? (Usually visible in the beginning of the log). How much disk-space is available, how many resources (CPU/Memory) are available when you run the MoM/EM? Are there any other errors in the logs?

You could of course get the https://github.com/CA-APM/cem-healthcheck-scripts and execute EM, SYS and PSQL log collection process (Note that for PSQL you will need to provide the credentials to access the PostgreSQL DB). Details on what the script does in the github repository's README.md file.

You should open a case and submit the log-files there. I'm sure our Support team will inform me and we'll check it out

Regards

------------------------------
Customer Solution Engineering
Broadcom Inc.
------------------------------

[Gabriel Orlando]

Hello and thanks for you help.
Broadcom Support asked me to re-generate the postgress database (cemdb). After that CEM works OK.

------------------------------
Cordinador Ingenieria y Soporte
Banco Patagonia
------------------------------

[Gabriel Orlando]

The solution steps:

• stop MOM, TIMS and EM Collectors
• XXXX/PostgresSQL/bin/dropdb -h localhost -p 5432 -U postgres cemdb
• XXXX/PostgresSQL/bin/createdb -U postgres cemdb
• XXXX/install/database-scripts/unixc/createschema.sh -databaseName cemdb -databaseType Postgres -host localhost -password xxxx -port 5432 -releaseVersion 10.7.0.0 -scriptsDir /applications/caapm/install/database-scripts -user admin
• start EM Collectors, MOM and TIMS

In my case i need to change the createschema.sh because on of the jars called by the .sh was older than the installation has. This was because an HF implementation.

This one postgresql-42.2.8.jar for postgresql-42.2.14.jar

------------------------------
Cordinador Ingenieria y Soporte
Banco Patagonia
------------------------------

[Jörg Mertin]

Good to know it works.
Thx for provide the solution. 
Note however that your way of doing it deletes all content, including the configuration.

------------------------------
Customer Solution Engineering
Broadcom Inc.
------------------------------