Service Operations Insight

  • Private Community

  • 1.  Unable To Update Alert Queue

    Posted Nov 22, 2014 07:44 PM

    Wondering if anyone has ever run into this issue.  This happened to us once in the past when we breached a limit in the size of the alert queue, but this time I was trying to simply remove an entry from our alert queue.

     

    Here are the following scenarios that have been tried:

     

    1. Opened 'OCC' alert queue

    2. Removed an entry

    3. Saved Queue.

    4. After 1 minute, checked alert queue ... the line that i removed is still

    there

    5. Eventually (about 5 - 10 minutes) i noticed an Exception in the

    soimgr-debug log

    6. Check Alert queue, the line I removed is now gone

     

    Last night we had to reboot every component of SOI due to a change in our

    environment, and I was hoping that it would fix the issue.  The interesting

    thing that I noticed however, is that the line that I had removed had

    RETURNED!.   I tried removing the line again (once SOI was back online and

    all connectors were green), and steps 1 - 6 occurred again.  Below is the

    exception

     

     

    
2014-11-22 19:25:04,636 ERROR [http-7090-6] def.AbstractFlushingEventListener.performExecutions(301)  - Could not synchronize database state with session
org.hibernate.exception.GenericJDBCException: could not update: [com.ca.sam.common.queue.AlertQueue#19]
  at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
  at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
  at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
  at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2430)
  at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2312)
  at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2612)
  at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:96)
  at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
  at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263)
  at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:168)
  at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
  at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
  at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
  at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)
  at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
  at com.ca.sam.manager.hibernate.HibernateUtil.commit(HibernateUtil.java:775)
  at com.ca.sam.manager.repository.DefaultDBTableManager.writeModelAttributes(DefaultDBTableManager.java:604)
  at com.ca.sam.manager.repository.SamModelRepository.writeModelAttributes(SamModelRepository.java:1538)
  at com.aprisma.spectrum.app.topo.web.repository.ModelRepository.writeModelAttributes(ModelRepository.java:1746)
  at com.aprisma.spectrum.app.topo.web.repository.ModelRepository.writeModelReqs(ModelRepository.java:1968)
  at com.aprisma.spectrum.app.topo.web.repository.ModelRepository.writeModels(ModelRepository.java:2076)
  at com.aprisma.spectrum.app.topo.web.servlet.ModelWriteServlet.doGet(ModelWriteServlet.java:163)
  at com.aprisma.spectrum.app.topo.web.servlet.ModelWriteServlet.doPost(ModelWriteServlet.java:78)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
  at com.ca.sam.common.HttpServlet.service(HttpServlet.java:89)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
  at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
  at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
  at java.lang.Thread.run(Thread.java:722)
Caused by: java.sql.SQLException: The query has timed out.
  at net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:3932)
  at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:1045)
  at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:537)
  at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeUpdate(JtdsPreparedStatement.java:505)
  at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:23)
  at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2408)
  ... 35 more
2014-11-22 19:25:04,637 INFO  [http-7090-6] hibernate.HibernateUtil.commit(780)  - Could not commit transaction:
org.hibernate.exception.GenericJDBCException: could not update: [com.ca.sam.common.queue.AlertQueue#19]
  at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
  at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
  at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
  at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2430)
  at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2312)
  at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2612)
  at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:96)
  at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
  at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263)
  at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:168)
  at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
  at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
  at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
  at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)
  at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
  at com.ca.sam.manager.hibernate.HibernateUtil.commit(HibernateUtil.java:775)
  at com.ca.sam.manager.repository.DefaultDBTableManager.writeModelAttributes(DefaultDBTableManager.java:604)
  at com.ca.sam.manager.repository.SamModelRepository.writeModelAttributes(SamModelRepository.java:1538)
  at com.aprisma.spectrum.app.topo.web.repository.ModelRepository.writeModelAttributes(ModelRepository.java:1746)
  at com.aprisma.spectrum.app.topo.web.repository.ModelRepository.writeModelReqs(ModelRepository.java:1968)
  at com.aprisma.spectrum.app.topo.web.repository.ModelRepository.writeModels(ModelRepository.java:2076)
  at com.aprisma.spectrum.app.topo.web.servlet.ModelWriteServlet.doGet(ModelWriteServlet.java:163)
  at com.aprisma.spectrum.app.topo.web.servlet.ModelWriteServlet.doPost(ModelWriteServlet.java:78)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
  at com.ca.sam.common.HttpServlet.service(HttpServlet.java:89)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
  at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
  at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
  at java.lang.Thread.run(Thread.java:722)
Caused by: java.sql.SQLException: The query has timed out.
  at net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:3932)
  at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:1045)
  at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:537)
  at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeUpdate(JtdsPreparedStatement.java:505)
  at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:23)
  at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2408)
  ... 35 more


2014-11-22 19:25:04,638 INFO  [http-7090-6] repository.DefaultDBTableManager.writeModelAttributes(615)  - DefaultDBTableManager: error writing table class com.ca.sam.common.queue.AlertQueue:
org.hibernate.exception.GenericJDBCException: could not update: [com.ca.sam.common.queue.AlertQueue#19]
  at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
  at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
  at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
  at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2430)
  at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2312)
  at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2612)
  at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:96)
  at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
  at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263)
  at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:168)
  at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
  at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
  at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
  at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)
  at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
  at com.ca.sam.manager.hibernate.HibernateUtil.commit(HibernateUtil.java:775)
  at com.ca.sam.manager.repository.DefaultDBTableManager.writeModelAttributes(DefaultDBTableManager.java:604)
  at com.ca.sam.manager.repository.SamModelRepository.writeModelAttributes(SamModelRepository.java:1538)
  at com.aprisma.spectrum.app.topo.web.repository.ModelRepository.writeModelAttributes(ModelRepository.java:1746)
  at com.aprisma.spectrum.app.topo.web.repository.ModelRepository.writeModelReqs(ModelRepository.java:1968)
  at com.aprisma.spectrum.app.topo.web.repository.ModelRepository.writeModels(ModelRepository.java:2076)
  at com.aprisma.spectrum.app.topo.web.servlet.ModelWriteServlet.doGet(ModelWriteServlet.java:163)
  at com.aprisma.spectrum.app.topo.web.servlet.ModelWriteServlet.doPost(ModelWriteServlet.java:78)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
  at com.ca.sam.common.HttpServlet.service(HttpServlet.java:89)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
  at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
  at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
  at java.lang.Thread.run(Thread.java:722)
Caused by: java.sql.SQLException: The query has timed out.
  at net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:3932)
  at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:1045)
  at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:537)
  at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeUpdate(JtdsPreparedStatement.java:505)
  at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:23)
  at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2408)
  ... 35 more


  • 2.  Re: Unable To Update Alert Queue

    Broadcom Employee
    Posted Nov 23, 2014 04:03 PM


    Hi Paimon, This discussion would be more suited for a support issue. We know by now that there was a hung query in your database that tried to collect historic alert information on this queue. Because of that the queue change could not be stored in the database. If you try again without the hung query you should not see this problem.



  • 3.  Re: Unable To Update Alert Queue

    Posted Nov 23, 2014 06:16 PM

    Hi Ralf;

     

    Thanks for the response.  I agree, I submitted a support ticket as well, just wanted to document it here in the communities in case anyone else had run into this.  Just an FYI for folks, looks like our issue was a long running query that spanned a few days.  Looks like it hosed the database and caused a bunch of random issues.  After killing the query, all was well.