Clarity PPM

Expand all | Collapse all

we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

Jump to Best Answer

Shalinee Chauhan04-26-2017 11:03 PM

Shalinee Chauhan04-27-2017 12:28 PM

  • 1.  we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 04:27 AM

    we are seeing an entry in PRLOCK table , on cmn_sch_job table .

    and all the scheduled jobs were not running .

     

    what is the reason for this lock .



  • 2.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 04:37 AM

    1. Have you checked the logs ?

    2. Have you checked if any report is running as well ?

     

    NJ



  • 3.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 05:08 AM

    Hi Ravinder,

     

    Is all your jobs scheduled/immediate not running? If it is then we have seen locks generated due to orphan records processing at the background. If its the case please let me know i can provide the workaround queries to overcome the same.

     

     

    Regards

    Suman Pramanik



  • 4.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 06:18 AM

    None of the schedule jobs were running , if we submit a job to run immediately also not running .

    but Processes were intiated were running .

     

    checked the logs not found any useful information .

     

    when we see the prlock , we found the locked time , with that if we trace the job log we haven't seen any related errors in log .

     

    we got to recycle the bg services to get this fixed , but this is happening once or twice in a week . no specific timings .

     

    in our system we have configured

       the rate matrix incremental to run every 15 mins in an hour .

       and rate matrix full load once in a day .

       also a data mart job which will copy the rates , obs informtion to datamart , this is configured hourly once . ( for this datamart we have added ratematrix as incompatible job )

     

    were these schedule jobs causing any problem ?

     

    also if we are restarting bg serices , what are the steps to take ?




  • 5.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 06:21 AM

    Hi suman,

      i have also read your previous post on this .  the scripts you have provided .

      in our case we haven't stopped the bg services , restarted bg services for any path .

      this is happening to us once or twice in a week .

     

    Thanks in advance,

    ravinder.



  • 6.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 06:25 AM

    small typo  path  instead of patch.
    in our case we haven't stopped the bg services , restarted bg services for any patch.



  • 7.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 06:29 AM

    Hi Ravinder,

     

    The restart is one main cause, but if there is no restart and there was some database disconnection due to network issues also, you may run into this. In case of cluster environment if the beacon is unable to poll other BG we have chances to have this problem. But the workaround is to run those queries and restart the BG.

     

    Regards

    Suman Pramanik



  • 8.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 06:43 AM

    Hi Suman,

      got the reason and answer .

      i have some more doubts .

      in our case we haven't restarted the services .

      so db disconnection or as our env is cluster there is a chance of beacon uable to poll other BG .

     

      we were doing the restart of BG services when ever this is happening , this is fixing our issue so far during this month period .

     

      do we need to run the scripts you have given with out missing ,  every time whenever this happened ?

     

       In that case why restaring the BG is solving our problem till now . are we breaking anything with this activity ?

     

      Is there any chance that excessive job load like ratematrix , datamart , other jobs will cause this problem ?



  • 9.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 06:48 AM

    Hi Ravinder,

     

    If restarting is fixing the problem then you don't need the script. Its the reason your BG is unstable or there is a thread hung and BG is unable to process once restart is done. All the hung thread in the pipeline is getting cleaned up. So the next point of analysis should be why there is a thread hung. It could be due to memory issue i.e. JVM memory allocated.

     

    Will it be possible to share the properties.xml after masking the sensitive information. I can take a look.

     

    Regards

    Suman Pramanik



  • 10.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 06:49 AM

    also during that time we have seen below error in log .. is this relevent to this issue ?

     

     

    ERROR 2014-10-20 01:48:11,034 [http-bio-8080-exec-548] view.ViewL10nSAXHandler (clarity:akum145:6321064__E7860BB4-0E1F-4DD0-94CE-7B21CBD7638D:odf.filterStateChange) Could not locate vxsl file '' in component odf



  • 11.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 06:55 AM

    Hi Ravinder,

     

    This error is from app and not BG, its a benign error occuring due to a bug. This cannot cause the BG to hung.


    Regards

    Suman Pramanik



  • 12.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 07:04 AM

    Hi Suman,

      we have two bg servers . restaring one of them is fixing the issue .

         below is the part of the properties.xml got from config directory of clarity installation on bg server .

     

      let me know if this helps ?

     

      also our calrity installatoin is bigger one , having 6 application servers , 2 bg servers .

      we have 30 k resources , 30 k project and many investments .

      we are just one month in this new system , piloting with few people around 200 .

     

     

    properties.xml is shown below

     

    <applicationServer vendor="tomcat" useLdap="false" home="E:\CA\apachetomcat7.0.42" adminPassword="admin" externalUrl="" tokenCacheCapacity="0" tokenCacheStrategy="none"> 

    <applicationServerInstance id="app" serviceName="CA Clarity PPM XOG Server" rmiPort="23791" jvmParameters="-Xms6144m -Xmx6144m -XX:-UseGCOverheadLimit -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:NewRatio=1 -DforceMemorySettings=false -XX:PermSize=96m -XX:MaxPermSize=256m" maxThreads="1000" programParameters="" distributed="false" cacheDistributedSession="false" runJobScheduler="false" useSSO="false" maxConcurrentJobs="10" runProcessEngine="false" messageTimeToLive="120" messageReceiverInterval="5" exceptionRunInterval="normal" serviceUser="" servicePassword="" autoStartService="false" maxXmlNodesLimit="150000" />

    <databaseServer vendor="oracle" home="/oracle/home" useMultilingualSort="false" sysPassword="change_on_install" largeTables="USERS_LARGE" smallTables="USERS_SMALL" largeIndex="INDX_LARGE" smallIndex="INDX_SMALL" highVolatilityParameters="PCTFREE 20 PCTUSED 60" lowVolatilityParameters="PCTFREE 5 PCTUSED 80" fetchSize="60" /> 

    <database id="Niku" useURL="true" jndiDatabaseId="jdbc/NikuDS" vendor="oracle" serviceName="xxxxx.com" serviceId="niku" instanceName="" driver="com.ca.clarity.jdbc.oracle.OracleDriver" url="jdbc:clarity:oracle://ora123:1521;ServiceName=xxxxx.com;BatchPerformanceWorkaround=true;InsensitiveResultSetBufferSize=0;ServerType=dedicated;supportLinks=true" host="localhost" username="xxxx" password="xxxxx" schemaName="xxxxx" />

    <processEngineMonitorConfig disable="false" numberOfThreads="1" appId="app"> 

    - <processEngineMonitorTask name="bpmMonitorLoopDetector" className="com.niku.bpm.utilities.BpmProcessMonitorLoopDetector" initialDelay="60" period="60" disable="false">


  • 13.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 07:07 AM

     

    <processEngineMonitorConfig disable="false" numberOfThreads="1" appId="app">

     

    <processEngineMonitorTask name="bpmMonitorLoopDetector" className="com.niku.bpm.utilities.BpmProcessMonitorLoopDetector" initialDelay="60" period="60" disable="false">

     

    <taskItems>

     

    <taskItem name="putOnHold" value="true" dataType="boolean"/>

     

    <taskItem name="sendNotification" value="true" dataType="boolean"/>

     

    <taskItem name="abortInstances" value="true" dataType="boolean"/>

     

    <taskItem name="timeThreshold" value="1" dataType="int"/>

     

    <taskItem name="loopLimit" value="100" dataType="int"/>

     

    <taskItem name="processExceptionList" value="" dataType="string"/>

     

    </taskItems>

     

    </processEngineMonitorTask>

     

    <processEngineMonitorTask name="bpmMonitorStormDetector" className="com.niku.bpm.utilities.BpmProcessMonitorStormDetector" initialDelay="60" period="60" disable="false">

     

    <taskItems>

     

    <taskItem name="putOnHold" value="true" dataType="boolean"/>

     

    <taskItem name="sendNotification" value="true" dataType="boolean"/>

     

    <taskItem name="timeThreshold" value="1" dataType="int"/>

     

    <taskItem name="processLimit" value="100" dataType="int"/>

     

    <taskItem name="processExceptionList" value="" dataType="string"/>

     

    </taskItems>

     

    </processEngineMonitorTask>

     

    </processEngineMonitorConfig>

     



  • 14.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 07:11 AM

    Error from BG log , yesterday when it hung..

     

    SYS   2014-10-30 04:46:16,330 [Thread-4] njs.SchedulerImpl (none:none:none:none) Clarity 13.3.0.286 Job Scheduler bg@*** stopping...

    SYS   2014-10-30 05:00:54,200 [WrapperSimpleAppMain] bgp.JobLogger (clarity:none:none:none) Tenant assigned: clarity

    SYS   2014-10-30 05:00:55,809 [WrapperSimpleAppMain] niku.union (clarity:scheduler:6371042__6D4DCF16-DBE5-45CE-89FE-4300A95B11F9:none) Initializing Event Manager on server instance: bg

    SYS   2014-10-30 05:00:56,294 [WrapperSimpleAppMain] niku.union (clarity:scheduler:6371042__6D4DCF16-DBE5-45CE-89FE-4300A95B11F9:none) Starting Process Engine on server instance: bg

    SYS   2014-10-30 05:00:56,419 [WrapperSimpleAppMain] niku.union (clarity:scheduler:6371042__6D4DCF16-DBE5-45CE-89FE-4300A95B11F9:none) Process Engine Name: bg-*** SYS   2014-10-30 05:00:56,512 [WrapperSimpleAppMain] niku.union (clarity:process_admin:6371043__076C04AE-F78D-4400-B426-8C795F16B658:none) Starting Message Receiver on Event Manager

    SYS   2014-10-30 05:00:56,559 [Event Interest Registration Thread] niku.union (clarity:none:none:none) Registering event interests...

    SYS   2014-10-30 05:00:56,591 [Thread-5] njs.SchedulerImpl (clarity:none:none:none) Niku Job Scheduler (tenant=clarity)

    SYS   2014-10-30 05:00:56,591 [Thread-5] njs.SchedulerImpl (clarity:none:none:none) Clarity 13.3.0.286 Job Scheduler bg@APSEP3912 initializing...

    SYS   2014-10-30 05:00:56,606 [Event Interest Registration Thread] niku.union (clarity:none:none:none) Event registration completed. Event manager started succesfully.

    SYS   2014-10-30 05:00:56,762 [Thread-5] njs.SchedulerImpl (clarity:scheduler:6371044__258876EB-CA19-4F74-9631-BC8834E45621:none) Clarity 13.3.0.286 Job Scheduler bg@xxxxx initialized

    ERROR 2014-10-30 05:00:59,684 [Dispatch pool-5-thread-2 : bg@xxxxx (tenant=clarity)] niku.reporting (clarity:admin:6371055__E6A1B713-28BA-495F-BCDB-AD5FDCBAF88E:D10 Create BusinessObjects Users) Login failed, unknown error.There was an error reading the shared secret from trusted principal configuration file. (FWM 02045)

    ERROR 2014-10-30 05:00:59,684 [Dispatch pool-5-thread-2 : bg@xxxxx (tenant=clarity)] niku.njs (clarity:admin:6371055__E6A1B713-28BA-495F-BCDB-AD5FDCBAF88E:D10 Create BusinessObjects Users) Error executing job: 5042001

    java.lang.Exception: reporting.loginException

    at com.niku.reporting.bo.BOReporting.login(BOReporting.java:167)

    at com.niku.reporting.bo.BOReporting.login(BOReporting.java:182)

    at com.niku.reporting.jobs.SyncBOUsersJob.CreateUsers(SyncBOUsersJob.java:82)

    at com.niku.reporting.jobs.SyncBOUsersJob.runCreateUsers(SyncBOUsersJob.java:65)

    at com.niku.reporting.jobs.SyncBOUsersJob.scheduledEventFired(SyncBOUsersJob.java:54)

    at com.niku.njs.Dispatcher$BGTask.run(Dispatcher.java:362)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:724)

    Caused by: com.crystaldecisions.sdk.exception.SDKException$TrustedPrincipalConfigError: There was an error reading the shared secret from trusted principal configuration file. (FWM 02045)

    at com.crystaldecisions.sdk.framework.internal.TrustedPrincipal.getPassword(TrustedPrincipal.java:154)

    at com.crystaldecisions.sdk.framework.internal.SessionMgr.logon(SessionMgr.java:657)

    at com.niku.reporting.bo.BOReporting.login(BOReporting.java:139)



  • 15.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 07:15 AM

    Hi Ravinder,

     

    This error says only BO trusted authentication is not correct? Are you not runnning reports?

     

    Regards

    Suman pramanik



  • 16.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 07:14 AM

    Hi Ravinder,

     

    I don't see the background server tag in the properties.xml. Also its better to have dedicated BG for job scheduler and process. Rather than running both on one service. Thats what i prefer and i have seen better performance also.

     

    Regards

    Suman



  • 17.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 07:19 AM

      may be the log was written when we restarted the BG after the jobs were struck . we have not yet running BO reports ..

     

    below is the background server tag ..

     

     

    <backgroundServer jvmParameters="-Xms3072m -Xmx6144m -XX:-UseGCOverheadLimit -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:NewRatio=1 -DforceMemorySettings=false -XX:PermSize=96m -XX:MaxPermSize=256m" programParameters=""/>

     



  • 18.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 07:22 AM

    Hi Suman,

    not understood what is your last post said exactly ..

     

    can we have one BG server for Job Scheduling   and  one more for Processes .

    is this what you mean by dedicated servers ?



  • 19.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 10-31-2014 07:27 AM

    Hi Ravinder,

     

    Not sure why PERM Size & MAX Perm size is there, it should be ideally below which i mentioned, you can change the memory as per your requirement. And yes one bg for Job and one for Process.

     

    -Xms512m -Xmx1024m -XX:-UseGCOverheadLimit -DforceMemorySettings=false -XX:MaxPermSize=192m

     

    Regard

    Suman Pramanik



  • 20.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 04-26-2017 10:18 PM

    Hi Suman,

    I am seeing the same issue in my production insatnce and none of the jobs are running schedule/immediately. I have restarted bg and removed, re-deployed but no improvement. Could you please provide us the query.

     

    Thanks,

    Shal.



  • 21.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .
    Best Answer

    Posted 04-26-2017 10:29 PM

    Hi Shal,

    Please try the queries in this tec doc

    Clarity: Scheduled jobs stuck in waiting or scheduled status 

     

    Thanks,

    Jerin



  • 22.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 04-26-2017 11:03 PM
      |   view attached

    Done.

     

    Working fine.

     

    Thanks.



  • 23.  Re: we are seeing an entry in PRLOCK table , on cmn_sch_job table .  and all the scheduled jobs were not running . what is the reason for this lock .

    Posted 04-27-2017 12:28 PM

    Thanks Jerin.