Workload Automation

Expand all | Collapse all

WCC showing conflicting results

Steve Carrobis08-01-2016 09:32 AM

  • 1.  WCC showing conflicting results

    Posted 02-10-2015 10:27 PM

    In some of our WCC views, the job status shows as one value, and on job details, another state. This is causing havoc, to say the least.

     

    Illustrations:

     

    (view shows job in starting state)

     

    EEM02-20150210-D1.png

    (detail shows job as success)

    EEM02-20150210-C1.png

     

    We have an issue open with support.

     

    Anyone know how this ambiguity could happen?

     

    The view definition is straightforward:

     

     

    // View JOBS-STARTING

    modify_view: JOBS-STARTING

        render_flow: true

        job_columns: job_name,job_type,status,server,run_machine,status_time

        alert_columns: alert,severity,status,job,server,create_time,status_time

     

    modify_filter: PROD

        view: JOBS-STARTING

        filter: status = STARTING

        active: true



  • 2.  Re: WCC showing conflicting results

    Posted 02-11-2015 08:29 PM

    The QVIEW/QEDIT data is populated live through a application server API/SDK call from the WAAE database, much like an autorep. Whereas, the dashboard, monitoring is from the WCC database which is populated by the WCC-Collector process. It is likely that the Collector process has stopped or has issues connecting to the database. Is WCC is HA mode (multiple cluster-members)? If yes, do you see the same status discrepancy on all WCC servers and between WCC servers?

    If you were to empty the view contents (using wcc_monitor CLI) and let it re-populate, do the jobs come back with correct statuses? Please check the collector application logs for any database related errors.



  • 3.  Re: WCC showing conflicting results

    Posted 07-12-2016 11:39 AM

    I currently have this exact issue with the latest SP for WCC (11.4 SP3) - I currently have a ticket opened and it seems that even if you run the wcc_monitor -d command the issue seems to pop up again.  We do have HA and I tried logging into each node and saw the issue persist.



  • 4.  Re: WCC showing conflicting results

    Posted 07-20-2016 07:18 AM

    Jim,

     

    check your collector.. make sure no errors about missing tables. ;-)



  • 5.  Re: WCC showing conflicting results

     
    Posted 07-20-2016 10:32 AM

    You may process these steps also just to make sure WCC is healthy

     

    1. Validate your credentials successfully

    2. Recycle Application Server in Scheduler machine

    3. Clear Cache

    4. Delete WCC services logs & recycle the WCC services & take a look into newly created services logs & upload them to case so CA Support will capture error messages walk through towards the solution.

    5. Clear Cache with server name specifically instead of using 'ALL' will be recommended.



  • 6.  Re: WCC showing conflicting results

    Posted 07-20-2016 10:42 AM
      |   view attached

    Why would you want to delete logs? ~

     

     

     

    Steve C.

    Batch System Engineer

    Batch Innovation Team

    steven.carrobis@gm.com

    T (313)553-1727|  C (770)502-5519

     

     



  • 7.  Re: WCC showing conflicting results

    Posted 07-26-2016 12:44 AM

    I have already opened a ticket with CA and they say that it is an elusive bug.  This was a problem in 11.3.5 and still is a problem in 11.3.6 SP1.  Except it seems to happen more often in 11.3.6 SP1.

     

    We do not use WCC in HA mode.  We use a physical load balancer in front of the two WCC servers.

     

    CA told us to do the following:

    • wcc_monitor –u username –p password –d %
    • the Dashboard views in the Jobs column should shows 0 and then rebuild
    • Recycle WCC services on each WCC server.  From CA: Recycling WCC is not required after
      executing -d, however I find recycling helps "refresh" WCC when
      experiencing this type of problem.
    • wcc_monitor –u userid –p password –r –v

     

    While this worked one time, we had the same issue last week and the views did not rebuild.  Only after rebooting WCC did it rebuild.



  • 8.  Re: WCC showing conflicting results

    Posted 07-26-2016 09:59 AM

    Hi Lester, are you using the default Derby database with WCC 11.3.6? We had similar issues with missing jobs a couple of years back and found out that CA does not recommend using Derby for large enterprise environments. We have since moved to Oracle and WCC 11.4 and have not experienced missing jobs in the WCC views anymore.

     

    Good luck with your case with CA and the resolution.

     

    Lisa



  • 9.  Re: WCC showing conflicting results

    Posted 07-26-2016 10:15 AM

    I tagged on to Lester's thread, so I don't know if he is still having the behavior, but my shop is Windows/SQL Server with the monitoring and reporting DBs apart of a Always On cluster.



  • 10.  Re: WCC showing conflicting results

    Posted 07-26-2016 10:25 PM

    Yes I am still having the behavior.  I had it less in 11.3.5 than in 11.3.6.  In the last 3 weeks, it has happened once each week.  But I upgraded to 11.3.6 SP1 back in April.  So I was somewhat surprised to have this issue so often in the last 3 weeks but was quiet prior to that.  To read that ckent's entry about being at SP4, was a bit disheartening. 

     

    I am not using Derby.  I am using SQL Server for AE and WCC.  The database is SQL Server clustered with a replication to a DR site.

     

    I just got a response from CA to upgrade to SP4, but I will point him to this posting to show that SP4 exhibits the problem as well.

     

    We did issue the wcc_monitor -d % command one time and there was no change in the Quick View.  We kind of panic and rebooted the two WCC servers.  We are also a small show with about 4,000 jobs.  So the command should not have taken too long.



  • 11.  Re: WCC showing conflicting results

    Posted 07-27-2016 03:12 AM

    It might be related to some kind of performance issue, please check/modify the WCC-collector intervals. Perhaps also increasing the memory on the java-process might help.

    Good luck



  • 12.  Re: WCC showing conflicting results

    Posted 07-26-2016 10:13 AM

    That is what CA gave us as a solution, but we are a small AutoSys shop and this only occurred after we upgraded to 11.4 SP3 WCC.

    It seems there are deadlocks that are occurring on specific PIDs that are causing the behavior.  Our current ticket is being upgraded to L2 support and being reviewed by Sustaining Engineering.

     

    In the meantime to make sure we are actively monitoring appropriately, we are clearing the cache every hour, however we are not recycling the WCC services as we have found it not to be needed to get the views in sync. 

     

    Obviously this is not a long term solution and we are hoping that this behavior can be fixed for the long term.

     

    Thanks,

    Chris Kent

    Dart Container Corp.



  • 13.  Re: WCC showing conflicting results

    Posted 08-22-2016 12:42 PM

    Just to update our current situation:  SE gave us a patch that updated the collector process to gather more data which in turn helped eliminate seeing the incorrect status.  The behavior has pretty much resolved.  I asked if the fix would be apart of the next SP, but was informed there was no guarantee.

     

    Let me know if anyone has any questions and I'll do what I can to address.

     

    Thanks,

    Chris Kent

    Dart Container Corp



  • 14.  Re: WCC showing conflicting results

    Posted 08-01-2016 07:33 AM

    Also look out for spontaneous table disappearances which then invalidate the collector which of course doesn't throw any errors.

    we have a few tables grow legs and walk out of the db.. One interesting feature I was shown. if you drop and recreate the user. WCC recreates the proper tables. That being said. I wonder if it nukes tables as well ;-)

    Some times a product can become TOO smart.

     

    Steve C.



  • 15.  Re: WCC showing conflicting results

    Posted 08-01-2016 09:27 AM

    Re: DB tables that are MIA - pardon the curiosity, is WCC using the native Derby or an external database such as Oracle?

    Thank you

    Chris <CJ>



  • 16.  Re: WCC showing conflicting results

    Posted 08-01-2016 09:32 AM

    ORACLE

     

     

    Steve C.



  • 17.  Re: WCC showing conflicting results

    Posted 08-22-2016 02:54 PM

    i would like to be able to stop and start the collector like the old days. having to stop the world just to give the collector a kick in the chest to get started is ludicrous.

     

    Steve C.