WCC showing conflicting results

[Jim Spath]

In some of our WCC views, the job status shows as one value, and on job details, another state. This is causing havoc, to say the least.

 

Illustrations:

 

(view shows job in starting state)

 

(detail shows job as success)

 

We have an issue open with support.

 

Anyone know how this ambiguity could happen?

 

The view definition is straightforward:

 

// View JOBS-STARTING modify_view: JOBS-STARTING     render_flow: true     job_columns: job_name,job_type,status,server,run_machine,status_time     alert_columns: alert,severity,status,job,server,create_time,status_time   modify_filter: PROD     view: JOBS-STARTING     filter: status = STARTING     active: true

[Chandru.Venkataraman]

The QVIEW/QEDIT data is populated live through a application server API/SDK call from the WAAE database, much like an autorep. Whereas, the dashboard, monitoring is from the WCC database which is populated by the WCC-Collector process. It is likely that the Collector process has stopped or has issues connecting to the database. Is WCC is HA mode (multiple cluster-members)? If yes, do you see the same status discrepancy on all WCC servers and between WCC servers?

If you were to empty the view contents (using wcc_monitor CLI) and let it re-populate, do the jobs come back with correct statuses? Please check the collector application logs for any database related errors.

[Chris Kent]

I currently have this exact issue with the latest SP for WCC (11.4 SP3) - I currently have a ticket opened and it seems that even if you run the wcc_monitor -d command the issue seems to pop up again.  We do have HA and I tried logging into each node and saw the issue persist.

[Steve Carrobis]

Jim,

 

check your collector.. make sure no errors about missing tables. ;-)

[Saicharan Palagiri]

You may process these steps also just to make sure WCC is healthy

 

1. Validate your credentials successfully

2. Recycle Application Server in Scheduler machine

3. Clear Cache

4. Delete WCC services logs & recycle the WCC services & take a look into newly created services logs & upload them to case so CA Support will capture error messages walk through towards the solution.

5. Clear Cache with server name specifically instead of using 'ALL' will be recommended.

[Steve Carrobis]

Why would you want to delete logs? ~

 

 

 

Steve C.

Batch System Engineer

Batch Innovation Team

[e-mail]

T [phone]|  C [phone]

 

 

[Lester Chew]

I have already opened a ticket with CA and they say that it is an elusive bug.  This was a problem in 11.3.5 and still is a problem in 11.3.6 SP1.  Except it seems to happen more often in 11.3.6 SP1.

 

We do not use WCC in HA mode.  We use a physical load balancer in front of the two WCC servers.

 

CA told us to do the following:

• wcc_monitor –u username –p password –d %
• the Dashboard views in the Jobs column should shows 0 and then rebuild
• Recycle WCC services on each WCC server.  From CA: Recycling WCC is not required after
  executing -d, however I find recycling helps "refresh" WCC when
  experiencing this type of problem.
• wcc_monitor –u userid –p password –r –v

 

While this worked one time, we had the same issue last week and the views did not rebuild.  Only after rebooting WCC did it rebuild.

[lisadunaway]

Hi Lester, are you using the default Derby database with WCC 11.3.6? We had similar issues with missing jobs a couple of years back and found out that CA does not recommend using Derby for large enterprise environments. We have since moved to Oracle and WCC 11.4 and have not experienced missing jobs in the WCC views anymore.

 

Good luck with your case with CA and the resolution.

 

Lisa

[Chris Kent]

I tagged on to Lester's thread, so I don't know if he is still having the behavior, but my shop is Windows/SQL Server with the monitoring and reporting DBs apart of a Always On cluster.

[Lester Chew]

Yes I am still having the behavior.  I had it less in 11.3.5 than in 11.3.6.  In the last 3 weeks, it has happened once each week.  But I upgraded to 11.3.6 SP1 back in April.  So I was somewhat surprised to have this issue so often in the last 3 weeks but was quiet prior to that.  To read that ckent's entry about being at SP4, was a bit disheartening. 

 

I am not using Derby.  I am using SQL Server for AE and WCC.  The database is SQL Server clustered with a replication to a DR site.

 

I just got a response from CA to upgrade to SP4, but I will point him to this posting to show that SP4 exhibits the problem as well.

 

We did issue the wcc_monitor -d % command one time and there was no change in the Quick View.  We kind of panic and rebooted the two WCC servers.  We are also a small show with about 4,000 jobs.  So the command should not have taken too long.

[LARS PERSSON]

It might be related to some kind of performance issue, please check/modify the WCC-collector intervals. Perhaps also increasing the memory on the java-process might help.

Good luck

[Chris Kent]

That is what CA gave us as a solution, but we are a small AutoSys shop and this only occurred after we upgraded to 11.4 SP3 WCC.

It seems there are deadlocks that are occurring on specific PIDs that are causing the behavior.  Our current ticket is being upgraded to L2 support and being reviewed by Sustaining Engineering.

 

In the meantime to make sure we are actively monitoring appropriately, we are clearing the cache every hour, however we are not recycling the WCC services as we have found it not to be needed to get the views in sync. 

 

Obviously this is not a long term solution and we are hoping that this behavior can be fixed for the long term.

 

Thanks,

Chris Kent

Dart Container Corp.

[Chris Kent]

Just to update our current situation:  SE gave us a patch that updated the collector process to gather more data which in turn helped eliminate seeing the incorrect status.  The behavior has pretty much resolved.  I asked if the fix would be apart of the next SP, but was informed there was no guarantee.

 

Let me know if anyone has any questions and I'll do what I can to address.

 

Thanks,

Chris Kent

Dart Container Corp

[Steve Carrobis]

Also look out for spontaneous table disappearances which then invalidate the collector which of course doesn't throw any errors.

we have a few tables grow legs and walk out of the db.. One interesting feature I was shown. if you drop and recreate the user. WCC recreates the proper tables. That being said. I wonder if it nukes tables as well ;-)

Some times a product can become TOO smart.

 

Steve C.

[Chris Trinidad]

Re: DB tables that are MIA - pardon the curiosity, is WCC using the native Derby or an external database such as Oracle?

Thank you

Chris <CJ>

[Steve Carrobis]

ORACLE

 

 

Steve C.

[Steve Carrobis]

i would like to be able to stop and start the collector like the old days. having to stop the world just to give the collector a kick in the chest to get started is ludicrous.

 

Steve C.