forced system traces

[Ron Gates]

Has anybody seen this in their V12 linux systems??  Seems like after migrating to linux we have had this type of error twice now..
Never had these type of messages on aix...

The error we are getting over and over non stop is

20190830/115720.249 - U00003631 Dump caused by:
20190830/115720.249 - U00005118 Program error: Database access via key 'AH_Idnr' with value 0 in module 'JPEXEC_R'.

We are running v12.2.3.hf1... but we saw this same type of error in v12.2.0.hf2..
Supports response... was to upgrade to the latest release.. well.. now I am seeing it in v12.2.3.hf1??
I have a ticket opened with support.. but hope their answer is to upgrade to v12.3.* as that's not acceptable..

Has anybody else been having these types of errors running V12?  on linux?
thanks

[Carsten Schmitz]

I have to do a little excursion here:

​With 10.x (on Linux), engine dumps were happening once every few weeks. We monitored them and if one happened, reported it to Automic.

With 12.0 and 12.1 (also on Linux), the engine was initially dumping every night, sometimes multiple times. We tried to go through with filing support tickets, but were either told to upgrade or no acceptable resolution could be offered.

We have since created an internal Wiki page were we merely catalog the dump messages and try to assess if they mean anything we need to worry about. From that page, I see that we had similar messages, but not the same:

U0005118 Program error: Database access via key 'RH_AH_Idnr' with value 0 in module 'UCUREP'.

From the comment on the Wiki page, I do believe a colleague filed a support ticket, but didn't get any conclusive answers (I can obviously not research this because all old support tickets are since inaccessible).

We are since ignoring this message with no immediately related negative consequences.

Not saying this is good, just saying this is the current state of affairs because with many dump messages, we didn't really get anywhere with Automic.

Br,
Carsten

[Christian Boeck]

Hi,
I watched the same bahavior on V12 and I opened a call.
Actually the bug should be fixed with 12.3 but it isn't.
I've extended my ticket, so the search will go on.
As soon as I get an answer from the Support I'll inform you.

------------------------------
Thx & rgds
Christian
------------------------------

[Christian Boeck]

Hi,

suppport told me that the fix will be included in version 12.3.2 and 12.4.

------------------------------
Thx & rgds
Christian
------------------------------

[Aicke Sandrock]

Hi @all,

we ​recognized the same behavior with our engine installation.
Nearly every day we're getting trace files in our engine directory. Our engine runs on SLES12 SP4 system
with Version 12.2. In previous engine versions (<12) didn't occur so many trace files.

Our current workaround ist a simple check job which reports new trace files via mail, so that we are 
able to take care of them. Unfortunately the "solution" ist mostly delete them to make sure that wie don't 
run out of space in our filesystem.

Regards
Aicke

[Frank Muffke]

Hi
we use RHEL7.6 with AE 12.3.GA+HF0 and do not have any issues with dumps so far...

On RHEL7.6 with AE 11.2.3 we had dumps regularly after a DB Reorg (From DB side, not UC4 Reorg)

KR Wolfgang

------------------------------
I know I do really know it!
------------------------------

[Krum Ganev]

Hello,

We are experiencing the same issue. We have currently an open ticket with Broadcom but so far nothing concreete.

The dumps are generated ~2times per day at different time.

AE - 12.3

OS - SLeS 12 SP4

P.S. The first update of the ticket was to adjust the AWI heap memory.

I dont believe this should be the case. We have it on two servers and the tomcat is started with 25GB RAM each.

Best Regards,

Krum Ganev

[Carsten Schmitz]

Sounds plenty to me.

We ran on 16GB for quite some time with no issues. We recently had a miss-configuration lowering that to 2GB, even that ran for a week or two before we got an OOM exception. 25G sounds plenty to me.

Also, from that occurence I can tell that a lack of Tomcat heap is clearly visible by an exception saying pretty much exactly that. So I'd wager this isn't the root cause here.

[Michael A. Lowry]

We've seen similiar forced traces in v12.0.5 HF3, and the response from Broadcom support was the same: upgrade to 12.0.8.
We asked if the forced traces contained enough information to conclusively identify the cause; the answer was no.
We asked if they were sure that the service pack would resolve the problem; again, the answer was no.
We will upgrade next month and hope for the best.

[Carsten Schmitz]

> We asked if the forced traces contained enough information to conclusively identify the cause; the answer was no.
> We asked if they were sure that the service pack would resolve the problem; again, the answer was no.

From my understanding and last I heard, support is pretty much prohibited from quizzing development for definitive statements on such things, except for well-documented cases they can immediately replicate. The descriptive documents for known issues are sometimes vague enough to not be applicable with enough certainty. Thus, they can oftentimes not reasonably say that anything is fixed in a new version. Therefore, the adopted stance is one of "try to update and see if that helps"; a stance that is not uncommon in software distribution, but nevertheless frustrating for the enterprise user.

This is, of course, based on information I received at some point and purely my own, subjective impression.

Since that is the state of affairs at this point, we elected to always try to track the latest version closely, even in production. It's in my mind the only way to deal with this and some of the user base had to begrudgingly agree.

Best,
Carsten