WPs core dumps on AIX after upgrade to versions 12.2.* or 12.3.*.

[Johnny Reed]

We have been attempting to upgrade from version 11.2.7 to 12.2 or 12.3 but are having issues with WPs core dumping. We're running the server processes on AIX 7.1 with an Oracle database also on AIX 7.1. 
Has anyone else experienced this issue? If so, how did you resolve it?

------------------------------
Eastman Chemical Co.
------------------------------

[Frank Muffke]

Hi
I bet you installed AE 12.3.GA - correct?

this seems to be fixed with 12.3.1

https://downloads.automic.com/downloads/advanced_mode?selected_tab=&lifecycle_entity_id=1409143059253&major_version_id=1548836251814&search=&no-default-values=on&show-patch-descriptions=on
cheers, Wolfgang

------------------------------
Support Info:
if you are using one of the latest version of UC4 / AWA / One Automation please get in contact with Support to open a ticket.
Otherwise update/upgrade your system and check if the problem still exists.
------------------------------

[Johnny Reed]

Hi Wolfgang,
I've actually tried AE 12.2.4, 12.2.6, 12.3.1, and yesterday 12.3.2. I still get the wp core dumps.

Johnny​

------------------------------
Eastman Chemical Co.
------------------------------

[Frank Muffke]

Hi Johhny,

damn that seems to be a more serious issue.
Honestly I doubt that all those versions have issues on AIX platform.

Can you see anything helpful in the logs/traces?
Otherwise I would open a support ticket and sent the the logs,traces&dumps

cheers, Wolfgang

------------------------------
Support Info:
if you are using one of the latest version of UC4 / AWA / One Automation please get in contact with Support to open a ticket.
Otherwise update/upgrade your system and check if the problem still exists.
------------------------------

[Carsten Schmitz]

​Hi.

I don't know about AIX, but if you get really desperate, let the core dump generate a core file (ulimit -c unlimited). You can load that file into a debugger like gcc. For gcc, give it the executable path and the core file, and then with the backtrace/bt commands you can see the call stack at the time of the crash. Unfortunately your Automic binaries will probably be stripped of debugging symbols, but sometimes the call stack can provide clues as to why things crash.

(edit: oh, and if it reliably core dumps, you can possibly also strace the process and look at what happens before the core dump happens. Substitute "strace" for whatever the comparable tool on AIX is called :)

edit (2): or if you're extra desparate, open a ticket and upload the core file to Automic support - you might need to be very convincing though to get them to get a developer to look at it ...)

Hth,

------------------------------
These contain very good advise on asking questions and describing supposed bugs (no, you do not need to go to StackExchange for Automic questions, but yes, the parts on asking detailed, useful questions ARE usually relevant):

http://www.catb.org/~esr/faqs/smart-questions.html

https://www.chiark.greenend.org.uk/~sgtatham/bugs.html

I will not respond to PM asking for help unless there's an actual reason to keep the discussion off of the public forums.
------------------------------

[Johnny Reed]

​The log files don't provide any useful information concerning the core dumps. I have opened a ticket and it has been escalated. I would like to know if other AIX users have upgraded to AE 12.* without these issues.

------------------------------
Eastman Chemical Co.
------------------------------

[Kordula Falk]

Hi Jonny,
we haven't such kind of issues on our environment. 
Current versions:
 AE-System: 12.2.2 HF4
Oracle-DB:  12.1
AIX:  7.2.3
Regards Kordula

------------------------------
Kordula
------------------------------

[Carsten Schmitz]

​I wager AIX has a concept of LD_LIBRARY_PATH?

Loading incompatible libraries via LD_LIBRARY_PATH is one common reason for core dumps with AE. Have a look at your LD_LIBRARY_PATH (it's best to look at /proc/environ if AIX has that to see the actual value), check if that includes any libraries that stem from other Automic versions than the one being used. The only LD_LIBRARY_PATH that you should be using for AE, at least on the UNIX variants I know, is "." - to ensure it only ever pulls the shared objects from the same directory as the binaries in that version being used.

This goes only for AE components. For LD_LIBRARY_PATH to e.g. database libraries, specifying a fully qualified path is usually fine.

Hth,

[Johnny Reed]

Hi Wolfgang,
I have tried several version - 12.2.4, 12.2.6, 12.3.1, and 12.3.2 and none of them fixed the issue.
Johnny

------------------------------
Eastman Chemical Co.
------------------------------

[Frank Muffke]

Hi
I would say thats an env issue - i doubt that all versions you tested have an issue with AIX.

I know that colleagues of mine run on AIX and upgraded their sandbox to 12.3.X lately.

did you doublecheck docu for AIX exceptions - just to be sure?

cheers, Wolfgang

------------------------------
Support Info:
if you are using one of the latest version of UC4 / AWA / One Automation please get in contact with Support to open a ticket.
Otherwise update/upgrade your system and check if the problem still exists.
------------------------------

[kay.koll]

Please file a support incident with the WP Trace Files. 
The engineering is currently working on solution and any additional information is useful.
Regards Kay

------------------------------
Sr. Solution Architect
Broadcom
------------------------------

[Johnny Reed]

I opened a ticket for this a couple of weeks ago. It has been escalated, so I hope to hear something soon.
Johnny

------------------------------
Eastman Chemical Co.
------------------------------

[Johnny Reed]

I have been testing a dev version (12.3.3) that has fixed the core dump issue. So far, I have had no WP core dumps.

------------------------------
Eastman Chemical Co.
------------------------------

[kay.koll]

This is good news, thanks for letting us know that the issue is fixed
Regards
Kay Koll

------------------------------
Sr. Solution Architect
Broadcom
------------------------------