We are currently seeking help on the autosys issue described below, so that we can resume running our autosys batch jobs without manual intervention.
On April 2018, an AIX 5.2 WPAR was installed on AIX 7.1. Since the hardware virtualization application batch jobs scheduled in autosys are hanging when a new container is invoked to run. The problem is intermittent and jobs resume running when resubmitted manually.
Software VersionsAIX 5.2 Versioned WPAR Runtime 22.214.171.124AIX 7.1 TL4 SP4 on the AIX host.Autosys 4.5Oracle 126.96.36.199.0
In log file SSAIX02:/autosysprd01/autotree/autouser/out/event_demon.*** the following Oracle error was found:
May 24th (Autsys Log) [03:53:48.0979]  -- ORACLE error -- [03:53:48.0979]  ORA-01115: IO error reading block from file 5 (block # 2466) ORA-01110: data file 5: '/oradataprd03/PRD02/data01.dbf' ORA-27091: skgfqio: unable to queue I/O IBM AIX RISC System/6000 Error: 5: I/O error [03:53:48.0980]  Processing OCI function not used (4)
Steps taken to resolve the issue.......
No errors in the Oracle alert log. Database review shows no issues with the database.
DB verify was clean.
DBVERIFY: Release 188.8.131.52.0 - Production on Thu May 24 11:58:54 2018
(c) Copyright 2000 Oracle Corporation. All rights reserved.
DBVERIFY - Verification starting : FILE = /oradataprd03/PRD02/data01.dbf
DBVERIFY - Verification complete
Total Pages Examined : 38400 Total Pages Processed (Data) : 20980 Total Pages Failing (Data) : 0 Total Pages Processed (Index): 0 Total Pages Failing (Index): 0 Total Pages Processed (Other): 38 Total Pages Empty : 17382 Total Pages Marked Corrupt : 0 Total Pages Influx : 0
Disk controller not reporting any errors,
Referenced Oracle Knowledge Article ( Doc ID 559697.1 ) : ORA-01115 ORA-01110 ORA-27091 ORA-27072 Error: 5: I/O error
Increased maxreqs to a value greater than or equal to 8192. (NOTE: Values that fixed the errors: 8192, 16384 or 32768.) The "MAXREQS" value currently set at 4096 was changed to 16384 and the server rebooted to take effect. The Oracle database and the autosys software was restarted as part of the normal reboot procedure.
Moved the datafile to a different filesystem.
Despite the aforementioned troubleshooting the problem still exists (same error message).
All recommendations with regards to this issue are greatly appreciated
Still using Autosys 4.5?
Is there any plan to move to latest versions for AIX, Oracle and Autosys?
As you know Autosys does not perform any direct IO operation to the database but uses an Oracle client with OCI calls.
These Oracle errors are returned to Autosys from the Oracle dataserver through the Oracle client
As this problem is related to IO errors, I would suggest to check this Oracle document to see if it helps
ORACLE-BASE - Direct and Asynchronous I/O
And on AIX with Sybase we had this requirement, not sure if it might help with Oracle:
When installing an Autosys 4.51 Agent on the AIX platform, 'Asynchronous IO' is an essential prerequisite.You will need to configure "AIO" and Not POSIX AIO with smit.
Please note AutoSys does not require AIO. AIO is required by Sybase.
Autosys 4.5 and Oracle 8? Did I read that properly?
Honestly, please upgrade. Your OS and your DBA will thank you.
Talk to the DBA IO errors have to deal with the database file system and the schema setup.
Nothing in this message is intended to constitute an electronic signature unless a specific statement to the contrary is included in this message.
Confidentiality Note: This message is intended only for the person or entity to which it is addressed. It may contain confidential and/or privileged material. Any review, transmission, dissemination or other use, or taking of any action in reliance upon this message by persons or entities other than the intended recipient is prohibited and may be unlawful. If you received this message in error, please contact the sender and delete it from your computer.
As much as I would like to there is no plan to upgrade AIX, Oracle and/or Autosys.
Need to maintain the application as is until it can be replaced.
AIO is enabled
$ lsdev | grep aio0aio0 Available Asynchronous I/O (Legacy)
Thanks for the suggestions. Will follow up on them.
I did some search in Google and I found the following community link
The problem has been resolved with the Oracle patch (metalink 390656.1)
Maybe this can help.
Thanks and Best regards,
Please spare a moment to update this thread with the status. Were you able to find the cause of the error and address?