Applications Manager v9.3.5 Oracle database 19.16 configuration using SCAN RAC help?

[Richard_Blumlein_8490]

Hi everyone,

We are currently runnng V9.3.5 with a Oracle 12.2 database. The DBAs want to migrate this to a 19c
database running Oracle RAC. With Oracle RAC from what I understand there is different modes you
can run it in active/active load balanced, or active/passive. The DBAs are pushing this becaue
of the asbility to patch the databases without having to take things down. It would allow the
applications to be kept up when they patch.

I opened a support case on this. The support person has been very helpful but I guess I would
like to get the opinion of people in the real world if they are doing this. The support person
told me AM supports RAC in a active/passive mode but not active/active.


I saw this in action when we tried to migrate the database from 12c to 19c on the SCAN RAC system.
The awinstall ran nicely. I was able to start, and stop the instance. When I logged in I tried
to start the agents. They would not start. I was told by the support person this is one of the
symptoms of using RAC in a load balanced environment.

We found this interesting Broadcom AM article that matches our release exactly.

https://techdocs.broadcom.com/us/en/ca-enterprise-software/intelligent-automation/applications-manager/
9-3-5/Administration_Guide/Applications_Manager_Security/Defining_Database_Logins/Configuring_Oracle_RAC.html

I asked support about it. I was told this is more related to normal RAC and not SCAN RAC. It was still
helpful to the DBAs.


The DBA's changed the tnsnames for the new environment a 2 node SCAN RAC system. I tried to start
the Appman instance which hung on the RMI servers.

The DBA's asked me to update the web/classes/Options.properties file as per the tech doc I mentioned.

By updating the Options.properties I was then able to start the Appman instance. I was also able
to start, and stop the agents. This was great.


I then tried to connection check the appworx user datbase logins we had. There is the one you cannot
change type ORACLE but we also have a few others we use like appworx@<some instance> also type ORACLE.
The connection checks were hanging. Then I would have to kill the Java GUI process.


I tried to run the TEST_JOB that comes with AM. It kept failing on a launch error connecting to the
database. It was using the default appworx user we have type ORACLE. I then created a new login
appworx@<instance>_rac with type ORACLE_RAC. It uses the tnsnames connection type information for
RAC. It connection checked great. I changed the test job to use that login. It still got the
launch error.


I just tried to run a awinstall which failed.

ERROR:
ORA-12154: TNS:could not resolve the connect identifier specified

SP2-0306: Invalid option.
Usage: CONN[ECT] [{logon|/|proxy} [AS {SYSDBA|SYSOPER|SYSASM|SYSBACKUP|SYSDG|SYSKM|SYSRAC}] [edition=value]]
where <logon> ::= <username>[/<password>][@<connect_identifier>]
<proxy> ::= <proxyuser>[<username>][/<password>][@<connect_identifier>]
SP2-0306: Invalid option.
Usage: CONN[ECT] [{logon|/|proxy} [AS {SYSDBA|SYSOPER|SYSASM|SYSBACKUP|SYSDG|SYSKM|SYSRAC}] [edition=value]]
where <logon> ::= <username>[/<password>][@<connect_identifier>]
<proxy> ::= <proxyuser>[<username>][/<password>][@<connect_identifier>]
SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus



The problem I have with modifying the Options.properties file is during the upgrades it usually
gets replaced. Running the cdinstall then calls awinstall which will probably fail when it tries t
o start components without the RAC change in place.


Seeing how my awinstall is failing I am at a loss for what to try next? The original suggestion
from support might very well be the one we need to fall back too. Running Appman in a single
database mode without RAC.


Thank you everyone, I appreciate the help.


Rich

[J Bruno]

The way we got around it is we had to create a virtual server that was dedicated to one RAC node only. In our awenv.ini file we have DB_SERVICE=APPMAN_ONENODE.ORARAC3.

[J Bruno]

I meant to say "virtual service" not "virtual server".

[Richard_Blumlein_8490]

Thank you J Bruno for chiming in on my question. I appreciate it.  Have a very nice holiday.  Rich

[Pat Richards]

the solution we use is to configure/modify the database service so that it preferentially connects to one instance rather than both. Your DBA's should be know how to do this, but I'll include the srvctl syntax for reference:

srvctl modify service -db <DatabaseName> -service <ServiceName> -modifyconfig -preferred "<Instance1>" -available "<Instance2>"

the syntax above would mean any connections to ServiceName (<instance>_rac in your example) would connect to Instance1 of your database.  If Instance1 were not available (e.g. its crashed) then any new connections would route to Instance2

It is worth noting that if/when you need to switch the services between instances (i.e. for patching) ApplicationsManager will need to be restarted after the service switch is done to ensure it's connected to the alternate instance - switching services does not transfer existing connections from one instance to the other.

hope this helps 

------------------------------
Cheers
Pat
------------------------------

[Richard_Blumlein_8490]

Thank you Pat for the information.  I appreciate it.  Have a nice holiday time. Rich

[Richard_Blumlein_8490]

Hi everyone,

This might be more J Bruno, and Pat Richards who replied to this question. I appreciate the advice.
I did find multiple Broadcom articles on not using Oracle RAC in active/active mode. This is the one
kb support gave me.

https://knowledge.broadcom.com/external/article?articleId=88170

I am not the DBA but they tell me we are using Oracle SCAN RAC where you would normally point
to the DNS entry for the SCAN IP. In this case we had to point to one of the nodes directly
to get around this.

I want to point out this is all part of the Oracle Exadata appliance we are using. It has 2 nodes.
Now the problem with this is when they patch there is no failover, and we need to take stuff down.
The DBAs tell me this is not as flexible as when the databases were on Oracle Linux VMs and we
could just do one DB at a time. Now they take down the whole node so all test DBA failover or
need to be shut.

Hope I dont have this wrong. I mentioned about the virtual service. The DBAs told me when we were on Exadata Gen1
that was the practice now with Gen2 things changed a bit. Either way they agree with the need for us to point to
just one node. The article was informative.

I was wondering about your configuration for running Oracle RAC if you are using an appliance or some other setup?

I want to give a big thank you to the Application Manager support person Phearith Mao who helped us understand this.

Thank you for the help.  

Rich

[Pat Richards]

Hi Rich

your DBA's are right; with exadata/SCAN and the appropriate database service config your connection should only need to point to the DNS entry of the SCAN IP as the listeners and services sitting behind it will control which instances the connection is routed to

did you try setting the database service as per my previous post ? 

Worth pointing out that if you've changed the tnsnames.ora of options.properties files as per the solution documented in the Administration Guide, these will need to be removed (otherwise they will be used in preference)









------------------------------
Cheers
Pat
------------------------------

[Richard_Blumlein_8490]

Hi Pat

I asked one of our DBA's about your reply. I was told that would work with Exadata Gen1. With the
Exadata Gen2 upgrade things changed. The DBA basically told me what we are doing with the Appman
database DNS entry pointing to one of the nodes is how we need to do it. I don't know enough to
question too much. We use this format for the DNS "db<database name?.<dns suffix. That points to
one of the nodes on the Exadata. Doing this worked, and allowed me to run the awinstall. When
I tried to use the Oracle SCAN RAC IP the awinstall completed but I could not start agents. I
could not run jobs, they all got launch errors. That is when I opened the case about it.

The problems is if they need to take down the Exadata test nodes for maintenance. There is no
failover. The DBA's perform quarterly patching. Without the active/active I expect to be stopping
all our test Appmans at the same time for test, and the prod instance when they do the prod nodes.


I appreciate the help Pat. Thank you

Rich

[Pat Richards]

Hi Rich

all i can say is that with our X8M exadata platform we are able to set up the database connectivity to make use of the SCAN address and allow the listeners and services to manage which instance connections are routed to.

this setup allows us to ensure connections go to just one instance of the database at a time: in normal operation connections will route to instance1, but if instance1 is not available (failure, maintenance etc) connections will automatically route to instance2



------------------------------
Cheers
Pat
------------------------------