Approach for Application Manager upgrade from 9.1.3 to 9.3.2

[Vinod SC]

Hi All,
   We are planning for the Application Manager upgrade from 9.1.3 to 9.3.2
    I am looking for steps on how to upgrade to 9.3.2:
             1) Is it going to be a direct upgrade to 9.3.2 or is there any intermediate upgrade path.
             2) Currently running Application Manager version 9.1.3 is using Oracle Database 11.2.0.4.0, May I run the AM 9.3.2 upgrade on the same database or does it require an Oracle  database 12c or later versions
             3) In case I have to use Database 12c or later for AM 9.3.2 upgrade, the database has to be upgraded first, then proceed with install or how is it, please suggest.
      Please excuse my ignorance on the understanding of the product, would like to explore more and understand the product better. We have upgraded to 9.3.2 is planned. Would appreciate it if I get all questions addressed. I am pretty new to this product. Reading on th installation guide of AM.

Kind Regards
-Vinod
<gdiv></gdiv>

------------------------------
Vinod Reddy
Ellucian
India
------------------------------

[Richard_Blumlein_8490]

Hi Vinod, 
I am tell you that we upgraded from V9.1.1 to V9.3.1. We were on 11.2.0.4 of the database, and later upgraded to 12.2.0.1 after we were already on V9.3.1
When I did the upgrade I pulled the compatibility docs for the upgrade. Items like database, and Java are always stand out for me. 

With V9.3.1 there was a lot of challenges for me. The new client, and customizing the connections.properties, and client.properties file. I customized these files, and modified the Intro.html file so the users wont try to download the new client.  We packaged up what they need, and had them download it from a Microsoft teams location. 

The other big challenge was the user_keystore files. It took me a while, but I finally understood it, and got it working. It all has to do with Oracle Java 1.8._201 or higher added the exceptions for the algorithims for anon, and null. We hit them while on V9.1.1, but there was no fixed for this other than removing anon, and NULL from your java.security file.  The affects the client, and hos. I purposely kept my host (Linux) at a lower level until after the upgrade, then I introduced the newer Java, and keystores. 

The other important item is the ojdbc jar file. I Was able to use ojdbc6.jar, but changed this to ojdbc8.jar when I went to 12.2. It is good checking the oracle sites for compatibility.  The other item we did was upgrade the Oracle client on our Appman systems.  We were able to use a 12.2 db, and 11g client, but later upgraded the client. 

I just recently solved an issue with some of our BANNER databases being moved to a Oracle Exadata appliance. The BAN* agents would not start because of an encryption incompatibility.  well, I was able to find parameters to fix this.   

Always something working on a product that interconnects into all we do. We do run the RA Banner agents, and that needs the ojdbc jar too in the web classes. 

Since we are at V9.3.1 we know there is a V9.3.2 out. We were not sure what the benefit would be at this time, but it is probably not as bad as going from V9.1.1 to V9.3.?

Good luck, let us know how you make out. 


Rich

[Lian Way Foon]

Hi Richard,

That would be awesome if you could give a step by step upgrade plan especially pre-install, upgrade and post upgrade steps.

What kind of verification task should we perform after the upgrade completed?

[Richard_Blumlein_8490]

Hi Lian, and everyone

I will keep this high level how I like to do things. I know everyone has there own way of performing installation. 
This is how I would do work on the IBM mainframe, AIX, Linux systems over the years. 

I always like to start with pulling down all of the manuals, and documents. I skim, and sometimes deep dive through the new features, changes, items dropped. I even a lot of times open tickets to ask more if something is gray to me. 

Going from V9.1.1 to V9.3.1 was a nice jump with a lot of challenges, I had to really pull that together. But it is not until you actually work on it that you really get to know it. 

I like to pull down the packages, and extract them into my software directory. I keep all kinds of software, and levels. With appman it is then easy to know what I need. For the Banner RA I put the jar on my PC in a directory for later. 

When I perform an install I start with my upgrade instance. I like to shutdown, and backup the master, and remote agent trees. and I have the DBA backup the DB. With V9.3.1 OAE changed to Advanced Queuing, and a lot of learning there.  Before I before the upgrade, 
I like to make sure I know what files we customized, I have it all tracked. This way I can check that after the upgrade.

You already know you upgrade the master first, then remote agents. If you previously stopped the remote agent you can just upgrade the master, and  leave the remotes for later. Just don't start it. I pay attention to make sure it uses the correct Java. If you can login, check the help/about for levels, and check as much as you can like the Automation Engine options to see if new items came up. 

With V9.3.? you will need to use the RunClient.jar, and the Client.jnlp gets removed. The odbc.jar file prompts you now for a location, and it will copy it to a new directory but I am pretty sure I had to copy it to web/classes because it could be used from there. I am using the ojdbc8.jar.  Same for the remote agents I copy it myself to web/classes. There is a note in the Banner manual about an agent doing a call it needs to be there. 

You will find with V9.3.? a learning curve with the user_keystore files. You actually don't need them in place until you use Java 1.8.0_201 or higher. Or on your PC you are using it. If lower that 201 on PC or host you will be ok. I have helped users with this by having them update there PC Java  java.security file. You remove anon, and NULL from the exclusions. This was when we were at V9.1.1 and users were getting AwE socket errors. The host was at Java 1.8 158 I believe. A lot to remember with Java. 

On the host if using Oracle Java change the java.security file setting for /dev/random to /dev/urandom. It fixes a Java awapi timeout. 

After I upgrade my AMUPGD instance we do basic testing, then move on to other components. We have OAE so you need to upgrade that, and Banner RA. Load the jar. Sometimes Banner RA changes a login setting or some feature. there was a few times where I have to change the Banner working files for a new env setting.

Once the upgrade instances is stable I move on to test, and then prod.  Basically it all comes together as we progress the systems. 

Sometimes it is a real learning curve, some feature changed or got tightened up. We used to share OAE agents between the two rest instances that changed. No can do now. Advanced Queuing setup did that. The DBA had to help me. 

You will have to decide how to roll out the new client. No more Client.jnlp. What I did was change the Intro.html page, I customized it. 
The users can't download the client.zip. I have the page say you need to get this from the Automaton team. And we have zip files we send you with the new client files customized, and the user_keystore files packaged into it. 

You use the java keytool command as the user you run the master as. Then you copy the files to the remote agents, and to your PC. You do this for each master you run, we run three.

Be careful with the directory on your PC. The manuals might show ".Appworx", or "Appworx" without the period. It is the one without. With 
With V9.3.1 +  it would be.

C:\Users\<user name>\Appworx\<instance>  

For instance 

C:\Users\rblumlei\Appworx\AMUPGD


Good luck with this.  I hope this helps.

Rich 

[Richard_Blumlein_8490]

Hi Lian, 
I mentioned in my last reply about pulling down the different manuals.  One of docs I really like is pulling down the AM Compatibility. I believe I have also pulled down the Compatibility guide for RA Banner. I like to cross reference the AM to RA, and all of the items mentioned  Like supported OS's, DB's, Java, things like that. 

Over the years when I have done installs I like to also see if there is any known issues with the release you are updating too. I like to see what the new release fixed, and also what known issues there are with the new release. 

When you look at the docs pay attention to what changed from V9.1 up to your new release. Sometimes you will find new features, and even one's that dropped, or changed. 

I mentioned we  controlled the new client software. We customized the connections.properties file, and client.properties file. 
The connections.properties file is where you define your instances, and is the basis for the pull down mention you get when running the 
RunClient.jar file.  Oh, important point. Don't try to use https, you need to use http. This was a concern for us, but then I know there is encryption parameters turned on in the appman env files for the logins.  

One of the features I like about the client.properties file is having debug turned on in the file there is a statement for that. 
And I really like having the ability to override the Java on your PC. I have Oracle Java defined in my file, but can easily point to 
OpenJDK to test. I like to have multiple Java's on my PC, and I test when I user has an issue just by pointing to that version. 

Good luck, Rich

[Vinod SC]

Hi Rich,
    If there is a change in the database server for AM, then what are the places/files we should update new database server details for existing AM.
    Recently, all AM schemas have been migrated from Oracle 11g to 19c database, now AM upgrade planned from 9.1 to 9.3.
    When I start the AM services, it's pointing to Oracle 11g, we should AM files to point to Oracle 19c and then proceed with an upgrade from 9.1.3 to 9.3.2.
   Please advise.
-Vinod<gdiv></gdiv>

------------------------------
Vinod Reddy
Ellucian
India
------------------------------

[Richard_Blumlein_8490]

Hi Vinod, 
As far as I know the awenv.ini, and sosite files have the DB related items. I do know that you will probably need to shutdown, and run a awinstall command. This will prompt you to change any of the definitions you need too including the Appman user, and password. From what I remember it will encrypt this information and update various files. I think it might be the data/Master.properties file. Don't try to update this manually.
Running the awinstall will do all of this for you. 

You will also have to make sure the tnsnames.ora file is correct for the DB you are using.  This is the Oracle client on the Appman master. 
We run a 12.1 client. 

We set various Oracle values in the user that runs Appman in the .bash_profile too. I also set the Java.

Hope this helps.

Rich 


export AW_ENV=test

export ORACLE_SID=am${AW_ENV}
export ORAENV_ASK="NO"
. oraenv
unset ORAENV_ASK
export TWO_TASK=am${AW_ENV}

export JAVA_HOME=/usr/java/default/jre/bin
export PATH=$JAVA_HOME:$PATH

export AW_BASE=/srv/automate/appworx/${AW_ENV}
export FPATH=${AW_BASE}/fun/prod

export AW_HOME=/srv/automate/appworx/${AW_ENV}/ammaster
. $AW_HOME/site/sosite

[Vinod SC]

Hi Rich,
     You are saying when I run upgrade scripts it will ask me for database information?
      Then I can input Oracle 19c information instead of old oracle database info...
       Please correct if my understanding is correct?  if this is the case it will solve my problem.
-Vinod<gdiv></gdiv>

------------------------------
Vinod Reddy
Ellucian
India
------------------------------

[Richard_Blumlein_8490]

Hi Vinod, 
I want to make sure I have this correct. You already migrated the DB from 11g to 19c.
now you want to upgrade Appman from V9.1.1 to V9.3.2?

I am not sure if you are already using 19c with V9.3.? or it's been down, and you need to now upgrade the app?

I know for a fact when you run the cdinst it does an awinstall to recompile everything into the DB.  I have run awinstalls 
by themselves too when we moved instances. 

I would say to make sure on the master system everything knows about the new location for the 19c DB.
You should be able to update the awenv.ini, and sosite files to the new information. 
Then when you run the cdinst it knows how to connect to the DB. Then verify the prompts, change anything you need too.
It will ask you for the oracle user, and password for the instance you are running. It saves it as encrypted in the DB, and in the 
Master.properties file. 

I can only give you the information I ran into, and did. But if you need more clarification open a support ticket to be sure. 
I don't want to send you into the wrong location. 

Thank you, 

Rich 

[Vinod SC]

<gdiv>Hi Rich,
  Here are my answers:
  
You already migrated the DB from 11g to 19c.
    VSC: Yes
now you want to upgrade Appman from V9.1.1 to V9.3.2?
    VSC:  Yes

I am not sure if you are already using 19c with V9.3.?
    VSC: No, appmgr is still pointing to old 11g DB. I thought to update to point to new 19c DB, however, updating awenv.ini is not working.
               So , asking you suggestion where else/files I must update 19c DB details, so that appmgr will start running using 19c. So that I can proceed with upgrade
or it's been down, and you need to now upgrade the app?
      VSC: Now, services will using Oracle 11g DB. I need to upgrade, post-upgrade 9.3.2 should start running using Oracle 19c. 
Please suggest

Vinod
</gdiv>

------------------------------
Vinod Reddy
Ellucian
India
------------------------------

[Richard_Blumlein_8490]

Hi Vinod, 
I get what you are saying. I think maybe it might be time to open a support ticket, and run this by them. 

I know in the past we had situations when we moved the DB from AIX to Linux, but kept it the same release. 
I had to have the tnsnames.ora file point to the new location once I had it down. 
I believe I did not update the awenv.ini file directly. I ran the awinstall while down. With the awinstall you get the menu to change any of the options DB port, sid, service name, password, and user. 

it should connect to the new DB that has the imported Appman DB, and recompile, and update the password. 

I think when we upgraded from 11g to 12g I ran an awinstall to the DB in place. It recompiled to 12.

What you are doing it going to 19c, and I don't believe V9.11 supports that but V9.3.2 will. 

So it sounds like you want to upgrade, and point to the new DB.. Since the cdinst does an awinstall I thought it should work.
Maybe not update the files directly, use the install process, which will do the awinstall. 

But like I said it is always best to ask the people that know best.     

Sorry if I could not be of mote help. I can only state what I have experienced myself. 

Rich

[Richard_Blumlein_8490]

I think you can shutdown, update the awenv.ini, and so site file, 
Then run a awinstall, change any values you need too. 
This will update all of the needed files, and inside the Appman DB.
It needs to be to the DB so this is where the tnsnames file comes in, and having your environment variables set correctly. 

Rich

[root@isift601 site]# more awenv.ini | egrep -i 'db'
DB_SERVICE=
DB_IP=
DB_PORT=
DirectDBConnection=
DB_ID=appworx

[root@isift601 site]# more sosite | egrep -i 'db'
SO_DB_ID=appworx;export SO_DB_ID

[Richard_Blumlein_8490]

During the awinstall you will be prompted for the Oracle DB password. It will save this encrypted into the Master.properties file. 

Rich

[Richard_Blumlein_8490]

Hi Vinod, 
I am tell you that we upgraded from V9.1.1 to V9.3.1. We were on 11.2.0.4 of the database, and later upgraded to 12.2.0.1 after we were already on V9.3.1
When I did the upgrade I pulled the compatibility docs for the upgrade. Items like database, and Java are always stand out for me. 

With V9.3.1 there was a lot of challenges for me. The new client, and customizing the connections.properties, and client.properties file. I customized these files, and modified the Intro.html file so the users wont try to download the new client.  We packaged up what they need, and had them download it from a Microsoft teams location. 

The other big challenge was the user_keystore files. It took me a while, but I finally understood it, and got it working. It all has to do with Oracle Java 1.8._201 or higher added the exceptions for the algorithims for anon, and null. We hit them while on V9.1.1, but there was no fixed for this other than removing anon, and NULL from your java.security file.  The affects the client, and hos. I purposely kept my host (Linux) at a lower level until after the upgrade, then I introduced the newer Java, and keystores. 

The other important item is the ojdbc jar file. I Was able to use ojdbc6.jar, but changed this to ojdbc8.jar when I went to 12.2. It is good checking the oracle sites for compatibility.  The other item we did was upgrade the Oracle client on our Appman systems.  We were able to use a 12.2 db, and 11g client, but later upgraded the client. 

I just recently solved an issue with some of our BANNER databases being moved to a Oracle Exadata appliance. The BAN* agents would not start because of an encryption incompatibility.  well, I was able to find parameters to fix this.   

Always something working on a product that interconnects into all we do. We do run the RA Banner agents, and that needs the ojdbc jar too in the web classes. 

Since we are at V9.3.1 we know there is a V9.3.2 out. We were not sure what the benefit would be at this time, but it is probably not as bad as going from V9.1.1 to V9.3.?

Good luck, let us know how you make out. 


Rich 

[Richard_Blumlein_8490]

Hi everyone, 

Like most things I do stop the application, backup the installation tree, and have the DBA backup the database first. 
I like to perform the Master installation, review/redo customizations, then upgrade my remote agents. 
Since we have RA Banner I usually have to load the upgraded jar for this, and review. 
I always review the AM docs, and RA docs for Banner before. Be sure to review the compatibility guides. 
And be sure to look for known bugs fixed by V9.3.2, and open bugs with it. It might be you need additional patches on top of V9.3.2. 

Yes I believe V9.3.? will work with 11.2.0.4, but double check the compat guides. The V9.3 installations will remove your ojdbc_signed.jar, and 
the client Client.jnlp file.  You will be asked during the installation for the location of your ojdbc jar file. The ojdbc6.jar should be ok for now. 
The install creates a new ojdbc directory, but be sure to copy the driver to web/classes too.2

The comment about the java.security file was to change /dev/random to /dev/urandom. This was the Java awapi timeout you probably saw I 
helped with that issue. 

The AM_Client was a little confusing but I finally got it working.  The format was a little confusing.  The connections.properties file is for your connections, and you see them when you run the RunClient.jar. The client.properties I like having debug on, and I like to point to a specific Java on my PC.      That was nice. 

Good Luck. 

Rich





Hi Richard,   Thanks for your response with a lot of information. I find it very useful.    These are my takeaways from your response:     
1) I can run directly upgrade AM from 9.1.3 to 9.3.2.                       Just unzip 9.3.2 binaries and run cdinst.sh and select upgrade when it prompts and follows on-screen instructions    
2) AM 9.3.2 will run on Oracle DB 11.2.0.4 as well. In other words 11.2.0.4 is supported for AM 9.3.2    
3) Will have to take care of the user Keystore section and editing java.security file.     4) ojdbc6.jar will do the work until I upgrade Oracle DB to 12c        Please do review my key takeaways from your response and correct me if there anything I misunderstood.    Upon your review, I will proceed with the upgrade. Vinod

[Vinod SC]

HI Rich,
     A few days ago I got a new proposal to upgrade AM 9.13 to 9.3.2
     Challenge here is that, Database and JobSub are running on 2 separate servers.
     How I should place Master and Agents here. Master and agents will go into the JobSub server OR it needs to be split to move Master on to JobSub server and agents to a Database server running Oracle 19c.
    Please suggest the best Architecture recommendation for AM master and agents installation acrossDatabase server and JobSub server.
-Vinod

<gdiv></gdiv>

------------------------------
Vinod Reddy
Ellucian
India
------------------------------

[Richard_Blumlein_8490]

Hi Vinod, I don't claim to have all the answers but I will tell you what we have.

We have 3 Appman instances, but all of them work the same.  I will just go over production. 

My prod system is on it's own VM. This is the master. 

We install a Remote agent on our Banner JobSub system, that talked to the master. Separate system for JobSub. 
Once you install the remote agent you define it to the master. We have standards on the user name we use, and file systems we share.
The BANNER agents use the remote agent for job submission. 

We also install a remote agent that is local. This is for one of our department. Same system as the master. It runs as a different user, and 
has it's own directory tree. It also gets defined to the master. 

The database is 12.2, but I do see 18 or 19 down the road. It too is on a different DB system. On the master system you have the Oracle Client installed, and have your tnsnames, and sqlnet.ora files that need to be set up.  Inside of the sosite, and awenv.ini files you point to the DB name, and port, and login information. 

As a side note for our test instances upgd, and test we have agents on a JobSub test system. You just run them as different users, and connect to different masters.  For production I only have one instance.    

When you go to V9.3.? there is a lot of changes with the client, and now using keystores.  It took e a while to get this, but now it is working nicely.  

I hope I didn't repeat too many things you already knew, and this was helpful. 

Thank you

Rich