Side by side upgrade cutover strategies and experience.

[Anon Anon]

We have a very large Nimbus environment with nearly a decade and a half of history. Upgrades have been done in place over that time and we're currently on a modified 5.1 environment with a beta of what became the first multi-threaded data engine with partitioning on MSSQL.

 

We are migrating to 8.x and our strategy is to build a new environment and cut over to it. Several strategies have been discussed. Uptime and the ability to work out the bugs before cutover is essential in our environment. Last time we copied our database from the old systems and installed old versions over the database in an isolated network. Then stepped through upgrades to get to the version we wanted, then cross posted production data to prove out and refine the system. Then shut down the old root infrastructure and switched out the domain on the new with secedit and some scripting.

 

Various changes appear to make that potentially more complicated and we've found that there are domain related keys in the new database. We're looking for recommendations and information on potential pitfalls to various strategies.

 

One strategy we're considering is to put both pieces of core infrastructure on the same domain. We believe we can do this by disabling all but the hub and robot probes on the new environment. Shutting the new robots down. Then switching the domain in the various configs, removing security.cfg, and restarting the robot so it syncs with the other hubs security.cfg. We may also have to merge new default ACLs from 8.0 into that security.cfg. We already hard code root probe addresses in various UMP components and most things that don't have it hard coded expect it on the local robot or hub anway. We intend to use the firewall to ensure new and old systems can't talk to the wrong database if something accidentally discovers the wrong data_engine.

 

Another approach being considered is leaving new infrastructure on a new domain, then changing the domain on all down stream hubs and robots in a phased cutover.

 

Does running two sets of root infrastructure in the same security domain work? What are the risks. I recall one unknown was that the install process could discover the old components and want to upgrade them, but I believe we have that mitigated.  What have others done?  Of course this is well outside of what we've found documented.

 

-Ray Ferguson

[NMS_Colorado]

Hi Ray,

 

How did your cutover to 8.x end up progressing?  Which approach did you go with?

 

We are running parallel environments and plan to start a phased cutover to 8.4 in the coming weeks so I'm interested in hearing about your experience.  If you did a phased cutover what approach did you use for repointing robots?

 

Any insights on lessons learned are much appreciated.

 

Thanks,

Les

[Greg Polk]

Hey guys I just did an upgrade from a 4.4 to max 5.7 version straight to 8.4 and its not as hard as it seems. I work for one of the largest hosting provider as a Nimsoft Domain Admin, well the only Nimsoft domain admin, and we had some major requirements like:
1. Existing hubs had to co-exist with new hubs because we could not upgrade or build 175 new servers (To be honest I did not want to) and customers could have no downtime

2. If your like us, you use the API and hopefully you are aware that any hub/robot after 5.7 has a different api so your existing api will need modifications.  (I am just an admin who did upgrade the coding of API I can not tell you zip about it but think it went from soap to rest.)

3. My advantage and the only way I was able to pull it off is have development create in our portal a selection for the api so our portal would send different instructions depending on the version installed and every (pod or you would call a server room) had an a and a b hub for fail-over.    T

 

The script I wrote is very simple and will post it in here but please don't laugh I do not do scripting for a living.  The great thing about the script is most robots never failed over to backup hub because the one being upgraded was completed so fast.   You run this as an admin in a command window

 

set hub=%1   <--- requests hubs name
set comp=%2  <--- request robot name

#---------------------------------------------------------------------------------------------------------------

#section 2 - this section found the nimsoft version we were looking for.  Over 8 years we changed our policies of where we installed items so it had to find it then set the directory to variable nimdir

If exist C:\Nimsoft\hub\hub.cfg set nimdir=c:\nimsoft\
If exist C:\"Program Files (x86)"\Nimsoft\hub\hub.cfg set nimdir=C:\Program Files (x86)\Nimsoft\
If exist C:\"Program Files (x86)"\"Nimsoft Infrastructure"\hub\hub.cfg set nimdir=C:\Program Files (x86)\Nimsoft Infrastructure\
if exist C:\"Nimsoft Infrastructure"\hub\hub.cfg set nimdir=C:\Nimsoft Infrastructure\
echo %nimdir%

#----------------------------------------------------------------------------------------------------------------

net use r: /delete   <--- deleted drive map if it had to be run again but only ever happened when there were two matches to previous section.

net use r: \\wsusdal0502.yourdomain.local\nimsoft_temp\hubupgrade /user:yourdomainr\gpolk  < folder with upgrade files
xcopy "%nimdir%archive\*.*" c:\nimsoftbkup\archive\ /s /v /c /k /o /x /y /i    <---- backedup entire directory before proceeding so a revert to old system was possible and fast. Notice how its only coping the archive first due to not being able to copy in use files.  The files are from a working hub installed manually.
sc stop nimsoft_shovel  <-- private app we wrote for nimsoft
waitfor God /t 8  <--- wait for service to quit before proceeding
sc stop nimbuswatcherservice  <--- stopped hub
waitfor God /t 25   <--- wait for service to quit before proceeding

xcopy "%nimdir%"*.*" c:\nimsoftbkup\archive\ /s /v /c /k /o /x /y /i   

<--- wait for service to quit before proceeding
xcopy  "%nimdir%""*.* /s /v /c /k /o /x /y /i /exclude:r:\exclude.txt  <--- copies n rest of files without archive
copy c:\nimsoftbkup\robot\robot.cfg "%nimdir%robot\robot.cfg" /y  <--Copies back over robot.cfg file
copy c:\nimsoftbkup\hub\hub.cfg "%nimdir%hub\hub.cfg" /y   <--Copies back over hub.cfg file
sc start nimbuswatcherservice   <-starts new nimsoft instance
waitfor God /t 25

#------------------------------------------------------------------------------------------------------------------------------------------

The following just activate the probes:
"%nimdir%bin\pu.exe" -u administrator -p password -I /softlayer/%hub%/%comp%/controller probe_verify controller
"%nimdir%bin\pu.exe" -u administrator -p ipassword -I /softlayer/%hub%/%comp%/controller probe_verify spooler
"%nimdir%bin\pu.exe" -u administrator -p password -I /softlayer/%hub%/%comp%/controller probe_verify hdb
"%nimdir%bin\pu.exe" -u administrator -p password -I /softlayer/%hub%/%comp%/controller probe_verify hub
"%nimdir%bin\pu.exe" -u administrator -p password -I /softlayer/%hub%/%comp%/controller probe_verify distsrv
#------------------------------------------------------------------------------------------------------------------------------------------------------
waitfor God /t8
sc start nimsoft_shovel
sc start slshovel
end

 

I you need someone to help you do it I always looking for side work and would be more than happy to talk about it.