ITMS Upgrade failing

[Martin Sapsed]

Hi all,
I'm trying to upgrade a server from 8.5 RU2 but getting nowhere. First problem was that I didn't see anything about the requirement for a version of SIM which spoke https so I had SIM failures until I found out about that and manually installed a newer version.
Having done that and got SIM working I then tried upgrading to RU4 as I believe that's the path from RU2 (even though SIM offered me 8.6?)
While attempting the Platform Core component I get error boxes (twice) saying
"Failed to configure NS.
Failed to run managed install."
before the upgrade fails and exist. I previously did the same upgrade on a dev box without issue so I'm a bit stumped as to why this is going wrong.
Anyone seen this or able to offer advice? (Platform is 2012 R2)
Thanks
Martin

[Igor Perevozchikov]

Hi MrSoapsud!

1. Yes, if SIM version is lower than 8.5 RU4 version (8.5.5711) then it can't download from https://solutionsam.com so need to manually update it

• https://knowledge.broadcom.com/external/article/199465/sim-is-no-longer-able-to-download-files.html

2. SIM always offers most latest released ITMS version to upgrade, so you saw offered 8.6 version although you have 8.5 RU2 and first need to update it to 8.5 RU4 then SIM will allows to upgrade to 8.6 release.
About supported upgrade paths to 8.5 RU4 or to 8.6 releases:

• https://techdocs.broadcom.com/us/en/symantec-security-software/endpoint-security-and-management/it-management-suite/ITMS/Getting-Started/Upgrading_IT_Management_Suite_11/about-upgrade-paths-of-it-management-suite-v71329564-d783e6171.html

3. About failed Upgrade (I don't see such problem on attempt to upgrade 8.5 RU2 to 8.5 RU4)
- Is there any useful information shows Altiris Log Viewer from gathered logs in C:\ProgramData\Symantec\SMP\Logs?
- Does "Application" Events logs have errors regarding installations?

- During manual upgrade of SIM, there was same installdir used like it was previously installed?


Best Regards,
IP.

[Martin Sapsed]

Hi IP,
1. Did that eventually so at least SIM is now working. Yes I installed to the same location.
2. Understood
3. Diagnostics and Common install OK, the problem is when it comes to do the Core package.
Event viewer suggests it installs ok but then decides there's a 1603 error.
Altiris Log Viewer suggests "Restart MSI service and try again." but it's already installed 2 MSIs?
It also subsequently states "Cannot get IIS service running. Uninstall cannot continue." but that would appear to be during the Rollback.
I also get 2 Application Errors in ReflectionProxy.exe and then an "error reading from the pipe" before SIM exits.
I've changed the firewall to only allow access from 127.0.0.1 in order to isolate the server from its clients, just in case that's an issue?
Regards
Martin

[Igor Perevozchikov]

MrSoapsud: 
Diagnostics and Common install OK, the problem is when it comes to do the Core package.
Event viewer suggests it installs ok but then decides there's a 1603 error.
Altiris Log Viewer suggests "Restart MSI service and try again." but it's already installed 2 MSIs?
It also subsequently states "Cannot get IIS service running. Uninstall cannot continue." but that would appear to be during the Rollback.
I also get 2 Application Errors in ReflectionProxy.exe and then an "error reading from the pipe" before SIM exits.
I've changed the firewall to only allow access from 127.0.0.1 in order to isolate the server from its clients, just in case that's an issue?

IP: 
About msi installation:
According msi exec 1603, there can be different reasons on this problem.
Cause. You may receive this error message if any one of the following conditions is true: Windows Installer is attempting to install an app that is already installed on your PC. The folder that you are trying to install the Windows Installer package to is encrypted.

Did you try to re open back SIM and try to repair/reinstall 8.5 RU4 packages again?

About firewall:
Well, probably if you enabled firewall and allowed only localhost 127.0.0.1 but in IIS your Default Web Site has binding on exact IPv4 address:80 then you maybe need to specify 127.0.0.1 yourNShostname YourNSFqdn in C:\Windows\system32\drivers\etc\hosts file and also set 127.0.0.1:80 in IIS binding of Defaut Web Site. If everything will be OK, then after upgrade return back previous original settings of NS server in IIS and hosts file.
Also need to consider firewall rules on NS to make sure that remote SQL Server is able to communicate with NS.

About blocking Agents communication:
Why don't you use "Blockout" communication settings in "Targeted Agent Settings" policies instead of firewall enabling on NS itself?
Also NS Activities are paused during upgrade of ITMS while msi packages are installing and then configuration phase runs. Actually you can manually force to disable NS activities from NS Settings page (but just do not forget to enable it back once all required ITMS upgrades will be completed, well even if you will forget to enable back NS activities, there will be a reminder about this in SMP Console)

Best regards,
IP.

[Martin Sapsed]

Hi again,
I've got the opportunity to look at this again. One the install fails I get
"The selected Product Listing file does not contain installed products definitions. Symantec Installation Manager functionality is limited."
I can't repair the individual MSI as "Platform Core" isn't in the list.
Full repair gives an Unhandled Exception.
I use the firewall to make sure things don't start talking immediately after the upgrade while I work out whether everything still works. The Agent blockout looks fiddled especially with 10 agent types!
I tried upgrading to RU3 this time in case it was any simpler...
I did have the Pointfix to DS Ru2 installed but removed it before I tried the upgrade.
Any more suggestions?
Thanks
Martin

[Martin Sapsed]

Just looking again and the Event viewer suggests that Platform Core installs correctly so the "Failed to configure NS" bit must be some post-install routine?
Thanks
Martin

[Igor Perevozchikov]

Hi MrSoapsud

First, it will be better to understand what was done step by step on this environment why we got this result now.

SIM version? SIM really uses https://solutionsam URL to get latest pl?
Database version is pointing to correct SQL Server Instance where DB is running? In opened SIM, open "Configure Settings" -> DB -> make sure that this is exact SQL and DB(version)
Database version is exact current RU version or isn't restored back to lower version to perform the update to next RU or 8.6? (SELECT * FROM DBSchema) --- check last record Id
Altiris Log Viewer shows something useful to understand?


--// You don't need to firewall enable etc to avoid NS activities with agents, because when NS is upgrading, agents anyways can't communicate with NS and send their data because NS is in paused activities while it is in upgrading process

[Martin Sapsed]

Hi IP,
I'll try and fill you in with some more details. This is a system that's been working happily for many months with 3-4000 clients which runs 8.5 RU2 with the DS pointfix.
I've been removing the pointfix before attempting the upgrade - do I need to?
I've been upgrading the WADK before attempting the upgrade - do I need to?
I have to update MSADO and add an IIS component in the pre-reqs for the upgrade - expected?
I had a similar dev server which I upgraded before Christmas without any problem.
Initially I had a problem with the SIM because it was looking for http: but after your advice I downloaded the new installer and SIM looks fine to start with. Version is 8.6.1045. I'm offered upgrades to 8.5 RU3, RU4 and 8.6 which, as you also said, won't work because I'm not on RU4 yet.
Last row of the DB Schema: 8 8.5.4249.0 0 2019-06-18 11:17:34.430.
The DB is as it was with 8.5 RU2 but with attempts at an upgrade being unsuccessful.
With the firewall I'm just trying to prevent clients getting ahead in case I need to roll back the upgrade (assuming I ever actually do it successfully)! I also have the package servers down. Should I just stop all the agents from upgrading? With the agent blockouts, do I need to specify 24 hours for several days to buy me time?
I've given up for today and in the morning I'll rewind the server to a snapshot (hurrah for VMs!) and try again with whatever advice you can give.
Thanks
Martin

[Igor Perevozchikov]

Hi Martin!

MrSoapsud: I've been removing the pointfix before attempting the upgrade - do I need to?
IP: There is no need to manually uninstall previously installed Point Fix or cumulative point fix if you are going to upgrade own ITMS to higher version.

MrSoapsud: I've been upgrading the WADK before attempting the upgrade - do I need to?
IP: No need to update WADK before ITMS upgrade, better to update it after ITMS upgrade and then create/re-create new WinPE 

MrSoapsud: I have to update MSADO and add an IIS component in the pre-reqs for the upgrade - expected?
IP: Yes, this is required by Install Readiness Check section so this should be done before ITMS upgrade starting.

MrSoapsud: Last row of the DB Schema: 8 8.5.4249.0 0 2019-06-18 11:17:34.430.
IP: Well, this is OK then that DBSchema version is 8.5.4249.0 (exactly 8.5 RU2) because sometimes user can forget to restore DB to previous version while his ITMS is higher version in this case SIM will show similar error messages in UI  "The selected Product Listing file does not contain installed products definitions. Symantec Installation Manager functionality is limited."

MrSoapsud: With the firewall I'm just trying to prevent clients getting ahead in case I need to roll back the upgrade (assuming I ever actually do it successfully)! I also have the package servers down.
IP: I would avoid such method with firewall usage, because as previously mentioned, when agents aren't in block-out period, they still will try to check available tasks via CTA, get policies, send basic inventory, and all these events amount on each client will grow and grow so suddenly when firewall will be disabled, large amount of nse will come from all agents immediately to NS.

MrSoapsud: Should I just stop all the agents from upgrading?
IP: After ITMS upgrade, all SMA, Site Server and other solution agents upgrade rollout policies are disabled by default, therefore agent will not be automatically upgraded to a newer version until user will enable required Agent upgrade rollout policies.

MrSoapsud:  With the agent blockouts, do I need to specify 24 hours for several days to buy me time?
IP: Sure, you can choose on which exact days and what time agents should have block-out period enabled

Some information from documentation about blockout period

Blockout periods

Lets you specify the blockout periods that you want to use. You can specify any number of blockout periods. If a blockout prevents a software delivery package download, the package download starts immediately when the blockout expires, according to the download options you selected: Download The package server and Symantec Management Agent do not download any software delivery packages. However, the Symantec Management Agent still sends events and gets Symantec Management Agent Settings policy requests from Notification Server. Events and Symantec Management Agent Settings policy requests are typically small amounts of information and have minimal effect on the network traffic. However, packages can be large and can affect the network load. This setting can help minimize the effect of package servers and Symantec Management Agents on the network during business hours. Total There is no communication between the package server or Symantec Management Agent and Notification Server during the specified time period. All events from the Symantec Management Agent are queued (on the Agent) and are sent after the blockout.

Best regards,
IP.

[Martin Sapsed]

Hi Igor,
So, I rolled back to the snapshot. I didn't bother with the pointfix or the WADK, upgraded the SIM, installed the pre-reqs, left the firewall alone, set blockouts, and got exactly the same result as every previous attempt - "Failed to configure NS" after installing Core, and then the ReflectionProxy crashes, and then SIM is broken.
Any more ideas?
Thanks
Martin

[Igor Perevozchikov]

Could you please open SIM -> click "Settings" -> Create support package and send it to me via private message?
Or zip all logs from C:\ProgramData\Symantec\SMP\Logs\ and send them to me?
Do you have any proxy server specified in SIM settings?

[twister12]

Hi,

check this
https://knowledge.broadcom.com/external/article/169455/failed-unable-to-run-proxy-for-reflectio.html

we had a similar issue. In our case "veeamagent.exe" or "HPInsightAgent.exe" i did not remember which process exactly causes the issue.
Killing this process before upgrade resolves the issue.

Best regards

[Martin Sapsed]

Hi,
@twister12 You might be onto something in that I can't start ReflectionProxy without it crashing. SIM has the proxy setting of autodetect.
The only references to net.pipe in Process Explorer are these. None use the same ID and the Microsoft ones sound important?
Thanks
Martin
​

[JOE.PS1]

If you've not examined the logs under C:\ProgramData\Symantec\SMP\Logs\Install[date], those normally provide you more detail on the cause of failure than the UI. 

I went through a similar upgrade process from 8.5 RU3 +PF rollup -> RU4 -> 8.6 and it went smooth.  I've never tried using firewall rules to block agents.  Our agents point to an alias of the server and I normally route that alias to a dead IP during the upgrade to avoid the traffic influx during the upgrade.  I then override that alias resolution using the HOSTS file on the NS and individual site servers to make sure all is working well before opening up the flood gates by restoring the DNS alias to the correct address.  The other thing that has caused upgrade failures in the past for me is our AV on the server.  So now, I shut down the AV on the NS before performing the upgrade.

[Martin Sapsed]

Thanks Joe. Igor and I have looked at the logs and I've tried all sorts of things. There's something really odd going on. One time the log referred to the wrong location for SIM! It doesn't help that when the install fails, it rolls back and removes a load of stuff including most of the .config files. The problem is with the 3rd component - altiris_ns_8_5_ru4_x64.msi. One time I tried installing manually and it failed but I then tried again and it worked, so I followed up with a repair which went quite well until it realised it had hardly any configuration! It's like it's finding a setting that it doesn't like which is confusing it. I've tried removing the PF and not removing it. I've stopped using the firewall and gone with Igor's recommendation of using Blockout.
I'm going to have one more try using all the tricks together - no AV and a few other things before I give up again for now. Perhaps I'll light a candle as well, just in case!!
Thanks
Martin​