SEP14.3 RU2 Issues

[MabundaG]

Hi, so I have upgraded the SEPM from SEP14.3 MP1 which was working fine to SEP14.3 RU2. The upgrade went through successfully. However when I log in, the Home page looks fine for 4 seconds then a Query Failed message pops up. Above the Query failed message are the following tabs "Summary, Logs, Command Status and Notifications" (This looks like the monitors tab). Is there any one else who have seen, or experienced these issues? See screenshots below:

After signing in. 

4 seconds later

After clicking "Refresh"




------------------------------
MabundaG
------------------------------

[John Owens]

I have not seen this but you would want to look at your Apache Reporting.log for the actual error.

------------------------------
John Owens
Strategic Support Engineer | Symantec Enterprise Division (SED)
Symantec
United States
------------------------------

[Cesar_Americo]

I did upgrade my DEV infra a week ago and not faced this issue or not facing. Just realized so far auto upgrade not working.

I have plans to update prod today

[Robert Christiansen]

Just tried an upgrade to 14.3 RU2 on our SEPM 14.3 RU1 MP1.  Getting blank pages on 'Home', 'Monitors', and 'Reports'.  Doing a refresh brings up the 'File or Directory Was Not Found', same as the screen shot from the Original Poster.

Apache Reporting.log is showing these type of lines:

2021-06-04 15:41:51 INFO:Login start
2021-06-04 15:41:52 ERROR:<br>QUERY: EXEC SEM_GET_GUI_PARAMETERS<br>EXCEPTION: com_exception: <b>Source:</b> Microsoft OLE DB Provider for ODBC Drivers<br/><b>Description:</b> [Microsoft][SQL Server Native Client 11.0][SQL Server]Could not find stored procedure 'SEM_GET_GUI_PARAMETERS'. in C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\ado.php:587
Stack trace:
#0 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\ado.php(587): com->Execute('EXEC SEM_GET_GU...', -1)
#1 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\ado.php(343): ado_doPreparedStatement('EXEC SEM_GET_GU...', Object(com))
#2 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\connectdb.php(234): ado_query('EXEC SEM_GET_GU...', Object(com), false)
#3 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Resources\globalvars.php(633): sav_query('EXEC SEM_GET_GU...', Object(com))
#4 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Resources\globalvars.php(664): recreateGUI_Cache(Array, Object(com), 'C:\\Program File...')
#5 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Inetpub\Reporting\System\response_events.php(26): require_once('C:\\Program File...')
#6 {main}Error message: <b>Source:</b> Microsoft OLE DB Provider for ODBC Drivers<br/><b>Description:</b> [Microsoft][SQL Server Native Client 11.0][SQL Server]Could not find stored procedure 'SEM_GET_GUI_PARAMETERS'.<br>Error code: -2147352567<br>File and line: C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\ado.php(587)<br>Trace: #0 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\ado.php(587): com->Execute('EXEC SEM_GET_GU...', -1)
#1 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\ado.php(343): ado_doPreparedStatement('EXEC SEM_GET_GU...', Object(com))
#2 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\connectdb.php(234): ado_query('EXEC SEM_GET_GU...', Object(com), false)
#3 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Resources\globalvars.php(633): sav_query('EXEC SEM_GET_GU...', Object(com))
#4 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Resources\globalvars.php(664): recreateGUI_Cache(Array, Object(com), 'C:\\Program File...')
#5 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Inetpub\Reporting\System\response_events.php(26): require_once('C:\\Program File...')
#6 {main}<br>
2021-06-04 15:41:58 ERROR:<br>QUERY: EXEC SEM_GET_GUI_PARAMETERS<br>EXCEPTION: com_exception: <b>Source:</b> Microsoft OLE DB Provider for ODBC Drivers<br/><b>Description:</b> [Microsoft][SQL Server Native Client 11.0][SQL Server]Could not find stored procedure 'SEM_GET_GUI_PARAMETERS'. in C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\ado.php:587


Opening case 32726727 now with Reporting.log file


------------------------------
Sr. Systems Administrator
ICU Medical
------------------------------

[buttadee]

Interesting. I upgraded my SEP Management console from 14.2 RU2 MP1 to 14.3 RU2 three days ago, and got the message that it succeeded, and was able to log into the Manager console. However, I get "Query Failed" messages when I click on the details for Endpoint Status in the Home tab, and when clicking the Monitors tab. (The Endpoint Status square does not show any data like it should either, and Reports didn't work.) Hitting refresh brings up the 'File or Directory Was Not Found', same as the screen shot from the Original Poster.

In my Apache reporting.log, there were several Microsoft OLE DB Provider for ODBC Drivers errors such as
EXCEPTION: com_exception: <b>Source:</b> Microsoft OLE DB Provider for ODBC Drivers<br/><b>Description:</b> [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]Invalid column name 'URL_ONOFF'

or "Invalid column name 'PRODUCTVERSIONFROM' "

Then I noticed later that the SEPM was downloading content updates, but failing to install any of them. (Symantec Endpoint Protection Manager Event log in Windows showed this.) And my clients were not getting any content/definition updates. By the next day  my SEPM server started to run low on disk space because of the content not getting handled and cleaned up properly, I think.

I had opened a case (# 32721508) the day of the upgrade, but it was useless because there was no troubleshooting/help/line of questioning in the ticket, only ill-timed attempts to reach me by phone (not their fault!) and never answering my pleas to contact me a different way instead to arrange a WebEx. I wound up reverting back to 14.2 RU2 MP1 via SQL database restore and snapshot reversion of my SEPM server late yesterday, right before I was finally connected by phone with support. So I learned nothing, and may  run into the same exact issue when I next attempt the upgrade. I was never sure if they even understood the problem I was having.

Possible contributing factors in my case:
1. I think I may have only stopped the main SEPM service before performing the upgrade for some reason, not the other two SEPM API and Web services. That's on me, and I will pay close attention next time and be sure to stop all three. This is a single SEPM server environment, with a remote SQL database. There is no Dev server.
2. During the "Upgrading database schema" step in the upgrade wizard (when only 1% done), I realized the remote Microsoft SQL server ran out of space on its data partition due to this upgrade. I quickly extended the disk on the fly by 10 GB and the upgrade proceeded and even gave a success message at the end. But did that temporary space crunch break something? If so, the success message is very misleading!



------------------------------
Whitehead Institute for Biomedical Research
------------------------------

[LUK JASON]

I have similar problem, case is raised to Broadcom

[MabundaG]

Hi Jason, thanks for the information. I have raised a case with Broadcom the same day I posted here. They are still investigating. My issue is still not resolved.

------------------------------
MabundaG
------------------------------

[LUK JASON]

Besides this issue, I encounter unexpected error to upgrade Windows 2019 SEPM, all other Windows 2012 R2 SEPM don't have such problem. Remote session with Broadcom is scheduled on next Monday.

[[e-mail]]

We opened an incident with Broadcom for this and resolved the issue.

We had the same issue with our test environment. For whatever reason the SQL Database collation was incorrect, though it was working just fine with with the 14.3 RU1-MP1. The database collation needs to be "SQL_Latin1_General_CP1_CI_AS" prior to upgrading to SEPM 14.3 RU2 or you will see the same issue. 

I have to do a recovery to SEPM 14.3 RU1-MP1, verified that the Database collation was correct (it's inherited from the SQL Instance) and then do the upgrade to SEPM 14.3 RU2. 




------------------------------
Tom Napier
Lead Analyst
End-user Technology Group
CBC/Radio-Canada
------------------------------

[LUK JASON]

SQL_Latin1_General_CP1_CI_AS is already used but same problem

[Torb]

We are experiencing the same on this version. Tried to roll back snapshot and db backup and the issue persistent on older SEP version.

------------------------------
Syscom AS
------------------------------

[LUK JASON]

agreed with Broadcom support to have remote session today, no one contact me. Poor support

[MabundaG]

As for me, the issue has been logged with support for more than a week now. The past Friday, they said I must give them some time. I am still awaiting for them.

------------------------------
MabundaG
------------------------------

[LUK JASON]

just have a remote session with Broadcom, they ask me to:
Change this
CREATE TABLE #TEMP_DOMAINS (DOMAIN_ID CHAR(32) )
To this:
CREATE TABLE #TEMP_DOMAINS (DOMAIN_ID CHAR(32) COLLATE DATABASE_DEFAULT)

In these 4 stored procedures
SEM_HOMEPAGE_DISABLED_COUNTS
SEM_HOMEPAGE_SONAR_REDANDYELLOW
SEM_HOMEPAGE_TOTAL_TECH
SEM_HOMEPAGE_UNMANAGED_MACHINES

[MabundaG]

Hi Jason, did it resolve your issue?

------------------------------
MabundaG
------------------------------

[LUK JASON]

my DBA require Broadcom support to provide SQL script to modify stored procedures instead of modify them manually, I already send this request to support

[Kristiansen]

Any update from Broadcom on the script?

------------------------------
Senior IT Security Engineer
Syscom AS
------------------------------

[LUK JASON]

they ask me to read script in SQL Maintenance script for the Symantec Management Platform database (broadcom.com) but they are related to SQL defragmentation, not related to any SQL script to modify 4 SEM_HOMEPAGE stored procedure

[Wintel Infraestructuras]

Same problem here.

Our database is using "Modern_Spanish_CI_AS" after the update...

[Wintel Infraestructuras]

Same issue here... any real solution?

[John Owens]

If you have a case opened please mention internal defect ESCRT-7187.  Could be related. It is being investigated now.

------------------------------
John Owens
Strategic Support Engineer | Symantec Enterprise Division (SED)
Symantec
United States
------------------------------

[[e-mail]]

Hi John,

Can you keep us updated on this issue? Anything you can share for what's been found so far for ESCRT-7187? We had the same failure with the upgrade in our test environment and had to do a full restore verify that the collation was correct and then do the upgrade again. I am hesitant to do the upgrade in production if the Collation is getting randomly changed from the standard.



------------------------------
Tom Napier
Lead Analyst
End-user Technology Group
CBC/Radio-Canada
------------------------------

[Torb]

Hi John,

Can you please also create a support KB. It's obvious now that this issue affects multiple customers. We've seen it in two separate environments and have stopped all 14.3 RU2 upgrades for our customers.

------------------------------
Syscom AS
------------------------------

[John Owens]

The error we are focusing on currently is - 

[Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Cannot resolve the collation conflict between "SQL_Latin1_General_CP1_CI_AS" and "Latin1_General_CI_AS" in the equal to operation. in E:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\ado.php:587

The symptom is SEPM homepage now loading.

It appears this thread has multiple issues being reported so I wanted to be clear regarding the issue we are investigating in ESCRT-7187

As we have more information a KB will be created.

------------------------------
John Owens
Strategic Support Engineer | Symantec Enterprise Division (SED)
Symantec
United States
------------------------------

[LUK JASON]

Problem suddenly is fixed by itself after I add a new SEPM

[ace1]

Once you have introduced a new DBMS like MS SQL, collation discrepancy is a thing that many new and beginners SQL developers get tripped up.
When you migrate from the former Sybase integrated database to new MS SQL you should use a particular locale to be used for the whole database and software business layer, it shouldn't be installation dependent.
Collation includes sorting and *comparison* rules, if you don't enforce and instead intermix different collations, you are ready for a software minefield. Because the DBMS is used only as support for the software layer, you should standardize on a collation expected by such software, and not by the "language" of the system (i..e. standardize on common UTF8 encoded standard rules collation).
thanks

------------------------------
Ace It
------------------------------

[John Owens]

https://knowledge.broadcom.com/external/article/217035/

------------------------------
John Owens
Strategic Support Engineer | Symantec Enterprise Division (SED)
Symantec
United States
------------------------------

[Rojopipe]

In Symantec days we could subscribe to a whitepaper to receive email updates, have you contemplated doing it again for critical situations like this ...?

 

******

AUH

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

[John Owens]

https://knowledge.broadcom.com/external/article/217035/ has been updated with a tool to resolve Collation issues.  The tool will only work on 14.3 RU2 SEPMs.

------------------------------
John Owens
Strategic Support Engineer | Symantec Enterprise Division (SED)
Symantec
United States
------------------------------

[Mauricio Silva]

Hello. I ran the .bat as per the documentation and it didn't work. Any different things to be done?

[John Owens]

Are you seeing the same error in upgrade log or something new? 

[Mauricio Silva]

what would the log be?
after the update the error appeared, I arrived on this forum and then performed the procedure informed in the article.

I'm thinking about backing up the database and snapshot, but I was wondering if there's anything to do before that

[John Owens]

Need to review the upgrade-0.log in tomcat/logs directory to see if this issue is what you are having first. Start from the bottom of the log and review.

[LUK JASON]

Will the fix be included in next release such as 14.3 RU3 or 14.3 RU2 MP1 ?

[John Owens]

Yes. 14.3 RU3.

------------------------------
John Owens
Strategic Support Engineer | Symantec Enterprise Division (SED)
Symantec
United States
------------------------------

[Sherri Austin]

Hi John,

Any ETA when 14.3 RU3 will be out? I need to upgrade our SEPMs but would like to wait since it seems there are a lot of issues with 14.3 RU2.

Thanks

[John Owens]

Mid to late September

------------------------------
John Owens
[JobTitle]
[CompanyName]
[Country]
------------------------------

[Mauricio Silva]

hello. have the same error

'<domainIds><domainId>2AF2C8360A01043001C8A9BEF7FADA89</domainId><domainId>CCB6AC0C0A01043001C3A3AF8D3D7CFA</domainId></domainIds>', '', ''<br>EXCEPTION: com_exception: <b>Source:</b> Microsoft OLE DB Provider for ODBC Drivers<br/><b>Description:</b> [Microsoft][SQL Server Native Client 10.0][SQL Server]Cannot resolve the collation conflict between "SQL_Latin1_General_CP1_CI_AS" and "Latin1_General_CI_AS" in the equal to operation. in C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\ado.php:587
Stack trace:
#0 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\ado.php(587): com->Execute('EXEC SEM_HOMEPA...', -1)
#1 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\ado.php(343): ado_doPreparedStatement('EXEC SEM_HOMEPA...', Object(com))
#2 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\connectdb.php(234): ado_query('EXEC SEM_HOMEPA...', Object(com), false)
#3 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Inetpub\Reporting\Dashboard\getOverallSecurityStatus.php(189): sav_query('EXEC SEM_HOMEPA...', Object(com))
#4 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Inetpub\Reporting\Dashboard\ajax.php(628): getOverallSecurityStatus()
#5 {main}Error message: <b>Source:</b> Microsoft OLE DB Provider for ODBC Drivers<br/><b>Description:</b> [Microsoft][SQL Server Native Client 10.0][SQL Server]Cannot resolve the collation conflict between "SQL_Latin1_General_CP1_CI_AS" and "Latin1_General_CI_AS" in the equal to operation.<br>Error code: -2147352567<br>File and line: C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\ado.php(587)<br>Trace: #0 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\ado.php(587): com->Execute('EXEC SEM_HOMEPA...', -1)
#1 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\ado.php(343): ado_doPreparedStatement('EXEC SEM_HOMEPA...', Object(com))
#2 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Php\Include\Common\connectdb.php(234): ado_query('EXEC SEM_HOMEPA...', Object(com), false)
#3 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Inetpub\Reporting\Dashboard\getOverallSecurityStatus.php(189): sav_query('EXEC SEM_HOMEPA...', Object(com))
#4 C:\Program Files (x86)\Symantec\Symantec Endpoint Protection Manager\Inetpub\Reporting\Dashboard\ajax.php(628): getOverallSecurityStatus()
#5 {main}<br>

[John Owens]

Can you post the following log?

ReportingSpUpdateFor14_3_RU2.log

It should be in tomcat/logs as well.

------------------------------
John Owens
Strategic Support Engineer | Symantec Enterprise Division (SED)
Symantec
United States
------------------------------

[Mauricio Silva]

Yes, I will attach. I ran it a few times thinking it might change something. This log was the last run

Attachment(s)

ReportingSpUpdateFor14_3_RU2.log   25 KB 1 version

[John Owens]

Hi Mauricio,

I am attaching an updated tool I made a change to. It doesn't look like the original tool is able to connect to DB to make the change. Can you try this tool and report back?

Thanks,

------------------------------
John Owens
Strategic Support Engineer | Symantec Enterprise Division (SED)
Symantec
United States
------------------------------

Attachment(s)

ReportingSpUpdateFor14_3_RU2 (1).7z   17 KB 1 version

[Mauricio Silva]

Thanks a lot for the help.
After running the script, it fixed the problem.
I believe I can help many others who are still struggling.
I'm going to run some more tests to see if everything is working.

Thanks again!

[John Owens]

Very welcome. I updated the tool attached to the article as well.

------------------------------
John Owens
Strategic Support Engineer | Symantec Enterprise Division (SED)
Symantec
United States
------------------------------

[Fares Istaitieh]

I have a google chrome extension issue after clients get the latest release, google drive & Facebook won't upload files using google chrome.
there are also many different sites just like https://www.shutterstock.com/.

Does anyone know how to resolve this?