Some clients lose connection after upgrade to 14.2

[alryder]

I recently upgraded SEPM from 14.0 RU1 MP2 to 14.2 and after pushing out the new install package three clients are now unable to connect to the server. Looking under Help --> Troubleshooting --> Server Connection Status on each client shows the error: "Peer certificate cannot be authenticated with given CA certificate". Clicking "Details" tells me to run the Symantec Diagnostics Tool.

After running SymDiag, the only "Requires Attention" item is "Service IPS Network Filter Driver is not configured and operating properly" with the details of "Service last exited with code 31" but I have that same item on working clients so I don't think that is the problem.

I have tried installing the client via Push, exporting the install and manually installing and installing the Unmanaged package from the download and then converting to a Managed client and I get the same error each way. Going back to 14.0 RU1 MP2 restores communication with the SEPM.

I'm not sure what certificate and CA certificate are in play here so I'm wondering if anyone has any suggestions?

[alryder]

Following the steps at https://support.symantec.com/en_US/article.HOWTO81059.html#v57256809 to disable secure connections, then updating the certificate by following https://support.symantec.com/en_US/article.HOWTO81146.html restored communication BUT after turning on enable secure connections again it breaks the communication between the client and the server with the original error message.

[ℬrίαη]

Looks like support needs to be enaged on this issue.

[John Owens]

Please open a case and report back the case number.

[alryder]

Case number 15010255 opened.

[alryder]

The solution to this was to disable encrypted communication (so your working endpoints don't lose communication), create a new certificate per https://support.symantec.com/en_US/article.HOWTO81085.html, push the new communication package to the non-working endpoints and reenable encrypted communication.

[Danny N]

i have the exact same issue, my 14.0 clients work, but when creating new 14.2 clients i get the "Peer certificate cannot be authenticated with given CA certificate" the thing about our setup, is that we are using a wildcard SSL certificate, and not a selfsigned.

And from the 14.2 mananger i am able to create a new 14.0 client that works. 

[alryder]

Do the 14.2 clients fail with that error when you upgrade a 14.0 client? That was also happening to me. Unfortunately I have no suggestions when using a custom certificate except to involve support.

Good luck and let us know what happens.

[Danny N]

yes in my case it fails no matter what i do, i tried symlink drop, and also tried to create a new 14.0 client from the 14.2 manager, this still works, and i tried to use the communication file from 14.0 and import it on a 14.2. that still getting me the error.

I have created a ticket for symantec that has been ongoing for a week now without any progress.

It could seem the client is bugged since its a very new release. the Symantec tools don't even support version 14.2, i tried cleanwipe and symlink monitor, where non of them work yet

[Danny N]

what i find out is that Apache has a Symantec Endpoint Protection\apache\conf\ssl\sslforclient.conf file, where it can be seen the why it handles certificates is different between versions. and is why one version can work and not another one.

at the moment i am not sure what the difference mean, but try to find out why its causing a problem on the newest version

 

#   connect with. Disable SSLv2 by default (cf. RFC 6176).
####APACHE_DIRECTIVE_MERGE for_14.0.0.0
SSLProtocol all -SSLv2 -SSLv3

#   SSL Strong Cipher Suite:
#   List the ciphers that the client is permitted to negotiate.
#   See the mod_ssl documentation for a complete list.
#   3DES is required in order to support WinXP, Win2003 as 
#   these OS do not have AES cipher support out of the box.
#   We no longer support these OSes without modern patches, re-enable
#   3DES if your environment does.
####APACHE_DIRECTIVE_OVERRIDE for_14.1.1.0
SSLCipherSuite HIGH:!MEDIUM:!LOW:!aNULL:!eNULL:!3DES:!RC4

#   Server Certificate:
#   Point SSLCertificateFile at a PEM encoded certificate.  If
#   the certificate is encrypted, then you will be prompted for a
#   pass phrase.  Note that a kill -HUP will prompt again.  Keep
#   in mind that if you have both an RSA and a DSA certificate you
#   can configure both in parallel (to also allow the use of DSA
#   ciphers, etc.)
SSLCertificateFile "conf/ssl/server.crt"

####APACHE_DIRECTIVE_REMOVE

#:Removed_by_14.2.0.0_upgrade_wizard Wed Jun 20 14:59:47 CEST 2018:# SSLDHParametersFile

#   Server Private Key:
#   If the key is not combined with the certificate, use this
#   directive to point at the key file.  Keep in mind that if
#   you've both a RSA and a DSA private key you can configure
#   both in parallel (to also allow the use of DSA ciphers, etc.)
SSLCertificateKeyFile conf/ssl/server.key

[John Owens]

Hi Danny,

What is your case number?  I would like to look into this for you.

Thanks,

John Owens

[Scott K]

Is this just a problem with self-sign certificates issues by SEPM or with real paid wildcard certificates? We use a wildcard certificate for communications on our SEPMs (like *.company.com) and wanted to see if this might be a problem when I am planning on upgrading our environment next week.

[alryder]

Hi Scott,

I experienced the issue with only some of my clients when I upgraded them to 14.2 with the self-signed certificate while Danny seems to be having the issue on all of his clients when he tries to upgrade them with a wildcard certificate.

[Danny N]

Hi John

my case number is 14885061

[Danny N]

Hi Scott

we are having the issue with wildcard certificates, you can see from the client cert config that there is a difference between the versions, that explains at least why my 14.0 works and not the 14.2, still i have not been able to find the solution.

Today i was trying to see if i could get a Lets Encrypt cert working. but am not that expierenced in how to create this for the Symantec Apache

[Danny N]

i just tried to test

https://www.ssllabs.com/ssltest/analyze.html?d=symantec.workbridge.com

and get 

Certificates provided	1 (1338 bytes)
Chain issues	Incomplete

 

This only counts for the SEPM, i imported a PFX file that we also use with other servers, that don't have any issues. but could think this error is causing the clients not to work correctly. 

[Lloyd Jones]

I am experiencing this issue as well, except using a trusted internal CA certificate. The chain is valid and we upgraded from 14 MP2 to 14.2. The certificate configuration on our SEPM server didn't appear to change after the upgrade but I see this exact error.

This has happened on Windows 10 and Windows 2012.

[Scott K]

Thanks for your responses. I think I'll wait on upgrading our SEPMs since we heavily use SSL for communication with our clients. Might need to wait for RU2 MP1.

[John Owens]

Are there any cases opened for this?  If not, please open a case if you are experiencing this issue.  I would like to start tracking it internally and help provide a fix.

John Owens

[alryder]

John,

My case number was 15010255 and Danny replied that his case number is 14885061.

[Danny N]

don't fear to upgrade, that part will work flawless, and your old clients will still work with the new manager, the issue only comes on newere clients. as they use different security

[PG_PG]

Hi all,

Not sure if this helps but have You tried adding this line below in sslforclients.conf. This did the trick in my case in handling some of Linux machines. 

SSLCertificateChainFile /your/path/to/CA_chain.crt 

Second thing You can try to add servername=yourservername.company.com since You are using CA cert

The last thing please check MSL and make sure that server with DNS name is on the list. 

Oh and one more are You sure that TLS 1.2 is turned on the client machine (Last steps with psexec tool https://support.symantec.com/en_US/article.TECH235995.html ) 

[Danny N]

You are my Symantec hero

just went to my D:\Symantec Endpoint Protection\apache\conf\ssl and added this line

SSLCertificateChainFile "conf/ssl/CA_chain.crt"

after renamning the gd_bundle-g2-g1.crt to CA_chain.crt and copied to D:\Symantec Endpoint Protection\apache\conf\ssl 

and restarted the Symantec Web Server the client start comming online. This sounds like symantec didn't read the PFX file correct or missed out adding that line.

Still i can see from ssllabs.com that the security needs to be tighten a bit. but now its a big improvement that it start to works.

Thanks once again.

[Danny N]

i also manage to fix the SSL security to get a A grade

with modifying sslforclient.conf

SSLCipherSuite HIGH:!MEDIUM:!LOW:!aNULL:!eNULL:!3DES:!RC4:!RSA

SSLProtocol all -SSLv2 -SSLv3 -TLSv1

 

[PG_PG]

I'm glad it helped :) It was a big fuss with some of our Linux machines even CA was from a respected vendor somehow not every Linux had those certs in store and that line rescued the day :) 

I don't think if SEPM will add this line while importing at least it didn't in my case. Btw there should be nice official TECH document about replacing to CA cert but I guess Symantec is devoted to self-sign too much  ;) 

Regarding ssllabs You can sniff network traffic and check what ciphers are in use in client-server communication and exclude weak ones if that is the case here. 

[Leon Poortvliet]

I had this problem too and solved it by adding the CA chain to the server.crt. Same as with Danny, I had imported the PFX using the proper methods and still I received the error.

I appended the server.crt with the CA chain in PEM format and saved the file. Then I restarted the Manager, API and Web Server services and the clients started to connect again.

[Danny N]

I was just called by the Symantec support again regarding the ticket, even after providing the solution they asked for log files, and basic trouble shooting, it seems they a following a complete basic troubleshooting guide even by i have provided them solution, area of issue, and root cause.

the first day i told that they shoud escalte this and go for troubleshooting the certificate part, but instead they went for 10 reinstallings and clean wipe procedures.

for me it took a lot of time when being a windows guy not that experienced in Linux and apache, but i got a lot of expierence i can say.

i manage to get a very high A grade on SSLlabs, and disabled all weak ciphers and only allowing TLS1.2

[PG_PG]

True..  Symantec support it's a different story and if a case is not escalated everything takes ages ://

Anyways be careful with TLS according to this article 1.0 should be enabled at OS level https://support.symantec.com/en_US/article.TECH240233.html but maybe it's outdated and not impacting latest releases. 

[Danny N]

I have added 1.0,1.1,1.2 on the OS level, but it sounds more like it relates to not having OS patched. i like the way Symantec priorities, performance in front of security as they write with database connection

[HotDonut600]

Hi John, I have just opened case 15153618 on this issue. I have tried the various workarounds including the "solution" with no luck.

[dvz]

 I had a ssl connection error with SEP14.2 too. The solution from PG_PG helped me. Thank!

[Danny N]

Symantec claims they have released a new version that fix the issue. but i would rather just keep it as it is right now and where everything works

[John Owens]

This is still being looked into by development on Symantec side.  We changed communication models in 14.2 and for some reason it seems 14.2 is more aggressively validating even when the validate over HTTPS is turned off.

Configuring Apache/Tomcat to follow certificate chains is the workaround for now.

I will keep everyone updated on when we have a fix.

Thanks,

John Owens

[JKisner]

Experiencing this issue now with what looks like 1 Windows 10 PC.  6 other PCs appear to have updated fine when the updated client package was pushed out to them (14.2.770.0000).    Glad I test things first....lol

[JKisner]

@John_Owens - why would some clients updgrade fine and connect to the SEPM, but some do not?   I even uninstalled SEP and reinstalled manually with an exported package from the SEPM and it will not connect.  Same package on a brand new machine installs fine and connects right up to the SEPM...

[JKisner]

Why would some clients updgrade fine and connect to the SEPM, but some do not?   I even uninstalled SEP and reinstalled manually with an exported package from the SEPM and it will not connect.  Same package on a brand new machine installs fine and connects right up to the SEPM...

[John Owens]

To track this specific issue:

https://support.symantec.com/en_US/article.TECH251024.html?