No alerts for invalid licenses/disabled antivirus

[ITS Technical Support]

Like many of you we've been migrating from Endpoint Small Business Edition to Endpoint Protection Cloud (for those that don't know, it's a forced migration for partners).  There have been some bugs with the migration, particularly of existing policies, and a few web portal features removed compared to SBE, but generally we were OK with things as we could move forward.

 

Thursday we happened to notice in the SEPC web portal that several PCs at one client that were in a yellow (not red) "At Risk" status because of the reason "Device license has expired."  Upon investigating we found events on the web portal that autoprotect, etc. had been disabled on the PCs.  Eventually we found if the PC is restarted it fixes itself within a couple minutes.

 

Symantec says that since we resolved this situation on the PCs, there is no way to find out why the licenses expired, and despite a multi-layered email alert system there is apparently no alert for "PC is unprotected" or "license expired."  (This is case 28936197, for anyone from Symantec.)

 

Does anyone know if there is a way to monitor via RMM for this situation?  I could not find a Windows Event Log entry on the PCs, and unlike the same condition in Small Business, no Windows services stop or become disabled.  Worse, we have a few clients who wanted antivirus but not our monitoring services so we can only look in the web portal to find out if any PCs are disabled...I guess multiple times per day if we are being thorough?

 

This is disturbing to me.  We've been using Symantec for 20 years, give or take, so were not looking to switch, but not alerting if PCs are unprotected is mind boggling.

 

--

 

Steve Yates

ITS, Inc.

[ITS Technical Support]

Made some progress with support today...

When upgrading from SEP SBE to SEPC, it seems that all the "preconfigured" alerts are set to Inactive until 100% of the devices have upgraded.  Ergo, if some laptop in a closet is off for a month or two, none of the default alerts will trigger during that time.  And, even if alerts are configured in the Partner Management Console (they are not, by default), the alerts will not trigger until activated on the client.  The alert rules can be manually activated.

It is still unclear whether we would need to add a custom alert rule to each client's SEPC portal in order to receive alerts about the antivirus being disabled.  For instance in the device's Event log we get this for Auto-Protect, SONAR, etc.:

AntiSpyware has been disabled     Security     Not Available (this is the username)     Protection     Mar 28, 2019 1:58:04 PM

It is not clear what Event Type to use when creating a custom alert rule, hence the lack of clarity.  For an event type of "Device Control" I can specify "Source EQUALS protection AND Severity GREATER THAN Minor" which seems fairly generic.  They're looking into that.  Still seems like it should be a default alert.

 

(all this was after support's first message today, trying to convince me that despite the PC showing everything as "disabled" that didn't mean the software wasn't working (!!):

"Because they had a notice about the licenses it does not mean they are unprotected either. 
They only way you can completely unprotected the PC is to uninstall the software.
Unless you caught a malware, there is no evidence to prove the computers were unprotected." )

[HarrisonP]

I'm going to cherry pick just one of your questions that's easiest to answer.

To monitor with RMM: https://www.symantec.com/docs/TECH251363

https://www.symantec.com/docs/HOWTO128567 (includes a link to our ConnectWise Integration package, and even if you don't choose ConnectWise as your RMM provider, it also includes CLI tools that may be helpful for whatever vendor you do choose)

I totally agree that the custom alert rules aren't as intuitive or as user friendly as I might hope, but you can check out https://www.symantec.com/docs/HOWTO124337, and https://www.symantec.com/docs/HOWTO127042 for the specific details I think that you're looking for. Otherwise, our help center has a whole section on alerts and events to check out: https://help.symantec.com/bucket/sepc/alerts_events?locale=EN_US

Otherwise, this would certainly make a great feature request.

I hope I've managed to address most everything, but there was certainly a lot there so I might have missed something

[ITS Technical Support]

Hi Harrison, thanks for posting...oddly I didn't get a notification of your post yesterday so didn't know you replied.

We finally got a different tech today (first one ended with, basically, "well then don't allow users to disable the antivirus and it's not a problem" obviously still not understanding the issue).

We (the new tech) think this is can be monitored via alert I posted:  Category "Security", Event Type "Device Control",  text/attributes "Source EQUALS protection AND Severity GREATER THAN Minor."

As far as I am concerned everyone needs to add this custom alert rule right away, or else check all your PCs on a regular basis to ensure they haven't falsely reverted to an expired license.

Note when creating alerts the interface can be really slow at times.  I've found it helps to type the first few letters of each attribute, otherwise it can take up to 30 seconds or so to populate the "dropdown" it creates.  And then hit either Tab (if you picked a prefilled item) or Enter (if you're entering a number on your custom rule) so it is accepted.

The ConnectWise integration package has several issues including the scripts running commands twice (regular and 64 bit, reported) and the Push script's installer not downloading files when we tried it.  The latter we have a case on also but it was with the tech from this issue and I told her were weren't going to spend more time on issues until we know if we had to move off SEPC, so it got closed.  Also, CW Automate did not catch this condition because the SCS service is left running so it doesn't know the a/v is disabled (unlike in SEP SBE).

[HarrisonP]

I'm not certain that alert will do what you intend, but I'll test it and see if I can get it to work.

You should be able to determine if autoprotect is on or off with the CLI tool: SEPCClientCLI64.exe -t 1

[ITS Technical Support]

I acknowlege that alert is kind of a slegehammer.  The other suggestion was to look for licenses with an expiration date in < 1 day but I'm not clear if that would actually apply.

It looks like SEPCClientCLI64.exe is only in the ConnectWise Integration zip?  It's not on my PC that I can find.  That's at least something scriptable but really it seems like the product should include alerting of disabled antivirus.

From our perspective we were being told that SEPC might disable itself, for reasons that are unknown and not possible to know (hence not preventable), and there was no way to know about it other than opening each and every client one at a time and looking at all the PCs for ones that are online but yellow.  And doing that hourly to approach real time monitoring is a massive time sink. 

I certainly hope the custom alert rule will work because our team already started talking about the loss of faith and finding another security solution.  Most of this forced migration from SEP SBE to SEPC has been very buggy...things that don't migrate properly include firewall rules, file exclusions, folder exclusions...I'll just stop there.  Those are fixable but broken trust in having the product even on is not.

[HarrisonP]

The CLI tool will need to be present on the device for that command to work. I believe if you use Connectwise to deploy SEPC it should be present. This video is the best demonstration of how to do this that I'm aware of: https://help.symantec.com/cs/PMC_SEPC/PMC_SEPC/v127236586_v123881503/Video:-Setting-up-ConnectWise-Automate-to-use-Symantec-Endpoint-Protection-Cloud/?locale=EN_US

As far as the rest of you comment? Unknown and unknowable? I think not.

Let me preface this by saying that this is a manifestation of an issue that is extremely common to SEP SBE cloud, and extremely rare in SEPC.

You got an error message on the agent showing that there was a device license issue correct? And the protection was disabled? https://www.symantec.com/docs/TECH237152

At its heart, this is the same issue as https://www.symantec.com/docs/TECH218313 for SEP SBE. This is actually the number one most common support call for SBE in my experience.

The underlying cause is an issue with the license activation with the agent itself (it has little to do with actual license expiration, so that proposed alert rule is unlikely to help either). There are several possible causes for this.

1. The agent is just plain unable to reach the activation servers. This is most often due to a firewall or proxy blocking the connection.

2. The license has recently been renewed or replaced, and the agent failed to remove the old activation information and replace it with the current information. In this case, this is easily corrected by booting into safe mode and removing the license activation folders.

3. The license has been recently replaced or renewed, and the agent grabbed the current license activation information, but failed to remove the old license activation info, resulting in 2 activation folders present.

However, once the license activation issue has been resolved, it stays resolved, so long as the computer stays active.

The most common times you'll run into this issue is immediately after install, or when bringing the computer back online for the first time after a long time offline.

But the point I'd like to stress is, this issue is much more common to SBE than it is to SEPC. SEPC has a much better activation logic, reducing the chances of this issue recurring. This is also made evident by the apparent ease with which you were able to resolve the issue, just rebooting the device instead of the much more time consuming and elaborate fixes we sometimes need to try in SBE (booting into safe mode and removing the activation folder, etc., etc.).

As near as I can tell, this is the sequence of events you experienced.

1. Upgrade to SEPC.

2. Noticed license issues on some devices. This was probably caused by leftover activation information from the SBE agent.

3. Rebooted the devices, thus clearing the old activation information and resolving the issue.

I highly doubt this is an issue you'll often run across.

[HarrisonP]

It looks like the activation process has vastly improved for SBE as well actually. I can't force the agents in either my SEPC lab or my SBE lab to actually have an activation problem anymore. It used to be extremely easy to do this.

[ITS Technical Support]

Hmm interesting, but I don't think it's exactly the same.  For us the issue happened around two weeks after upgrading from SBE (I am not looking at the actual date).  All four that lost their license were the same day, but over several hours (we can see it in the logs online).  The PCs were fine before that, so it was totally unexpected.  I don't know if the PCs had ever restarted again after SEPC was installed, but I know they didn't that day.  One was even logged out and idle.

Per case 28936197, "It is not possible to know exactly what the problem was because devices are working normally now.  This could have been several things for example, connectivity issues, a temporary issue in our servers that verify the license, a software issue in the computer, incompatibility with a software that is installed etc."  To us that is pretty clear there are multiple causes.  Now, that tech is the same one that later suggested we simply turn off the option to allow users to disable autoprotect (which we didn't have enabled and clearly wasn't the case here), so I am now a bit suspicous of everything that tech is telling me, but that's why we went ballistic.

We did have occasional slow activations with SBE, but 99% of the time it was within minutes. Normally when we had PCs lose their license it was either because they were off for a long time (months) or were on but had no connectivity.  In the past we had to uninstall, restart, and reinstall, but starting earlier this year sometimes they usually reactivated themselves (and deleted PCs rejoined on their own).

I did try manually disabling autoprotect last night, but doing that logs a totally different event, so my alert did not trigger. Hopefully it will for a "real" event.

 

re: SEPCClientCLI64.exe, I think "-t 1" shows firewall status, while "-t 0" shows autoprotect?  At least I''m guessing that by the output:

>SEPCClientCLI64.exe -t 1
Executing IS_STATUS::eGetFWStatus... IS_STATUS::eStatusEnabled

>SEPCClientCLI64.exe -t 0
Executing IS_STATUS::eGetAPStatus... IS_STATUS::eStatusEnabled

I found that by running it without the "-t" which, for any lurkers, be aware that shows like 75 things but also runs everything it possibly can including updates and scans and fixes. :)  This morning though I get an error for some reason:

>SEPCClientCLI64.exe -t 0
Executing IS_STATUS::eGetAPStatus... Failed (_com_error): 0x80040300 - IDispatch error #256
(shrug)

 

The SEPC KB you linked mentions "Unenrolling the agent from the management console fails to uninstall the agent from the Windows machine causing it to still consume a license."  I'm not sure if that's saying uninstalling should remove the software and isn't?  Per our case 28930670 unenrolling will not uninstall the software: "In the future, we are planing to add a function to the SEPC portal that will allow administrators to uninstall the software remotely.
However, at this moment this function is not available." (same tech as above, though, so...)

Thanks,
Steve

[HarrisonP]

Unenrolling from the portal will not uninstall the software from the computer. That's an edit we'll need to make to that document.

[ITS Technical Support]

Happened again today at a different client.  The custom alert did NOT email us as the partner and the Alerts tab shows no alerts.  Suggestions?

	Auto-Protect has been disabled	Security	Not Available	Protection	Apr 9, 2019 7:49:26 AM
Event Details Auto-Protect has been disabled Event ID: 4355:e8728ff0-5ac5-11e9-d6cf-0000001c3d4f Source: Protection Security Client: SEP Cloud Source Version: Not Available Security Client Version: 22.16.4.15

[ITS Technical Support]

So to follow up I have heard from support and another Symantec partner that this self-deactivation issue was resolved in early May.  Due to this and various other issues surrounding the new software and distribution model, we moved on to other software for all but one of our clients, but at least it will be fixed going forward.