Have you tried this:
Please back-up the registry before modifying any keys
1. Open and registry editor and browse to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\semsrv\Parameters
2. Make sure value data for JVM Option Number 0 is set to -Xms512m
3. Also Make sure value data for JVM Option Number 1 is set to -Xmx512m
4. If it is already set to mentioned numbers go to C:\Program Files\Symantec\Symantec Endpoint Protection Manager\bin and Edit sesm.bat
5. Confirm if the values are -Xms512m -Xmx512m that match with the registry entries
6. If not change the vaues to -Xms512m -Xmx512m then save the file
7. Restart the SEPM and Embedded database service
8. Login to SEPM and should work as expected.