Thanks for the info, Gene.
We decided to go with separating the UMP, this has resolved the intermittent UMP issue, buuuuut, we originally gave the wasp:
java_mem_max = -Xmx16384m
java_mem_init = -Xms4096m
and it was all taken, so I gave it:
java_mem_max = -Xmx24576m
java_mem_init = -Xms4096m
That is now all taken.
The log file is still spammed with these logs, and I cannot find any custom scopes, but the IP addresses of the uim/ump/sql boxes are 172.31..17.x
I have the log set to 5, and I do not think these display in 3 and lower. Maybe having the log on level 5 is hitting the resources, but I wouldnt have thought 20+GB worth
Jul 04 08:58:49:475 DBLOW [GroupCacheThread, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.1.0/24. Starting address=172.16.1.1, ending address=172.16.1.254.
Jul 04 08:58:49:475 DBLOW [GroupCacheThread, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.1.0/24. Starting address=172.16.1.1, ending address=172.16.1.254.
Jul 04 08:58:49:475 DBLOW [GroupCacheThread, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.2.0/24. Starting address=172.16.2.1, ending address=172.16.2.254.
Jul 04 08:58:49:475 DBLOW [GroupCacheThread, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.3.0/24. Starting address=172.16.3.1, ending address=172.16.3.254.
Jul 04 08:58:49:475 DBLOW [GroupCacheThread, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.4.0/24. Starting address=172.16.4.1, ending address=172.16.4.254.
Jul 04 08:58:51:039 DBLOW [GroupCacheThread, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.1.0/24. Starting address=172.16.1.1, ending address=172.16.1.254.
Jul 04 08:58:51:039 DBLOW [GroupCacheThread, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.1.0/24. Starting address=172.16.1.1, ending address=172.16.1.254.
Jul 04 08:58:51:039 DBLOW [GroupCacheThread, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.2.0/24. Starting address=172.16.2.1, ending address=172.16.2.254.
Jul 04 08:58:51:039 DBLOW [GroupCacheThread, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.3.0/24. Starting address=172.16.3.1, ending address=172.16.3.254.
Jul 04 08:58:51:039 DBLOW [GroupCacheThread, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.4.0/24. Starting address=172.16.4.1, ending address=172.16.4.254.
Jul 04 08:58:55:617 DBLOW [https-jsse-nio-443-exec-95, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.1.0/24. Starting address=172.16.1.1, ending address=172.16.1.254.
Jul 04 08:58:55:617 DBLOW [https-jsse-nio-443-exec-95, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.2.0/24. Starting address=172.16.2.1, ending address=172.16.2.254.
Jul 04 08:58:55:617 DBLOW [https-jsse-nio-443-exec-95, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.3.0/24. Starting address=172.16.3.1, ending address=172.16.3.254.
Jul 04 08:58:55:617 DBLOW [https-jsse-nio-443-exec-95, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.4.0/24. Starting address=172.16.4.1, ending address=172.16.4.254.
Jul 04 08:58:55:617 DBLOW [https-jsse-nio-443-exec-95, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.1.0/24. Starting address=172.16.1.1, ending address=172.16.1.254.
Jul 04 08:58:55:617 DBLOW [https-jsse-nio-443-exec-38, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.1.0/24. Starting address=172.16.1.1, ending address=172.16.1.254.
Jul 04 08:58:55:617 DBLOW [https-jsse-nio-443-exec-38, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.2.0/24. Starting address=172.16.2.1, ending address=172.16.2.254.
Jul 04 08:58:55:617 DBLOW [https-jsse-nio-443-exec-38, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.3.0/24. Starting address=172.16.3.1, ending address=172.16.3.254.
Jul 04 08:58:55:617 DBLOW [https-jsse-nio-443-exec-38, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.4.0/24. Starting address=172.16.4.1, ending address=172.16.4.254.
Jul 04 08:58:55:617 DBLOW [https-jsse-nio-443-exec-38, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.1.0/24. Starting address=172.16.1.1, ending address=172.16.1.254.
Jul 04 08:58:56:086 DBLOW [https-jsse-nio-443-exec-11, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.1.0/24. Starting address=172.16.1.1, ending address=172.16.1.254.
Jul 04 08:58:56:086 DBLOW [https-jsse-nio-443-exec-11, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.2.0/24. Starting address=172.16.2.1, ending address=172.16.2.254.
Jul 04 08:58:56:086 DBLOW [https-jsse-nio-443-exec-11, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.3.0/24. Starting address=172.16.3.1, ending address=172.16.3.254.
Jul 04 08:58:56:086 DBLOW [https-jsse-nio-443-exec-11, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.4.0/24. Starting address=172.16.4.1, ending address=172.16.4.254.
Jul 04 08:58:56:086 DBLOW [https-jsse-nio-443-exec-11, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.1.0/24. Starting address=172.16.1.1, ending address=172.16.1.254.
Jul 04 08:58:57:242 DBLOW [https-jsse-nio-443-exec-39, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.1.0/24. Starting address=172.16.1.1, ending address=172.16.1.254.
Jul 04 08:58:57:242 DBLOW [https-jsse-nio-443-exec-39, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.2.0/24. Starting address=172.16.2.1, ending address=172.16.2.254.
Jul 04 08:58:57:242 DBLOW [https-jsse-nio-443-exec-39, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.3.0/24. Starting address=172.16.3.1, ending address=172.16.3.254.
Jul 04 08:58:57:242 DBLOW [https-jsse-nio-443-exec-39, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.4.0/24. Starting address=172.16.4.1, ending address=172.16.4.254.
Jul 04 08:58:57:242 DBLOW [https-jsse-nio-443-exec-39, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.1.0/24. Starting address=172.16.1.1, ending address=172.16.1.254.
Original Message:
Sent: 07-01-2019 11:09 AM
From: Gene HOWARD
Subject: Downgrade from 9.1 to 9.0.2
Hi,
So there is no test /supported method to moving back from 9.10 to 9.02 other than doing a complete restore of the OS and the database.
That being said you can try the following if you would like. The worst case scenario will be 1) you restore to a good known back if 9.1 before doing the below process. 2) you install a clean database and a clean install of 9.02
1) make a backup of the UIM server and test it
2) make a backup of the database and test it.
3) delete the C:\Program Files\Zero G Registry\.com.zerog.registry.xml or the /var/.com.zerog.registry.xml file
4) deactivate the wasp probe. and rename the folder
5) delete the wasp probe from IM or AC2
6) make a backup of the robot and hub directories.
7) run the UIM 9.02 installer again. It will act as if it is a clean install of 9.02. Make sure you probe the EXACT domain host and robot name as done previously and install into the same location.
8) this should get you back to a 9.02 probe installation.
9) run the UMP installer.
This process should get you to have uim9.02/ump9.02.
If this does not resolve the issue or you run into major problems your best bet will be to uninstall and do a clean install of 9.02
------------------------------
Gene Howard
Principal Support Engineer
Broadcom
Original Message:
Sent: 07-01-2019 03:57 AM
From: David Givens
Subject: Downgrade from 9.1 to 9.0.2
I can also see that 9.1 is still using 8.50 schema. Does this suggest, from a database perspective, a downgrade is possible ?
data_engine schema version: 8.50(0)
Original Message:
Sent: 07-01-2019 03:34 AM
From: David Givens
Subject: Downgrade from 9.1 to 9.0.2
Hi Gene,
We do run the UMP on the primary and have also ran it on multiple systems this way for years on 851 , we also have a 901 using this configuration with no issues. We have a high spec server to run this.
The main reason we run this is to get around the issue of he Admin Console attempting to connect to the private IP of the UIM server when access via public UMP, or is there another way around it ?
A downgrade to 902 would still be a preferred solution as this is working on another system with no issues, and is actually a lower spec build.
Thanks
Original Message:
Sent: 06-28-2019 12:10 PM
From: Gene HOWARD
Subject: Downgrade from 9.1 to 9.0.2
HI,
First, the new portal is different than the old SFDC. Emails used to be copied into the case comments for us.
now they are not.
You should have a tab for emails where you can see the emails incoming and outgoing and you can go into those to see the messages you received via email.
We are still working out our processes to try to have a unified view of comments and emails.
Hopefully, this will be worked out soon.
As to the database. As this is just another application on another server it can change over time.
This should not be discounted as a possible root issue because you did not use to have an issue with 8.51.
I am not saying it IS the issue just that it should be checked as well.
I would double check your indexes and fragmentation on your database. these are the usual culprits of your description.
A good reference is below:
https://community.broadcom.com/communities/community-home/librarydocuments/viewdocument?DocumentKey=bb816ff1-331f-4b53-a868-45435e1966ec
this does not list 9.10 but it is still relevant.
the MCS issues should not be causing an issue with UMP loading.
I would suggest you set the loglevel to 3 and logsize to 35000. setting your logsize above 3 in UMP wasp will only cause you to miss important information.
Currently, the logsize is set to 100 and loglevel is set to 5 so the logs cover less than 1 minute in time.
next, it looks like you might have some large discovery_scopes setup. if these are set up to run on a schedule you might want to disable these to they are only run manually while troubleshooting.
one way to check if you are having database issues is to set the data_engine loglevel to 3 and logsize to 75000.
When the problem is happening to check the commit lines in the data_engine logs and see how long it is taking the database to do inserts.
If these values are very high 3-10 seconds then you know you have an issue on the back end database that needs to be addressed.
You are not running the UMP on the primary hub, are you?
I see in the 27th folder a wasp and portal logs but the controller.cfg shows a data_engine probe in it.
if you are trying to run UMP on the primary in 9.x I would suggest the first thing you do it move this to a dedicated machine and see if that does not resolve your issue.
running ump on the primary should only be done in a dev environment as the primary hub and UMP are really too resource intensive to run on the same machine.
hope this helps
------------------------------
Gene Howard
Principal Support Engineer
Broadcom
Original Message:
Sent: 06-28-2019 11:42 AM
From: David Givens
Subject: Downgrade from 9.1 to 9.0.2
Hi Gene,
Thanks for the quick reply.
The issue look to be just USM , it can be fine for hours and then the browser tab will start spinning, like a refresh icon. No folders, qos etc can be viewed via USM.
Is a refresh of the page is performed or a new login is performed, USM attempts to load but sites on the final (dark blue/light blue) "Loading" bar.
If open the wasp via AC, stop/start usm webapp, USM is then restored.
We are not using secure setup. At one stage we were on robot 9.10 and hub 9.10HF1.
After reading the post here, I downgraded to robot 7.97HF3 and hub 7.97. The issue is still occuring
The case is also logged under 20014504 , the suspicion from support is that it is database related. I have doubts due to it performing with no issues under 851, and also during the usm outage, any query made via the wasp (dashboard for example) returns with no issues.
This error exists in MCS log and is also added to the case, may not be related ... ?: MCS is not used, other than default system setup.
Jun 27 11:53:15:818 [Thread-3, mon_config_service] TemplateController.getActiveTemplateVersion:713: !!!Contact Support!!!: Invalid configuration: More than one template is already active for template Profile Copy
Maybe unrelated too, but wasp.log is constantly being spammed by (below) and we do not have any discovery scopes for this.
Jun 28 16:36:39:870 DBLOW [GroupCacheThread, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.1.0/24. Starting address=172.16.1.1, ending address=172.16.1.254.
Jun 28 16:36:39:870 DBLOW [GroupCacheThread, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.1.0/24. Starting address=172.16.1.1, ending address=172.16.1.254.
Jun 28 16:36:39:870 DBLOW [GroupCacheThread, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.2.0/24. Starting address=172.16.2.1, ending address=172.16.2.254.
Jun 28 16:36:39:870 DBLOW [GroupCacheThread, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for 172.16.3.0/24. Starting address=172.16.3.1, ending address=172.16.3.254.
Jun 28 16:36:39:870 DBLOW [GroupCacheThread, com.nimsoft.discovery.common.net.NetworkV4SubnetRange] Created subnet range network scope for
This information is with support, under the case I mentioned.
I also think there is an issue with the customer portal as I cannot see any updates in the portal, only emails. I dont know if this is an option that is not being used by support ?
Thanks
Original Message:
Sent: 06-28-2019 11:16 AM
From: Gene HOWARD
Subject: Downgrade from 9.1 to 9.0.2
Hi can you provide some more details on the hand issue?
There is no real good option for moving back to 9.02 from 9.10 if you do not have a full backup and database restore point.
Is the problem IM hanging, ump hanging?
If you can provide some additional details we might be able to help you fix it.
Also, are you running controller and hub 9.10S as this adds another level of complexity to the issue.
------------------------------
Gene Howard
Principal Support Engineer
Broadcom
Original Message:
Sent: 06-28-2019 11:13 AM
From: David Givens
Subject: Downgrade from 9.1 to 9.0.2
hi all,
We are having issues in which usm hangs intermittently and we are unable to find the cause.
The same system was running on 851 for years without any issues.
The upgrade path was 851 -> 902 -> 910
We currently have a working system on 902, so I would like to revert back down to that version. is 901 to 902 possible?
The other option is to restore 851 but that involved 1 weeks worth of QoS loss
Thanks