Hi Shiv,this could be linked to the other issue you see with the TIM & Defect Storm.The one thing we do not tend to see, is that the aggregator is spawned as its own instance. Hence requiring the same amount of memory as usually the EM/MoM itself.
This in turn means that if you MoM is using 1Gb, when the aggregator runs (hourly aggregator) it will use one additional GB of Ram.If you now have the daily aggregator running because it is catching up (due to the TIM not providing data for a while) to handle the statistics and the defects, this 1Gb will also add up. If in the worst case you are at the end of a Month and have also the Weekly and Monthly aggregators running at the same time, you'll have 4 additional aggregators running, each using another 1Gb of RAM.So - only thing you can do here - if this is a Virtual machine, add enough resources (here RAM and CPU to speed up the process). If not as stated in the other thread (https://community.broadcom.com/enterprisesoftware/communities/community-home/digestviewer/viewthread?MessageKey=32ffb92e-41ea-4af0-ae83-42b19dff0c8a&CommunityKey=be08e336-5d32-4176-96fe-a778ffe72115&tab=digestviewer&bm=32ffb92e-41ea-4af0-ae83-42b19dff0c8a#bm32ffb92e-41ea-4af0-ae83-42b19dff0c8a) make sure you stop the defect storm condition so the MoM/EMs can run normally.Note: I also see that you have one of the first releases. Using (applying) SP3 at least would help increase stability.
Hi Jorg,No error is showing as of now in CEM, only error getting now is Enterprise manager restarted and with every instances of error in CEM getting log message java. lang.OutofMemoryError : Java heap space.1) I doubt if initial Java heap memory & maximum java heap memory setting needs to be done .
I see that we can change the value of wrapper.java.initmemory & wrapper.java.maxmemory in EMService.conf file. Current value set as below:
# Initial Java Heap Size (in MB)wrapper.java.initmemory=4096# Maximum Java Heap Size (in MB)wrapper.java.maxmemory=102402) Also i have seen one file is there SmartStorTools.bat.Edit JAVA_OPTS to increase the heap size if needed. Current value set as below:
set JAVA_HOME=%INSTALLDIR%\jreset JAVA_OPTS=-Xmx512m
Currently the value set to 512m should we increase or decrease to resolve Java heap space issue3) Also there is one another file Introscope_Enterprise_Manager.lax where i have seen option to set the lax java option additional values like Xms & Xmx setting.
lax.nl.java.option.additional=-Xms8192m -Xmx8192m -Djava.awt.headless=false -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./logs/ -Xss512kPlz let me know the exact setting needs to be done to all these to resolve java heap space issue if we have 32GB of RAM available in the server .
Hi Shin,From the case 32979151 logs I see you have not applied Hotfix SP3 yet. That would help because the condition you run into seems to be due to an issue causing the OOM.If after you applied SP3 this problem still occurs, please see https://knowledge.broadcom.com/external/article?articleId=140703 - just in case you do not apply the memory changes at the right location.
I also found in the readme.txt , Configuration steps for the Cluster. So, does this step need to be done?
If yes, when we need to do this before or after installing hotfix. PFA screenshot below. Also let me know is it to be done after applying the hotfix on all 3 servers i.e., including a MoM & an agent collector and TIM collector server .
Hi SHiv,the Hotfix is to be applied on the EM (Collectors) and MoM only.From your explanation, you have one MoM, one EM (for Agents) and one EM for the TIM collection service. You have to apply the Hotfix to all these 3, and also on the machine (if separate) running the PostgreSQL Server.
You also have 3 TIM's running. These do not need to have the Hotfix applied. If there are any Hotfixes for the TIMs, it will come in a separate hotfix package.
Hi All,We have applied hotfix HF 60 to our MoM server & not applied currently to collectors in Non Prod Env.getting below error in WebView 1. Error retrieving permissions. Status code: 500
Enterprise Manager and WebView start up with no errors. But When logging into Enterprise Manager Team Center, seeing the following error in the IntroscopeWebview.log and there isn't any data showing up at all under metric view, agent view, mapAlso in New APM status console , collectors are not reflecting , I suspect it was due to not applying HF60 patch on collectors or due to some other issue . Also one of our Agent collector is on 10.5 & other TIM collector is on 10.7 version so how do we proceed ?
Hi Shiv,it becomes difficult to read the thread - as we are dealing with consecutive different situations here.So - first of all: Please update all MoM and Collectors with HF60 prior starting them.
The workstation needs to be of the same version as the MoM. Means, you will have to get the correct HF 60 version of the workstation to have a seamless integration with the updated MoM. Please ask support to provide you the Workstation for your current installation.Regards
We have troubleshooted it with Broadcom support team & found that ts_defect_meta_values tables creating huge space in the Postgre database ,finally support team suggested the below steps & it worked .1) Drop tablesDROP TABLE ts_defect_meta_values_20130405 -- Drop these first (sample date)Then drop tables for the same exact dateDROP TABLE ts_defects_20130405;;DROP TABLE ts_tran_comp_details_201304052) Vacuum Appmap tablesvacuum full analyze appmap_edges;vacuum full analyze appmap_vertices;vacuum full analyze appmap_attribs;We performed above steps & able to free up the space of 285GB from the drive . Also we ran full vacuum & able to clear another 40GB of space.EM crash issue also resolved after removing the file com.wily.apm.webservices_10.7.0.jar from the <EM Home>/product/enterprisemanager/plugins folder as suggested by Broadcom support .
15GB heap occupied by one jetty thread serving a Web Service call.
This is a Web Service SOAP call to MetricsDataService.getMetricData(String agentRegex, String metricRegex, ...) with huge regex parameters.
- agentRegex is a 7k long String:
- metricRegex is a 40k long String:
JSP\\|_newAgentDetails:Average Response Time\\(ms\\)|Backends\\|vpaspr dpdcplvpasdb1-1641\\(Oracle DB\\)\\|Commits:Responses Per Interval|Backends\\|nbfpr dpdcplnbfdb1-1661\\(Oracle DB\\)\\|SQL\\|Prepared\\|Query\\|SELECT ROLEMENUXR0_\\.MENU_ID AS COL_0_0_, MENUDO1_\\.MENU_NAME AS COL_1_0_, ROLEMENUXR0_\\.EFFECTIVE_DATE AS COL_2_0_, MENUDO1_\\.SEQUENCE_NUMBER AS COL_3_0_ FROM QACONFIGADMIN\\.ROLE_MENU_XREF ROLEMENUXR0_, QACONFIGADMIN\\.MENU MENUDO1_ WHERE ROLEMENUXR0_\\.USER_ROLE_ID = \\? AND ROLEMENUXR0_\\.MENU_ID = MENUDO1_\\.MENU_ID AND ROLEMENUXR0_\\.VIEW_IND = \\?:Concurrent Invocations|Servlets\\|_InboxDataEntryLayout2:Concurrent Invocations|JSP\\|_ConversionDataEntry:Average Response Time\\(ms\\)|Servlets\\|_TicketPopup:Errors Per Interval|JSP\\|_revivalProcessData_2D_QualityChecker:Average Response Time\\(ms\\)|Variance\\|Servlets\\|_DeathClaimProcessDataApproval:Average Response Time\\(ms\\) Deviation|JSP\\|_selectRO:Responses Per Interval|Servlets\\|_SurvivalClaimProcessData:Errors Per Interval|Variance\\|Servlets\\|_DuplicatePolicyBondApprover:Average Response Time\\(ms\\) Prediction|JSP\\|...